Step 8: Write a Regression Test

Test Setup

TestHarness requires that all test related files be organized in a standard format. By now, the reader must have noticed the test directory in their application root. All tests should be grouped logically into individual directories in test/tests and have a consistent naming convention. The folders in test/tests are usually hierarchically structured, similarly as to how those in the src and include directories are. For example, test files for the Diffusion class are located in a subdirectory of test/test/kernels, since it is a Kernel object. The object's designated test directory will then contain the following:

1. Input files designed to test the application.

2. A gold directory containing the certified results to be matched by outputs.

3. A test specification file that provides the instructions to TestHarness.

Test Specification

Test specification files are Hierarchical Input Text (HIT) formatted ones that define the tests to be run. By default, TestHarness searches the directory tree for a file named "tests" to read the specs from. The familiar HIT syntax used in these files is demonstrated by the specification for the simple diffusion test:

[Tests]
  [test]
    type = 'Exodiff'
    input = 'simple_diffusion.i'
    exodiff = 'simple_diffusion_out.e'
  []
[]

The type parameter specifies one of the available test methods. In the above case, the Exodiff object is used, which requires an input file be specified with the input parameter and an ExodusII formatted gold file be specified with the exodiff parameter. The Demonstration section will show how to use the Exodiff and RunException testers on the DarcyPressure class. For a complete list of test objects, please see the discussion about testers.

Run Tests

The run_tests Python script located in the application root can be executed to invoke TestHarness on all test specifications it finds in test/tests. The basic command was already demonstrated in Step 1:

cd ~/projects/babbler
./run_tests -j4

This will execute the simple_diffusion.i file and conduct the Exodiff procedure. When the Test section introduces two new specifications, it will be shown that the above command will cause those tests to run as well.

cd ~/projects/babbler
./run_tests -j4 -p2 --n-threads=2

Source Code

To be sure, one can confirm that the "viscosity = 0" input leads to a crash by adding it to the [diffusion] block in pressure_diffusion.i and executing the input file. Upon doing so, the following message is reported to the terminal:

Nonlinear solve did not converge due to DIVERGED_FNORM_NAN iterations 0
 Solve Did NOT Converge!
Aborting as solve did not converge

The status DIVERGED_FNORM_NAN indicates that the residual was indeed NaN due to division by zero. Please be sure to remove this erroneous input, or simply enter git checkout pressure_diffusion.i, before proceeding.

Since it is clear that the zero-valued viscosity input is invalid, an error should occur from within the DarcyPressure class by invoking the paramError() method on the "viscosity" parameter. Add this call to the constructor method in DarcyPressure.C:

DarcyPressure::DarcyPressure(const InputParameters & parameters)
  : ADKernelGrad(parameters),

    // Get the parameters from the input file
    _permeability(getParam<Real>("permeability")),
    _viscosity(getParam<Real>("viscosity"))
{
  // check that viscosity value is not zero
  if (_viscosity == 0)
    paramError("viscosity", "The viscosity must be a non-zero real number.");
}

From now on, if a user tries to input viscosity = 0, the message printed to the terminal will be more explicit about what went wrong: The viscosity must be a non-zero real number. Do not modify any other parts of DarcyPressure.C. Lastly, be sure to recompile the application:

cd ~/projects/babbler
make -j4

Input File

The simple diffusion test is a great example of a regression test that is simple, cheap, and provides sufficient verification of the class being tested, i.e., Diffusion. Specifically, it creates very few objects besides a Diffusion one, the mesh is coarse enough to produce a number of degrees-of-freedom (DOFs) that can be solved in almost no time, but fine enough to result in a fairly complex system of equations, and the correct solution to the boundary value problem (BVP) (zero on the left and unity on the right) is obvious, because the geometry (a 1-by-1 square) is a direct mapping of it. Therefore, to make matters easy, the simple_diffusion.i file will be copied and then modified to use a DarcyPressure object.

Start by creating a directory to store test files related to the DarcyPressure class:

cd ~/projects/babbler
mkdir test/tests/kernels/darcy_pressure

In this folder, create a file named darcy_pressure_test.i and add the inputs given in Listing 1. Here, the value input for the permeability is that for the 1 mm steel sphere medium and the viscosity is left as the default value . The "exodus" parameter was set to true in the [Outputs] block so that an Exodiff object can reference the output file.

[Mesh]
  type = GeneratedMesh
  dim = 2
  nx = 10
  ny = 10
[]

[Variables]
  [pressure]
  []
[]

[Kernels]
  [diffusion]
    type = DarcyPressure
    variable = pressure
    permeability = 0.8451e-09
  []
[]

[BCs]
  [left]
    type = ADDirichletBC
    variable = pressure
    boundary = left
    value = 0
  []
  [right]
    type = ADDirichletBC
    variable = pressure
    boundary = right
    value = 1
  []
[]

[Executioner]
  type = Steady
  solve_type = PJFNK
  petsc_options_iname = '-pc_type -pc_hypre_type'
  petsc_options_value = 'hypre boomeramg'
[]

[Outputs]
  exodus = true
[]

Execute the input file in Listing 1, confirm that the solver completes, and that a file named darcy_pressure_test_out.e is generated:

cd ~/projects/babbler/test/tests/kernels/darcy_pressure
../../../../babbler-opt -i darcy_pressure_test.i

Next, create a file named zero_viscosity_error.i in the same folder and add the inputs given in Listing 2. Here, viscosity = 0 was input to deliberately incur the expected error. Also, since no actual finite element (FE) problem is going to be solved, the "solve" parameter was set to false in the [Problem] block, which will increase performance by not processing tasks that are unnecessary for this particular test.

[Mesh]
  type = GeneratedMesh
  dim = 1
[]

[Problem]
  solve = false
[]

[Variables]
  [u]
  []
[]

[Kernels]
  [zero_viscosity]
    type = DarcyPressure
    variable = u
    permeability = 0.8451e-09
    viscosity = 0 # The viscosity must be a non-zero number, so this input should invoke an error
  []
[]

[Executioner]
  type = Steady
[]

Execute the input file in Listing 2 and confirm that the application reports the error message that was passed to the paramError() method:

cd ~/projects/babbler/test/tests/kernels/darcy_pressure
../../../../babbler-opt -i zero_viscosity_error.i

Results

Visualize the results of the darcy_pressure_test.i input file with PEACOCK:

cd ~/projects/babbler/test/tests/kernels/darcy_pressure
peacock -r darcy_pressure_test_out.e

Confirm that the solution resembles that which is shown in Figure 1. Notice that the solution satisfies the BVP: and , . Also, notice that the solution directly maps to the spatial coordinates along the -direction as expected. This result is clearly indicated by the contour colors displayed by Figure 1. Thus, one may declare that the solution is good and, therefore, that the DarcyPressure class is functioning properly.

Figure 1: Rendering of the solution produced by the DarcyPressure test.

Test

Since the results of the darcy_pressure_test.i input file have been deemed good, the ExodusII output can now become a certified gold file:

cd ~/projects/babbler/test/tests/kernels/darcy_pressure
mkdir gold
mv darcy_pressure_test_out.e gold

Next, create a file named tests in test/tests/kernels/darcy_pressure and add the inputs given in Listing 3. The first test is defined in the [test] block, which creates an Exodiff object that will attempt to match the output from darcy_pressure_test.i to the gold file just created. The second test is defined in the [zero_viscosity_error] block, which creates a RunException object that attempts to match the error message incurred by zero_viscosity_error.i to the string specified for the expect_err parameter.

[Tests]
  [test]
    type = Exodiff
    input = darcy_pressure_test.i
    exodiff = darcy_pressure_test_out.e
  []
  [zero_viscosity_error]
    type = RunException
    input = zero_viscosity_error.i
    expect_err = "The viscosity must be a non-zero real number."
  []
[]

Finally, all tests, including the new ones, can be run by TestHarness:

cd ~/projects/babbler
./run_tests -j4

If the tests passed, the terminal output should look something like that shown below.

test:kernels/darcy_pressure.test .......................................................................... OK
test:kernels/simple_diffusion.test ........................................................................ OK
test:kernels/darcy_pressure.zero_viscosity_error .......................................................... OK
--------------------------------------------------------------------------------------------------------------
Ran 3 tests in 0.4 seconds. Average test time 0.1 seconds, maximum test time 0.1 seconds.
3 passed, 0 skipped, 0 pending, 0 failed

Commit

Update the DarcyPressure.C file and add all of the new test files to the git tracker:

cd ~/projects/babbler
git add src/kernels/DarcyPressure.C test/tests/kernels/darcy_pressure

Now, commit and push the changes to the remote repository:

git commit -m "implemented a zero-viscosity error and created tests for the Darcy pressure kernel"
git push