Stochastic Tools Requirement Traceability Matrix

Introduction

The Requirement Traceability Matrix (RTM) for Stochastic Tools captures all requirements and maps each to the associated design documentation and associated test case.

Dependencies

The Stochastic Tools application is developed using MOOSE and is based on various modules, as such the RTM for Stochastic Tools is dependent upon the following documents.

• Framework Requirements Traceability Matrix

Requirements

The following is a complete list for all the functional requirements including links to the design documents and test cases for Stochastic Tools.

• stochastic_tools: Distributions
• F3.1.1The system shall provide distribution function including
  1. uniform,
  2. Weibull (3 parameter),
  3. normal,
  4. truncated normal,
  5. Johnson Special Bounded (SB), and
  6. logistic distributions.

  Specification: distributions:distributions

  Design: UniformDistributionWeibullDistributionNormalDistributionTruncatedNormalDistributionJohnsonSBDistributionLogisticDistribution

  Issue(s): #12720

  202020202020
• F3.1.2The system shall provide distribution functions based on Boost library including
  1. Weibull (2 parameter),
  2. normal, and
  3. lognormal distributions.

  Specification: distributions:boost

  Design: BoostNormalDistributionBoostWeibullDistribution

  Issue(s): #12688

  333
• F3.1.3The system shall error if a distribution defined using Boost is used but the system has not been configured to utilize the Boost library:
  1. for a Weibull distribion,
  2. a normal distribution, and
  3. a lognormal distribution.

  Specification: distributions:no_boost

  Design: BoostWeibullDistributionBoostNormalDistributionBoostLognormalDistribution

  Issue(s): #12688

  171717
• F3.1.4The system shall provide a normal distribution with the ability to directly call methods with distribution inputs.

  Specification: distributions:normal_direct

  Design: NormalDistribution

  Issue(s): #12720

  20
• F3.1.5The system shall produce an error if a distribution is retrieved with the differing type than supplied.

  Specification: distributions:normal_direct_type_error

  Design: NormalDistribution

  Issue(s): #12720

  20

• stochastic_tools: Ics
• F3.2.1The system shall generate parallel agnostic random initial conditions using a distribution function.

  Specification: ics/random_ic_distribution_test:generate

  Design: RandomIC

  Issue(s): #5567#11901#9710

  20
• F3.2.2The system shall generate an error the random initial condition is used with both a distribution and min or max value defined.

  Specification: ics/random_ic_distribution_test:test_err_distribution_and_min_max

  Design: RandomIC

  Issue(s): #5567#11901#9710

  Prerequisite(s): F3.2.1

  20

• stochastic_tools: Multiapps
• F3.3.1The system shall be able to set command line parameter(s) for sub-application that executes completely from a sample distribution for
  1. for a single parameter,
  2. for a single parameter for a batch of sub-applications,
  3. for multiple parameters, and
  4. for multiple parameters for a batch of sub-applications.

  Specification: multiapps/batch_commandline_control:master

  Design: SamplerFullSolveMultiAppMultiAppCommandLineControl

  Issue(s): #13320

  20202020
• F3.3.2The system shall error when the supplied sampler object operates in a mode that does not allow for command line arguments to be modified.

  Specification: multiapps/batch_commandline_control:wrong_batch_mode

  Design: SamplerFullSolveMultiAppMultiAppCommandLineControl

  Issue(s): #13320

  Prerequisite(s):

  20
• F3.3.3The system shall error when the supplied sampler does not use the correct execution flags.

  Specification: multiapps/batch_commandline_control:wrong_execute_on

  Design: SamplerFullSolveMultiAppMultiAppCommandLineControl

  Issue(s): #13320

  Prerequisite(s):

  20
• F3.3.4The system shall support pulling postprocessor data from a sub-application for each row of sampled data.

  Specification: multiapps/batch_full_solve_multiapp:normal

  Design: SamplerPostprocessorTransferSamplerFullSolveMultiApp

  Issue(s): #8863

  20
• F3.3.5The system shall support running sub-applications in a batches on a
  1. on a single processor and
  2. on a multiple processors.

  Specification: multiapps/batch_full_solve_multiapp:batch

  Design: SamplerPostprocessorTransferSamplerFullSolveMultiApp

  Issue(s): #8863

  2020
• F3.3.6The stochastic tools module shall support pulling postprocessor data from a single sub-application running a batch of sampled data
  1. on a single processor,
  2. on multiple processors, and
  3. on multiple processors using in-memory backup.

  Specification: multiapps/batch_full_solve_multiapp:batch_csv

  Design: SamplerPostprocessorTransferSamplerFullSolveMultiApp

  Issue(s): #8863

  202020
• F3.3.7The system shall support running sub-applications with input parameters varying at each time step
  1. with individual sub-applications,
  2. with sub-applications batches using in-memory restore functionality.

  Specification: multiapps/batch_sampler_transient_multiapp:transient

  Design: SamplerTransientMultiApp

  Issue(s): #13320

  2020
• F3.3.8The SamplerTransientMultiApp object shall error if the 'batch-reset' mode is supplied.

  Specification: multiapps/batch_sampler_transient_multiapp:mode_error

  Design: SamplerTransientMultiApp

  Issue(s): #13320

  20
• F3.3.9The system shall be able to set a sub-application command line parameters from a sample distribution
  1. for a single parameter and
  2. for a multiple parameters.

  Specification: multiapps/commandline_control:master

  Design: MultiApp System

  Issue(s): #12576

  2020
• F3.3.10The system shall error when the number of samplers differs from the number of sub applications.

  Specification: multiapps/commandline_control:wrong_size

  Design: MultiAppCommandLineControl

  Issue(s): #12576

  20
• F3.3.11The system shall error when the number of samples differs from the number of command line parameters.

  Specification: multiapps/commandline_control:wrong_num_params

  Design: MultiAppCommandLineControl

  Issue(s): #12576

  20
• F3.3.12The system shall provide the ability to create a full-solve type sub-application from sampled data from distributions.

  Specification: multiapps/sampler_full_solve_multiapp:full_solve

  Design: SamplerFullSolveMultiApp

  Issue(s): #9923

  20
• F3.3.13The system shall provide the ability to create a transient sub-application from the sample data generated from distributions.

  Specification: multiapps/sampler_transient_multiapp:transient

  Design: SamplerTransientMultiApp

  Issue(s): #9923

  20

• stochastic_tools: Samplers
• F3.4.1The system shall support the creation of data sampled from distribution during the initial setup of a simulation.

  Specification: samplers/execute_on:initial

  Design: Sampler

  Issue(s): #8065

  20
• F3.4.2The system shall include a Monte Carlo method for sampling distribution data including
  1. a uniform distribution distributed across processors,
  2. a uniform distribution replicated across processors,
  3. a Wiebull distribution distributed across processors, and
  4. a Weibull distribution replicated across processors.

  Specification: samplers/monte_carlo:monte_carlo

  Design: MonteCarloSampler

  Issue(s): #8065

  202033
• F3.4.3The system shall include a SOBOL method for sampling distribution data.

  Specification: samplers/sobol:sobol

  Design: SobolSampler

  Issue(s): #8065

  20

• stochastic_tools: Transfers
• F3.5.1The system shall include the ability to modify parameters for sub-applications using values from a distribution
  1. on a single processor and
  2. on multiple processors.

  Specification: transfers/batch_sampler_transfer:normal

  Design: SamplerParameterTransfer

  Issue(s): #8863

  2020
• F3.5.2The system shall include the ability to modify parameters for sub-applications executed in batches using values from a distribution
  1. on a single processor,
  2. on multiple processors, and
  3. on multiple processors using in-memory sub-application restore.

  Specification: transfers/batch_sampler_transfer:batch

  Design: SamplerParameterTransfer

  Issue(s): #8863

  202020
• F3.5.3The 'StochasticToolsTransfer object shall error if the 'execute_on' parameter is defined when the corresponding MultiApp object is running in batch mode.

  Specification: transfers/batch_sampler_transfer:StochasticToolsTransfer_execute_on_error

  Design: SamplerParameterTransfer

  Issue(s): #8863

  20
• F3.5.4The 'StochasticToolsTransfer' object shall error if the 'execute_on' parameter does not match the corresponding MultiApp object is running in normal mode.

  Specification: transfers/batch_sampler_transfer:StochasticToolsTransfer_execute_on_check

  Design: SamplerParameterTransfer

  Issue(s): #8863

  20
• F3.5.5The system shall produce an error if sub-application type is not correct for performing a transfer of stochastic data.

  Specification: transfers/errors:multiapp_type

  Design: SamplerParameterTransfer

  Issue(s): #11363

  20
• F3.5.6The system shall produce an error if the sampler sub-application does not contain control object with the correct name.

  Specification: transfers/errors:control_missing

  Design: SamplerTransientMultiApp

  Issue(s): #11363

  20
• F3.5.7The system produce an error if the sampler sub-application does not have a correct object for receiving stochastic data.

  Specification: transfers/errors:wrong_control

  Design: SamplerTransientMultiApp

  Issue(s): #11363

  20
• F3.5.8The system shall produce an error if supplied vector of real values is not sized correctly.

  Specification: transfers/errors:num_pararameters_wrong

  Design: SamplerParameterTransfer

  Issue(s): #11363

  20
• F3.5.9The system shall support the creation of a sub-application for each row of the stochastic data.

  Specification: transfers/monte_carlo:monte_carlo

  Design: MonteCarloSamplerSamplerParameterTransfer

  Issue(s): #8863

  20
• F3.5.10MOOSE shall produce an error if neither a 'SamplerTransientMultiApp' nor SamplerFullSolveMultiApp is provided in SamplerPostprocessorTransfer.

  Specification: transfers/sampler_postprocessor/errors:wrong_multi_app

  Design: StochasticResults

  Issue(s): #9419

  20
• F3.5.11MOOSE shall produce an error if the 'result' object in 'SamplerPostprocessorTransfer' is not a 'StochasticResults object'.

  Specification: transfers/sampler_postprocessor/errors:require_stochastic_results

  Design: StochasticResults

  Issue(s): #9419

  20
• F3.5.12The system shall support the ability to transfer a single value from each sub-application for a set of stochastic data.

  Specification: transfers/sampler_postprocessor:sobol_from_multiapp

  Design: SamplerPostprocessorTransfer

  Issue(s): #8065

  20
• F3.5.13MOOSE shall produce an error if neither a 'SamplerTransientMultiApp' nor SamplerFullSolveMultiApp is provided in SamplerParameterTransfer.

  Specification: transfers/sampler_transfer/errors:multiapp_type

  Design: SamplerParameterTransfer

  Issue(s): #11363

  20
• F3.5.14MOOSE shall produce an error if the sampler sub-application does not contain a Control object with the name 'stochastic'.

  Specification: transfers/sampler_transfer/errors:control_missing

  Design: SamplerTransientMultiApp

  Issue(s): #11363

  20
• F3.5.15MOOSE shall produce an error if the sampler sub-application does not have a correct Control object with 'to_control' parameter being 'SamplerReceiver' type.

  Specification: transfers/sampler_transfer/errors:wrong_control

  Design: SamplerTransientMultiApp

  Issue(s): #11363

  20
• F3.5.16MOOSE shall produce an error if supplied vector of real values is not sized correctly.

  Specification: transfers/sampler_transfer/errors:num_pararameters_wrong

  Design: SamplerParameterTransfer

  Issue(s): #11363

  20
• F3.5.17MOOSE shall produce an error if sampling method differs between the sub-application and the associated sub-application data transfer.

  Specification: transfers/sampler_transfer/errors:sampler_mismatch

  Design: SamplerParameterTransfer

  Issue(s): #11363

  20
• F3.5.18The system shall be capable of transferring scalar data to sub-applications for each row of the stochastic data
  1. using a Monte Carlo and
  2. Sobol sampling scheme.

  Specification: transfers/sampler_transfer:transfer

  Design: SamplerParameterTransfer

  Issue(s): #8065

  2020
• F3.5.19The system shall be capable of transferring vector data to sub-applications for each row of the stochastic data.

  Specification: transfers/sampler_transfer_vector:monte_carlo

  Design: SamplerParameterTransfer

  Issue(s): #8065

  20
• F3.5.20The system shall error if the transferred vector to a sub-application
  1. is not sized correctly for stochastic data,
  2. is not sized uniformily across sub-applications,
  3. if the vector parameter does not exist, and
  4. if the sub-application does not consume all of the supplied data.

  Specification: transfers/sampler_transfer_vector:errors

  Design: SamplerParameterTransfer

  Issue(s): #8065

  20202020
• F3.5.21The system shall support the creation of a sub-application for each row sampled data generated from a Sobol scheme.

  Specification: transfers/sobol:sobol

  Design: SobolSamplerSamplerParameterTransfer

  Issue(s): #8863

  20

• stochastic_tools: Vectorpostprocessors
• F3.6.1The system shall support the collection of stochastic data from multiple sub-applications.

  Specification: vectorpostprocessors/multiple_stochastic_results:multiple

  Design: StochasticResults

  Issue(s): #14414

  20
• F3.6.2The system shall error if stochastic data is distributed across processors and output of the data is enabled.

  Specification: vectorpostprocessors/stochastic_results:warning

  Design: StochasticResults

  Issue(s): #14410

  20
• F3.6.3The system shall error when stochastic data is set to output on a specific processor with
  1. the supplied operation mode is replicated and
  2. the provided rank number is too large for the number of processors being executed.

  Specification: vectorpostprocessors/stochastic_results:errors

  Design: StochasticResults

  Issue(s): #14410

  2020
• F3.6.4The system shall support the collection of stochastic data that is
  1. replicated on all processors and
  2. distributed across one,
  3. two, and
  4. three processors.

  Specification: vectorpostprocessors/stochastic_results:parallel_type

  Design: StochasticResults

  Issue(s): #14410

  20202020
• F3.6.5The system shall support the collection of stochastic data that
  1. can be appended into a single data set or
  2. or contain a single file per timestep.

  Specification: vectorpostprocessors/stochastic_results_complete_history:data

  Design: StochasticResults

  Issue(s): #14412

  2020

[distributions]
  requirement = 'The system shall provide distribution function including'
  design = 'UniformDistribution.md WeibullDistribution.md NormalDistribution.md TruncatedNormalDistribution.md JohnsonSBDistribution.md LogisticDistribution.md'
  issues = '#12720'

  [uniform]
    type = CSVDiff
    input = uniform.i
    csvdiff = 'uniform_out.csv'
    allow_test_objects = true

    detail = 'uniform,'
  []
  [weibull]
    type = CSVDiff
    input = weibull.i
    csvdiff = 'weibull_out.csv'
    allow_test_objects = true

    detail = 'Weibull (3 parameter),'
  []

  [normal]
    type = CSVDiff
    input = normal.i
    csvdiff = 'normal_out.csv'
    allow_test_objects = true

    detail = 'normal,'
  []
  [truncated_normal]
    type = CSVDiff
    input = truncated_normal.i
    csvdiff = 'truncated_normal_out.csv'
    allow_test_objects = true

    detail = 'truncated normal,'
  []

  [johnsonsb]
    type = CSVDiff
    input = johnsonsb.i
    csvdiff = 'johnsonsb_out.csv'
    allow_test_objects = true

    detail = 'Johnson Special Bounded (SB), and'
  []
  [logistic]
    type = CSVDiff
    input = logistic.i
    csvdiff = 'logistic_out.csv'
    allow_test_objects = true

    detail = 'logistic distributions.'
  []
[]

[boost]
  requirement = 'The system shall provide distribution functions based on Boost library including'
  design = 'BoostNormalDistribution.md BoostWeibullDistribution.md'

  [weibull]
    type = CSVDiff
    input = boost_weibull.i
    csvdiff = 'weibull_out.csv'
    boost = true
    allow_test_objects = true
    rel_err = 1e-8
    abs_zero = 1e-8
    prereq = 'distributions/weibull'

    detail = 'Weibull (2 parameter),'
  []
  [normal]
    type = CSVDiff
    input = boost_normal.i
    csvdiff = 'normal_out.csv'
    boost = true
    allow_test_objects = true
    prereq = 'distributions/normal'
    rel_err = 1e-6
    abs_zero = 1e-7

    detail = 'normal, and'
  []

  [lognormal]
    type = CSVDiff
    input = boost_lognormal.i
    csvdiff = 'boost_lognormal_out.csv'
    boost = true
    allow_test_objects = true

    detail = 'lognormal distributions.'
  []

[]

[no_boost]
  requirement = 'The system shall error if a distribution defined using Boost is used but the system has not been configured to utilize the Boost library:'
  design = 'BoostWeibullDistribution.md BoostNormalDistribution.md BoostLognormalDistribution.md'

  [weibull]
    type = RunException
    input = boost_weibull.i
    expect_err = "The BoostWeibullDistribution distribution named 'weibull' requires that libMesh"
    boost = false
    allow_test_objects = true

    detail = 'for a Weibull distribion,'
  []

  [normal]
    type = RunException
    input = boost_normal.i
    expect_err = "The BoostNormalDistribution distribution named 'normal_test' requires that libMesh"
    boost = false
    allow_test_objects = true

    detail = 'a normal distribution, and'
  []

  [lognormal]
    type = RunException
    input = boost_lognormal.i
    expect_err = "The BoostLognormalDistribution distribution named 'lognormal_test' requires that libMesh"
    boost = false
    allow_test_objects = true

    detail = 'a lognormal distribution.'
  []
[]

[normal_direct]
  type = CSVDiff
  input = normal_direct.i
  csvdiff = 'normal_direct_out.csv'
  allow_test_objects = true
  requirement = 'The system shall provide a normal distribution with the ability to directly call methods with distribution inputs.'
  design = 'NormalDistribution.md'
  issues = '#12720'
[]

[normal_direct_type_error]
  type = RunException
  input = normal_direct_type_error.i
  allow_test_objects = true
  expect_err = "The 'cdf' object failed to retrieve 'this_is_the_wrong_type' distribution with the desired type."
  requirement = 'The system shall produce an error if a distribution is retrieved with the differing type than supplied.'
  design = 'NormalDistribution.md'
  issues = '#12720'
[]

[generate]
  type = 'CSVDiff'
  input = 'random_ic_distribution_test.i'
  csvdiff = 'random_ic_distribution_test_initial_histo_0000.csv'
  mesh_mode = 'replicated' # sensitive to difference in partitioning

  requirement = 'The system shall generate parallel agnostic random initial conditions using a distribution function.'
[]

[test_err_distribution_and_min_max]
  type = 'RunException'
  input = 'random_ic_distribution_test.i'
  cli_args = 'ICs/u_aux/min=0 ICs/u_aux/max=1'
  prereq = 'generate'
  expect_err = "Cannot use together with 'min' or 'max' parameter"

  requirement = 'The system shall generate an error the random initial condition is used with both a distribution and min or max value defined.'
[]

[master]
  requirement = "The system shall be able to set command line parameter(s) for sub-application that executes completely from a sample distribution for"

  [single_normal]
    type = CSVDiff
    input = master_single.i
    cli_args = MultiApps/sub/mode=normal
    csvdiff = 'master_single_out_storage_0002.csv'

    detail = 'for a single parameter, '
  []

  [single_batch]
    type = CSVDiff
    input = master_single.i
    csvdiff = 'master_single_out_storage_0002.csv'
    prereq = master/single_normal

    detail = 'for a single parameter for a batch of sub-applications,'
  []

  [multiple_normal]
    type = CSVDiff
    input = master_multiple.i
    cli_args = MultiApps/sub/mode=normal
    csvdiff = 'master_multiple_out_storage_0002.csv'
    prereq = master/single_batch

    detail = 'for multiple parameters, and'
  []

  [multiple_batch]
    type = CSVDiff
    input = master_multiple.i
    csvdiff = 'master_multiple_out_storage_0002.csv'
    prereq = master/multiple_normal

    detail = 'for multiple parameters for a batch of sub-applications.'
  []

[]

[wrong_batch_mode]
  type = RunException
  input = master_single.i
  cli_args = MultiApps/sub/mode=batch-restore
  expect_err = "The MultiApp object, 'sub', supplied to the 'cmdline' object is setup to run in 'batch-restore' mode, when using this mode command line arguments cannot be modified; batch-reset mode should be used instead."
  requirement = "The system shall error when the supplied sampler object operates in a mode that does not allow for command line arguments to be modified."
  prereq = master/multiple_batch
[]

[wrong_execute_on]
  type = RunException
  input = master_single.i
  cli_args = Samplers/sample/execute_on=INITIAL
  expect_err = "The sampler object, 'sample', is being used by the 'cmdline' object, thus the 'execute_on' of the sampler must include 'PRE_MULTIAPP_SETUP'."
  requirement = "The system shall error when the supplied sampler does not use the correct execution flags."
  prereq = master/multiple_batch
[]

[normal]
  type = CSVDiff
  input = master_full_solve.i
  csvdiff = master_full_solve_out_storage_0002.csv
  cli_args = MultiApps/runner/mode=normal
  requirement = "The system shall support pulling postprocessor data from a sub-application for each row of sampled data."
[]

[batch]
  requirement = "The system shall support running sub-applications in a batches on a"

  [exodus_1]
    type = CheckFiles
    input = master_full_solve.i
    check_files = master_full_solve_out_runner0.e
    check_not_exists = master_full_solve_out_runner1.e
    max_parallel = 1
    prereq = normal

    detail = 'on a single processor and'
  []
  [exodus_2]
    type = CheckFiles
    input = master_full_solve.i
    check_files = 'master_full_solve_out_runner0.e master_full_solve_out_runner1.e'
    check_not_exists = master_full_solve_out_runner2.e
    max_parallel = 2
    min_parallel = 2
    prereq = batch/exodus_1

    detail = 'on a multiple processors.'
  []
[]

[batch_csv]
  requirement = "The stochastic tools module shall support pulling postprocessor data from a single sub-application running a batch of sampled data"

  [n1]
    type = CSVDiff
    input = master_full_solve.i
    csvdiff = master_full_solve_out_storage_0002.csv
    prereq = batch/exodus_2

    detail = 'on a single processor,'
  []
  [n2]
    type = CSVDiff
    input = master_full_solve.i
    csvdiff = master_full_solve_out_storage_0002.csv
    max_parallel = 2
    min_parallel = 2
    prereq = batch_csv/n1

    detail = 'on multiple processors, and'
  []
  [n2_restore]
    type = CSVDiff
    input = master_full_solve.i
    csvdiff = master_full_solve_out_storage_0002.csv
    max_parallel = 2
    min_parallel = 2
    prereq = batch_csv/n2
    cli_args = MultiApps/runner/batch-restore

    detail = 'on multiple processors using in-memory backup.'
  []
[]

[transient]
  requirement = "The system shall support running sub-applications with input parameters varying at each time step"

  [normal]
    type = CSVDiff
    input = master_transient.i
    cli_args = MultiApps/runner/mode=normal
    csvdiff = 'master_transient_out_storage_0000.csv master_transient_out_storage_0001.csv master_transient_out_storage_0002.csv master_transient_out_storage_0003.csv'

    detail = 'with individual sub-applications,'
  []
  [batch]
    type = CSVDiff
    input = master_transient.i
    csvdiff = 'master_transient_out_storage_0000.csv master_transient_out_storage_0001.csv master_transient_out_storage_0002.csv master_transient_out_storage_0003.csv'
    prereq = transient/normal

    detail = 'with sub-applications batches using in-memory restore functionality.'
  []
[]

[mode_error]
  type = RunException
  input = master_transient.i
  cli_args = MultiApps/runner/mode=batch-reset
  expect_err = "The supplied mode, 'batch-reset', currently"
  requirement = "The SamplerTransientMultiApp object shall error if the 'batch-reset' mode is supplied."
[]

[master]
  requirement = "The system shall be able to set a sub-application command line parameters from a sample distribution"

  [single]
    type = CSVDiff
    input = master_single.i
    csvdiff = 'master_single_out_sub0.csv master_single_out_sub1.csv master_single_out_sub2.csv'

    detail = 'for a single parameter and'
  []

  [multiple]
    type = CSVDiff
    input = master_multiple.i
    csvdiff = 'master_multiple_out_sub0.csv master_multiple_out_sub1.csv master_multiple_out_sub2.csv'

    detail = 'for a multiple parameters.'
  []
[]

[wrong_size]
  type = RunException
  input = master_wrong_size.i
  expect_err = "The number of sub apps \(2\) created by MultiApp object 'sub' must be equal to the number for rows \(10\) for the 'sample' Sampler object"
  design = 'source/controls/MultiAppCommandLineControl.md'

  requirement = "The system shall error when the number of samplers differs from the number of sub applications."
[]

[wrong_num_params]
  type = RunException
  input = master_wrong_num_params.i
  expect_err = "The number of columns \(2\) must match the number of parameters \(3\)"
  design = 'source/controls/MultiAppCommandLineControl.md'

  requirement = "The system shall error when the number of samples differs from the number of command line parameters."
[]

[full_solve]
  type = CheckFiles
  input = master_full_solve.i
  check_files = 'master_full_solve_out_runner0.e master_full_solve_out_runner1.e master_full_solve_out_runner2.e master_full_solve_out_runner3.e master_full_solve_out_runner4.e'

  requirement = 'The system shall provide the ability to create a full-solve type sub-application from sampled data from distributions.'
[]

[transient]
  type = CheckFiles
  input = master_transient.i
  check_files = 'master_transient_out_runner0.e master_transient_out_runner1.e master_transient_out_runner2.e master_transient_out_runner3.e master_transient_out_runner4.e'

  requirement = 'The system shall provide the ability to create a transient sub-application from the sample data generated from distributions.'
[]

[initial]
  type = CSVDiff
  input = initial.i
  csvdiff = 'initial_out_data_0000.csv initial_out_data_0001.csv initial_out_data_0002.csv initial_out_data_0003.csv'

  requirement = 'The system shall support the creation of data sampled from distribution during the initial setup of a simulation.'
[]

[monte_carlo]
  requirement = 'The system shall include a Monte Carlo method for sampling distribution data including'

  [uniform]
    type = 'CSVDiff'
    input = 'monte_carlo_uniform.i'
    csvdiff = 'monte_carlo_uniform_out_data_0000.csv monte_carlo_uniform_out_data_0001.csv'

    detail = 'a uniform distribution distributed across processors,'
  []

  [uniform_global]
    type = 'CSVDiff'
    input = 'monte_carlo_uniform.i'
    csvdiff = 'monte_carlo_uniform_out_data_0000.csv monte_carlo_uniform_out_data_0001.csv'
    prereq = monte_carlo/uniform
    cli_args = VectorPostprocessors/data/sampler_method=get_global_samples

    detail = 'a uniform distribution replicated across processors,'
  []

  [weibull]
    type = 'CSVDiff'
    input = 'monte_carlo_weibull.i'
    csvdiff = 'monte_carlo_weibull_out_data_0000.csv monte_carlo_weibull_out_data_0001.csv'
    boost = true

    detail = 'a Wiebull distribution distributed across processors, and'
  []
  [weibull_global]
    type = 'CSVDiff'
    input = 'monte_carlo_weibull.i'
    csvdiff = 'monte_carlo_weibull_out_data_0000.csv monte_carlo_weibull_out_data_0001.csv'
    boost = true
    prereq = monte_carlo/weibull
    cli_args = VectorPostprocessors/data/sampler_method=get_global_samples

    detail = 'a Weibull distribution replicated across processors.'
  []
[]

[sobol]
  type = CSVDiff
  input = sobol.i
  csvdiff = 'sobol_out_data_0000.csv'

  requirement = 'The system shall include a SOBOL method for sampling distribution data.'
[]

[normal]
  requirement = "The system shall include the ability to modify parameters for sub-applications using values from a distribution"

  [n1]
    type = CSVDiff
    input = master.i
    csvdiff = 'master_out_storage_0001.csv master_out_data_0001.csv'
    cli_args = 'MultiApps/runner/mode=normal'

    detail = "on a single processor and"
  []
  [n2]
    type = CSVDiff
    input = master.i
    csvdiff = 'master_out_storage_0001.csv master_out_data_0001.csv'
    cli_args = 'MultiApps/runner/mode=normal'
    prereq = normal/n1
    min_parallel = 2
    max_parallel = 2

    detail = "on multiple processors."
  []
[]

[batch]
  requirement = "The system shall include the ability to modify parameters for sub-applications executed in batches using values from a distribution"

  [n1]
    type = CSVDiff
    input = master.i
    csvdiff = 'master_out_storage_0001.csv master_out_data_0001.csv'
    max_parallel = 1
    prereq = normal/n2

    detail = "on a single processor,"
  []
  [n2]
    type = CSVDiff
    input = master.i
    csvdiff = 'master_out_storage_0001.csv master_out_data_0001.csv'
    min_parallel = 2
    max_parallel = 2
    prereq = batch/n1

    detail = "on multiple processors, and"
  []
  [n2_restore]
    type = CSVDiff
    input = master.i
    csvdiff = 'master_out_storage_0001.csv master_out_data_0001.csv'
    min_parallel = 2
    max_parallel = 2
    prereq = batch/n2
    cli_args = MultiApps/runner/mode=batch-restore

    detail = "on multiple processors using in-memory sub-application restore."
  []
[]

[StochasticToolsTransfer_execute_on_error]
  type = RunException
  input = master.i
  cli_args = Transfers/data/execute_on=INITIAL
  expect_err = "The 'execute_on' parameter for the 'data' transfer was set"
  requirement = "The 'StochasticToolsTransfer object shall error if the 'execute_on' parameter is defined when the corresponding MultiApp object is running in batch mode."
[]

[StochasticToolsTransfer_execute_on_check]
  type = RunException
  input = master.i
  cli_args = "Transfers/data/execute_on=INITIAL MultiApps/runner/mode=normal"
  expect_err = "MultiAppTransfer execute_on flags do not match associated Multiapp execute_on flags"
  requirement = "The 'StochasticToolsTransfer' object shall error if the 'execute_on' parameter does not match the corresponding MultiApp object is running in normal mode."
[]

[multiapp_type]
  type = RunException
  input = master_multiapp_type_error.i
  expect_err = "The 'multi_app' parameter must provide either a 'SamplerTransientMultiApp' or 'SamplerFullSolveMultiApp' object."
  design = 'SamplerParameterTransfer.md'
  requirement = "The system shall produce an error if sub-application type is not correct for performing a transfer of stochastic data."
[]

[control_missing]
  type = RunException
  input = master_missing_control.i
  design = 'SamplerTransientMultiApp.md'
  expect_err = "The sub-application \(sub\) does not contain a Control object with the name 'stochastic'."
  requirement = "The system shall produce an error if the sampler sub-application does not contain control object with the correct name."
[]

[wrong_control]
  type = RunException
  input = master_wrong_control.i
  expect_err = "The sub-application \(sub\) Control object for the 'to_control' parameter must be of type 'SamplerReceiver'."
  design = 'SamplerTransientMultiApp.md'
  requirement = "The system produce an error if the sampler sub-application does not have a correct object for receiving stochastic data."
[]

[num_pararameters_wrong]
  type = RunException
  input = master_num_parameters_wrong.i
  expect_err = "The supplied vector of Real values is not sized correctly, the Real parameter 'BCs/right/value requires a value but no more values are available in the supplied values which have a size of 2."
  design = 'SamplerParameterTransfer.md'
  requirement = "The system shall produce an error if supplied vector of real values is not sized correctly."
[]

[monte_carlo]
  type = CSVDiff
  input = monte_carlo.i
  csvdiff = 'monte_carlo_out_sub0.csv monte_carlo_out_sub1.csv monte_carlo_out_sub2.csv monte_carlo_out_sub3.csv monte_carlo_out_sub4.csv'
  requirement = "The system shall support the creation of a sub-application for each row of the stochastic data."
[]

[wrong_multi_app]
  type = RunException
  input = wrong_multi_app.i
  expect_err = "The 'multi_app' parameter must provide either a 'SamplerTransientMultiApp' or 'SamplerFullSolveMultiApp' object."
  requirement = "MOOSE shall produce an error if neither a 'SamplerTransientMultiApp' nor `SamplerFullSolveMultiApp` is provided in SamplerPostprocessorTransfer."
[]

[require_stochastic_results]
  type = RunException
  input = require_stochastic_results.i
  expect_err = "The 'results' object must be a 'StochasticResults' object."
  requirement = "MOOSE shall produce an error if the 'result' object in 'SamplerPostprocessorTransfer' is not a 'StochasticResults object'."
[]

[sobol_from_multiapp]
  type = CSVDiff
  input = master.i
  csvdiff = 'master_out_storage_0001.csv master_out_storage_0002.csv master_out_storage_0003.csv master_out_storage_0004.csv master_out_storage_0005.csv'

  requirement = "The system shall support the ability to transfer a single value from each sub-application for a set of stochastic data."
[]

[multiapp_type]
  type = RunException
  input = master_multiapp_type_error.i
  expect_err = "The 'multi_app' parameter must provide either a 'SamplerTransientMultiApp' or 'SamplerFullSolveMultiApp' object."
  design = 'SamplerParameterTransfer.md'
  requirement = "MOOSE shall produce an error if neither a 'SamplerTransientMultiApp' nor `SamplerFullSolveMultiApp` is provided in SamplerParameterTransfer."
[]

[control_missing]
  type = RunException
  input = master_missing_control.i
  design = 'SamplerTransientMultiApp.md'
  expect_err = "The sub-application \(sub\) does not contain a Control object with the name 'stochastic'."
  requirement = "MOOSE shall produce an error if the sampler sub-application does not contain a Control object with the name 'stochastic'."
[]

[wrong_control]
  type = RunException
  input = master_wrong_control.i
  expect_err = "The sub-application \(sub\) Control object for the 'to_control' parameter must be of type 'SamplerReceiver'."
  design = 'SamplerTransientMultiApp.md'
  requirement = "MOOSE shall produce an error if the sampler sub-application does not have a correct Control object with 'to_control' parameter being 'SamplerReceiver' type."
[]

[num_pararameters_wrong]
  type = RunException
  input = master_num_parameters_wrong.i
  expect_err = "The supplied vector of Real values is not sized correctly, the Real parameter 'BCs/right/value requires a value but no more values are available in the supplied values which have a size of 2."
  design = 'SamplerParameterTransfer.md'
  requirement = "MOOSE shall produce an error if supplied vector of real values is not sized correctly."
[]

[sampler_mismatch]
  type = RunException
  input = master_transfer_wrong_sampler.i
  expect_err = "The supplied 'multi_app' must have the same Sampler object as this Transfer."
  design = 'SamplerParameterTransfer.md'
  requirement = "MOOSE shall produce an error if sampling method differs between the sub-application and the associated sub-application data transfer."
[]

[transfer]
  requirement = "The system shall be capable of transferring scalar data to sub-applications for each row of the stochastic data"

  [monte_carlo]
    type = CSVDiff
    input = monte_carlo.i
    csvdiff = 'monte_carlo_out_sub0.csv monte_carlo_out_sub1.csv monte_carlo_out_sub2.csv monte_carlo_out_sub3.csv monte_carlo_out_sub4.csv'

    detail = 'using a Monte Carlo and'
  []
  [sobol]
    type = CSVDiff
    input = sobol.i
    csvdiff = 'sobol_out_sub00.csv sobol_out_sub01.csv sobol_out_sub02.csv sobol_out_sub03.csv sobol_out_sub04.csv sobol_out_sub05.csv sobol_out_sub06.csv sobol_out_sub07.csv sobol_out_sub08.csv sobol_out_sub09.csv sobol_out_sub10.csv sobol_out_sub11.csv'

    detail = 'Sobol sampling scheme.'
  []
[]

[monte_carlo]
  type = CSVDiff
  input = master.i
  csvdiff = 'master_out_sub0.csv master_out_sub1.csv master_out_sub2.csv master_out_sub3.csv master_out_sub4.csv'

  requirement = "The system shall be capable of transferring vector data to sub-applications for each row of the stochastic data."

[]

[errors]

  requirement = "The system shall error if the transferred vector to a sub-application"

  [not_enough_data]
    type = RunException
    input = master_not_enough_data.i
    expect_err = "The supplied vector of Real values is not sized correctly, the std::vector<Real> parameter 'Materials/mat/prop_values requires 2 values but only 1 are available in the supplied vector."

    detail = "is not sized correctly for stochastic data,"
  []
  [size_mismatch]
    type = RunException
    input = master_size_mismatch.i
    expect_err = "The std::vector<Real> parameters being controlled must all be the same size:"

    detail = "is not sized uniformily across sub-applications,"
  []
  [invalid_name]
    type = RunException
    input = master.i
    cli_args = "Transfers/sub/parameters=foo :Outputs/csv=false"
    expect_err = "The desired parameter 'foo' was not located for the 'stochastic' object, it either does not exist or has not been declared as controllable."

    detail = "if the vector parameter does not exist, and"
  []
  [extra_data]
    type = RunException
    input = master.i
    cli_args = "Transfers/sub/parameters='BCs/left/value' :Outputs/csv=false"
    expect_err = "The number of values supplied \(4\) does not match the number of values consumed by setting parameter values \(1\)."

    detail = "if the sub-application does not consume all of the supplied data."
  []
[]

[sobol]
  type = CSVDiff
  input = sobol.i
  csvdiff = 'sobol_out_sub00.csv sobol_out_sub01.csv sobol_out_sub02.csv sobol_out_sub03.csv sobol_out_sub04.csv sobol_out_sub05.csv sobol_out_sub06.csv sobol_out_sub07.csv sobol_out_sub08.csv sobol_out_sub09.csv sobol_out_sub10.csv sobol_out_sub11.csv'
  requirement = "The system shall support the creation of a sub-application for each row sampled data generated from a Sobol scheme."
[]

[multiple]
  type = CSVDiff
  input = master.i
  csvdiff = master_out_storage_0002.csv

  requirement = "The system shall support the collection of stochastic data from multiple sub-applications."
[]

[warning]
  type = RunApp
  input = master.i
  expect_out = "The parallel_type was set to DISTRIBUTED and output is enabled"
  min_parallel = 2 # warning is only triggered with MPI
  allow_warnings = true # the warning is being tested

  requirement = "The system shall error if stochastic data is distributed across processors and output of the data is enabled."
[]

[errors]
  requirement = "The system shall error when stochastic data is set to output on a specific processor with"
  [wrong_parallel_type]
    type = RunException
    input = master.i
    cli_args = 'VectorPostprocessors/storage/output_distributed_rank=1 VectorPostprocessors/storage/parallel_type=REPLICATED'
    expect_err = "The output rank cannot be used with"

    detail = "the supplied operation mode is replicated and"
  []
  [bad_output_rank]
    type = RunException
    input = master.i
    expect_err = "The supplied value is greater than"
    cli_args = 'VectorPostprocessors/storage/output_distributed_rank=2'
    max_parallel = 2

    detail = "the provided rank number is too large for the number of processors being executed."
  []
[]

[parallel_type]
  requirement = "The system shall support the collection of stochastic data that is"
  [replicated]
    type = CSVDiff
    input = master.i
    cli_args = 'VectorPostprocessors/storage/parallel_type=REPLICATED Outputs/file_base=replicated_out'
    csvdiff = replicated_out_storage_0002.csv

    detail = "replicated on all processors and"
  []

  [distributed_one]
    type = CSVDiff
    input = master.i
    min_parallel = 3
    max_parallel = 3
    cli_args = 'Outputs/file_base=distributed_one_out VectorPostprocessors/storage/output_distributed_rank=0'
    csvdiff = distributed_one_out_storage_0002.csv

    detail = "distributed across one,"
  []

  [distributed_two]
    type = CSVDiff
    input = master.i
    min_parallel = 3
    max_parallel = 3
    cli_args = 'Outputs/file_base=distributed_two_out VectorPostprocessors/storage/output_distributed_rank=1'
    csvdiff = distributed_two_out_storage_0002.csv

    detail = "two, and"
  []

  [distributed_three]
    type = CSVDiff
    input = master.i
    min_parallel = 3
    max_parallel = 3
    cli_args = 'Outputs/file_base=distributed_three_out VectorPostprocessors/storage/output_distributed_rank=2'
    csvdiff = distributed_three_out_storage_0002.csv

    detail = "three processors."
  []
[]

[data]
  requirement = "The system shall support the collection of stochastic data that"

  [complete]
    type = CSVDiff
    input = master.i
    csvdiff = master_out_storage.csv

    detail = "can be appended into a single data set or"
  []

  [time]
    type = CSVDiff
    input = master.i
    cli_args = VectorPostprocessors/storage/contains_complete_history=false
    csvdiff = 'master_out_storage_0000.csv master_out_storage_0001.csv master_out_storage_0002.csv'

    detail = "or contain a single file per timestep."
  []
[]