HardwareIDAux

Creates a field showing the assignment of partitions to physical nodes in the cluster.

Description

One of the main purposes of this object is to aid in the diagnostic of mesh partitioners. One metric to look at for mesh partitioners is how well they keep down inter-node (compute node) communication. HardwareIDAux allows you to visually see the mapping of elements to compute nodes in your job.

This is particularly interesting in the case of the PetscExternalPartitioner which has the capability to do "hierarchical" partitioning. Hierarchical partitioning makes it possible to partition over compute-nodes first... then within compute nodes, in order to better respect the physical topology of the compute cluster.

One important aspect of that is that how you launch your parallel job can matter quite a bit to partitioning. In-general, it's better for partitioners if all of the ranks of your job are contiguously assigned to each compute node. Here are four different ways, and the outcome using HardwareIDAux, to launch a job using a 100x100 generated mesh on 16 processes and 4 ndoes with two different partitioner...

Top left (METIS):


mpiexec -n 16 -host lemhi0002,lemhi0003,lemhi0004,lemhi0005 ../../../moose_test-opt -i hardware_id_aux.i

Top right (Hierarchic):


mpiexec -n 16 -host lemhi0002,lemhi0003,lemhi0004,lemhi0005 ../../../moose_test-opt -i hardware_id_aux.i -mat_partitioning_hierarchical_nfineparts 4

Bottom left (METIS):


mpiexec -n 16 -host lemhi0002,lemhi0003,lemhi0004,lemhi0005 -ppn 4 ../../../moose_test-opt -i hardware_id_aux.i

Bottom right (Hierarchic):


mpiexec -n 16 -host lemhi0002,lemhi0003,lemhi0004,lemhi0005 -ppn 4 ../../../moose_test-opt -i hardware_id_aux.i -mat_partitioning_hierarchical_nfineparts 4

It should be immediately apparent that the bottom right partitioning is best (will reduce the amount of inter-node communication). That result was achieved by using hierarchical partitioning and using -ppn 4 to tell mpiexec to put 4 processes on each compute node... which will cause those four processes to be contiguous on each node. The top two examples, which omit the -ppn option, end up getting "striped" mpi processes (one process is placed on each node and then it wraps around) causing a jumbly mess of partitioning which will increase the communication cost for the job (and decrease scalability).

Input Parameters

variableThe name of the variable that this object applies to
C++ Type:AuxVariableName
Options:
Description:The name of the variable that this object applies to

Required Parameters

blockThe list of block ids (SubdomainID) that this object will be applied
C++ Type:std::vector
Options:
Description:The list of block ids (SubdomainID) that this object will be applied
boundaryThe list of boundary IDs from the mesh where this boundary condition applies
C++ Type:std::vector
Options:
Description:The list of boundary IDs from the mesh where this boundary condition applies
execute_onLINEAR TIMESTEP_ENDThe list of flag(s) indicating when this object should be executed, the available options include NONE, INITIAL, LINEAR, NONLINEAR, TIMESTEP_END, TIMESTEP_BEGIN, FINAL, CUSTOM, PRE_DISPLACE.
Default:LINEAR TIMESTEP_END
C++ Type:ExecFlagEnum
Options:NONE INITIAL LINEAR NONLINEAR TIMESTEP_END TIMESTEP_BEGIN FINAL CUSTOM PRE_DISPLACE
Description:The list of flag(s) indicating when this object should be executed, the available options include NONE, INITIAL, LINEAR, NONLINEAR, TIMESTEP_END, TIMESTEP_BEGIN, FINAL, CUSTOM, PRE_DISPLACE.

Optional Parameters

control_tagsAdds user-defined labels for accessing object parameters via control logic.
C++ Type:std::vector
Options:
Description:Adds user-defined labels for accessing object parameters via control logic.
enableTrueSet the enabled status of the MooseObject.
Default:True
C++ Type:bool
Options:
Description:Set the enabled status of the MooseObject.
seed0The seed for the master random number generator
Default:0
C++ Type:unsigned int
Options:
Description:The seed for the master random number generator
use_displaced_meshFalseWhether or not this object should use the displaced mesh for computation. Note that in the case this is true but no displacements are provided in the Mesh block the undisplaced mesh will still be used.
Default:False
C++ Type:bool
Options:
Description:Whether or not this object should use the displaced mesh for computation. Note that in the case this is true but no displacements are provided in the Mesh block the undisplaced mesh will still be used.

Advanced Parameters

Input Files

test/tests/auxkernels/hardware_id_aux/hardware_id_aux.i

test/tests/auxkernels/hardware_id_aux/hardware_id_aux.i

[Mesh]
  type = GeneratedMesh
  dim = 2
  nx = 10
  ny = 10
[]

[Variables]
  [./u]
  [../]
[]

[AuxVariables]
  [./hardware_id]
    family = MONOMIAL
    order = CONSTANT
  [../]
[]

[Kernels]
  [./diff]
    type = Diffusion
    variable = u
  [../]
[]

[AuxKernels]
  [./hardware_id]
    type = HardwareIDAux
    variable = hardware_id
  [../]
[]

[BCs]
  [./left]
    type = DirichletBC
    variable = u
    boundary = left
    value = 0
  [../]
  [./right]
    type = DirichletBC
    variable = u
    boundary = right
    value = 1
  [../]
[]

[Executioner]
  type = Steady
  solve_type = 'PJFNK'
  petsc_options_iname = '-pc_type -pc_hypre_type'
  petsc_options_value = 'hypre boomeramg'
[]

[Outputs]
  exodus = true
[]

References

Description
Input Parameters
Input Files
References

Application Usage

Physics and Syntax

Application Development

Framework Development

MOOSEDocs

Infrastructure

Questions

Information and Tools