libMesh
|
This class defines a wrapper around the PETSc DM infrastructure. More...
#include <petsc_dm_wrapper.h>
Public Member Functions | |
PetscDMWrapper ()=default | |
~PetscDMWrapper () | |
void | clear () |
Destroys and clears all build DM-related data. More... | |
void | init_and_attach_petscdm (System &system, SNES snes) |
void | init_and_attach_petscdm (System &system, KSP ksp) |
Private Member Functions | |
unsigned int | init_petscdm (System &system) |
Initialize the PETSc DM and return the number of geometric multigrid levels. More... | |
void | init_dm_data (unsigned int n_levels, const Parallel::Communicator &comm) |
Init all the n_mesh_level dependent data structures. More... | |
DM & | get_dm (unsigned int level) |
Get reference to DM for the given mesh level. More... | |
PetscSection & | get_section (unsigned int level) |
Get reference to PetscSection for the given mesh level. More... | |
PetscSF & | get_star_forest (unsigned int level) |
Get reference to PetscSF for the given mesh level. More... | |
void | build_section (const System &system, PetscSection §ion) |
Takes System, empty PetscSection and populates the PetscSection. More... | |
void | build_sf (const System &system, PetscSF &star_forest) |
Takes System, empty PetscSF and populates the PetscSF. More... | |
void | set_point_range_in_section (const System &system, PetscSection §ion, std::unordered_map< dof_id_type, dof_id_type > &node_map, std::unordered_map< dof_id_type, dof_id_type > &elem_map, std::map< dof_id_type, unsigned int > &scalar_map) |
Helper function for build_section. More... | |
void | add_dofs_to_section (const System &system, PetscSection §ion, const std::unordered_map< dof_id_type, dof_id_type > &node_map, const std::unordered_map< dof_id_type, dof_id_type > &elem_map, const std::map< dof_id_type, unsigned int > &scalar_map) |
Helper function for build_section. More... | |
dof_id_type | check_section_n_dofs (PetscSection §ion) |
Helper function to sanity check PetscSection construction The PetscSection contains local dof information. More... | |
void | add_dofs_helper (const System &system, const DofObject &dof_object, dof_id_type local_id, PetscSection §ion) |
Private Attributes | |
std::vector< WrappedPetsc< DM > > | _dms |
Vector of DMs for all grid levels. More... | |
std::vector< WrappedPetsc< PetscSection > > | _sections |
Vector of PETScSections for all grid levels. More... | |
std::vector< PetscSF > | _star_forests |
Vector of star forests for all grid levels. More... | |
std::vector< std::unique_ptr< PetscMatrixBase< Number > > > | _pmtx_vec |
Vector of projection matrixes for all grid levels. More... | |
std::vector< std::unique_ptr< PetscMatrixBase< Number > > > | _subpmtx_vec |
Vector of sub projection matrixes for all grid levels for fieldsplit. More... | |
std::vector< PetscDMContext > | _ctx_vec |
Vector of internal PetscDM context structs for all grid levels Pointers to these C++ objects are passed to DMShellSetContext(), they are cleaned up automatically by their destructors. More... | |
std::vector< std::unique_ptr< PetscVector< Number > > > | _vec_vec |
Vector of solution vectors for all grid levels. More... | |
std::vector< unsigned int > | _mesh_dof_sizes |
Stores n_dofs for each grid level, to be used for projection matrix sizing. More... | |
std::vector< unsigned int > | _mesh_dof_loc_sizes |
Stores n_local_dofs for each grid level, to be used for projection vector sizing. More... | |
This class defines a wrapper around the PETSc DM infrastructure.
By coordinating DM data structures with libMesh, we can use libMesh mesh hierarchies for geometric multigrid. Additionally, by setting the DM data, we can additionally (with or without multigrid) define recursive fieldsplits of our variables.
Definition at line 97 of file petsc_dm_wrapper.h.
|
default |
|
default |
|
private |
Definition at line 962 of file petsc_dm_wrapper.C.
References libMesh::ParallelObject::comm(), libMesh::make_range(), libMesh::DofObject::n_dofs(), libMesh::System::n_vars(), and libMesh::System::number().
Referenced by add_dofs_to_section().
|
private |
Helper function for build_section.
This function will set the DoF info for each "point" in the PetscSection.
Definition at line 908 of file petsc_dm_wrapper.C.
References add_dofs_helper(), libMesh::ParallelObject::comm(), libMesh::System::get_mesh(), libMesh::OrderWrapper::get_order(), libMesh::libmesh_assert(), mesh, libMesh::ParallelObject::n_processors(), libMesh::FEType::order, libMesh::ParallelObject::processor_id(), libMesh::Variable::type(), and libMesh::System::variable().
Referenced by build_section().
|
private |
Takes System, empty PetscSection and populates the PetscSection.
Take the System in its current state and an empty PetscSection and then populate the PetscSection. The PetscSection is comprised of global "point" numbers, where a "point" in PetscDM parlance is a geometric entity: node, edge, face, or element. Then, we also add the DoF numbering for each variable for each of the "points". The PetscSection, together the with PetscSF will allow for recursive fieldsplits from the command line using PETSc.
Definition at line 671 of file petsc_dm_wrapper.C.
References add_dofs_to_section(), check_section_n_dofs(), libMesh::ParallelObject::comm(), TIMPI::Communicator::get(), libMesh::make_range(), libMesh::System::n_local_dofs(), libMesh::System::n_vars(), libMesh::on_command_line(), set_point_range_in_section(), and libMesh::System::variable_name().
Referenced by init_petscdm().
|
private |
Takes System, empty PetscSF and populates the PetscSF.
The PetscSF (star forest) is a cousin of PetscSection. PetscSection has the DoF info, and PetscSF gives the parallel distribution of the DoF info. So PetscSF should only be necessary when we have more than one MPI rank. Essentially, we are copying the DofMap.send_list(): we are specifying the local dofs, what rank communicates that dof info (for off-processor dofs that are communicated) and the dofs local index on that rank.
https://jedbrown.org/files/StarForest.pdf
Definition at line 728 of file petsc_dm_wrapper.C.
References libMesh::ParallelObject::comm(), libMesh::DofMap::dof_owner(), libMesh::DofMapBase::first_dof(), TIMPI::Communicator::get(), libMesh::System::get_dof_map(), libMesh::DofMap::get_send_list(), libMesh::index_range(), and libMesh::DofMap::n_local_dofs().
Referenced by init_petscdm().
|
private |
Helper function to sanity check PetscSection construction The PetscSection contains local dof information.
This helper function just facilitates sanity checking that in fact it only has n_local_dofs.
Definition at line 991 of file petsc_dm_wrapper.C.
Referenced by build_section().
void libMesh::PetscDMWrapper::clear | ( | ) |
Destroys and clears all build DM-related data.
Definition at line 386 of file petsc_dm_wrapper.C.
References _ctx_vec, _dms, _mesh_dof_loc_sizes, _mesh_dof_sizes, _pmtx_vec, _sections, _star_forests, _subpmtx_vec, and _vec_vec.
Referenced by libMesh::PetscDiffSolver::clear().
|
inlineprivate |
Get reference to DM for the given mesh level.
init_dm_data() should be called before this function.
Definition at line 184 of file petsc_dm_wrapper.h.
References _dms.
Referenced by init_and_attach_petscdm(), and init_petscdm().
|
inlineprivate |
Get reference to PetscSection for the given mesh level.
init_dm_data() should be called before this function.
Definition at line 192 of file petsc_dm_wrapper.h.
References _sections.
Referenced by init_petscdm().
|
inlineprivate |
Get reference to PetscSF for the given mesh level.
init_dm_data() should be called before this function.
Definition at line 200 of file petsc_dm_wrapper.h.
References _star_forests.
Referenced by init_petscdm().
void libMesh::PetscDMWrapper::init_and_attach_petscdm | ( | System & | system, |
SNES | snes | ||
) |
Definition at line 653 of file petsc_dm_wrapper.C.
References libMesh::ParallelObject::comm(), get_dm(), init_petscdm(), and libMesh::MeshTools::n_levels().
Referenced by libMesh::PetscDiffSolver::setup_petsc_data().
void libMesh::PetscDMWrapper::init_and_attach_petscdm | ( | System & | system, |
KSP | ksp | ||
) |
Definition at line 662 of file petsc_dm_wrapper.C.
References libMesh::ParallelObject::comm(), get_dm(), init_petscdm(), and libMesh::MeshTools::n_levels().
|
private |
Init all the n_mesh_level dependent data structures.
Definition at line 1015 of file petsc_dm_wrapper.C.
References _ctx_vec, _dms, _mesh_dof_loc_sizes, _mesh_dof_sizes, _pmtx_vec, _sections, _star_forests, _subpmtx_vec, _vec_vec, and libMesh::MeshTools::n_levels().
Referenced by init_petscdm().
Initialize the PETSc DM and return the number of geometric multigrid levels.
Definition at line 407 of file petsc_dm_wrapper.C.
References _ctx_vec, _dms, _mesh_dof_loc_sizes, _mesh_dof_sizes, _pmtx_vec, _subpmtx_vec, _vec_vec, build_section(), build_sf(), libMesh::ParallelObject::comm(), libMesh::command_line_next(), libMesh::DofMap::distribute_dofs(), libMesh::DofMapBase::end_old_dof(), libMesh::DofMapBase::first_old_dof(), TIMPI::Communicator::get(), get_dm(), libMesh::System::get_dof_map(), libMesh::System::get_mesh(), get_section(), get_star_forest(), init_dm_data(), libMesh::libmesh_assert(), libMesh::libmesh_petsc_DMCoarsen(), libMesh::libmesh_petsc_DMCreateInterpolation(), libMesh::libmesh_petsc_DMCreateRestriction(), libMesh::libmesh_petsc_DMCreateSubDM(), libMesh::libmesh_petsc_DMRefine(), libMesh::DofMap::local_variable_indices(), mesh, libMesh::DofMap::n_dofs(), libMesh::MeshTools::n_levels(), libMesh::DofMap::n_local_dofs(), libMesh::ParallelObject::n_processors(), n_vars, libMesh::System::n_vars(), libMesh::DofMap::prepare_send_list(), libMesh::System::projection_matrix(), libMesh::System::reinit_constraints(), libMesh::MeshRefinement::uniformly_coarsen(), and libMesh::MeshRefinement::uniformly_refine().
Referenced by init_and_attach_petscdm().
|
private |
Helper function for build_section.
This function will count how many "points" on the current processor have DoFs associated with them and give that count to PETSc. We need to cache a mapping between the global node id and our local count that we do in this function because we will need the local number again in the add_dofs_to_section function.
Definition at line 794 of file petsc_dm_wrapper.C.
References libMesh::ParallelObject::comm(), libMesh::DofMap::dof_indices(), libMesh::FEType::family, libMesh::System::get_dof_map(), libMesh::System::get_mesh(), libMesh::libmesh_assert(), libMesh::DofMap::local_index(), libMesh::make_range(), mesh, libMesh::DofMap::n_local_dofs(), libMesh::ParallelObject::n_processors(), libMesh::DofMap::n_SCALAR_dofs(), libMesh::System::n_vars(), libMesh::System::number(), libMesh::ParallelObject::processor_id(), libMesh::SCALAR, libMesh::Variable::type(), and libMesh::System::variable().
Referenced by build_section().
|
private |
Vector of internal PetscDM context structs for all grid levels Pointers to these C++ objects are passed to DMShellSetContext(), they are cleaned up automatically by their destructors.
Definition at line 162 of file petsc_dm_wrapper.h.
Referenced by clear(), init_dm_data(), and init_petscdm().
|
private |
Vector of DMs for all grid levels.
These are PETSc objects created by calling DMShellCreate(), so we are responsible for cleaning them up.
Definition at line 122 of file petsc_dm_wrapper.h.
Referenced by clear(), get_dm(), init_dm_data(), and init_petscdm().
|
private |
Stores n_local_dofs for each grid level, to be used for projection vector sizing.
Definition at line 175 of file petsc_dm_wrapper.h.
Referenced by clear(), init_dm_data(), and init_petscdm().
|
private |
Stores n_dofs for each grid level, to be used for projection matrix sizing.
Definition at line 172 of file petsc_dm_wrapper.h.
Referenced by clear(), init_dm_data(), and init_petscdm().
|
private |
Vector of projection matrixes for all grid levels.
These are C++ objects, they are cleaned up automatically by their destructors.
Definition at line 148 of file petsc_dm_wrapper.h.
Referenced by clear(), init_dm_data(), and init_petscdm().
|
private |
Vector of PETScSections for all grid levels.
These are PETSc objects which are attached to the DM by calling DMSetLocalSection(). While the DM takes care of destroying existing PetscSections in calls to DMSetLocalSection(), it does not appear to clean up PetscSections it holds when it is destroyed, so we manage their lifetimes using WrappedPetsc objects.
Definition at line 132 of file petsc_dm_wrapper.h.
Referenced by clear(), get_section(), and init_dm_data().
|
private |
Vector of star forests for all grid levels.
These are PETSc objects which are attached to the DM by calling DMSetSectionSF(). The DM seems to take care of cleaning these up itself as far as I can tell, so we do not try to manage their lifetime in any way.
Definition at line 141 of file petsc_dm_wrapper.h.
Referenced by clear(), get_star_forest(), and init_dm_data().
|
private |
Vector of sub projection matrixes for all grid levels for fieldsplit.
These are C++ objects, they are cleaned up automatically by their destructors.
Definition at line 155 of file petsc_dm_wrapper.h.
Referenced by clear(), init_dm_data(), and init_petscdm().
|
private |
Vector of solution vectors for all grid levels.
These are C++ objects, they are cleaned up automatically by their destructors.
Definition at line 169 of file petsc_dm_wrapper.h.
Referenced by clear(), init_dm_data(), and init_petscdm().