libMesh::MeshCommunication Class Reference

This is the MeshCommunication class. More...

#include <mesh_communication.h>

Public Member Functions
	MeshCommunication ()
	Constructor. More...
	~MeshCommunication ()
	Destructor. More...
void	clear ()
	Clears all data structures and resets to a pristine state. More...
void	broadcast (MeshBase &) const
void	redistribute (DistributedMesh &mesh, bool newly_coarsened_only=false) const
	This method takes a parallel distributed mesh and redistributes the elements. More...
void	gather_neighboring_elements (DistributedMesh &) const
void	send_coarse_ghosts (MeshBase &) const
	Examine a just-coarsened mesh, and for any newly-coarsened elements, send the associated ghosted elements to the processor which needs them. More...
void	gather (const processor_id_type root_id, DistributedMesh &) const
	This method takes an input DistributedMesh which may be distributed among all the processors. More...
void	allgather (DistributedMesh &mesh) const
	This method takes an input DistributedMesh which may be distributed among all the processors. More...
void	delete_remote_elements (DistributedMesh &, const std::set< Elem * > &) const
	This method takes an input DistributedMesh which may be distributed among all the processors. More...
void	assign_global_indices (MeshBase &) const
	This method assigns globally unique, partition-agnostic indices to the nodes and elements in the mesh. More...
void	check_for_duplicate_global_indices (MeshBase &) const
	Throw an error if we have any index clashes in the numbering used by assign_global_indices. More...
template<typename ForwardIterator >
void	find_local_indices (const libMesh::BoundingBox &, const ForwardIterator &, const ForwardIterator &, std::unordered_map< dof_id_type, dof_id_type > &) const
	This method determines a locally unique, contiguous index for each object in the input range. More...
template<typename ForwardIterator >
void	find_global_indices (const Parallel::Communicator &communicator, const libMesh::BoundingBox &, const ForwardIterator &, const ForwardIterator &, std::vector< dof_id_type > &) const
	This method determines a globally unique, partition-agnostic index for each object in the input range. More...
void	make_elems_parallel_consistent (MeshBase &)
	Copy ids of ghost elements from their local processors. More...
void	make_p_levels_parallel_consistent (MeshBase &)
	Copy p levels of ghost elements from their local processors. More...
void	make_node_ids_parallel_consistent (MeshBase &)
	Assuming all ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all other ids parallel consistent. More...
void	make_node_unique_ids_parallel_consistent (MeshBase &)
	Assuming all unique_ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all ghost unique_ids parallel consistent. More...
void	make_node_proc_ids_parallel_consistent (MeshBase &)
	Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes all other processor ids parallel consistent as well. More...
void	make_new_node_proc_ids_parallel_consistent (MeshBase &)
	Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes processor ids on new nodes on other processors parallel consistent as well. More...
void	make_nodes_parallel_consistent (MeshBase &)
	Copy processor_ids and ids on ghost nodes from their local processors. More...
void	make_new_nodes_parallel_consistent (MeshBase &)
	Copy processor_ids and ids on new nodes from their local processors. More...

Detailed Description

This is the MeshCommunication class.

It handles all the details of communicating mesh information from one processor to another. All parallelization of the Mesh data structures is done via this class.

Author

Benjamin S. Kirk

Date

2003

Definition at line 50 of file mesh_communication.h.

Constructor & Destructor Documentation

MeshCommunication()

libMesh::MeshCommunication::MeshCommunication ( )

inline

Constructor.

Definition at line 57 of file mesh_communication.h.

{}

~MeshCommunication()

libMesh::MeshCommunication::~MeshCommunication ( )

inline

Destructor.

Definition at line 62 of file mesh_communication.h.

{}

Member Function Documentation

allgather()

void libMesh::MeshCommunication::allgather ( DistributedMesh &  mesh ) const

inline

This method takes an input DistributedMesh which may be distributed among all the processors.

Each processor then sends its local nodes and elements to the other processors. The end result is that a previously distributed DistributedMesh will be serialized on each processor. Since this method is collective it must be called by all processors.

Definition at line 135 of file mesh_communication.h.

  { MeshCommunication::gather(DofObject::invalid_processor_id, mesh); }

References gather(), libMesh::DofObject::invalid_processor_id, and mesh.

assign_global_indices()

void libMesh::MeshCommunication::assign_global_indices

(

MeshBase &

mesh

)

const

This method assigns globally unique, partition-agnostic indices to the nodes and elements in the mesh.

The approach is to compute the Hilbert space-filling curve key and use its value to assign an index in [0,N_global). Since the Hilbert key is unique for each spatial location, two objects occupying the same location will be assigned the same global id. Thus, this method can also be useful for identifying duplicate nodes which may occur during parallel refinement.

Definition at line 178 of file mesh_communication_global_indices.C.

{
  LOG_SCOPE ("assign_global_indices()", "MeshCommunication");
 
  // This method determines partition-agnostic global indices
  // for nodes and elements.
 
  // Algorithm:
  // (1) compute the Hilbert key for each local node/element
  // (2) perform a parallel sort of the Hilbert key
  // (3) get the min/max value on each processor
  // (4) determine the position in the global ranking for
  //     each local object
 
  const Parallel::Communicator & communicator (mesh.comm());
 
  // Global bounding box.  We choose the nodal bounding box for
  // backwards compatibility; the element bounding box may be looser
  // on curved elements.
  BoundingBox bbox =
    MeshTools::create_nodal_bounding_box (mesh);
 
  //-------------------------------------------------------------
  // (1) compute Hilbert keys
  std::vector<Parallel::DofObjectKey>
    node_keys, elem_keys;
 
  {
    // Nodes first
    {
      ConstNodeRange nr (mesh.local_nodes_begin(),
                         mesh.local_nodes_end());
      node_keys.resize (nr.size());
      Threads::parallel_for (nr, ComputeHilbertKeys (bbox, node_keys));
 
      // // It's O(N^2) to check that these keys don't duplicate before the
      // // sort...
      // MeshBase::const_node_iterator nodei = mesh.local_nodes_begin();
      // for (std::size_t i = 0; i != node_keys.size(); ++i, ++nodei)
      //   {
      //     MeshBase::const_node_iterator nodej = mesh.local_nodes_begin();
      //     for (std::size_t j = 0; j != i; ++j, ++nodej)
      //       {
      //         if (node_keys[i] == node_keys[j])
      //           {
      //             CFixBitVec icoords[3], jcoords[3];
      //             get_hilbert_coords(**nodej, bbox, jcoords);
      //             libMesh::err << "node " << (*nodej)->id() << ", " << static_cast<Point &>(**nodej) << " has HilbertIndices " << node_keys[j] << std::endl;
      //             get_hilbert_coords(**nodei, bbox, icoords);
      //             libMesh::err << "node " << (*nodei)->id() << ", " << static_cast<Point &>(**nodei) << " has HilbertIndices " << node_keys[i] << std::endl;
      //             libmesh_error_msg("Error: nodes with duplicate Hilbert keys!");
      //           }
      //       }
      //   }
    }
 
    // Elements next
    {
      ConstElemRange er (mesh.local_elements_begin(),
                         mesh.local_elements_end());
      elem_keys.resize (er.size());
      Threads::parallel_for (er, ComputeHilbertKeys (bbox, elem_keys));
 
      // // For elements, the keys can be (and in the case of TRI, are
      // // expected to be) duplicates, but only if the elements are at
      // // different levels
      // MeshBase::const_element_iterator elemi = mesh.local_elements_begin();
      // for (std::size_t i = 0; i != elem_keys.size(); ++i, ++elemi)
      //   {
      //     MeshBase::const_element_iterator elemj = mesh.local_elements_begin();
      //     for (std::size_t j = 0; j != i; ++j, ++elemj)
      //       {
      //         if ((elem_keys[i] == elem_keys[j]) &&
      //             ((*elemi)->level() == (*elemj)->level()))
      //           {
      //             libMesh::err << "level " << (*elemj)->level()
      //                          << " elem\n" << (**elemj)
      //                          << " centroid " << (*elemj)->centroid()
      //                          << " has HilbertIndices " << elem_keys[j]
      //                          << " or " << get_dofobject_key((*elemj), bbox)
      //                          << std::endl;
      //             libMesh::err << "level " << (*elemi)->level()
      //                          << " elem\n" << (**elemi)
      //                          << " centroid " << (*elemi)->centroid()
      //                          << " has HilbertIndices " << elem_keys[i]
      //                          << " or " << get_dofobject_key((*elemi), bbox)
      //                          << std::endl;
      //             libmesh_error_msg("Error: level " << (*elemi)->level() << " elements with duplicate Hilbert keys!");
      //           }
      //       }
      //  }
    }
  } // done computing Hilbert keys
 
 
 
  //-------------------------------------------------------------
  // (2) parallel sort the Hilbert keys
  Parallel::Sort<Parallel::DofObjectKey> node_sorter (communicator,
                                                      node_keys);
  node_sorter.sort(); /* done with node_keys */ //node_keys.clear();
 
  const std::vector<Parallel::DofObjectKey> & my_node_bin =
    node_sorter.bin();
 
  Parallel::Sort<Parallel::DofObjectKey> elem_sorter (communicator,
                                                      elem_keys);
  elem_sorter.sort(); /* done with elem_keys */ //elem_keys.clear();
 
  const std::vector<Parallel::DofObjectKey> & my_elem_bin =
    elem_sorter.bin();
 
 
 
  //-------------------------------------------------------------
  // (3) get the max value on each processor
  std::vector<Parallel::DofObjectKey>
    node_upper_bounds(communicator.size()),
    elem_upper_bounds(communicator.size());
 
  { // limit scope of temporaries
    std::vector<Parallel::DofObjectKey> recvbuf(2*communicator.size());
    std::vector<unsigned short int> /* do not use a vector of bools here since it is not always so! */
      empty_nodes (communicator.size()),
      empty_elem  (communicator.size());
    std::vector<Parallel::DofObjectKey> my_max(2);
 
    communicator.allgather (static_cast<unsigned short int>(my_node_bin.empty()), empty_nodes);
    communicator.allgather (static_cast<unsigned short int>(my_elem_bin.empty()),  empty_elem);
 
    if (!my_node_bin.empty()) my_max[0] = my_node_bin.back();
    if (!my_elem_bin.empty()) my_max[1] = my_elem_bin.back();
 
    communicator.allgather (my_max, /* identical_buffer_sizes = */ true);
 
    // Be careful here.  The *_upper_bounds will be used to find the processor
    // a given object belongs to.  So, if a processor contains no objects (possible!)
    // then copy the bound from the lower processor id.
    for (auto p : IntRange<processor_id_type>(0, communicator.size()))
      {
        node_upper_bounds[p] = my_max[2*p+0];
        elem_upper_bounds[p] = my_max[2*p+1];
 
        if (p > 0) // default hilbert index value is the OK upper bound for processor 0.
          {
            if (empty_nodes[p]) node_upper_bounds[p] = node_upper_bounds[p-1];
            if (empty_elem[p])  elem_upper_bounds[p] = elem_upper_bounds[p-1];
          }
      }
  }
 
 
 
  //-------------------------------------------------------------
  // (4) determine the position in the global ranking for
  //     each local object
  {
    //----------------------------------------------
    // Nodes first -- all nodes, not just local ones
    {
      // Request sets to send to each processor
      std::map<processor_id_type, std::vector<Parallel::DofObjectKey>>
        requested_ids;
      // Results to gather from each processor - kept in a map so we
      // do only one loop over nodes after all receives are done.
      std::map<dof_id_type, std::vector<dof_id_type>>
        filled_request;
 
      // build up list of requests
      for (const auto & node : mesh.node_ptr_range())
        {
          libmesh_assert(node);
          const Parallel::DofObjectKey hi =
            get_dofobject_key (node, bbox);
          const processor_id_type pid =
            cast_int<processor_id_type>
            (std::distance (node_upper_bounds.begin(),
                            std::lower_bound(node_upper_bounds.begin(),
                                             node_upper_bounds.end(),
                                             hi)));
 
          libmesh_assert_less (pid, communicator.size());
 
          requested_ids[pid].push_back(hi);
        }
 
      // The number of objects in my_node_bin on each processor
      std::vector<dof_id_type> node_bin_sizes(communicator.size());
      communicator.allgather (static_cast<dof_id_type>(my_node_bin.size()), node_bin_sizes);
 
      // The offset of my first global index
      dof_id_type my_offset = 0;
      for (auto pid : IntRange<processor_id_type>(0, communicator.rank()))
        my_offset += node_bin_sizes[pid];
 
      auto gather_functor =
        [
#ifndef NDEBUG
         & node_upper_bounds,
         & communicator,
#endif
         & my_node_bin,
         my_offset
        ]
        (processor_id_type,
         const std::vector<Parallel::DofObjectKey> & keys,
         std::vector<dof_id_type> & global_ids)
        {
          // Fill the requests
          const std::size_t keys_size = keys.size();
          global_ids.reserve(keys_size);
          for (std::size_t idx=0; idx != keys_size; idx++)
            {
              const Parallel::DofObjectKey & hi = keys[idx];
              libmesh_assert_less_equal (hi, node_upper_bounds[communicator.rank()]);
 
              // find the requested index in my node bin
              std::vector<Parallel::DofObjectKey>::const_iterator pos =
                std::lower_bound (my_node_bin.begin(), my_node_bin.end(), hi);
              libmesh_assert (pos != my_node_bin.end());
              libmesh_assert_equal_to (*pos, hi);
 
              // Finally, assign the global index based off the position of the index
              // in my array, properly offset.
              global_ids.push_back(cast_int<dof_id_type>(std::distance(my_node_bin.begin(), pos) + my_offset));
            }
        };
 
      auto action_functor =
        [&filled_request]
        (processor_id_type pid,
         const std::vector<Parallel::DofObjectKey> &,
         const std::vector<dof_id_type> & global_ids)
        {
          filled_request[pid] = global_ids;
        };
 
      // Trade requests with other processors
      const dof_id_type * ex = nullptr;
      Parallel::pull_parallel_vector_data
        (communicator, requested_ids, gather_functor, action_functor, ex);
 
      // We now have all the filled requests, so we can loop through our
      // nodes once and assign the global index to each one.
      {
        std::map<dof_id_type, std::vector<dof_id_type>::const_iterator>
          next_obj_on_proc;
        for (auto & p : filled_request)
          next_obj_on_proc[p.first] = p.second.begin();
 
        for (auto & node : mesh.node_ptr_range())
          {
            libmesh_assert(node);
            const Parallel::DofObjectKey hi =
              get_dofobject_key (node, bbox);
            const processor_id_type pid =
              cast_int<processor_id_type>
              (std::distance (node_upper_bounds.begin(),
                              std::lower_bound(node_upper_bounds.begin(),
                                               node_upper_bounds.end(),
                                               hi)));
 
            libmesh_assert_less (pid, communicator.size());
            libmesh_assert (next_obj_on_proc[pid] != filled_request[pid].end());
 
            const dof_id_type global_index = *next_obj_on_proc[pid];
            libmesh_assert_less (global_index, mesh.n_nodes());
            node->set_id() = global_index;
 
            ++next_obj_on_proc[pid];
          }
      }
    }
 
    //---------------------------------------------------
    // elements next -- all elements, not just local ones
    {
      // Request sets to send to each processor
      std::map<processor_id_type, std::vector<Parallel::DofObjectKey>>
        requested_ids;
      // Results to gather from each processor - kept in a map so we
      // do only one loop over elements after all receives are done.
      std::map<dof_id_type, std::vector<dof_id_type>>
        filled_request;
 
      for (const auto & elem : mesh.element_ptr_range())
        {
          libmesh_assert(elem);
          const Parallel::DofObjectKey hi =
            get_dofobject_key (elem, bbox);
          const processor_id_type pid =
            cast_int<processor_id_type>
            (std::distance (elem_upper_bounds.begin(),
                            std::lower_bound(elem_upper_bounds.begin(),
                                             elem_upper_bounds.end(),
                                             hi)));
 
          libmesh_assert_less (pid, communicator.size());
 
          requested_ids[pid].push_back(hi);
        }
 
      // The number of objects in my_elem_bin on each processor
      std::vector<dof_id_type> elem_bin_sizes(communicator.size());
      communicator.allgather (static_cast<dof_id_type>(my_elem_bin.size()), elem_bin_sizes);
 
      // The offset of my first global index
      dof_id_type my_offset = 0;
      for (auto pid : IntRange<processor_id_type>(0, communicator.rank()))
        my_offset += elem_bin_sizes[pid];
 
      auto gather_functor =
        [
#ifndef NDEBUG
         & elem_upper_bounds,
         & communicator,
#endif
         & my_elem_bin,
         my_offset
        ]
        (processor_id_type,
         const std::vector<Parallel::DofObjectKey> & keys,
         std::vector<dof_id_type> & global_ids)
        {
          // Fill the requests
          const std::size_t keys_size = keys.size();
          global_ids.reserve(keys_size);
          for (std::size_t idx=0; idx != keys_size; idx++)
            {
              const Parallel::DofObjectKey & hi = keys[idx];
              libmesh_assert_less_equal (hi, elem_upper_bounds[communicator.rank()]);
 
              // find the requested index in my elem bin
              std::vector<Parallel::DofObjectKey>::const_iterator pos =
                std::lower_bound (my_elem_bin.begin(), my_elem_bin.end(), hi);
              libmesh_assert (pos != my_elem_bin.end());
              libmesh_assert_equal_to (*pos, hi);
 
              // Finally, assign the global index based off the position of the index
              // in my array, properly offset.
              global_ids.push_back (cast_int<dof_id_type>(std::distance(my_elem_bin.begin(), pos) + my_offset));
            }
        };
 
      auto action_functor =
        [&filled_request]
        (processor_id_type pid,
         const std::vector<Parallel::DofObjectKey> &,
         const std::vector<dof_id_type> & global_ids)
        {
          filled_request[pid] = global_ids;
        };
 
      // Trade requests with other processors
      const dof_id_type * ex = nullptr;
      Parallel::pull_parallel_vector_data
        (communicator, requested_ids, gather_functor, action_functor, ex);
 
      // We now have all the filled requests, so we can loop through our
      // elements once and assign the global index to each one.
      {
        std::vector<std::vector<dof_id_type>::const_iterator>
          next_obj_on_proc; next_obj_on_proc.reserve(communicator.size());
        for (auto pid : IntRange<processor_id_type>(0, communicator.size()))
          next_obj_on_proc.push_back(filled_request[pid].begin());
 
        for (auto & elem : mesh.element_ptr_range())
          {
            libmesh_assert(elem);
            const Parallel::DofObjectKey hi =
              get_dofobject_key (elem, bbox);
            const processor_id_type pid =
              cast_int<processor_id_type>
              (std::distance (elem_upper_bounds.begin(),
                              std::lower_bound(elem_upper_bounds.begin(),
                                               elem_upper_bounds.end(),
                                               hi)));
 
            libmesh_assert_less (pid, communicator.size());
            libmesh_assert (next_obj_on_proc[pid] != filled_request[pid].end());
 
            const dof_id_type global_index = *next_obj_on_proc[pid];
            libmesh_assert_less (global_index, mesh.n_elem());
            elem->set_id() = global_index;
 
            ++next_obj_on_proc[pid];
          }
      }
    }
  }
}

References libMesh::Parallel::Sort< KeyType, IdxType >::bin(), libMesh::ParallelObject::comm(), libMesh::MeshTools::create_nodal_bounding_box(), distance(), libMesh::MeshBase::element_ptr_range(), end, libMesh::MeshTools::Generation::Private::idx(), libMesh::libmesh_assert(), libMesh::MeshBase::local_elements_begin(), libMesh::MeshBase::local_elements_end(), libMesh::MeshBase::local_nodes_begin(), libMesh::MeshBase::local_nodes_end(), mesh, libMesh::MeshBase::n_elem(), libMesh::MeshBase::n_nodes(), libMesh::MeshBase::node_ptr_range(), libMesh::Threads::parallel_for(), and libMesh::Parallel::Sort< KeyType, IdxType >::sort().

Referenced by libMesh::MeshTools::Private::globally_renumber_nodes_and_elements().

broadcast()

void libMesh::MeshCommunication::broadcast

(

MeshBase &

mesh

)

const

      Finds all the processors that may contain
      elements that neighbor my elements.  This list
      is guaranteed to include all processors that border
      any of my elements, but may include additional ones as
      well.  This method computes bounding boxes for the
      elements on each processor and checks for overlaps.
     ‍/

void find_neighboring_processors(const MeshBase &);

/** This method takes a mesh (which is assumed to reside on processor 0) and broadcasts it to all the other processors. It also broadcasts any boundary information the mesh has associated with it.

Definition at line 1084 of file mesh_communication.C.

{
  // no MPI == one processor, no need for this method...
  return;
}

Referenced by libMesh::NameBasedIO::read(), libMesh::CheckpointIO::read(), MeshInputTest::testDynaReadElem(), MeshInputTest::testDynaReadPatch(), MeshInputTest::testExodusCopyElementSolution(), and MeshInputTest::testExodusWriteElementDataFromDiscontinuousNodalData().

check_for_duplicate_global_indices()

void libMesh::MeshCommunication::check_for_duplicate_global_indices

(

MeshBase &

mesh

)

const

Throw an error if we have any index clashes in the numbering used by assign_global_indices.

Definition at line 577 of file mesh_communication_global_indices.C.

{
  LOG_SCOPE ("check_for_duplicate_global_indices()", "MeshCommunication");
 
  // Global bounding box.  We choose the nodal bounding box for
  // backwards compatibility; the element bounding box may be looser
  // on curved elements.
  BoundingBox bbox =
    MeshTools::create_nodal_bounding_box (mesh);
 
 
  std::vector<Parallel::DofObjectKey>
    node_keys, elem_keys;
 
  {
    // Nodes first
    {
      ConstNodeRange nr (mesh.local_nodes_begin(),
                         mesh.local_nodes_end());
      node_keys.resize (nr.size());
      Threads::parallel_for (nr, ComputeHilbertKeys (bbox, node_keys));
 
      // It's O(N^2) to check that these keys don't duplicate before the
      // sort...
      MeshBase::const_node_iterator nodei = mesh.local_nodes_begin();
      for (std::size_t i = 0; i != node_keys.size(); ++i, ++nodei)
        {
          MeshBase::const_node_iterator nodej = mesh.local_nodes_begin();
          for (std::size_t j = 0; j != i; ++j, ++nodej)
            {
              if (node_keys[i] == node_keys[j])
                {
                  CFixBitVec icoords[3], jcoords[3];
                  get_hilbert_coords(**nodej, bbox, jcoords);
                  libMesh::err <<
                    "node " << (*nodej)->id() << ", " <<
                    *(Point *)(*nodej) << " has HilbertIndices " <<
                    node_keys[j] << std::endl;
                  get_hilbert_coords(**nodei, bbox, icoords);
                  libMesh::err <<
                    "node " << (*nodei)->id() << ", " <<
                    *(Point *)(*nodei) << " has HilbertIndices " <<
                    node_keys[i] << std::endl;
                  libmesh_error_msg("Error: nodes with duplicate Hilbert keys!");
                }
            }
        }
    }
 
    // Elements next
    {
      ConstElemRange er (mesh.local_elements_begin(),
                         mesh.local_elements_end());
      elem_keys.resize (er.size());
      Threads::parallel_for (er, ComputeHilbertKeys (bbox, elem_keys));
 
      // For elements, the keys can be (and in the case of TRI, are
      // expected to be) duplicates, but only if the elements are at
      // different levels
      MeshBase::const_element_iterator elemi = mesh.local_elements_begin();
      for (std::size_t i = 0; i != elem_keys.size(); ++i, ++elemi)
        {
          MeshBase::const_element_iterator elemj = mesh.local_elements_begin();
          for (std::size_t j = 0; j != i; ++j, ++elemj)
            {
              if ((elem_keys[i] == elem_keys[j]) &&
                  ((*elemi)->level() == (*elemj)->level()))
                {
                  libMesh::err <<
                    "level " << (*elemj)->level() << " elem\n" <<
                    (**elemj) << " centroid " <<
                    (*elemj)->centroid() << " has HilbertIndices " <<
                    elem_keys[j] << " or " <<
                    get_dofobject_key((*elemj), bbox) <<
                    std::endl;
                  libMesh::err <<
                    "level " << (*elemi)->level() << " elem\n" <<
                    (**elemi) << " centroid " <<
                    (*elemi)->centroid() << " has HilbertIndices " <<
                    elem_keys[i] << " or " <<
                    get_dofobject_key((*elemi), bbox) <<
                    std::endl;
                  libmesh_error_msg("Error: level " << (*elemi)->level() << " elements with duplicate Hilbert keys!");
                }
            }
        }
    }
  } // done checking Hilbert keys
}

References libMesh::MeshTools::create_nodal_bounding_box(), libMesh::err, libMesh::MeshBase::local_elements_begin(), libMesh::MeshBase::local_elements_end(), libMesh::MeshBase::local_nodes_begin(), libMesh::MeshBase::local_nodes_end(), mesh, and libMesh::Threads::parallel_for().

clear()

void libMesh::MeshCommunication::clear

(

)

Clears all data structures and resets to a pristine state.

Definition at line 276 of file mesh_communication.C.

{
  //  _neighboring_processors.clear();
}

delete_remote_elements()

void libMesh::MeshCommunication::delete_remote_elements	(	DistributedMesh &	mesh,
		const std::set< Elem * > &	extra_ghost_elem_ids
	)		const

This method takes an input DistributedMesh which may be distributed among all the processors.

Each processor deletes all elements which are neither local elements nor "ghost" elements which touch local elements, and deletes all nodes which are not contained in local or ghost elements. The end result is that a previously serial DistributedMesh will be distributed between processors. Since this method is collective it must be called by all processors.

The std::set is a list of extra elements that you don't want to delete. These will be left on the current processor along with local elements and ghosted neighbors.

Definition at line 1852 of file mesh_communication.C.

{
  // The mesh should know it's about to be parallelized
  libmesh_assert (!mesh.is_serial());
 
  LOG_SCOPE("delete_remote_elements()", "MeshCommunication");
 
#ifdef DEBUG
  // We expect maximum ids to be in sync so we can use them to size
  // vectors
  libmesh_assert(mesh.comm().verify(mesh.max_node_id()));
  libmesh_assert(mesh.comm().verify(mesh.max_elem_id()));
  const dof_id_type par_max_node_id = mesh.parallel_max_node_id();
  const dof_id_type par_max_elem_id = mesh.parallel_max_elem_id();
  libmesh_assert_equal_to (par_max_node_id, mesh.max_node_id());
  libmesh_assert_equal_to (par_max_elem_id, mesh.max_elem_id());
#endif
 
  std::set<const Elem *, CompareElemIdsByLevel> elements_to_keep;
 
  // Don't delete elements that we were explicitly told not to
  for (const auto & elem : extra_ghost_elem_ids)
    {
      std::vector<const Elem *> active_family;
#ifdef LIBMESH_ENABLE_AMR
      if (!elem->subactive())
        elem->active_family_tree(active_family);
      else
#endif
        active_family.push_back(elem);
 
      for (const auto & f : active_family)
        elements_to_keep.insert(f);
    }
 
  // See which elements we still need to keep ghosted, given that
  // we're keeping local and unpartitioned elements.
  query_ghosting_functors
    (mesh, mesh.processor_id(),
     mesh.active_pid_elements_begin(mesh.processor_id()),
     mesh.active_pid_elements_end(mesh.processor_id()),
     elements_to_keep);
  query_ghosting_functors
    (mesh, DofObject::invalid_processor_id,
     mesh.active_pid_elements_begin(DofObject::invalid_processor_id),
     mesh.active_pid_elements_end(DofObject::invalid_processor_id),
     elements_to_keep);
 
  // The inactive elements we need to send should have their
  // immediate children present.
  connect_children(mesh, mesh.pid_elements_begin(mesh.processor_id()),
                   mesh.pid_elements_end(mesh.processor_id()),
                   elements_to_keep);
  connect_children(mesh,
                   mesh.pid_elements_begin(DofObject::invalid_processor_id),
                   mesh.pid_elements_end(DofObject::invalid_processor_id),
                   elements_to_keep);
 
  // The elements we need should have their ancestors and their
  // subactive children present too.
  connect_families(elements_to_keep);
 
  // Don't delete nodes that our semilocal elements need
  std::set<const Node *> connected_nodes;
  reconnect_nodes(elements_to_keep, connected_nodes);
 
  // Delete all the elements we have no reason to save,
  // starting with the most refined so that the mesh
  // is valid at all intermediate steps
  unsigned int n_levels = MeshTools::n_levels(mesh);
 
  for (int l = n_levels - 1; l >= 0; --l)
    for (auto & elem : as_range(mesh.level_elements_begin(l),
                                mesh.level_elements_end(l)))
      {
        libmesh_assert (elem);
        // Make sure we don't leave any invalid pointers
        const bool keep_me = elements_to_keep.count(elem);
 
        if (!keep_me)
          elem->make_links_to_me_remote();
 
        // delete_elem doesn't currently invalidate element
        // iterators... that had better not change
        if (!keep_me)
          mesh.delete_elem(elem);
      }
 
  // Delete all the nodes we have no reason to save
  for (auto & node : mesh.node_ptr_range())
    {
      libmesh_assert(node);
      if (!connected_nodes.count(node))
        mesh.delete_node(node);
    }
 
  // If we had a point locator, it's invalid now that some of the
  // elements it pointed to have been deleted.
  mesh.clear_point_locator();
 
  // Much of our boundary info may have been for now-remote parts of
  // the mesh, in which case we don't want to keep local copies.
  mesh.get_boundary_info().regenerate_id_sets();
 
  // We now have all remote elements and nodes deleted; our ghosting
  // functors should be ready to delete any now-redundant cached data
  // they use too.
  for (auto & gf : as_range(mesh.ghosting_functors_begin(), mesh.ghosting_functors_end()))
    gf->delete_remote_elements();
 
#ifdef DEBUG
  MeshTools::libmesh_assert_valid_refinement_tree(mesh);
#endif
}

References libMesh::Elem::active_family_tree(), libMesh::MeshBase::active_pid_elements_begin(), libMesh::MeshBase::active_pid_elements_end(), libMesh::as_range(), libMesh::MeshBase::clear_point_locator(), libMesh::ParallelObject::comm(), libMesh::connect_children(), libMesh::connect_families(), libMesh::MeshBase::delete_elem(), libMesh::MeshBase::delete_node(), libMesh::MeshBase::get_boundary_info(), libMesh::MeshBase::ghosting_functors_begin(), libMesh::MeshBase::ghosting_functors_end(), libMesh::DofObject::invalid_processor_id, libMesh::MeshBase::is_serial(), libMesh::MeshBase::level_elements_begin(), libMesh::MeshBase::level_elements_end(), libMesh::libmesh_assert(), libMesh::MeshTools::libmesh_assert_valid_refinement_tree(), libMesh::Elem::make_links_to_me_remote(), libMesh::MeshBase::max_elem_id(), libMesh::MeshBase::max_node_id(), mesh, libMesh::MeshTools::n_levels(), libMesh::MeshBase::node_ptr_range(), libMesh::MeshBase::pid_elements_begin(), libMesh::MeshBase::pid_elements_end(), libMesh::ParallelObject::processor_id(), libMesh::query_ghosting_functors(), libMesh::reconnect_nodes(), libMesh::BoundaryInfo::regenerate_id_sets(), and libMesh::Elem::subactive().

find_global_indices()

template<typename ForwardIterator >

void libMesh::MeshCommunication::find_global_indices	(	const Parallel::Communicator &	communicator,
		const libMesh::BoundingBox &	bbox,
		const ForwardIterator &	begin,
		const ForwardIterator &	end,
		std::vector< dof_id_type > &	index_map
	)		const

This method determines a globally unique, partition-agnostic index for each object in the input range.

Definition at line 710 of file mesh_communication_global_indices.C.

{
  LOG_SCOPE ("find_global_indices()", "MeshCommunication");
 
  // This method determines partition-agnostic global indices
  // for nodes and elements.
 
  // Algorithm:
  // (1) compute the Hilbert key for each local node/element
  // (2) perform a parallel sort of the Hilbert key
  // (3) get the min/max value on each processor
  // (4) determine the position in the global ranking for
  //     each local object
  index_map.clear();
  std::size_t n_objects = std::distance (begin, end);
  index_map.reserve(n_objects);
 
  //-------------------------------------------------------------
  // (1) compute Hilbert keys
  // These aren't trivial to compute, and we will need them again.
  // But the binsort will sort the input vector, trashing the order
  // that we'd like to rely on.  So, two vectors...
  std::vector<Parallel::DofObjectKey>
    sorted_hilbert_keys,
    hilbert_keys;
  sorted_hilbert_keys.reserve(n_objects);
  hilbert_keys.reserve(n_objects);
  {
    LOG_SCOPE("compute_hilbert_indices()", "MeshCommunication");
    for (ForwardIterator it=begin; it!=end; ++it)
      {
        const Parallel::DofObjectKey hi(get_dofobject_key (*it, bbox));
        hilbert_keys.push_back(hi);
 
        if ((*it)->processor_id() == communicator.rank())
          sorted_hilbert_keys.push_back(hi);
 
        // someone needs to take care of unpartitioned objects!
        if ((communicator.rank() == 0) &&
            ((*it)->processor_id() == DofObject::invalid_processor_id))
          sorted_hilbert_keys.push_back(hi);
      }
  }
 
  //-------------------------------------------------------------
  // (2) parallel sort the Hilbert keys
  START_LOG ("parallel_sort()", "MeshCommunication");
  Parallel::Sort<Parallel::DofObjectKey> sorter (communicator,
                                                 sorted_hilbert_keys);
  sorter.sort();
  STOP_LOG ("parallel_sort()", "MeshCommunication");
  const std::vector<Parallel::DofObjectKey> & my_bin = sorter.bin();
 
  // The number of objects in my_bin on each processor
  std::vector<unsigned int> bin_sizes(communicator.size());
  communicator.allgather (static_cast<unsigned int>(my_bin.size()), bin_sizes);
 
  // The offset of my first global index
  unsigned int my_offset = 0;
  for (auto pid : IntRange<processor_id_type>(0, communicator.rank()))
    my_offset += bin_sizes[pid];
 
  //-------------------------------------------------------------
  // (3) get the max value on each processor
  std::vector<Parallel::DofObjectKey>
    upper_bounds(1);
 
  if (!my_bin.empty())
    upper_bounds[0] = my_bin.back();
 
  communicator.allgather (upper_bounds, /* identical_buffer_sizes = */ true);
 
  // Be careful here.  The *_upper_bounds will be used to find the processor
  // a given object belongs to.  So, if a processor contains no objects (possible!)
  // then copy the bound from the lower processor id.
  for (auto p : IntRange<processor_id_type>(1, communicator.size()))
    if (!bin_sizes[p]) upper_bounds[p] = upper_bounds[p-1];
 
 
  //-------------------------------------------------------------
  // (4) determine the position in the global ranking for
  //     each local object
  {
    //----------------------------------------------
    // all objects, not just local ones
 
    // Request sets to send to each processor
    std::map<processor_id_type, std::vector<Parallel::DofObjectKey>>
      requested_ids;
    // Results to gather from each processor
    std::map<processor_id_type, std::vector<dof_id_type>>
      filled_request;
 
    // build up list of requests
    std::vector<Parallel::DofObjectKey>::const_iterator hi =
      hilbert_keys.begin();
 
    for (ForwardIterator it = begin; it != end; ++it)
      {
        libmesh_assert (hi != hilbert_keys.end());
 
        std::vector<Parallel::DofObjectKey>::iterator lb =
          std::lower_bound(upper_bounds.begin(), upper_bounds.end(),
                           *hi);
 
        const processor_id_type pid =
          cast_int<processor_id_type>
          (std::distance (upper_bounds.begin(), lb));
 
        libmesh_assert_less (pid, communicator.size());
 
        requested_ids[pid].push_back(*hi);
 
        ++hi;
        // go ahead and put pid in index_map, that way we
        // don't have to repeat the std::lower_bound()
        index_map.push_back(pid);
      }
 
  auto gather_functor =
    [
#ifndef NDEBUG
     & upper_bounds,
     & communicator,
#endif
     & bbox,
     & my_bin,
       my_offset
    ]
    (processor_id_type, const std::vector<Parallel::DofObjectKey> & keys,
     std::vector<dof_id_type> & global_ids)
    {
      // Ignore unused lambda capture warnings in devel mode
      libmesh_ignore(bbox);
 
      // Fill the requests
      const std::size_t keys_size = keys.size();
      global_ids.clear();
      global_ids.reserve(keys_size);
      for (std::size_t idx=0; idx != keys_size; idx++)
        {
          const Parallel::DofObjectKey & hilbert_indices = keys[idx];
          libmesh_assert_less_equal (hilbert_indices, upper_bounds[communicator.rank()]);
 
          // find the requested index in my node bin
          std::vector<Parallel::DofObjectKey>::const_iterator pos =
            std::lower_bound (my_bin.begin(), my_bin.end(), hilbert_indices);
          libmesh_assert (pos != my_bin.end());
#ifdef DEBUG
          // If we could not find the requested Hilbert index in
          // my_bin, something went terribly wrong, possibly the
          // Mesh was displaced differently on different processors,
          // and therefore the Hilbert indices don't agree.
          if (*pos != hilbert_indices)
            {
              // The input will be hilbert_indices.  We convert it
              // to BitVecType using the operator= provided by the
              // BitVecType class. BitVecType is a CBigBitVec!
              Hilbert::BitVecType input;
#ifdef LIBMESH_ENABLE_UNIQUE_ID
              input = hilbert_indices.first;
#else
              input = hilbert_indices;
#endif
 
              // Get output in a vector of CBigBitVec
              std::vector<CBigBitVec> output(3);
 
              // Call the indexToCoords function
              Hilbert::indexToCoords(output.data(), 8*sizeof(Hilbert::inttype), 3, input);
 
              // The entries in the output racks are integers in the
              // range [0, Hilbert::inttype::max] which can be
              // converted to floating point values in [0,1] and
              // finally to actual values using the bounding box.
              const Real max_int_as_real =
                static_cast<Real>(std::numeric_limits<Hilbert::inttype>::max());
 
              // Get the points in [0,1]^3.  The zeroth rack of each entry in
              // 'output' maps to the normalized x, y, and z locations,
              // respectively.
              Point p_hat(static_cast<Real>(output[0].racks()[0]) / max_int_as_real,
                          static_cast<Real>(output[1].racks()[0]) / max_int_as_real,
                          static_cast<Real>(output[2].racks()[0]) / max_int_as_real);
 
              // Convert the points from [0,1]^3 to their actual (x,y,z) locations
              Real
                xmin = bbox.first(0),
                xmax = bbox.second(0),
                ymin = bbox.first(1),
                ymax = bbox.second(1),
                zmin = bbox.first(2),
                zmax = bbox.second(2);
 
              // Convert the points from [0,1]^3 to their actual (x,y,z) locations
              Point p(xmin + (xmax-xmin)*p_hat(0),
                      ymin + (ymax-ymin)*p_hat(1),
                      zmin + (zmax-zmin)*p_hat(2));
 
              libmesh_error_msg("Could not find hilbert indices: "
                                << hilbert_indices
                                << " corresponding to point " << p);
            }
#endif
 
          // Finally, assign the global index based off the position of the index
          // in my array, properly offset.
          global_ids.push_back (cast_int<dof_id_type>(std::distance(my_bin.begin(), pos) + my_offset));
        }
    };
 
  auto action_functor =
    [&filled_request]
    (processor_id_type pid,
     const std::vector<Parallel::DofObjectKey> &,
     const std::vector<dof_id_type> & global_ids)
    {
      filled_request[pid] = global_ids;
    };
 
  const dof_id_type * ex = nullptr;
  Parallel::pull_parallel_vector_data
    (communicator, requested_ids, gather_functor, action_functor, ex);
 
    // We now have all the filled requests, so we can loop through our
    // nodes once and assign the global index to each one.
    {
      std::vector<std::vector<dof_id_type>::const_iterator>
        next_obj_on_proc; next_obj_on_proc.reserve(communicator.size());
      for (auto pid : IntRange<processor_id_type>(0, communicator.size()))
        next_obj_on_proc.push_back(filled_request[pid].begin());
 
      unsigned int cnt=0;
      for (ForwardIterator it = begin; it != end; ++it, cnt++)
        {
          const processor_id_type pid = cast_int<processor_id_type>
            (index_map[cnt]);
 
          libmesh_assert_less (pid, communicator.size());
          libmesh_assert (next_obj_on_proc[pid] != filled_request[pid].end());
 
          const dof_id_type global_index = *next_obj_on_proc[pid];
          index_map[cnt] = global_index;
 
          ++next_obj_on_proc[pid];
        }
    }
  }
 
  libmesh_assert_equal_to(index_map.size(), n_objects);
}

References libMesh::Parallel::Sort< KeyType, IdxType >::bin(), distance(), end, libMesh::MeshTools::Generation::Private::idx(), libMesh::DofObject::invalid_processor_id, libMesh::libmesh_assert(), libMesh::libmesh_ignore(), libMesh::Real, and libMesh::Parallel::Sort< KeyType, IdxType >::sort().

Referenced by libMesh::ParmetisPartitioner::initialize(), libMesh::MetisPartitioner::partition_range(), and libMesh::Partitioner::partition_unpartitioned_elements().

find_local_indices()

template<typename ForwardIterator >

void libMesh::MeshCommunication::find_local_indices	(	const libMesh::BoundingBox &	bbox,
		const ForwardIterator &	begin,
		const ForwardIterator &	end,
		std::unordered_map< dof_id_type, dof_id_type > &	index_map
	)		const

This method determines a locally unique, contiguous index for each object in the input range.

Definition at line 674 of file mesh_communication_global_indices.C.

{
  LOG_SCOPE ("find_local_indices()", "MeshCommunication");
 
  // This method determines id-agnostic local indices
  // for nodes and elements by sorting Hilbert keys.
 
  index_map.clear();
 
  //-------------------------------------------------------------
  // (1) compute Hilbert keys
  // These aren't trivial to compute, and we will need them again.
  // But the binsort will sort the input vector, trashing the order
  // that we'd like to rely on.  So, two vectors...
  std::map<Parallel::DofObjectKey, dof_id_type> hilbert_keys;
  {
    LOG_SCOPE("local_hilbert_indices", "MeshCommunication");
    for (ForwardIterator it=begin; it!=end; ++it)
      {
        const Parallel::DofObjectKey hi(get_dofobject_key ((*it), bbox));
        hilbert_keys.emplace(hi, (*it)->id());
      }
  }
 
  {
    dof_id_type cnt = 0;
    for (auto key_val : hilbert_keys)
      index_map[key_val.second] = cnt++;
  }
}

References end.

Referenced by libMesh::Partitioner::_find_global_index_by_pid_map().

gather()

void libMesh::MeshCommunication::gather	(	const processor_id_type	root_id,
		DistributedMesh &	mesh
	)		const

This method takes an input DistributedMesh which may be distributed among all the processors.

Each processor then sends its local nodes and elements to processor root_id. The end result is that a previously distributed DistributedMesh will be serialized on processor root_id. Since this method is collective it must be called by all processors. For the special case of root_id equal to DofObject::invalid_processor_id this function performs an allgather.

Definition at line 1168 of file mesh_communication.C.

{
  // no MPI == one processor, no need for this method...
  return;
}

Referenced by allgather().

gather_neighboring_elements()

void libMesh::MeshCommunication::gather_neighboring_elements

(

DistributedMesh &

mesh

)

const

Definition at line 540 of file mesh_communication.C.

{
  // no MPI == one processor, no need for this method...
  return;
}

Referenced by libMesh::Nemesis_IO::read().

make_elems_parallel_consistent()

void libMesh::MeshCommunication::make_elems_parallel_consistent

(

MeshBase &

mesh

)

Copy ids of ghost elements from their local processors.

Definition at line 1513 of file mesh_communication.C.

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
  LOG_SCOPE ("make_elems_parallel_consistent()", "MeshCommunication");
 
  SyncIds syncids(mesh, &MeshBase::renumber_elem);
  Parallel::sync_element_data_by_parent_id
    (mesh, mesh.active_elements_begin(),
     mesh.active_elements_end(), syncids);
 
#ifdef LIBMESH_ENABLE_UNIQUE_ID
  SyncUniqueIds<Elem> syncuniqueids(mesh, &MeshBase::query_elem_ptr);
  Parallel::sync_dofobject_data_by_id
    (mesh.comm(), mesh.active_elements_begin(),
     mesh.active_elements_end(), syncuniqueids);
#endif
}

References libMesh::MeshBase::active_elements_begin(), libMesh::MeshBase::active_elements_end(), libMesh::ParallelObject::comm(), mesh, libMesh::MeshBase::query_elem_ptr(), libMesh::MeshBase::renumber_elem(), libMesh::Parallel::sync_dofobject_data_by_id(), and libMesh::Parallel::sync_element_data_by_parent_id().

Referenced by libMesh::MeshRefinement::_refine_elements().

make_new_node_proc_ids_parallel_consistent()

void libMesh::MeshCommunication::make_new_node_proc_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes processor ids on new nodes on other processors parallel consistent as well.

Definition at line 1693 of file mesh_communication.C.

{
  LOG_SCOPE ("make_new_node_proc_ids_parallel_consistent()", "MeshCommunication");
 
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // Local nodes should have unique authoritative ids,
  // and new nodes should be unpartitioned.
  //
  // New ghost nodes touching local elements should be unpartitioned.
 
  // We may not have consistent processor ids for new nodes (because a
  // node may be old and partitioned on one processor but new and
  // unpartitioned on another) when we start
#ifdef DEBUG
  MeshTools::libmesh_assert_parallel_consistent_procids<Node>(mesh);
  // MeshTools::libmesh_assert_parallel_consistent_new_node_procids(mesh);
#endif
 
  // We have two kinds of new nodes.  *NEW* nodes are unpartitioned on
  // all processors: we need to use a id-independent (i.e. dumb)
  // heuristic to partition them.  But "new" nodes are newly created
  // on some processors (when ghost elements are refined) yet
  // correspond to existing nodes on other processors: we need to use
  // the existing processor id for them.
  //
  // A node which is "new" on one processor will be associated with at
  // least one ghost element, and we can just query that ghost
  // element's owner to find out the correct processor id.
 
  auto node_unpartitioned =
    [](const Elem * elem, unsigned int local_node_num)
    { return elem->node_ref(local_node_num).processor_id() ==
        DofObject::invalid_processor_id; };
 
  SyncProcIds sync(mesh);
 
  sync_node_data_by_element_id_once
    (mesh, mesh.not_local_elements_begin(),
     mesh.not_local_elements_end(), Parallel::SyncEverything(),
     node_unpartitioned, sync);
 
  // Nodes should now be unpartitioned iff they are truly new; those
  // are the *only* nodes we will touch.
#ifdef DEBUG
  MeshTools::libmesh_assert_parallel_consistent_new_node_procids(mesh);
#endif
 
  NodeWasNew node_was_new(mesh);
 
  // Set the lowest processor id we can on truly new nodes
  for (auto & elem : mesh.element_ptr_range())
    for (auto & node : elem->node_ref_range())
      if (node_was_new.was_new.count(&node))
        {
          processor_id_type & pid = node.processor_id();
          pid = std::min(pid, elem->processor_id());
        }
 
  // Then finally see if other processors have a lower option
  Parallel::sync_node_data_by_element_id
    (mesh, mesh.elements_begin(), mesh.elements_end(),
     ElemNodesMaybeNew(), node_was_new, sync);
 
  // We should have consistent processor ids when we're done.
#ifdef DEBUG
  MeshTools::libmesh_assert_parallel_consistent_procids<Node>(mesh);
  MeshTools::libmesh_assert_parallel_consistent_new_node_procids(mesh);
#endif
}

References libMesh::ParallelObject::comm(), libMesh::MeshBase::element_ptr_range(), libMesh::MeshBase::elements_begin(), libMesh::MeshBase::elements_end(), libMesh::DofObject::invalid_processor_id, libMesh::MeshTools::libmesh_assert_parallel_consistent_new_node_procids(), libMesh::MeshTools::libmesh_assert_parallel_consistent_procids< Node >(), mesh, libMesh::Elem::node_ref(), libMesh::Elem::node_ref_range(), libMesh::MeshBase::not_local_elements_begin(), libMesh::MeshBase::not_local_elements_end(), libMesh::DofObject::processor_id(), libMesh::Parallel::sync_node_data_by_element_id(), and libMesh::Parallel::sync_node_data_by_element_id_once().

Referenced by make_new_nodes_parallel_consistent().

make_new_nodes_parallel_consistent()

void libMesh::MeshCommunication::make_new_nodes_parallel_consistent

(

MeshBase &

mesh

)

Copy processor_ids and ids on new nodes from their local processors.

Definition at line 1813 of file mesh_communication.C.

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // All nodes should have the exact same physical location on every
  // processor where they exist.
  //
  // Local nodes should have unique authoritative ids,
  // and new nodes should be unpartitioned.
  //
  // New ghost nodes touching local elements should be unpartitioned.
  //
  // New ghost nodes should have ids which are either already correct
  // or which are in the "unpartitioned" id space.
  //
  // Non-new nodes should have correct ids and processor ids already.
 
  // First, let's sync up new nodes' processor ids.
 
  this->make_new_node_proc_ids_parallel_consistent(mesh);
 
  // Second, sync up dofobject ids.
  this->make_node_ids_parallel_consistent(mesh);
 
  // Third, sync up dofobject unique_ids if applicable.
  this->make_node_unique_ids_parallel_consistent(mesh);
 
  // Finally, correct the processor ids to make DofMap happy
  MeshTools::correct_node_proc_ids(mesh);
}

References libMesh::ParallelObject::comm(), libMesh::MeshTools::correct_node_proc_ids(), make_new_node_proc_ids_parallel_consistent(), make_node_ids_parallel_consistent(), make_node_unique_ids_parallel_consistent(), and mesh.

Referenced by libMesh::MeshRefinement::_refine_elements().

make_node_ids_parallel_consistent()

void libMesh::MeshCommunication::make_node_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all other ids parallel consistent.

Definition at line 1460 of file mesh_communication.C.

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
  // We need to agree on which processor owns every node, but we can't
  // easily assert that here because we don't currently agree on which
  // id every node has, and some of our temporary ids on unrelated
  // nodes will "overlap".
//#ifdef DEBUG
//  MeshTools::libmesh_assert_parallel_consistent_procids<Node> (mesh);
//#endif // DEBUG
 
  LOG_SCOPE ("make_node_ids_parallel_consistent()", "MeshCommunication");
 
  SyncNodeIds syncids(mesh);
  Parallel::sync_node_data_by_element_id
    (mesh, mesh.elements_begin(), mesh.elements_end(),
     Parallel::SyncEverything(), Parallel::SyncEverything(), syncids);
 
  // At this point, with both ids and processor ids synced, we can
  // finally check for topological consistency of node processor ids.
#ifdef DEBUG
  MeshTools::libmesh_assert_topology_consistent_procids<Node> (mesh);
#endif
}

References libMesh::ParallelObject::comm(), libMesh::MeshBase::elements_begin(), libMesh::MeshBase::elements_end(), libMesh::MeshTools::libmesh_assert_topology_consistent_procids< Node >(), mesh, and libMesh::Parallel::sync_node_data_by_element_id().

Referenced by make_new_nodes_parallel_consistent(), and make_nodes_parallel_consistent().

make_node_proc_ids_parallel_consistent()

void libMesh::MeshCommunication::make_node_proc_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes all other processor ids parallel consistent as well.

Definition at line 1664 of file mesh_communication.C.

{
  LOG_SCOPE ("make_node_proc_ids_parallel_consistent()", "MeshCommunication");
 
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // All nodes should have the exact same physical location on every
  // processor where they exist.
  //
  // Local nodes should have unique authoritative ids,
  // and processor ids consistent with all processors which own
  // an element touching them.
  //
  // Ghost nodes touching local elements should have processor ids
  // consistent with all processors which own an element touching
  // them.
  SyncProcIds sync(mesh);
  Parallel::sync_node_data_by_element_id
    (mesh, mesh.elements_begin(), mesh.elements_end(),
     Parallel::SyncEverything(), Parallel::SyncEverything(), sync);
}

References libMesh::ParallelObject::comm(), libMesh::MeshBase::elements_begin(), libMesh::MeshBase::elements_end(), mesh, and libMesh::Parallel::sync_node_data_by_element_id().

Referenced by make_nodes_parallel_consistent().

make_node_unique_ids_parallel_consistent()

void libMesh::MeshCommunication::make_node_unique_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all unique_ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all ghost unique_ids parallel consistent.

Definition at line 1489 of file mesh_communication.C.

{
  // Avoid unused variable warnings if unique ids aren't enabled.
  libmesh_ignore(mesh);
 
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
#ifdef LIBMESH_ENABLE_UNIQUE_ID
  LOG_SCOPE ("make_node_unique_ids_parallel_consistent()", "MeshCommunication");
 
  SyncUniqueIds<Node> syncuniqueids(mesh, &MeshBase::query_node_ptr);
  Parallel::sync_dofobject_data_by_id(mesh.comm(),
                                      mesh.nodes_begin(),
                                      mesh.nodes_end(),
                                      syncuniqueids);
 
#endif
}

References libMesh::ParallelObject::comm(), libMesh::libmesh_ignore(), mesh, libMesh::MeshBase::nodes_begin(), libMesh::MeshBase::nodes_end(), libMesh::MeshBase::query_node_ptr(), and libMesh::Parallel::sync_dofobject_data_by_id().

Referenced by libMesh::BoundaryInfo::add_elements(), make_new_nodes_parallel_consistent(), make_nodes_parallel_consistent(), and libMesh::Nemesis_IO::read().

make_nodes_parallel_consistent()

void libMesh::MeshCommunication::make_nodes_parallel_consistent

(

MeshBase &

mesh

)

Copy processor_ids and ids on ghost nodes from their local processors.

This is useful for code which wants to add nodes to a distributed mesh.

Definition at line 1771 of file mesh_communication.C.

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // All nodes should have the exact same physical location on every
  // processor where they exist.
  //
  // Local nodes should have unique authoritative ids,
  // and processor ids consistent with all processors which own
  // an element touching them.
  //
  // Ghost nodes touching local elements should have processor ids
  // consistent with all processors which own an element touching
  // them.
  //
  // Ghost nodes should have ids which are either already correct
  // or which are in the "unpartitioned" id space.
 
  // First, let's sync up processor ids.  Some of these processor ids
  // may be "wrong" from coarsening, but they're right in the sense
  // that they'll tell us who has the authoritative dofobject ids for
  // each node.
 
  this->make_node_proc_ids_parallel_consistent(mesh);
 
  // Second, sync up dofobject ids.
  this->make_node_ids_parallel_consistent(mesh);
 
  // Third, sync up dofobject unique_ids if applicable.
  this->make_node_unique_ids_parallel_consistent(mesh);
 
  // Finally, correct the processor ids to make DofMap happy
  MeshTools::correct_node_proc_ids(mesh);
}

References libMesh::ParallelObject::comm(), libMesh::MeshTools::correct_node_proc_ids(), make_node_ids_parallel_consistent(), make_node_proc_ids_parallel_consistent(), make_node_unique_ids_parallel_consistent(), and mesh.

Referenced by libMesh::MeshRefinement::_coarsen_elements(), and libMesh::MeshTools::Modification::all_tri().

make_p_levels_parallel_consistent()

void libMesh::MeshCommunication::make_p_levels_parallel_consistent

(

MeshBase &

mesh

)

Copy p levels of ghost elements from their local processors.

Definition at line 1537 of file mesh_communication.C.

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());
 
  LOG_SCOPE ("make_p_levels_parallel_consistent()", "MeshCommunication");
 
  SyncPLevels syncplevels(mesh);
  Parallel::sync_dofobject_data_by_id
    (mesh.comm(), mesh.elements_begin(), mesh.elements_end(),
     syncplevels);
}

References libMesh::ParallelObject::comm(), libMesh::MeshBase::elements_begin(), libMesh::MeshBase::elements_end(), mesh, and libMesh::Parallel::sync_dofobject_data_by_id().

Referenced by libMesh::MeshRefinement::_coarsen_elements(), and libMesh::MeshRefinement::_refine_elements().

redistribute()

void libMesh::MeshCommunication::redistribute	(	DistributedMesh &	mesh,
		bool	newly_coarsened_only = false
	)		const

This method takes a parallel distributed mesh and redistributes the elements.

Specifically, any elements stored on a given processor are sent to the processor which "owns" them. Similarly, any elements assigned to the current processor but stored on another are received. Once this step is completed any required ghost elements are updated. The final result is that each processor stores only the elements it actually owns and any ghost elements required to satisfy data dependencies. This method can be invoked after a partitioning step to affect the new partitioning.

Redistribution can also be done with newly coarsened elements' neighbors only.

Definition at line 285 of file mesh_communication.C.

{
  // no MPI == one processor, no redistribution
  return;
}

Referenced by libMesh::DistributedMesh::redistribute().

send_coarse_ghosts()

void libMesh::MeshCommunication::send_coarse_ghosts

(

MeshBase &

mesh

)

const

Examine a just-coarsened mesh, and for any newly-coarsened elements, send the associated ghosted elements to the processor which needs them.

Definition at line 915 of file mesh_communication.C.

{
  // no MPI == one processor, no need for this method...
  return;
}

Referenced by libMesh::MeshRefinement::_coarsen_elements().

---

The documentation for this class was generated from the following files:

• include/mesh/mesh_communication.h
• src/mesh/mesh_communication.C
• src/mesh/mesh_communication_global_indices.C