libMesh::MeshCommunication Class Reference

This is the MeshCommunication class. More...

#include <mesh_communication.h>

Public Member Functions
	MeshCommunication ()=default
	Constructor. More...
	~MeshCommunication ()=default
	Destructor. More...
void	clear ()
	Clears all data structures and resets to a pristine state. More...
void	broadcast (MeshBase &) const
	Finds all the processors that may contain elements that neighbor my elements. More...
void	redistribute (DistributedMesh &mesh, bool newly_coarsened_only=false) const
	This method takes a parallel distributed mesh and redistributes the elements. More...
void	gather_neighboring_elements (DistributedMesh &) const
void	send_coarse_ghosts (MeshBase &) const
	Examine a just-coarsened mesh, and for any newly-coarsened elements, send the associated ghosted elements to the processor which needs them. More...
void	gather (const processor_id_type root_id, MeshBase &) const
	This method takes an input DistributedMesh which may be distributed among all the processors. More...
void	allgather (MeshBase &mesh) const
	This method takes an input DistributedMesh which may be distributed among all the processors. More...
void	delete_remote_elements (DistributedMesh &, const std::set< Elem *> &) const
	This method takes an input DistributedMesh which may be distributed among all the processors. More...
void	assign_global_indices (MeshBase &) const
	This method assigns globally unique, partition-agnostic indices to the nodes and elements in the mesh. More...
void	check_for_duplicate_global_indices (MeshBase &) const
	Throw an error if we have any index clashes in the numbering used by assign_global_indices. More...
template<typename ForwardIterator >
void	find_local_indices (const libMesh::BoundingBox &, const ForwardIterator &, const ForwardIterator &, std::unordered_map< dof_id_type, dof_id_type > &) const
	This method determines a locally unique, contiguous index for each object in the input range. More...
template<typename ForwardIterator >
void	find_global_indices (const Parallel::Communicator &communicator, const libMesh::BoundingBox &, const ForwardIterator &, const ForwardIterator &, std::vector< dof_id_type > &) const
	This method determines a globally unique, partition-agnostic index for each object in the input range. More...
void	make_elems_parallel_consistent (MeshBase &)
	Copy ids of ghost elements from their local processors. More...
void	make_p_levels_parallel_consistent (MeshBase &)
	Copy p levels of ghost elements from their local processors. More...
void	make_node_ids_parallel_consistent (MeshBase &)
	Assuming all ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all other ids parallel consistent. More...
void	make_node_unique_ids_parallel_consistent (MeshBase &)
	Assuming all unique_ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all ghost unique_ids parallel consistent. More...
void	make_node_bcids_parallel_consistent (MeshBase &)
	Assuming all processor ids are parallel consistent, this function makes all ghost boundary ids on nodes parallel consistent. More...
void	make_node_proc_ids_parallel_consistent (MeshBase &)
	Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes all other processor ids parallel consistent as well. More...
void	make_new_node_proc_ids_parallel_consistent (MeshBase &)
	Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes processor ids on new nodes on other processors parallel consistent as well. More...
void	make_nodes_parallel_consistent (MeshBase &)
	Copy processor_ids and ids on ghost nodes from their local processors. More...
void	make_new_nodes_parallel_consistent (MeshBase &)
	Copy processor_ids and ids on new nodes from their local processors. More...

Detailed Description

This is the MeshCommunication class.

It handles all the details of communicating mesh information from one processor to another. All parallelization of the Mesh data structures is done via this class.

Author

Benjamin S. Kirk

Date

2003

Definition at line 53 of file mesh_communication.h.

Constructor & Destructor Documentation

MeshCommunication()

libMesh::MeshCommunication::MeshCommunication ( )

default

Constructor.

~MeshCommunication()

libMesh::MeshCommunication::~MeshCommunication ( )

default

Destructor.

Member Function Documentation

allgather()

void libMesh::MeshCommunication::allgather ( MeshBase &  mesh ) const

inline

This method takes an input DistributedMesh which may be distributed among all the processors.

Each processor then sends its local nodes and elements to the other processors. The end result is that a previously distributed DistributedMesh will be serialized on each processor. Since this method is collective it must be called by all processors.

Definition at line 138 of file mesh_communication.h.

References gather(), libMesh::DofObject::invalid_processor_id, and mesh.

Referenced by libMesh::Nemesis_IO::read().

  { MeshCommunication::gather(DofObject::invalid_processor_id, mesh); }

assign_global_indices()

void libMesh::MeshCommunication::assign_global_indices

(

MeshBase &

mesh

)

const

This method assigns globally unique, partition-agnostic indices to the nodes and elements in the mesh.

The approach is to compute the Hilbert space-filling curve key and use its value to assign an index in [0,N_global). Since the Hilbert key is unique for each spatial location, two objects occupying the same location will be assigned the same global id. Thus, this method can also be useful for identifying duplicate nodes which may occur during parallel refinement.

Definition at line 204 of file mesh_communication_global_indices.C.

References libMesh::Parallel::Sort< KeyType, IdxType >::bin(), libMesh::ParallelObject::comm(), communicator, libMesh::MeshTools::create_nodal_bounding_box(), distance(), libMesh::MeshTools::Generation::Private::idx(), libMesh::libmesh_assert(), libMesh::MeshTools::libmesh_assert_equal_connectivity(), libMesh::MeshTools::libmesh_assert_equal_points(), libMesh::make_range(), mesh, libMesh::MeshBase::n_elem(), libMesh::MeshBase::n_nodes(), libMesh::Threads::parallel_for(), and libMesh::Parallel::Sort< KeyType, IdxType >::sort().

Referenced by libMesh::MeshTools::Private::globally_renumber_nodes_and_elements().

{
  LOG_SCOPE ("assign_global_indices()", "MeshCommunication");

  // This method determines partition-agnostic global indices
  // for nodes and elements.

  // Algorithm:
  // (1) compute the Hilbert key for each local node/element
  // (2) perform a parallel sort of the Hilbert key
  // (3) get the min/max value on each processor
  // (4) determine the position in the global ranking for
  //     each local object

  const Parallel::Communicator & communicator (mesh.comm());

#ifdef DEBUG
  // This is going to be a mess if geometry is out of sync
  MeshTools::libmesh_assert_equal_points(mesh);
  MeshTools::libmesh_assert_equal_connectivity(mesh);
#endif

  // Global bounding box.  We choose the nodal bounding box for
  // backwards compatibility; the element bounding box may be looser
  // on curved elements.
  const BoundingBox bbox =
    MeshTools::create_nodal_bounding_box (mesh);

  const Point bboxinv = invert_bbox(bbox);

  //-------------------------------------------------------------
  // (1) compute Hilbert keys
  std::vector<Parallel::DofObjectKey>
    node_keys, elem_keys;

  {
    // Nodes first
    {
      ConstNodeRange nr (mesh.local_nodes_begin(),
                         mesh.local_nodes_end());
      node_keys.resize (nr.size());
      Threads::parallel_for (nr, ComputeHilbertKeys (bbox, bboxinv, node_keys));

      // // It's O(N^2) to check that these keys don't duplicate before the
      // // sort...
      //
      // MeshBase::const_node_iterator nodei = mesh.local_nodes_begin();
      // for (std::size_t i = 0; i != node_keys.size(); ++i, ++nodei)
      //   {
      //     MeshBase::const_node_iterator nodej = mesh.local_nodes_begin();
      //     for (std::size_t j = 0; j != i; ++j, ++nodej)
      //       {
      //         if (node_keys[i] == node_keys[j])
      //           {
      //             CFixBitVec icoords[3], jcoords[3];
      //             get_hilbert_coords(**nodej, bbox, bboxinv, jcoords);
      //             libMesh::err << "node " << (*nodej)->id() << ", " << static_cast<Point &>(**nodej) << " has HilbertIndices " << node_keys[j] << std::endl;
      //             get_hilbert_coords(**nodei, bbox, bboxinv, icoords);
      //             libMesh::err << "node " << (*nodei)->id() << ", " << static_cast<Point &>(**nodei) << " has HilbertIndices " << node_keys[i] << std::endl;
      //             libmesh_error_msg("Error: nodes with duplicate Hilbert keys!");
      //           }
      //       }
      //   }
    }

    // Elements next
    {
      ConstElemRange er (mesh.local_elements_begin(),
                         mesh.local_elements_end());
      elem_keys.resize (er.size());
      Threads::parallel_for (er, ComputeHilbertKeys (bbox, bboxinv, elem_keys));

      // // For elements, the keys can be (and in the case of TRI, are
      // // expected to be) duplicates, but only if the elements are at
      // // different levels
      // MeshBase::const_element_iterator elemi = mesh.local_elements_begin();
      // for (std::size_t i = 0; i != elem_keys.size(); ++i, ++elemi)
      //   {
      //     MeshBase::const_element_iterator elemj = mesh.local_elements_begin();
      //     for (std::size_t j = 0; j != i; ++j, ++elemj)
      //       {
      //         if ((elem_keys[i] == elem_keys[j]) &&
      //             ((*elemi)->level() == (*elemj)->level()))
      //           {
      //             libMesh::err << "level " << (*elemj)->level()
      //                          << " elem\n" << (**elemj)
      //                          << " vertex average " << (*elemj)->vertex_average()
      //                          << " has HilbertIndices " << elem_keys[j]
      //                          << " or " << get_dofobject_key((**elemj), bbox, bboxinv)
      //                          << std::endl;
      //             libMesh::err << "level " << (*elemi)->level()
      //                          << " elem\n" << (**elemi)
      //                          << " vertex average " << (*elemi)->vertex_average()
      //                          << " has HilbertIndices " << elem_keys[i]
      //                          << " or " << get_dofobject_key((**elemi), bbox, bboxinv)
      //                          << std::endl;
      //             libmesh_error_msg("Error: level " << (*elemi)->level() << " elements with duplicate Hilbert keys!");
      //           }
      //       }
      //  }
    }
  } // done computing Hilbert keys



  //-------------------------------------------------------------
  // (2) parallel sort the Hilbert keys
  Parallel::Sort<Parallel::DofObjectKey> node_sorter (communicator,
                                                      node_keys);
  node_sorter.sort(); /* done with node_keys */ //node_keys.clear();

  const std::vector<Parallel::DofObjectKey> & my_node_bin =
    node_sorter.bin();

  Parallel::Sort<Parallel::DofObjectKey> elem_sorter (communicator,
                                                      elem_keys);
  elem_sorter.sort(); /* done with elem_keys */ //elem_keys.clear();

  const std::vector<Parallel::DofObjectKey> & my_elem_bin =
    elem_sorter.bin();



  //-------------------------------------------------------------
  // (3) get the max value on each processor
  std::vector<Parallel::DofObjectKey>
    node_upper_bounds(communicator.size()),
    elem_upper_bounds(communicator.size());

  { // limit scope of temporaries
    std::vector<Parallel::DofObjectKey> recvbuf(2*communicator.size());
    std::vector<unsigned short int> /* do not use a vector of bools here since it is not always so! */
      empty_nodes (communicator.size()),
      empty_elem  (communicator.size());
    std::vector<Parallel::DofObjectKey> my_max(2);

    communicator.allgather (static_cast<unsigned short int>(my_node_bin.empty()), empty_nodes);
    communicator.allgather (static_cast<unsigned short int>(my_elem_bin.empty()),  empty_elem);

    if (!my_node_bin.empty()) my_max[0] = my_node_bin.back();
    if (!my_elem_bin.empty()) my_max[1] = my_elem_bin.back();

    communicator.allgather (my_max, /* identical_buffer_sizes = */ true);

    // Be careful here.  The *_upper_bounds will be used to find the processor
    // a given object belongs to.  So, if a processor contains no objects (possible!)
    // then copy the bound from the lower processor id.
    for (auto p : make_range(communicator.size()))
      {
        node_upper_bounds[p] = my_max[2*p+0];
        elem_upper_bounds[p] = my_max[2*p+1];

        if (p > 0) // default hilbert index value is the OK upper bound for processor 0.
          {
            if (empty_nodes[p]) node_upper_bounds[p] = node_upper_bounds[p-1];
            if (empty_elem[p])  elem_upper_bounds[p] = elem_upper_bounds[p-1];
          }
      }
  }



  //-------------------------------------------------------------
  // (4) determine the position in the global ranking for
  //     each local object
  {
    //----------------------------------------------
    // Nodes first -- all nodes, not just local ones
    {
      // Request sets to send to each processor
      std::map<processor_id_type, std::vector<Parallel::DofObjectKey>>
        requested_ids;
      // Results to gather from each processor - kept in a map so we
      // do only one loop over nodes after all receives are done.
      std::map<dof_id_type, std::vector<dof_id_type>>
        filled_request;

      // build up list of requests
      for (const auto & node : mesh.node_ptr_range())
        {
          libmesh_assert(node);
          const Parallel::DofObjectKey hi =
            get_dofobject_key (*node, bbox, bboxinv);
          const processor_id_type pid =
            cast_int<processor_id_type>
            (std::distance (node_upper_bounds.begin(),
                            std::lower_bound(node_upper_bounds.begin(),
                                             node_upper_bounds.end(),
                                             hi)));

          libmesh_assert_less (pid, communicator.size());

          requested_ids[pid].push_back(hi);
        }

      // The number of objects in my_node_bin on each processor
      std::vector<dof_id_type> node_bin_sizes(communicator.size());
      communicator.allgather (static_cast<dof_id_type>(my_node_bin.size()), node_bin_sizes);

      // The offset of my first global index
      dof_id_type my_offset = 0;
      for (auto pid : make_range(communicator.rank()))
        my_offset += node_bin_sizes[pid];

      auto gather_functor =
        [
#ifndef NDEBUG
         & node_upper_bounds,
         & communicator,
#endif
         & my_node_bin,
         my_offset
        ]
        (processor_id_type,
         const std::vector<Parallel::DofObjectKey> & keys,
         std::vector<dof_id_type> & global_ids)
        {
          // Fill the requests
          const std::size_t keys_size = keys.size();
          global_ids.reserve(keys_size);
          for (std::size_t idx=0; idx != keys_size; idx++)
            {
              const Parallel::DofObjectKey & hi = keys[idx];
              libmesh_assert_less_equal (hi, node_upper_bounds[communicator.rank()]);

              // find the requested index in my node bin
              std::vector<Parallel::DofObjectKey>::const_iterator pos =
                std::lower_bound (my_node_bin.begin(), my_node_bin.end(), hi);
              libmesh_assert (pos != my_node_bin.end());
              libmesh_assert_equal_to (*pos, hi);

              // Finally, assign the global index based off the position of the index
              // in my array, properly offset.
              global_ids.push_back(cast_int<dof_id_type>(std::distance(my_node_bin.begin(), pos) + my_offset));
            }
        };

      auto action_functor =
        [&filled_request]
        (processor_id_type pid,
         const std::vector<Parallel::DofObjectKey> &,
         const std::vector<dof_id_type> & global_ids)
        {
          filled_request[pid] = global_ids;
        };

      // Trade requests with other processors
      const dof_id_type * ex = nullptr;
      Parallel::pull_parallel_vector_data
        (communicator, requested_ids, gather_functor, action_functor, ex);

      // We now have all the filled requests, so we can loop through our
      // nodes once and assign the global index to each one.
      {
        std::map<dof_id_type, std::vector<dof_id_type>::const_iterator>
          next_obj_on_proc;
        for (auto & p : filled_request)
          next_obj_on_proc[p.first] = p.second.begin();

        for (auto & node : mesh.node_ptr_range())
          {
            libmesh_assert(node);
            const Parallel::DofObjectKey hi =
              get_dofobject_key (*node, bbox, bboxinv);
            const processor_id_type pid =
              cast_int<processor_id_type>
              (std::distance (node_upper_bounds.begin(),
                              std::lower_bound(node_upper_bounds.begin(),
                                               node_upper_bounds.end(),
                                               hi)));

            libmesh_assert_less (pid, communicator.size());
            libmesh_assert (next_obj_on_proc[pid] != filled_request[pid].end());

            const dof_id_type global_index = *next_obj_on_proc[pid];
            libmesh_assert_less (global_index, mesh.n_nodes());
            node->set_id() = global_index;

            ++next_obj_on_proc[pid];
          }
      }
    }

    //---------------------------------------------------
    // elements next -- all elements, not just local ones
    {
      // Request sets to send to each processor
      std::map<processor_id_type, std::vector<Parallel::DofObjectKey>>
        requested_ids;
      // Results to gather from each processor - kept in a map so we
      // do only one loop over elements after all receives are done.
      std::map<dof_id_type, std::vector<dof_id_type>>
        filled_request;

      for (const auto & elem : mesh.element_ptr_range())
        {
          libmesh_assert(elem);
          const Parallel::DofObjectKey hi =
            get_dofobject_key (*elem, bbox, bboxinv);
          const processor_id_type pid =
            cast_int<processor_id_type>
            (std::distance (elem_upper_bounds.begin(),
                            std::lower_bound(elem_upper_bounds.begin(),
                                             elem_upper_bounds.end(),
                                             hi)));

          libmesh_assert_less (pid, communicator.size());

          requested_ids[pid].push_back(hi);
        }

      // The number of objects in my_elem_bin on each processor
      std::vector<dof_id_type> elem_bin_sizes(communicator.size());
      communicator.allgather (static_cast<dof_id_type>(my_elem_bin.size()), elem_bin_sizes);

      // The offset of my first global index
      dof_id_type my_offset = 0;
      for (auto pid : make_range(communicator.rank()))
        my_offset += elem_bin_sizes[pid];

      auto gather_functor =
        [
#ifndef NDEBUG
         & elem_upper_bounds,
         & communicator,
#endif
         & my_elem_bin,
         my_offset
        ]
        (processor_id_type,
         const std::vector<Parallel::DofObjectKey> & keys,
         std::vector<dof_id_type> & global_ids)
        {
          // Fill the requests
          const std::size_t keys_size = keys.size();
          global_ids.reserve(keys_size);
          for (std::size_t idx=0; idx != keys_size; idx++)
            {
              const Parallel::DofObjectKey & hi = keys[idx];
              libmesh_assert_less_equal (hi, elem_upper_bounds[communicator.rank()]);

              // find the requested index in my elem bin
              std::vector<Parallel::DofObjectKey>::const_iterator pos =
                std::lower_bound (my_elem_bin.begin(), my_elem_bin.end(), hi);
              libmesh_assert (pos != my_elem_bin.end());
              libmesh_assert_equal_to (*pos, hi);

              // Finally, assign the global index based off the position of the index
              // in my array, properly offset.
              global_ids.push_back (cast_int<dof_id_type>(std::distance(my_elem_bin.begin(), pos) + my_offset));
            }
        };

      auto action_functor =
        [&filled_request]
        (processor_id_type pid,
         const std::vector<Parallel::DofObjectKey> &,
         const std::vector<dof_id_type> & global_ids)
        {
          filled_request[pid] = global_ids;
        };

      // Trade requests with other processors
      const dof_id_type * ex = nullptr;
      Parallel::pull_parallel_vector_data
        (communicator, requested_ids, gather_functor, action_functor, ex);

      // We now have all the filled requests, so we can loop through our
      // elements once and assign the global index to each one.
      {
        std::vector<std::vector<dof_id_type>::const_iterator>
          next_obj_on_proc; next_obj_on_proc.reserve(communicator.size());
        for (auto pid : make_range(communicator.size()))
          next_obj_on_proc.push_back(filled_request[pid].begin());

        for (auto & elem : mesh.element_ptr_range())
          {
            libmesh_assert(elem);
            const Parallel::DofObjectKey hi =
              get_dofobject_key (*elem, bbox, bboxinv);
            const processor_id_type pid =
              cast_int<processor_id_type>
              (std::distance (elem_upper_bounds.begin(),
                              std::lower_bound(elem_upper_bounds.begin(),
                                               elem_upper_bounds.end(),
                                               hi)));

            libmesh_assert_less (pid, communicator.size());
            libmesh_assert (next_obj_on_proc[pid] != filled_request[pid].end());

            const dof_id_type global_index = *next_obj_on_proc[pid];
            libmesh_assert_less (global_index, mesh.n_elem());
            elem->set_id() = global_index;

            ++next_obj_on_proc[pid];
          }
      }
    }
  }
}

broadcast()

void libMesh::MeshCommunication::broadcast

(

MeshBase &

mesh

)

const

Finds all the processors that may contain elements that neighbor my elements.

This list is guaranteed to include all processors that border any of my elements, but may include additional ones as well. This method computes bounding boxes for the elements on each processor and checks for overlaps. This method takes a mesh (which is assumed to reside on processor 0) and broadcasts it to all the other processors. It also broadcasts any boundary information the mesh has associated with it.

Definition at line 1177 of file mesh_communication.C.

Referenced by main(), libMesh::NameBasedIO::read(), libMesh::CheckpointIO::read(), ExodusTest< elem_type >::test_read_gold(), ExodusTest< elem_type >::test_write(), MeshInputTest::testAbaqusRead(), MeshInputTest::testCopyElementSolutionImpl(), MeshInputTest::testCopyElementVectorImpl(), MeshInputTest::testCopyNodalSolutionImpl(), MeshInputTest::testDynaFileMappings(), MeshInputTest::testDynaNoSplines(), MeshInputTest::testDynaReadElem(), MeshInputTest::testDynaReadPatch(), MeshInputTest::testExodusFileMappings(), MeshInputTest::testExodusIGASidesets(), MeshInputTest::testExodusWriteElementDataFromDiscontinuousNodalData(), MeshInputTest::testGmshBCIDOverlap(), MeshInputTest::testGoodGmsh(), MeshInputTest::testGoodSTL(), MeshInputTest::testGoodSTLBinary(), MeshInputTest::testLowOrderEdgeBlocks(), MeshInputTest::testTetgenIO(), and libMesh::NetGenMeshInterface::triangulate().

{
  // no MPI == one processor, no need for this method...
  return;
}

check_for_duplicate_global_indices()

void libMesh::MeshCommunication::check_for_duplicate_global_indices

(

MeshBase &

mesh

)

const

Throw an error if we have any index clashes in the numbering used by assign_global_indices.

Definition at line 612 of file mesh_communication_global_indices.C.

References libMesh::MeshTools::create_nodal_bounding_box(), libMesh::err, mesh, and libMesh::Threads::parallel_for().

{
  LOG_SCOPE ("check_for_duplicate_global_indices()", "MeshCommunication");

  // Global bounding box.  We choose the nodal bounding box for
  // backwards compatibility; the element bounding box may be looser
  // on curved elements.
  const BoundingBox bbox =
    MeshTools::create_nodal_bounding_box (mesh);

  const Point bboxinv = invert_bbox(bbox);

  std::vector<Parallel::DofObjectKey>
    node_keys, elem_keys;

  {
    // Nodes first
    {
      ConstNodeRange nr (mesh.local_nodes_begin(),
                         mesh.local_nodes_end());
      node_keys.resize (nr.size());
      Threads::parallel_for (nr, ComputeHilbertKeys (bbox, bboxinv, node_keys));

      // It's O(N^2) to check that these keys don't duplicate before the
      // sort...
      MeshBase::const_node_iterator nodei = mesh.local_nodes_begin();
      for (std::size_t i = 0; i != node_keys.size(); ++i, ++nodei)
        {
          MeshBase::const_node_iterator nodej = mesh.local_nodes_begin();
          for (std::size_t j = 0; j != i; ++j, ++nodej)
            {
              if (node_keys[i] == node_keys[j])
                {
                  CFixBitVec icoords[3], jcoords[3];
                  get_hilbert_coords(**nodej, bbox, bboxinv, jcoords);
                  libMesh::err <<
                    "node " << (*nodej)->id() << ", " <<
                    *(const Point *)(*nodej) << " has HilbertIndices " <<
                    node_keys[j] << std::endl;
                  get_hilbert_coords(**nodei, bbox, bboxinv, icoords);
                  libMesh::err <<
                    "node " << (*nodei)->id() << ", " <<
                    *(const Point *)(*nodei) << " has HilbertIndices " <<
                    node_keys[i] << std::endl;
                  libmesh_error_msg("Error: nodes with duplicate Hilbert keys!");
                }
            }
        }
    }

    // Elements next
    {
      ConstElemRange er (mesh.local_elements_begin(),
                         mesh.local_elements_end());
      elem_keys.resize (er.size());
      Threads::parallel_for (er, ComputeHilbertKeys (bbox, bboxinv, elem_keys));

      // For elements, the keys can be (and in the case of TRI, are
      // expected to be) duplicates, but only if the elements are at
      // different levels
      MeshBase::const_element_iterator elemi = mesh.local_elements_begin();
      for (std::size_t i = 0; i != elem_keys.size(); ++i, ++elemi)
        {
          MeshBase::const_element_iterator elemj = mesh.local_elements_begin();
          for (std::size_t j = 0; j != i; ++j, ++elemj)
            {
              if ((elem_keys[i] == elem_keys[j]) &&
                  ((*elemi)->level() == (*elemj)->level()))
                {
                  libMesh::err <<
                    "level " << (*elemj)->level() << " elem\n" <<
                    (**elemj) << " vertex average " <<
                    (*elemj)->vertex_average() << " has HilbertIndices " <<
                    elem_keys[j] << " or " <<
                    get_dofobject_key((**elemj), bbox, bboxinv) <<
                    std::endl;
                  libMesh::err <<
                    "level " << (*elemi)->level() << " elem\n" <<
                    (**elemi) << " vertex average " <<
                    (*elemi)->vertex_average() << " has HilbertIndices " <<
                    elem_keys[i] << " or " <<
                    get_dofobject_key((**elemi), bbox, bboxinv) <<
                    std::endl;
                  libmesh_error_msg("Error: level " << (*elemi)->level() << " elements with duplicate Hilbert keys!");
                }
            }
        }
    }
  } // done checking Hilbert keys
}

clear()

void libMesh::MeshCommunication::clear

(

)

Clears all data structures and resets to a pristine state.

Definition at line 442 of file mesh_communication.C.

{
  //  _neighboring_processors.clear();
}

delete_remote_elements()

void libMesh::MeshCommunication::delete_remote_elements	(	DistributedMesh &	mesh,
		const std::set< Elem *> &	extra_ghost_elem_ids
	)		const

This method takes an input DistributedMesh which may be distributed among all the processors.

Each processor deletes all elements which are neither local elements nor "ghost" elements which touch local elements, and deletes all nodes which are not contained in local or ghost elements. The end result is that a previously serial DistributedMesh will be distributed between processors. Since this method is collective it must be called by all processors.

The std::set is a list of extra elements that you don't want to delete. These will be left on the current processor along with local elements and ghosted neighbors.

Definition at line 2147 of file mesh_communication.C.

References libMesh::Elem::active_family_tree(), libMesh::as_range(), libMesh::MeshBase::clear_point_locator(), libMesh::ParallelObject::comm(), libMesh::connect_children(), libMesh::connect_element_dependencies(), libMesh::MeshBase::delete_elem(), libMesh::MeshBase::delete_node(), libMesh::MeshBase::ghosting_functors_begin(), libMesh::MeshBase::ghosting_functors_end(), libMesh::DofObject::invalid_processor_id, libMesh::MeshBase::is_serial(), libMesh::libmesh_assert(), libMesh::MeshTools::libmesh_assert_valid_constraint_rows(), libMesh::MeshTools::libmesh_assert_valid_refinement_tree(), libMesh::Elem::make_links_to_me_remote(), libMesh::MeshBase::max_elem_id(), libMesh::MeshBase::max_node_id(), mesh, libMesh::MeshBase::n_constraint_rows(), libMesh::MeshTools::n_levels(), libMesh::ParallelObject::processor_id(), libMesh::DofObject::processor_id(), libMesh::query_ghosting_functors(), libMesh::Elem::subactive(), and TIMPI::Communicator::verify().

{
  // The mesh should know it's about to be parallelized
  libmesh_assert (!mesh.is_serial());

  LOG_SCOPE("delete_remote_elements()", "MeshCommunication");

#ifdef DEBUG
  // We expect maximum ids to be in sync so we can use them to size
  // vectors
  libmesh_assert(mesh.comm().verify(mesh.max_node_id()));
  libmesh_assert(mesh.comm().verify(mesh.max_elem_id()));
  const dof_id_type par_max_node_id = mesh.parallel_max_node_id();
  const dof_id_type par_max_elem_id = mesh.parallel_max_elem_id();
  libmesh_assert_equal_to (par_max_node_id, mesh.max_node_id());
  libmesh_assert_equal_to (par_max_elem_id, mesh.max_elem_id());
  const dof_id_type n_constraint_rows = mesh.n_constraint_rows();
#endif

  connected_elem_set_type elements_to_keep;

  // Don't delete elements that we were explicitly told not to
  for (const auto & elem : extra_ghost_elem_ids)
    {
      std::vector<const Elem *> active_family;
#ifdef LIBMESH_ENABLE_AMR
      if (!elem->subactive())
        elem->active_family_tree(active_family);
      else
#endif
        active_family.push_back(elem);

      for (const auto & f : active_family)
        elements_to_keep.insert(f);
    }

  // See which elements we still need to keep ghosted, given that
  // we're keeping local and unpartitioned elements.
  query_ghosting_functors
    (mesh, mesh.processor_id(),
     mesh.active_pid_elements_begin(mesh.processor_id()),
     mesh.active_pid_elements_end(mesh.processor_id()),
     elements_to_keep);
  query_ghosting_functors
    (mesh, DofObject::invalid_processor_id,
     mesh.active_pid_elements_begin(DofObject::invalid_processor_id),
     mesh.active_pid_elements_end(DofObject::invalid_processor_id),
     elements_to_keep);

  // The inactive elements we need to send should have their
  // immediate children present.
  connect_children(mesh, mesh.pid_elements_begin(mesh.processor_id()),
                   mesh.pid_elements_end(mesh.processor_id()),
                   elements_to_keep);
  connect_children(mesh,
                   mesh.pid_elements_begin(DofObject::invalid_processor_id),
                   mesh.pid_elements_end(DofObject::invalid_processor_id),
                   elements_to_keep);

  // And see which elements and nodes they depend on
  connected_node_set_type connected_nodes;
  connect_element_dependencies(mesh, elements_to_keep, connected_nodes);

  // Delete all the elements we have no reason to save,
  // starting with the most refined so that the mesh
  // is valid at all intermediate steps
  unsigned int n_levels = MeshTools::n_levels(mesh);

  for (int l = n_levels - 1; l >= 0; --l)
    for (auto & elem : as_range(mesh.level_elements_begin(l),
                                mesh.level_elements_end(l)))
      {
        libmesh_assert (elem);
        // Make sure we don't leave any invalid pointers
        const bool keep_me = elements_to_keep.count(elem);

        if (!keep_me)
          elem->make_links_to_me_remote();

        // delete_elem doesn't currently invalidate element
        // iterators... that had better not change
        if (!keep_me)
          mesh.delete_elem(elem);
      }

  // Delete all the nodes we have no reason to save
  for (auto & node : mesh.node_ptr_range())
    {
      libmesh_assert(node);
      if (!connected_nodes.count(node))
        {
          libmesh_assert_not_equal_to(node->processor_id(),
                                      mesh.processor_id());
          mesh.delete_node(node);
        }
    }

  // If we had a point locator, it's invalid now that some of the
  // elements it pointed to have been deleted.
  mesh.clear_point_locator();

  // We now have all remote elements and nodes deleted; our ghosting
  // functors should be ready to delete any now-redundant cached data
  // they use too.
  for (auto & gf : as_range(mesh.ghosting_functors_begin(), mesh.ghosting_functors_end()))
    gf->delete_remote_elements();

#ifdef DEBUG
  const dof_id_type n_new_constraint_rows = mesh.n_constraint_rows();
  libmesh_assert_equal_to(n_constraint_rows, n_new_constraint_rows);

  MeshTools::libmesh_assert_valid_refinement_tree(mesh);
  MeshTools::libmesh_assert_valid_constraint_rows(mesh);
#endif
}

find_global_indices()

template<typename ForwardIterator >

void libMesh::MeshCommunication::find_global_indices	(	const Parallel::Communicator &	communicator,
		const libMesh::BoundingBox &	bbox,
		const ForwardIterator &	begin,
		const ForwardIterator &	end,
		std::vector< dof_id_type > &	index_map
	)		const

This method determines a globally unique, partition-agnostic index for each object in the input range.

Definition at line 748 of file mesh_communication_global_indices.C.

References libMesh::Parallel::Sort< KeyType, IdxType >::bin(), communicator, distance(), libMesh::MeshTools::Generation::Private::idx(), libMesh::DofObject::invalid_processor_id, libMesh::libmesh_assert(), libMesh::libmesh_ignore(), libMesh::make_range(), libMesh::Real, and libMesh::Parallel::Sort< KeyType, IdxType >::sort().

Referenced by libMesh::ParmetisPartitioner::initialize(), libMesh::MetisPartitioner::partition_range(), and libMesh::Partitioner::partition_unpartitioned_elements().

{
  LOG_SCOPE ("find_global_indices()", "MeshCommunication");

  // This method determines partition-agnostic global indices
  // for nodes and elements.

  // Algorithm:
  // (1) compute the Hilbert key for each local node/element
  // (2) perform a parallel sort of the Hilbert key
  // (3) get the min/max value on each processor
  // (4) determine the position in the global ranking for
  //     each local object
  index_map.clear();
  std::size_t n_objects = std::distance (begin, end);
  index_map.reserve(n_objects);

  const Point bboxinv = invert_bbox(bbox);

  //-------------------------------------------------------------
  // (1) compute Hilbert keys
  // These aren't trivial to compute, and we will need them again.
  // But the binsort will sort the input vector, trashing the order
  // that we'd like to rely on.  So, two vectors...
  std::vector<Parallel::DofObjectKey>
    sorted_hilbert_keys,
    hilbert_keys;
  sorted_hilbert_keys.reserve(n_objects);
  hilbert_keys.reserve(n_objects);
  {
    LOG_SCOPE("compute_hilbert_indices()", "MeshCommunication");
    const processor_id_type my_pid = communicator.rank();
    for (ForwardIterator it=begin; it!=end; ++it)
      {
        const Parallel::DofObjectKey hi(get_dofobject_key (**it, bbox, bboxinv));
        hilbert_keys.push_back(hi);

        const processor_id_type pid = (*it)->processor_id();

        if (pid == my_pid)
          sorted_hilbert_keys.push_back(hi);

        // someone needs to take care of unpartitioned objects!
        if ((my_pid == 0) &&
            (pid == DofObject::invalid_processor_id))
          sorted_hilbert_keys.push_back(hi);
      }
  }

  //-------------------------------------------------------------
  // (2) parallel sort the Hilbert keys
  START_LOG ("parallel_sort()", "MeshCommunication");
  Parallel::Sort<Parallel::DofObjectKey> sorter (communicator,
                                                 sorted_hilbert_keys);
  sorter.sort();
  STOP_LOG ("parallel_sort()", "MeshCommunication");
  const std::vector<Parallel::DofObjectKey> & my_bin = sorter.bin();

  // The number of objects in my_bin on each processor
  std::vector<unsigned int> bin_sizes(communicator.size());
  communicator.allgather (static_cast<unsigned int>(my_bin.size()), bin_sizes);

  // The offset of my first global index
  unsigned int my_offset = 0;
  for (auto pid : make_range(communicator.rank()))
    my_offset += bin_sizes[pid];

  //-------------------------------------------------------------
  // (3) get the max value on each processor
  std::vector<Parallel::DofObjectKey>
    upper_bounds(1);

  if (!my_bin.empty())
    upper_bounds[0] = my_bin.back();

  communicator.allgather (upper_bounds, /* identical_buffer_sizes = */ true);

  // Be careful here.  The *_upper_bounds will be used to find the processor
  // a given object belongs to.  So, if a processor contains no objects (possible!)
  // then copy the bound from the lower processor id.
  for (auto p : IntRange<processor_id_type>(1, communicator.size()))
    if (!bin_sizes[p]) upper_bounds[p] = upper_bounds[p-1];


  //-------------------------------------------------------------
  // (4) determine the position in the global ranking for
  //     each local object
  {
    //----------------------------------------------
    // all objects, not just local ones

    // Request sets to send to each processor
    std::map<processor_id_type, std::vector<Parallel::DofObjectKey>>
      requested_ids;
    // Results to gather from each processor
    std::map<processor_id_type, std::vector<dof_id_type>>
      filled_request;

    // build up list of requests
    std::vector<Parallel::DofObjectKey>::const_iterator hi =
      hilbert_keys.begin();

    for (ForwardIterator it = begin; it != end; ++it)
      {
        libmesh_assert (hi != hilbert_keys.end());

        std::vector<Parallel::DofObjectKey>::iterator lb =
          std::lower_bound(upper_bounds.begin(), upper_bounds.end(),
                           *hi);

        const processor_id_type pid =
          cast_int<processor_id_type>
          (std::distance (upper_bounds.begin(), lb));

        libmesh_assert_less (pid, communicator.size());

        requested_ids[pid].push_back(*hi);

        ++hi;
        // go ahead and put pid in index_map, that way we
        // don't have to repeat the std::lower_bound()
        index_map.push_back(pid);
      }

  auto gather_functor =
    [
#ifndef NDEBUG
     & upper_bounds,
     & communicator,
#endif
     & bbox,
     & my_bin,
       my_offset
    ]
    (processor_id_type, const std::vector<Parallel::DofObjectKey> & keys,
     std::vector<dof_id_type> & global_ids)
    {
      // Ignore unused lambda capture warnings in devel mode
      libmesh_ignore(bbox);

      // Fill the requests
      const std::size_t keys_size = keys.size();
      global_ids.clear();
      global_ids.reserve(keys_size);
      for (std::size_t idx=0; idx != keys_size; idx++)
        {
          const Parallel::DofObjectKey & hilbert_indices = keys[idx];
          libmesh_assert_less_equal (hilbert_indices, upper_bounds[communicator.rank()]);

          // find the requested index in my node bin
          std::vector<Parallel::DofObjectKey>::const_iterator pos =
            std::lower_bound (my_bin.begin(), my_bin.end(), hilbert_indices);
          libmesh_assert (pos != my_bin.end());
#ifdef DEBUG
          // If we could not find the requested Hilbert index in
          // my_bin, something went terribly wrong, possibly the
          // Mesh was displaced differently on different processors,
          // and therefore the Hilbert indices don't agree.
          if (*pos != hilbert_indices)
            {
              // The input will be hilbert_indices.  We convert it
              // to BitVecType using the operator= provided by the
              // BitVecType class. BitVecType is a CBigBitVec!
              Hilbert::BitVecType input;
#ifdef LIBMESH_ENABLE_UNIQUE_ID
              input = hilbert_indices.first;
#else
              input = hilbert_indices;
#endif

              // Get output in a vector of CBigBitVec
              std::vector<CBigBitVec> output(3);

              // Call the indexToCoords function
              Hilbert::indexToCoords(output.data(), 8*sizeof(Hilbert::inttype), 3, input);

              // The entries in the output racks are integers in the
              // range [0, Hilbert::inttype::max] which can be
              // converted to floating point values in [0,1] and
              // finally to actual values using the bounding box.
              const Real max_int_as_real =
                static_cast<Real>(std::numeric_limits<Hilbert::inttype>::max());

              // Get the points in [0,1]^3.  The zeroth rack of each entry in
              // 'output' maps to the normalized x, y, and z locations,
              // respectively.
              Point p_hat(static_cast<Real>(output[0].racks()[0]) / max_int_as_real,
                          static_cast<Real>(output[1].racks()[0]) / max_int_as_real,
                          static_cast<Real>(output[2].racks()[0]) / max_int_as_real);

              // Convert the points from [0,1]^3 to their actual (x,y,z) locations
              Real
                xmin = bbox.first(0),
                xmax = bbox.second(0),
                ymin = bbox.first(1),
                ymax = bbox.second(1),
                zmin = bbox.first(2),
                zmax = bbox.second(2);

              // Convert the points from [0,1]^3 to their actual (x,y,z) locations
              Point p(xmin + (xmax-xmin)*p_hat(0),
                      ymin + (ymax-ymin)*p_hat(1),
                      zmin + (zmax-zmin)*p_hat(2));

              libmesh_error_msg("Could not find hilbert indices: "
                                << hilbert_indices
                                << " corresponding to point " << p);
            }
#endif

          // Finally, assign the global index based off the position of the index
          // in my array, properly offset.
          global_ids.push_back (cast_int<dof_id_type>(std::distance(my_bin.begin(), pos) + my_offset));
        }
    };

  auto action_functor =
    [&filled_request]
    (processor_id_type pid,
     const std::vector<Parallel::DofObjectKey> &,
     const std::vector<dof_id_type> & global_ids)
    {
      filled_request[pid] = global_ids;
    };

  const dof_id_type * ex = nullptr;
  Parallel::pull_parallel_vector_data
    (communicator, requested_ids, gather_functor, action_functor, ex);

    // We now have all the filled requests, so we can loop through our
    // nodes once and assign the global index to each one.
    {
      std::vector<std::vector<dof_id_type>::const_iterator>
        next_obj_on_proc; next_obj_on_proc.reserve(communicator.size());
      for (auto pid : make_range(communicator.size()))
        next_obj_on_proc.push_back(filled_request[pid].begin());

      unsigned int cnt=0;
      for (ForwardIterator it = begin; it != end; ++it, cnt++)
        {
          const processor_id_type pid = cast_int<processor_id_type>
            (index_map[cnt]);

          libmesh_assert_less (pid, communicator.size());
          libmesh_assert (next_obj_on_proc[pid] != filled_request[pid].end());

          const dof_id_type global_index = *next_obj_on_proc[pid];
          index_map[cnt] = global_index;

          ++next_obj_on_proc[pid];
        }
    }
  }

  libmesh_assert_equal_to(index_map.size(), n_objects);
}

find_local_indices()

template<typename ForwardIterator >

void libMesh::MeshCommunication::find_local_indices	(	const libMesh::BoundingBox &	bbox,
		const ForwardIterator &	begin,
		const ForwardIterator &	end,
		std::unordered_map< dof_id_type, dof_id_type > &	index_map
	)		const

This method determines a locally unique, contiguous index for each object in the input range.

Definition at line 710 of file mesh_communication_global_indices.C.

Referenced by libMesh::Partitioner::_find_global_index_by_pid_map().

{
  LOG_SCOPE ("find_local_indices()", "MeshCommunication");

  // This method determines id-agnostic local indices
  // for nodes and elements by sorting Hilbert keys.

  index_map.clear();

  const Point bboxinv = invert_bbox(bbox);

  //-------------------------------------------------------------
  // (1) compute Hilbert keys
  // These aren't trivial to compute, and we will need them again.
  // But the binsort will sort the input vector, trashing the order
  // that we'd like to rely on.  So, two vectors...
  std::map<Parallel::DofObjectKey, dof_id_type> hilbert_keys;
  {
    LOG_SCOPE("local_hilbert_indices", "MeshCommunication");
    for (ForwardIterator it=begin; it!=end; ++it)
      {
        const Parallel::DofObjectKey hi(get_dofobject_key ((**it), bbox, bboxinv));
        hilbert_keys.emplace(hi, (*it)->id());
      }
  }

  {
    dof_id_type cnt = 0;
    for (auto key_val : hilbert_keys)
      index_map[key_val.second] = cnt++;
  }
}

gather()

void libMesh::MeshCommunication::gather	(	const processor_id_type	root_id,
		MeshBase &	mesh
	)		const

This method takes an input DistributedMesh which may be distributed among all the processors.

Each processor then sends its local nodes and elements to processor root_id. The end result is that a previously distributed DistributedMesh will be serialized on processor root_id. Since this method is collective it must be called by all processors. For the special case of root_id equal to DofObject::invalid_processor_id this function performs an allgather.

Definition at line 1318 of file mesh_communication.C.

Referenced by allgather().

{
  // no MPI == one processor, no need for this method...
  return;
}

gather_neighboring_elements()

void libMesh::MeshCommunication::gather_neighboring_elements

(

DistributedMesh &

mesh

)

const

Definition at line 676 of file mesh_communication.C.

Referenced by libMesh::Nemesis_IO::read().

{
  // no MPI == one processor, no need for this method...
  return;
}

make_elems_parallel_consistent()

void libMesh::MeshCommunication::make_elems_parallel_consistent

(

MeshBase &

mesh

)

Copy ids of ghost elements from their local processors.

Definition at line 1802 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), mesh, libMesh::MeshBase::query_elem_ptr(), libMesh::MeshBase::renumber_elem(), libMesh::Parallel::sync_dofobject_data_by_id(), and libMesh::Parallel::sync_element_data_by_parent_id().

Referenced by libMesh::MeshRefinement::_refine_elements().

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  LOG_SCOPE ("make_elems_parallel_consistent()", "MeshCommunication");

  SyncIds syncids(mesh, &MeshBase::renumber_elem);
  Parallel::sync_element_data_by_parent_id
    (mesh, mesh.active_elements_begin(),
     mesh.active_elements_end(), syncids);

#ifdef LIBMESH_ENABLE_UNIQUE_ID
  SyncUniqueIds<Elem> syncuniqueids(mesh, &MeshBase::query_elem_ptr);
  Parallel::sync_dofobject_data_by_id
    (mesh.comm(), mesh.active_elements_begin(),
     mesh.active_elements_end(), syncuniqueids);
#endif
}

make_new_node_proc_ids_parallel_consistent()

void libMesh::MeshCommunication::make_new_node_proc_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes processor ids on new nodes on other processors parallel consistent as well.

Definition at line 1982 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), libMesh::DofObject::invalid_processor_id, libMesh::MeshTools::libmesh_assert_parallel_consistent_new_node_procids(), libMesh::MeshTools::libmesh_assert_parallel_consistent_procids< Node >(), mesh, libMesh::Elem::node_ref(), libMesh::Elem::node_ref_range(), libMesh::DofObject::processor_id(), libMesh::Parallel::sync_node_data_by_element_id(), and libMesh::Parallel::sync_node_data_by_element_id_once().

Referenced by make_new_nodes_parallel_consistent().

{
  LOG_SCOPE ("make_new_node_proc_ids_parallel_consistent()", "MeshCommunication");

  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // Local nodes should have unique authoritative ids,
  // and new nodes should be unpartitioned.
  //
  // New ghost nodes touching local elements should be unpartitioned.

  // We may not have consistent processor ids for new nodes (because a
  // node may be old and partitioned on one processor but new and
  // unpartitioned on another) when we start
#ifdef DEBUG
  MeshTools::libmesh_assert_parallel_consistent_procids<Node>(mesh);
  // MeshTools::libmesh_assert_parallel_consistent_new_node_procids(mesh);
#endif

  // We have two kinds of new nodes.  *NEW* nodes are unpartitioned on
  // all processors: we need to use a id-independent (i.e. dumb)
  // heuristic to partition them.  But "new" nodes are newly created
  // on some processors (when ghost elements are refined) yet
  // correspond to existing nodes on other processors: we need to use
  // the existing processor id for them.
  //
  // A node which is "new" on one processor will be associated with at
  // least one ghost element, and we can just query that ghost
  // element's owner to find out the correct processor id.

  auto node_unpartitioned =
    [](const Elem * elem, unsigned int local_node_num)
    { return elem->node_ref(local_node_num).processor_id() ==
        DofObject::invalid_processor_id; };

  SyncProcIds sync(mesh);

  sync_node_data_by_element_id_once
    (mesh, mesh.not_local_elements_begin(),
     mesh.not_local_elements_end(), Parallel::SyncEverything(),
     node_unpartitioned, sync);

  // Nodes should now be unpartitioned iff they are truly new; those
  // are the *only* nodes we will touch.
#ifdef DEBUG
  MeshTools::libmesh_assert_parallel_consistent_new_node_procids(mesh);
#endif

  NodeWasNew node_was_new(mesh);

  // Set the lowest processor id we can on truly new nodes
  for (auto & elem : mesh.element_ptr_range())
    for (auto & node : elem->node_ref_range())
      if (node_was_new.was_new.count(&node))
        {
          processor_id_type & pid = node.processor_id();
          pid = std::min(pid, elem->processor_id());
        }

  // Then finally see if other processors have a lower option
  Parallel::sync_node_data_by_element_id
    (mesh, mesh.elements_begin(), mesh.elements_end(),
     ElemNodesMaybeNew(), node_was_new, sync);

  // We should have consistent processor ids when we're done.
#ifdef DEBUG
  MeshTools::libmesh_assert_parallel_consistent_procids<Node>(mesh);
  MeshTools::libmesh_assert_parallel_consistent_new_node_procids(mesh);
#endif
}

make_new_nodes_parallel_consistent()

void libMesh::MeshCommunication::make_new_nodes_parallel_consistent

(

MeshBase &

mesh

)

Copy processor_ids and ids on new nodes from their local processors.

Definition at line 2105 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), libMesh::MeshTools::correct_node_proc_ids(), make_new_node_proc_ids_parallel_consistent(), make_node_bcids_parallel_consistent(), make_node_ids_parallel_consistent(), make_node_unique_ids_parallel_consistent(), and mesh.

Referenced by libMesh::MeshRefinement::_refine_elements().

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // All nodes should have the exact same physical location on every
  // processor where they exist.
  //
  // Local nodes should have unique authoritative ids,
  // and new nodes should be unpartitioned.
  //
  // New ghost nodes touching local elements should be unpartitioned.
  //
  // New ghost nodes should have ids which are either already correct
  // or which are in the "unpartitioned" id space.
  //
  // Non-new nodes should have correct ids and processor ids already.

  // First, let's sync up new nodes' processor ids.

  this->make_new_node_proc_ids_parallel_consistent(mesh);

  // Second, sync up dofobject ids.
  this->make_node_ids_parallel_consistent(mesh);

  // Third, sync up dofobject unique_ids if applicable.
  this->make_node_unique_ids_parallel_consistent(mesh);

  // Fourth, sync up any nodal boundary conditions
  this->make_node_bcids_parallel_consistent(mesh);

  // Finally, correct the processor ids to make DofMap happy
  MeshTools::correct_node_proc_ids(mesh);
}

make_node_bcids_parallel_consistent()

void libMesh::MeshCommunication::make_node_bcids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all processor ids are parallel consistent, this function makes all ghost boundary ids on nodes parallel consistent.

Definition at line 1780 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), libMesh::libmesh_ignore(), mesh, and libMesh::Parallel::sync_dofobject_data_by_id().

Referenced by make_new_nodes_parallel_consistent().

{
  // Avoid unused variable warnings if unique ids aren't enabled.
  libmesh_ignore(mesh);

  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  LOG_SCOPE ("make_node_bcids_parallel_consistent()", "MeshCommunication");

  SyncBCIds<Node> syncbcids(mesh);
  Parallel::sync_dofobject_data_by_id(mesh.comm(),
                                      mesh.nodes_begin(),
                                      mesh.nodes_end(),
                                      syncbcids);
}

make_node_ids_parallel_consistent()

void libMesh::MeshCommunication::make_node_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all other ids parallel consistent.

Definition at line 1730 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), libMesh::MeshTools::libmesh_assert_topology_consistent_procids< Node >(), mesh, and libMesh::Parallel::sync_node_data_by_element_id().

Referenced by make_new_nodes_parallel_consistent(), and make_nodes_parallel_consistent().

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  // We need to agree on which processor owns every node, but we can't
  // easily assert that here because we don't currently agree on which
  // id every node has, and some of our temporary ids on unrelated
  // nodes will "overlap".
//#ifdef DEBUG
//  MeshTools::libmesh_assert_parallel_consistent_procids<Node> (mesh);
//#endif // DEBUG

  LOG_SCOPE ("make_node_ids_parallel_consistent()", "MeshCommunication");

  SyncNodeIds syncids(mesh);
  Parallel::sync_node_data_by_element_id
    (mesh, mesh.elements_begin(), mesh.elements_end(),
     Parallel::SyncEverything(), Parallel::SyncEverything(), syncids);

  // At this point, with both ids and processor ids synced, we can
  // finally check for topological consistency of node processor ids.
#ifdef DEBUG
  MeshTools::libmesh_assert_topology_consistent_procids<Node> (mesh);
#endif
}

make_node_proc_ids_parallel_consistent()

void libMesh::MeshCommunication::make_node_proc_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all processor ids on nodes touching local elements are parallel consistent, this function makes all other processor ids parallel consistent as well.

Definition at line 1953 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), mesh, and libMesh::Parallel::sync_node_data_by_element_id().

Referenced by make_nodes_parallel_consistent(), and libMesh::SimplexRefiner::refine_via_edges().

{
  LOG_SCOPE ("make_node_proc_ids_parallel_consistent()", "MeshCommunication");

  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // All nodes should have the exact same physical location on every
  // processor where they exist.
  //
  // Local nodes should have unique authoritative ids,
  // and processor ids consistent with all processors which own
  // an element touching them.
  //
  // Ghost nodes touching local elements should have processor ids
  // consistent with all processors which own an element touching
  // them.
  SyncProcIds sync(mesh);
  Parallel::sync_node_data_by_element_id
    (mesh, mesh.elements_begin(), mesh.elements_end(),
     Parallel::SyncEverything(), Parallel::SyncEverything(), sync);
}

make_node_unique_ids_parallel_consistent()

void libMesh::MeshCommunication::make_node_unique_ids_parallel_consistent

(

MeshBase &

mesh

)

Assuming all unique_ids on local nodes are globally unique, and assuming all processor ids are parallel consistent, this function makes all ghost unique_ids parallel consistent.

Definition at line 1759 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), libMesh::libmesh_ignore(), mesh, libMesh::MeshBase::query_node_ptr(), and libMesh::Parallel::sync_dofobject_data_by_id().

Referenced by libMesh::BoundaryInfo::add_elements(), make_new_nodes_parallel_consistent(), make_nodes_parallel_consistent(), and libMesh::Nemesis_IO::read().

{
  // Avoid unused variable warnings if unique ids aren't enabled.
  libmesh_ignore(mesh);

  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

#ifdef LIBMESH_ENABLE_UNIQUE_ID
  LOG_SCOPE ("make_node_unique_ids_parallel_consistent()", "MeshCommunication");

  SyncUniqueIds<Node> syncuniqueids(mesh, &MeshBase::query_node_ptr);
  Parallel::sync_dofobject_data_by_id(mesh.comm(),
                                      mesh.nodes_begin(),
                                      mesh.nodes_end(),
                                      syncuniqueids);

#endif
}

make_nodes_parallel_consistent()

void libMesh::MeshCommunication::make_nodes_parallel_consistent

(

MeshBase &

mesh

)

Copy processor_ids and ids on ghost nodes from their local processors.

This is useful for code which wants to add nodes to a distributed mesh.

Definition at line 2060 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), libMesh::MeshTools::correct_node_proc_ids(), make_node_ids_parallel_consistent(), make_node_proc_ids_parallel_consistent(), make_node_unique_ids_parallel_consistent(), and mesh.

Referenced by libMesh::MeshRefinement::_coarsen_elements(), and libMesh::MeshTools::Modification::all_tri().

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  // When this function is called, each section of a parallelized mesh
  // should be in the following state:
  //
  // Element ids and locations should be consistent on every processor
  // where they exist.
  //
  // All nodes should have the exact same physical location on every
  // processor where they exist.
  //
  // Local nodes should have unique authoritative ids,
  // and processor ids consistent with all processors which own
  // an element touching them.
  //
  // Ghost nodes touching local elements should have processor ids
  // consistent with all processors which own an element touching
  // them.
  //
  // Ghost nodes should have ids which are either already correct
  // or which are in the "unpartitioned" id space.

  // First, let's sync up processor ids.  Some of these processor ids
  // may be "wrong" from coarsening, but they're right in the sense
  // that they'll tell us who has the authoritative dofobject ids for
  // each node.

  this->make_node_proc_ids_parallel_consistent(mesh);

  // Second, sync up dofobject ids.
  this->make_node_ids_parallel_consistent(mesh);

  // Third, sync up dofobject unique_ids if applicable.
  this->make_node_unique_ids_parallel_consistent(mesh);

  // Finally, correct the processor ids to make DofMap happy
  MeshTools::correct_node_proc_ids(mesh);
}

make_p_levels_parallel_consistent()

void libMesh::MeshCommunication::make_p_levels_parallel_consistent

(

MeshBase &

mesh

)

Copy p levels of ghost elements from their local processors.

Definition at line 1826 of file mesh_communication.C.

References libMesh::ParallelObject::comm(), mesh, and libMesh::Parallel::sync_dofobject_data_by_id().

Referenced by libMesh::MeshRefinement::_coarsen_elements(), and libMesh::MeshRefinement::_refine_elements().

{
  // This function must be run on all processors at once
  libmesh_parallel_only(mesh.comm());

  LOG_SCOPE ("make_p_levels_parallel_consistent()", "MeshCommunication");

  SyncPLevels syncplevels(mesh);
  Parallel::sync_dofobject_data_by_id
    (mesh.comm(), mesh.elements_begin(), mesh.elements_end(),
     syncplevels);
}

redistribute()

void libMesh::MeshCommunication::redistribute	(	DistributedMesh &	mesh,
		bool	newly_coarsened_only = false
	)		const

This method takes a parallel distributed mesh and redistributes the elements.

Specifically, any elements stored on a given processor are sent to the processor which "owns" them. Similarly, any elements assigned to the current processor but stored on another are received. Once this step is completed any required ghost elements are updated. The final result is that each processor stores only the elements it actually owns and any ghost elements required to satisfy data dependencies. This method can be invoked after a partitioning step to affect the new partitioning.

Redistribution can also be done with newly coarsened elements' neighbors only.

Definition at line 451 of file mesh_communication.C.

Referenced by libMesh::DistributedMesh::redistribute().

{
  // no MPI == one processor, no redistribution
  return;
}

send_coarse_ghosts()

void libMesh::MeshCommunication::send_coarse_ghosts

(

MeshBase &

mesh

)

const

Examine a just-coarsened mesh, and for any newly-coarsened elements, send the associated ghosted elements to the processor which needs them.

Definition at line 1055 of file mesh_communication.C.

Referenced by libMesh::MeshRefinement::_coarsen_elements().

{
  // no MPI == one processor, no need for this method...
  return;
}

---

The documentation for this class was generated from the following files:

• include/mesh/mesh_communication.h
• src/mesh/mesh_communication.C
• src/mesh/mesh_communication_global_indices.C