dmlcloud.root_first

dmlcloud.root_first(group=None)

Context manager that ensures that the root rank executes the code first before all other ranks.

This is realized by inserting a barrier before or after the code block depending on the rank. Notice, that only a regular barrier is used, and, hence, the default timeout of 1800000 seconds applies for nccl.