You can try to look at

  

      github.com
  

  
    tensorflow/tensorflow/blob/c35883e15a675767c15b8f3c5ed619bd9e051af4/tensorflow/python/distribute/distribute_lib.py#L16:L25 2



      
      
          #
          #     http://www.apache.org/licenses/LICENSE-2.0
          #
          # Unless required by applicable law or agreed to in writing, software
          # distributed under the License is distributed on an "AS IS" BASIS,
          # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          # See the License for the specific language governing permissions and
          # limitations under the License.
          # ==============================================================================
          # pylint: disable=line-too-long
          """Library for running a computation across multiple devices.
          
          
The intent of this library is that you can write an algorithm in a stylized way
          and it will be usable with a variety of different `tf.distribute.Strategy`
          implementations. Each descendant will implement a different strategy for
          distributing the algorithm across multiple devices/machines.  Furthermore, these
          changes can be hidden inside the specific layers and other library classes that
          need special treatment to run in a distributed setting, so that most users'
          model definition code can run unchanged. The `tf.distribute.Strategy` API works
          the same way with eager and graph execution.