0hhdZddlZddlZddlmcmZdZdZ d dZ d dZ dZ Gdd Z y) zUtils for running models in a distribution setting. Mostly from https://github.com/tensorflow/models/blob/master/official/utils/misc/distribution_utils.py. Nc\tjjjjtjjjj tjjjj d}||vrtdj|||S)aKReturn a CollectiveCommunication based on all_reduce_alg. Args: all_reduce_alg: a string specifying which collective communication to pick, or None. Returns: tf.distribute.experimental.CollectiveCommunication object Raises: ValueError: if `all_reduce_alg` not in [None, "ring", "nccl"] )NringncclzqWhen used with `multi_worker_mirrored`, valid values for all_reduce_alg are [`ring`, `nccl`]. Supplied value: {}) tf distribute experimentalCollectiveCommunicationAUTORINGNCCL ValueErrorformat)all_reduce_alg collective_communication_optionss c/var/www/html/engine/venv/lib/python3.12/site-packages/tf_keras/src/benchmarks/distribution_util.py_collective_communicationrsmm((@@EE **BBGG **BBGG($ == GGMvH  ,N ;;c|ytjjtjjd}||vrt dj |||}||S)aReturn a CrossDeviceOps based on all_reduce_alg and num_packs. Args: all_reduce_alg: a string specifying which cross device op to pick, or None. num_packs: an integer specifying number of packs for the cross device op. Returns: tf.distribute.CrossDeviceOps object or None. Raises: ValueError: if `all_reduce_alg` not in [None, "nccl", "hierarchical_copy"]. N)rhierarchical_copyzqWhen used with `mirrored`, valid values for all_reduce_alg are [`nccl`, `hierarchical_copy`]. Supplied value: {}) num_packs)rr NcclAllReduceHierarchicalCopyAllReducer r)rrmirrored_all_reduce_optionscross_device_ops_classs r_mirrored_cross_device_opsr7ss ++]]DD#88 AAGB  9H !I 66rcz|dkr td|j}|dk(r |dkDrtdj|y|dk(r3tjj j t|S|d k(rS|dk(rtjjd S|dkDr td tjjd S|d k(rO|dk(rd g}nt|Dcgc]}d|z }}tjj|t||Std|cc}w)aKReturn a DistributionStrategy for running the model. Args: distribution_strategy: a string specifying which distribution strategy to use. Accepted values are "off", "one_device", "mirrored", and "multi_worker_mirrored" -- case insensitive. "off" means not to use Distribution Strategy. num_gpus: Number of GPUs to run this model. Returns: tf.distribute.DistibutionStrategy object. Raises: ValueError: if `distribution_strategy` is "off" or "one_device" and `num_gpus` is larger than 1; or `num_gpus` is negative. rz`num_gpus` can not be negative.offzNWhen {} GPUs are specified, distribution_strategy flag cannot be set to `off`.Nmulti_worker_mirrored) communication one_devicez device:CPU:0z=`OneDeviceStrategy` can not be used for more than one device.z device:GPU:0mirroredz device:GPU:%d)devicescross_device_opsz$Unrecognized Distribution Strategy: ) r lowerrrrrMultiWorkerMirroredStrategyrOneDeviceStrategyrangeMirroredStrategyr)distribution_strategynum_gpusrrr#is rget_distribution_strategyr-Ws^*!|:;;1779% a<//5vh/?  77}}))EE3NCF   , q===22>B B a<O }}..~>> * q=%&G49(ODq*DGD}}--7 .    ./D.EF Es1 D8ctjtjj dd}|r?t |dj dgt |dj dgz}|S|rj|j d}t |}|dkDr|dkr td |dk(rdn|}tjd|id|d d tjd<|Sd}|S) zSet multi-worker cluster spec in TF_CONFIG environment variable. Args: worker_hosts: comma-separated list of worker ip:port pairs. Returns: Number of workers in the cluster. TF_CONFIGz{}clusterchiefworker,rrz2Must specify task_index when number of workers > 1)typeindex)r0task) jsonloadsosenvirongetlensplitr dumps) worker_hosts task_index tf_config num_workersworkerss rconfigure_clusterrDs 2::>>+t<=I)I.227B?@3 i $ $Xr 2D  & ! $$S)'l ?zA~D &*Q "&**$g.!)J? #  ;  rcB|r|j}|St}|SN)scopeDummyContextManager)strategystrategy_scopes rget_strategy_scoperKs)!) -. rceZdZdZdZy)rHcyrF)selfs r __enter__zDummyContextManager.__enter__ rcyrFrN)rOargss r__exit__zDummyContextManager.__exit__rQrN)__name__ __module__ __qualname__rPrTrNrrrHrHs   rrH)r"rNr)N) __doc__r7r9tensorflow.compat.v2compatv2rrrr-rDrKrHrNrrr]sL  !!<87B%  >BB  r