0h)NdZddlZddlmcmZddlmZgdZ dZ dZ dZ dZ y) z(Keras Utilities for DTensor related API.N) dtensor_api) alphabetabias depthwise embeddingsgammakernel moving_meanmoving_variance pointwise recurrentcdfd}tjjj|S)aA decorator for injecting layout information to layer.__init__. Layout will be a new param for any of the weights for all the keras layers. Adding the param to all the __init__ method will be a big/duplicated work. This decorator is design to reduce and code duplication and make it easy to add/remove the dtensor feature if needed. Sample usage: ```python class Dense(tf.keras.layer.Layer): @allow_initializer_layout def __init__(self, units, kernel_initializer='zeros', bias_initializer='zeros', **kwargs): super().__init__(**kwargs) d = Dense(units=8, kernel_layout=layout1, bias_layout=layout2) d.kernel_layout == layout1 d.bias_layout == layout2 ``` By adding this annotation, it will: 1. Filter out the kwargs based on some keywords, eg if the 'kernel_initialzer' appears in method signature, then it will try to pop the 'kernel_layout' if it presents. Same for "bias" and "recurrent_kernel", etc. This will make sure the layout related param is not passed to `BaseLayer.__init__`, which will raise error about unexpect keyword args. 2. Set the self.kernel/bias_layout attribute after the `__init__` method is called. TF-Keras framework will use those fields to create weights down the stream. Args: init_method: the `__init__` method of the TF-Keras layer to annotate. Returns: the annotated __init__ method. ctj}i}tD]4}|dz|jvs|j |dzd}|s-|||dz<6|g|i||j D]\}}t |||y)N _initializer_layout)inspect signatureKERAS_VARIABLE_NAMES parameterspopitemssetattr) layer_instanceargskwargsr layout_args variable_namelayoutlayout_param_name init_methods T/var/www/html/engine/venv/lib/python3.12/site-packages/tf_keras/src/dtensor/utils.py_wrap_functionz0allow_initializer_layout.._wrap_functionUs%%k2  2 DM~-1E1EEMI$=tD=CK 9:  D N4T4V4*5):):)< ? % v N$5v > ?targetdecorator_functf __internal__ decoratormake_decoratorr!r#s` r"allow_initializer_layoutr.)s2X?$ ?? $ $ 3 3> 4 r$cdfd}tjjj|S)aInject DTensor mesh information to an object. This is useful for keras object like `Metric` and `Optimizer` which need DTensor mesh to create the weights, but doesn't want to change the current public API interface. This is for temporary usage and eventually the mesh/layout information will be public arguments in the `__init__` method. Sample usage: ```python class Accuracy(tf.keras.metrics.Metric): @inject_mesh def __init__(self, name='accuracy', dtype=None): super().__init__(**kwargs) acc = Accuracy(mesh=mesh) assert acc._mesh == mesh ``` Args: init_method: the `__init__` method of the TF-Keras class to annotate. Returns: the annotated __init__ method. cT|jdd}|||_|g|i|y)Nmesh)r_mesh)instancerrr1r!s r"r#z#inject_mesh.._wrap_functions4zz&$'  !HNH.t.v.r$r%r(r-s` r" inject_meshr4ls1:/ ?? $ $ 3 3> 4 r$c|rGtj|j5||i|}tj||cdddS||i|S#1swYxYw)aInvoke the function with inputs and relayout the result. Args: fn: the function to invoke. layout: if not None, the output of the fn will be relayout with this. *args: positional arguments to be called with fn. **kwargs: keyword arguments to be called with fn. Returns: The output of fn, with potential relayout with the layout specified. N)dtensor default_meshr1relayout)fnrrrresults r"call_with_layoutr;sd  ! !&++ . 4((F##FF3 4 4 t v  4 4s AActjjsytjj}t |ddduS)aCheck whether running with a `Strategy` that is backed by DTensor. In the DTensor based training, all the tensors are in global context, which is different from the local context. Some keras components need to behave differently, e.g. BatchNormalization and SyncBatchNormalization, as well as optimizers. This check will help those layer to branch the logic and keep the correct behavior between different context. Fr2N)r) distribute has_strategy get_strategygetattr)strategys r"running_with_dtensor_strategyrBs? == % % '}}))+H 8Wd +4 77r$)__doc__rtensorflow.compat.v2compatv2r)tf_keras.src.dtensorrr6rr.r4r;rBr$r"rIs5/!!7 @F(V&8r$