o hV$@sddlZddlmZddlmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZmZdZGd d d e Zd ed ed efddZGdddZeed dde jdeeej jd e jfddZde jd efddZdS)N)Iterable)AnyNoReturnOptional)_State)DistributedDataParallel) _get_registrycontractc seZdZUejed<dfdd Zefdej de ej de ej d e ddf d d Z dd d Zdej de ej ddfddZdddZdddZdej deedfdee efdefddZdej deejdejdejfddZZS)_ReplicateState _ddp_weakrefreturnNcsNtt|_d|_t|_|j|_g|_d|_ d|_ i|_ g|_ dS)NF) super__init__nnZ ParameterListmodulehas_initialized _param_listZ _orig_module _param_names_no_sync _init_args _init_kwargs_comm_hook_argsself __class__m/home/www/facesmatcher.com/frenv_anti/lib/python3.10/site-packages/torch/distributed/_composable/replicate.pyrs    z_ReplicateState.__init__rignored_modulesignored_paramsprefixc Cst|rdS||vr dS|tkr|dnt}|jddD]\}}||vr5|j||j||q|D]\}} |j| ||||dq:dS)N.F)Zrecurse)r")_is_fully_sharded_ROOT_MODULE_PREFIXZnamed_parametersrappendrZnamed_children_collect_params) rrr r!r"Zrecurse_prefixnpnameZ child_modulerrrr'"s&  z_ReplicateState._collect_paramscs"tjddfdd}|dS)NT) recursivecs:jdusJjjijd_i_dS)Nr)rinitrregister_comm_hookrrrr _lazy_initCs  z-_ReplicateState.lazy_init.._lazy_init)torchZ_disable_dynamo)rr.rrr lazy_initBs  z_ReplicateState.lazy_initcKs|jrdSd|_||_dd|D}|D] }t|r#||qddlm}|||d||||d|vrd|ddur[|d}t |t j rU|j dkrUd|d <q_|g|d <nd|d <| dt|jfi||_t|jt|j_dS) NTcSsh|] }|D]}|qqSr) parameters).0mr)rrr Xsz'_ReplicateState.init..r)_localize_dtensor)r! device_idcpuZ device_ids)rrmodulesr$updater1%torch.distributed.tensor.parallel.ddpr5r' isinstancer/devicetypepoprr_ddpweakrefref replicatestater )rrr kwargsr! submoduler5r6rrrr,Ms,       z_ReplicateState.initcCs0|jD] \}}|jj|i|q|jdSN)rr?r-clear)rZ comm_argsZ comm_kwargsrrrr-wsz"_ReplicateState.register_comm_hookcOs||_||_dSrF)rrrargsrDrrrrecord_init_args|s z _ReplicateState.record_init_argsrI.rDcCs2|js|jr ||j |j_|jj|i|SrF)rrr0rr?Zrequire_backward_grad_syncZ _pre_forward)rrrIrDrrrforward_pre_hooks  z _ReplicateState.forward_pre_hookinputoutputcCs |j|SrF)r?Z _post_forward)rrrLrMrrrforward_post_hooks z!_ReplicateState.forward_post_hookrN)__name__ __module__ __qualname__r@rA__annotations__rr%rModuleset Parameterstrr'r0r,r-rJtuplerdictrKr/ZTensorrN __classcell__rrrrr sV    *    r rIrDrcOstd)NzGDDP does not support deepcopy. Please use state dict for serialization.)AssertionError)rIrDrrrunimplemented_deepcopysr\c@s0eZdZddZdeddfddZd dd ZdS) DDPcOs"|jd}|j|g|Ri|S)z Override ``__new__`` to remove the DDP class and directly construct the original class for cases like indexing into a container module. )__mro____new__)clsrIrDZorig_clsrrrr`s z DDP.__new__requires_gradient_syncrNcCs| t|_dS)a Sets if the module should sync gradients. This can be used to implement gradient accumulation without communication. Args: requires_gradient_sync (bool): Whether to reduce gradients for the module's parameters. N)rBrCr)rrbrrrset_requires_gradient_syncs zDDP.set_requires_gradient_synccOst|j||fdSrF)rBrCrr&rHrrrr-szDDP.register_comm_hookrO)rPrQrRr`boolrcr-rrrrr]s  r])Z state_clsrr c Ks$tjdd|vrt|dttjfstdt|dt|r'td|dur.i}nt |}t |}|j |j dd|dd}|durjd d lm}||}||krjd d lm}m}| |||||j|j||fi||j} d ti} td | jt| f| } | |_|S)zReplicates a module Args: module (torch.nn.Module): module to replicate Example:: >>> # xdoctest: +REQUIRES(module:torch._C._distributed_c10d) >>> module = nn.Linear(3, 3) >>> replicate(module) ztorch.distributed.replicater6z6Expected device_id to be int or torch.device, but got zGCannot apply `replicate()` on a Module already managed by `fully_shard`NT)Z with_kwargs device_meshr)_mesh_resources)r5_reconstruct_dtensor __deepcopy__r])r/Z_CZ_log_api_usage_oncer;intr< RuntimeErrorr=r$rUrBrCZregister_forward_pre_hookrKgetZtorch.distributed.device_meshrfZ get_root_meshr:r5rgZregister_forward_hookrNrJrr\rPr]) rr rDrCrerfZ root_meshr5rgradctZnew_clsrrrrBs@         rBcCst|}|dur dSd|vS)z+Check if module is marked with fully_shard.NFZ fully_shard)r )rregistryrrrr$sr$rF)r@collections.abcrtypingrrrr/Ztorch.nnrZ#torch.distributed._composable_staterZtorch.nn.parallelrr r r%r r\r]rTrBrdr$rrrrs.     H