קg/%ddlZddlmZmZddlZgdZejjejjejj hZ ej jj j ej jj jej jj jhZe eZddZddZdefdZdZd Zdd Zdd Zdd ZdZddZdZdZddZ ddZ!dS)N)AnyDict)set_module_weightset_module_biashas_biasget_module_weightget_module_bias max_over_ndim min_over_ndim channel_rangeget_name_by_modulecross_layer_equalization#process_paired_modules_list_to_name$expand_groups_in_paired_modules_listequalize convergedreturnct|tvr&tj||_dStj||d_dSNr)type_supported_typestorchnn Parameterweight)modulers [/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/ao/quantization/_equalize.pyrr"sQ F||'''**622  8--f55q ct|tvr&tj||_dStj||d_dSr)rrrrrbias)rr s rrr)sN F||'''h((.. ++D11q rc^t|tvr |jduS|djduSrrrr rs rrr0s2 F||'''{$&&ay~T))rcVt|tvr|jS|djSr)rrrr#s rrr7s) F||'''}ayrcVt|tvr|jS|djSrr"r#s rr r >s( F||'''{ay~rFcn|d|D]}|||\}}|S)z&Apply 'torch.max' over the given axes.Treverse)sortmaxinput axis_listkeepdimaxis_s rr r EC NN4N   ,,99T7++qq Lrcn|d|D]}|||\}}|S)z&Apply 'torch.min' over the given axes.Tr')r)minr+s rr r Mr1rc(|j}tt|}||t ||}t ||}|d||ks Jd||z S)z=Find the range of weights associated with a specific channel.rzKDimensions of resultant channel range does not match size of requested axis)ndimlistrangeremover r size)r,r/size_of_tensor_dimr-minsmaxss rr r UsU-..//I T  * *D  * *D 99Q<<5::    T    $;rcd|D] \}}||ur|cStd)zGet the name of a module within a model. Args: model: a model (nn.module) that equalization is to be applied on module: a module within the model Returns: name: the name of the module within the model zmodule is not in the model) named_modules ValueError)modelrnamems rr r dsI&&((a ;;KKK  1 2 22rct|tvst|tvr,tdt|dt|t|}d}t |}t |}||||krt d|rt|}t||}t||} | dz } tj || z } tj | } |r|| z}dg|j z} ||| |<dg|j z} ||| |<tj | | } tj | | } || z}|| z}t|||rt||t||dS)zScale the range of Tensor1.output to equal Tensor2.input. Given two adjacent tensors', the weights are scaled such that the ranges of the first tensors' output channel are equal to the ranges of the second tensors' input channel zmodule type not supported: Nz_Number of output channels of first arg do not match number input channels of second argg& .>rC)r_all_supported_typesr?rrr9 TypeErrorr r rsqrt reciprocalr5reshaperr)module1module2 output_axis input_axisconv1_has_biasr weight1weight2 weight1_range weight2_rangescaling_factorsinverse_scaling_factorssize1size2s rrrts W 111 == 4 4 4 ($w--d7mm   g&&N D((G((G||K  GLL$<$<<< -   (w''!';77M!':66MTMj!>??O#.??.--C', E k22E+ C', E Z00E*mOU;;O#m,CUKK//G'Ggw''''&&&gw'''''rc|D]n}t|D]\\}}t|tjjrt ||||<8t|t std]o|S)zHProcesses a list of paired modules to a list of names of paired modules.z$item must be a nn.Module or a string) enumerate isinstancerrModuler strrG)r@paired_modules_listgroupiitems rrrs%HH '' H HGAt$00 H-eT::ac** H FGGG H H rc`g}|D]}t|dkrtdt|dkr||Mt|dkrHtt|dz D](}|||||dzg)|S)zFExpands module pair groups larger than two into groups of two modules.rCz$Group must have at least two modules)lenr?appendr7)r]new_listr^r_s rrrsH$:: u::??CDD D ZZ1__ OOE " " " " ZZ!^^3u::>** : :q5Q< 89999 Or-C6?Tc"t||}|stj|}t|}i}i}d|D}|D]\}}||vr |||<d||<t |||s|D]|} tj|| d|| d<tj|| d|| d<t || d|| d}t ||||S)aEqualize modules until convergence is achieved. Given a list of adjacent modules within a model, equalization will be applied between each pair, this will repeated until convergence is achieved Keeps a copy of the changing modules from the previous iteration, if the copies are not that different than the current modules (determined by converged_test), then the modules have converged enough that further equalizing is not necessary Reference is section 4.1 of this paper https://arxiv.org/pdf/1906.04721.pdf Args: model: a model (nn.Module) that equalization is to be applied on paired_modules_list (List(List[nn.module || str])): a list of lists where each sublist is a pair of two submodules found in the model, for each pair the two modules have to be adjacent in the model, with only piece-wise-linear functions like a (P)ReLU or LeakyReLU in between to get expected results. The list can contain either modules, or names of modules in the model. If you pass multiple modules in the same list, they will all be equalized together. threshold (float): a number used by the converged function to determine what degree of similarity between models is necessary for them to be called equivalent inplace (bool): determines if function is inplace or not ch|] }|D]}| Sri).0pairrAs r zequalize..s%FFFFFFFFFrNrrC)rcopydeepcopyrr>rr) r@r] thresholdinplacename_to_moduleprevious_name_to_modulename_setrArrks rrrsS4> " % e$$>?RSS13N.0FF!4FFFH++--11 f 8  #)N4 ,0 #D )(?KKW' W WD/3}^DQRG=T/U/U #DG ,/3}^DQRG=T/U/U #DG , $^DG%U V V V V (?KKW Lrc||krtdtjd}d|vrdS|D]X}t ||}t ||}||}|tj|z }Yt||kS)aTest whether modules are converged to a specified threshold. Tests for the summed norm of the differences between each set of modules being less than the given threshold Takes two dictionaries mapping names to modules, the set of names for each dictionary should be the same, looping over the set of names, for each name take the difference between the associated modules in each dictionary zIThe keys to the given mappings must have the same set of names of modulesgNF) keysr?rtensorvaluesrsubnormbool) curr_modules prev_modulesro summed_normsrA curr_weight prev_weight differences rrrsl//1111 W   <$$L |""$$$$u!!##//' T(:;; ' T(:;;  __[11  :...  y( ) ))r)rN)F)r)rrC)rfT)rf)"rmtypingrrr__all__rConv2dLinearConv1drao intrinsic ConvReLU2d LinearReLU ConvReLU1d_supported_intrinsic_typesunionrFrrrzrrr r r r r rrrrrrirrrs     "HOUX_ehoF HK$ HK$ HK$ (--.HII66662222*****        3 3 3 9(9(9(9(x       2222j******r