gddlZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddZdZdZdeeejfd eee jffd ZGd d eZGd deZd eee jfdeefdZdS)N)Mapping)DictListOptionalUnion) safe_openc"d}t|jdkr!|tj}d}|}tj ||d}|0|t|j}|t|j d||<|j dkr|d}tj||jd|j }|dd|dd<||S)Nztorch.bfloat16bfloat16.dat)dtypeshaperzw+)r moder )strr viewtorchint16cpunumpyospathjoinlistr ndimnpmemmapflush)weight weight_nameoffload_folderindexr array tensor_file file_arrays T/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/utils/offload.pyoffload_weightr%s E 6<,,,U[)) JJLL   E',,~+/C/C/CDDK  = $$E',tEK7H7HIIk zQd ;ekEKXXXJ!!!HJqqqM LcPt|d}|dkrd}|d}|dkrd}tj|||d}t|dd kr|d }t j|}|ddkr|tj}|S) Nr )r r rr)r r rr)tuplerrlenrtensorrr ) weight_file weight_infor r rs r$load_offloaded_weightr0.s +g& ' 'E {{  E  Y{%u3 G G GF ;w   A%% \& ! !F7z))U^,, Mr&c|t|dkrdStj|d}tj|r>t |d5}t j|}dddn #1swxYwYni}||t |dd5}t j ||dddddS#1swxYwYdS)Nr index.jsonzutf-8)encodingw)indent) r,rrrisfileopenjsonloadupdatedump)r roffload_index_filef current_indexs r$save_offload_indexr@Dsd }E anlCC w~~()) $w 7 7 7 )1 IaLLM ) ) ) ) ) ) ) ) ) ) ) ) ) ) )   # 8 8 8.A -1----..................s$(B  B B >C##C'*C'save_dir state_dictctj|di}|D]\}}t||||}t ||dS)a Offload a state dict in a given folder. Args: save_dir (`str` or `os.PathLike`): The directory in which to offload the state dict. state_dict (`Dict[str, torch.Tensor]`): The dictionary of tensors to offload. T)exist_ok)r N)rmakedirsitemsr%r@)rArBr name parameters r$offload_state_dictrIUsoK4(((( E%++--GGiy$FFFuh'''''r&c4eZdZdZdedefdZdZdZdZ dS) PrefixedDatasetz Will access keys in a given dataset by adding a prefix. Args: dataset (`Mapping`): Any map with string keys. prefix (`str`): A prefix to add when trying to access any element in the underlying dataset. datasetprefixc"||_||_dSNrLrM)selfrLrMs r$__init__zPrefixedDataset.__init__qs  r&c.|j|j|SrOrP)rQkeys r$ __getitem__zPrefixedDataset.__getitem__us|t{1C1122r&cDtfdjDS)NcHg|]}|j|Sr() startswithrM.0rTrQs r$ z,PrefixedDataset.__iter__..ys,PPPSCNN4;4O4OPSPPPr&)iterrLrQs`r$__iter__zPrefixedDataset.__iter__xs(PPPPDLPPPQQQr&c*t|jSrO)r,rLr]s r$__len__zPrefixedDataset.__len__{s4<   r&N) __name__ __module__ __qualname____doc__rrrRrUr^r`r(r&r$rKrKhsp333RRR!!!!!r&rKceZdZdZ d deeejfdee ee j fde fdZ defdZd Zd ZdS) OffloadedWeightsLoaderaC A collection that loads weights stored in a given state dict or memory-mapped on disk. Args: state_dict (`Dict[str, torch.Tensor]`, *optional*): A dictionary parameter name to tensor. save_folder (`str` or `os.PathLike`, *optional*): The directory in which the weights are stored (by `offload_state_dict` for instance). index (`Dict`, *optional*): A dictionary from weight name to their information (`dtype`/ `shape` or safetensors filename). Will default to the index saved in `save_folder`. NrB save_folderr c|||td|in|_|_|[|Yttj|d5}tj|}dddn #1swxYwY|in|_ tj _ j fdj D|_dS)NzWNeed either a `state_dict`, a `save_folder` or an `index` containing offloaded weights.r2c&g|] }|jv |Sr()all_keysrYs r$r[z3OffloadedWeightsLoader.__init__..s%TTTc3dm;S;Sc;S;S;Sr&) ValueErrorrBrgr8rrrr9r:r rkeysrjextenddevice)rQrBrgr rnr>s` r$rRzOffloadedWeightsLoader.__init__s&  +"5%-vww w * 2"" & =[4bgll; ==>> %! !  % % % % % % % % % % % % % % % =RRe T_113344  TTTTTZTTTUUU sA;;A?A?rTc\||jvr |j|S|j|}|d<|jdn|j}d} t |dd|5}||d|}dddn #1swxYwYni#t $r\t |ddd5}||d|}dddn #1swxYwYYnwxYwd|vr.|tt|d}|jtj|kr||}|Stj |j |d}t||S)Nsafetensors_filerpt) frameworkrnrr r )rBr getrnr get_tensor TypeErrortogetattrrrrrrgr0)rQrTr/rnr-r>r.s r$rUz"OffloadedWeightsLoader.__getitem__s= $/ ! !?3' 'jo ??- . . :"k1UUt{FF O{+=>$W]^^^Obc\\+//-*M*MNNFOOOOOOOOOOOOOOO O O O{+=>$W\]]]Oab\\+//-*M*MNNFOOOOOOOOOOOOOOO O +%%75+g2F#G#GHH} V 4 4446**Mgll4#3\\\BB $[+>>>sZ B'%*B B'BB'"B#B''"D  *C?3 D ?D D D D  D c*t|jSrO)r\rjr]s r$r^zOffloadedWeightsLoader.__iter__sDM"""r&c*t|jSrO)r,rjr]s r$r`zOffloadedWeightsLoader.__len__s4=!!!r&)NNNN)rarbrcrdrrrTensorrrrPathLikerrRrUr^r`r(r&r$rfrfs  /39= el*+eC$456 (?s????4###"""""r&rfsubmodule_namesc|i}|D]5|fd|D6|S)a Extract the sub state-dict corresponding to a list of given submodules. Args: state_dict (`Dict[str, torch.Tensor]`): The state dict to extract from. submodule_names (`List[str]`): The list of submodule names we want to extract. cXi|]&\}}|ks|dz#||'S).)rX)rZrTparam module_names r$ z1extract_submodules_state_dict..sJ   C+%% c8I)J)J%U%%%r&)r;rF)rBr|resultrs @r$extract_submodules_state_dictrsoF&         ","2"2"4"4        Mr&rO)r9rcollections.abcrtypingrrrrrrr safetensorsrr%r0r@rr{rzrIrKrfrr(r&r$rs ######............ !!!!!!*,..."(sBK'7!8(d3PUP\K\F]((((&!!!!!g!!!.@"@"@"@"@"W@"@"@"Fd3 3D.EX\]`Xar&