gg d Z ddlZddlZddlZddlZddlmZ ddlmZ ddl m
Z
ddl m Z m
Z
mZmZmZmZmZmZ ddlmZmZ d d
lmZmZmZ ej e Z e rddl Z dd edd d d
ddee e
f dee de dee!e f deee e f de fdZ"dd edd d dee df dee e
f dee de dee!e f deee e f de ddfdZ#ej$ eddee df de dee!e f defdZ%d ddee!ee
d f f fd Z&d ddeed ee!ee
d f f e!f fd Z'd dde!fd!Z( e d1d" Z)d ddee!ee
d f f fd#Z* d1dee df dee e f de fd$Z+d dde!fd%Z,d&eee dee df deee fd'Z-dee df deee fd(Z.d dde fd)Z/ddd*dee df d+eee d,eee dee ee f fd-Z0 e d.d/de!fd0 Z1dS )2z"Contains pytorch-specific helpers. N) defaultdict) lru_cache)Path)
TYPE_CHECKINGAnyDictListOptionalSetTupleUnion ) constantslogging )MAX_SHARD_SIZEStateDictSplit$split_state_dict_into_shards_factoryT)filename_patternforce_contiguousmax_shard_sizemetadatasafe_serializationmodelztorch.nn.Modulesave_directoryr r r r r c V t | |||||| dS )a
Saves a given torch model to disk, handling sharding and shared tensors issues.
See also [`save_torch_state_dict`] to save a state dict with more flexibility.
For more information about tensor sharing, check out [this guide](https://huggingface.co/docs/safetensors/torch_shared_tensors).
The model state dictionary is split into shards so that each shard is smaller than a given size. The shards are
saved in the `save_directory` with the given `filename_pattern`. If the model is too big to fit in a single shard,
an index file is saved in the `save_directory` to indicate where each tensor is saved. This helper uses
[`split_torch_state_dict_into_shards`] under the hood. If `safe_serialization` is `True`, the shards are saved as
safetensors (the default). Otherwise, the shards are saved as pickle.
Before saving the model, the `save_directory` is cleaned from any previous shard files.
If one of the model's tensor is bigger than `max_shard_size`, it will end up in its own shard which will have a
size greater than `max_shard_size`.
Args:
model (`torch.nn.Module`):
The model to save on disk.
save_directory (`str` or `Path`):
The directory in which the model will be saved.
filename_pattern (`str`, *optional*):
The pattern to generate the files names in which the model will be saved. Pattern must be a string that
can be formatted with `filename_pattern.format(suffix=...)` and must contain the keyword `suffix`
Defaults to `"model{suffix}.safetensors"` or `pytorch_model{suffix}.bin` depending on `safe_serialization`
parameter.
force_contiguous (`boolean`, *optional*):
Forcing the state_dict to be saved as contiguous tensors. This has no effect on the correctness of the
model, but it could potentially change performance if the layout of the tensor was chosen specifically for
that reason. Defaults to `True`.
max_shard_size (`int` or `str`, *optional*):
The maximum size of each shard, in bytes. Defaults to 5GB.
metadata (`Dict[str, str]`, *optional*):
Extra information to save along with the model. Some metadata will be added for each dropped tensors.
This information will not be enough to recover the entire shared structure but might help understanding
things.
safe_serialization (`bool`, *optional*):
Whether to save as safetensors, which is the default behavior. If `False`, the shards are saved as pickle.
Safe serialization is recommended for security reasons. Saving as pickle is deprecated and will be removed
in a future version.
Example:
```py
>>> from huggingface_hub import save_torch_model
>>> model = ... # A PyTorch model
# Save state dict to "path/to/folder". The model will be split into shards of 5GB each and saved as safetensors.
>>> save_torch_model(model, "path/to/folder")
# Load model back
>>> from huggingface_hub import load_torch_model # TODO
>>> load_torch_model(model, "path/to/folder")
>>>
```
)
state_dictr r r r r r N)save_torch_state_dictr )r r r r r r r s `/var/www/html/ai-engine/env/lib/python3.11/site-packages/huggingface_hub/serialization/_torch.pysave_torch_modelr # sG P ##%%))%-% r z torch.Tensorreturnc t | }||r t j n t j }|r* ddlm} nB# t $ r}t
d |d}~ww xY wddlm} t
d |i }|rt || t || } t j | d
d z }
t! j | D ]} |
| r t d | d
t! j t j || i# t. $ r+}t
d| d| d Y d}~d}~ww xY wddi} | j s| | |rd| ini }
| j D ]X\ } } fd|D } ||t j || fi |
t d| Y| j r| d
dz }i | j || j d}t= t j || d 5 }t? j ||d ddd n # 1 swxY w Y t ! d | d tE | j d | d t ! d | d! dS )"a`
Save a model state dictionary to the disk, handling sharding and shared tensors issues.
See also [`save_torch_model`] to directly save a PyTorch model.
For more information about tensor sharing, check out [this guide](https://huggingface.co/docs/safetensors/torch_shared_tensors).
The model state dictionary is split into shards so that each shard is smaller than a given size. The shards are
saved in the `save_directory` with the given `filename_pattern`. If the model is too big to fit in a single shard,
an index file is saved in the `save_directory` to indicate where each tensor is saved. This helper uses
[`split_torch_state_dict_into_shards`] under the hood. If `safe_serialization` is `True`, the shards are saved as
safetensors (the default). Otherwise, the shards are saved as pickle.
Before saving the model, the `save_directory` is cleaned from any previous shard files.
If one of the model's tensor is bigger than `max_shard_size`, it will end up in its own shard which will have a
size greater than `max_shard_size`.
Args:
state_dict (`Dict[str, torch.Tensor]`):
The state dictionary to save.
save_directory (`str` or `Path`):
The directory in which the model will be saved.
filename_pattern (`str`, *optional*):
The pattern to generate the files names in which the model will be saved. Pattern must be a string that
can be formatted with `filename_pattern.format(suffix=...)` and must contain the keyword `suffix`
Defaults to `"model{suffix}.safetensors"` or `pytorch_model{suffix}.bin` depending on `safe_serialization`
parameter.
force_contiguous (`boolean`, *optional*):
Forcing the state_dict to be saved as contiguous tensors. This has no effect on the correctness of the
model, but it could potentially change performance if the layout of the tensor was chosen specifically for
that reason. Defaults to `True`.
max_shard_size (`int` or `str`, *optional*):
The maximum size of each shard, in bytes. Defaults to 5GB.
metadata (`Dict[str, str]`, *optional*):
Extra information to save along with the model. Some metadata will be added for each dropped tensors.
This information will not be enough to recover the entire shared structure but might help understanding
things.
safe_serialization (`bool`, *optional*):
Whether to save as safetensors, which is the default behavior. If `False`, the shards are saved as pickle.
Safe serialization is recommended for security reasons. Saving as pickle is deprecated and will be removed
in a future version.
Example:
```py
>>> from huggingface_hub import save_torch_state_dict
>>> model = ... # A PyTorch model
# Save state dict to "path/to/folder". The model will be split into shards of 5GB each and saved as safetensors.
>>> state_dict = model_to_save.state_dict()
>>> save_torch_state_dict(state_dict, "path/to/folder")
```
Nr ) save_filezjPlease install `safetensors` to use safe serialization. You can install it with `pip install safetensors`.)savea You are using unsafe serialization. Due to security reasons, it is recommended not to load pickled models from untrusted sources. If you intend to share your model, we strongly recommend using safe serialization by installing `safetensors` with `pip install safetensors`.)r r r z(-\d{5}-of-\d{5})?)suffixz(\.index\.json)?zRemoving existing file 'z' from folder.z&Error when trying to remove existing 'z' from folder: z. Continuing...formatptr c " i | ] }|| S r+ ).0tensorr s r
z)save_torch_state_dict.. s BBBF+BBBr! zShard saved to z .index.json)r
weight_mapwr )indentz:The model is bigger than the maximum size per checkpoint (z#). Model weighs have been saved in z^ checkpoint shards. You can find where each parameters has been saved in the index located at .z$Model weights successfully saved to !)#strr SAFETENSORS_WEIGHTS_FILE_PATTERN PYTORCH_WEIGHTS_FILE_PATTERNsafetensors.torchr$ ImportErrortorchr% loggerwarning!_clean_state_dict_for_safetensors"split_torch_state_dict_into_shardsrecompiler( oslistdirmatchdebugremovepathjoin Exception
is_shardedupdatefilename_to_tensorsitemsr tensor_to_filenameopenjsondumpinfolen)r r r r r r r save_file_fnestate_dict_splitexisting_files_regexfilenameper_file_metadatasafe_file_kwargstensorsshard
index_pathindexfs ` r r r v s1 H ((N "
8I 6 67
C C C C C C C E
/.....
c
p6z8^nooo
:%5n
:&6&=&=EZ&=&[&[^q&qrrJ~.. u u % %h / / u
u P P P PQQQ "',,~x@@AAAA
u
u
u s s sab s s stttttttt
u u "4( & + ***:LT
$566RT - A G G I I 3 3'BBBB'BBB UBGLL B BWWFVWWW 1x112222 "
%,,B,77-G
A+4AA*=
"',,~z::C
@
@ *A IeQq ) ) ) ) * * * * * * * * * * * * * * *
g
g
g/23C3W/X/X
g
gYc
g
g
g
KKH~HHHIIIIIs@ 7
AAA:AE
F !E;;F ?J##J'*J'r& c > t | ||t t S )a>
Split a model state dictionary in shards so that each shard is smaller than a given size.
The shards are determined by iterating through the `state_dict` in the order of its keys. There is no optimization
made to make each shard as close as possible to the maximum size passed. For example, if the limit is 10GB and we
have tensors of sizes [6GB, 6GB, 2GB, 6GB, 2GB, 2GB] they will get sharded as [6GB], [6+2GB], [6+2+2GB] and not
[6+2+2GB], [6+2GB], [6GB].
To save a model state dictionary to the disk, see [`save_torch_state_dict`]. This helper uses
`split_torch_state_dict_into_shards` under the hood.
If one of the model's tensor is bigger than `max_shard_size`, it will end up in its own shard which will have a
size greater than `max_shard_size`.
Args:
state_dict (`Dict[str, torch.Tensor]`):
The state dictionary to save.
filename_pattern (`str`, *optional*):
The pattern to generate the files names in which the model will be saved. Pattern must be a string that
can be formatted with `filename_pattern.format(suffix=...)` and must contain the keyword `suffix`
Defaults to `"model{suffix}.safetensors"`.
max_shard_size (`int` or `str`, *optional*):
The maximum size of each shard, in bytes. Defaults to 5GB.
Returns:
[`StateDictSplit`]: A `StateDictSplit` object containing the shards and the index to retrieve them.
Example:
```py
>>> import json
>>> import os
>>> from safetensors.torch import save_file as safe_save_file
>>> from huggingface_hub import split_torch_state_dict_into_shards
>>> def save_state_dict(state_dict: Dict[str, torch.Tensor], save_directory: str):
... state_dict_split = split_torch_state_dict_into_shards(state_dict)
... for filename, tensors in state_dict_split.filename_to_tensors.items():
... shard = {tensor: state_dict[tensor] for tensor in tensors}
... safe_save_file(
... shard,
... os.path.join(save_directory, filename),
... metadata={"format": "pt"},
... )
... if state_dict_split.is_sharded:
... index = {
... "metadata": state_dict_split.metadata,
... "weight_map": state_dict_split.tensor_to_filename,
... }
... with open(os.path.join(save_directory, "model.safetensors.index.json"), "w") as f:
... f.write(json.dumps(index, indent=2))
```
)r r get_storage_sizeget_storage_id)r get_torch_storage_sizeget_torch_storage_id)r r r s r r> r> s+ F 0%)/+ r! r- .c J ddl m} | r2 \ }}t fd|D S n# t $ r Y nw xY w j j dk r-t r ddl}|j
}nt }|S )zReturns a unique id for plain tensor
or a (potentially nested) Tuple of unique id for the flattened Tensor
if the input is a wrapper tensor subclass Tensor
r is_traceable_wrapper_subclassc 3 R K | ]!}t t | V "d S N)_get_unique_idgetattrr, attrr- s r z!_get_unique_id.._ s5 QQ4 (=(=>>QQQQQQr! xlaN) torch.utils._python_dispatchrf __tensor_flatten__tupler9 devicetypeis_torch_tpu_available torch_xla_XLAC_xla_get_tensor_id storage_ptr)r- rf attrs_ru unique_ids ` r ri ri S s
NNNNNN ( ( 0 0 R 0022HE1QQQQ5QQQQQ Q R
}U""'='?'?"
O66v>> '' AA
AAz torch.devicec n | j j dk rdS | j t | t | fS )a,
Return unique identifier to a tensor storage.
Multiple different tensors can share the same underlying storage. This identifier is
guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with
non-overlapping lifetimes may have the same id.
In the case of meta tensors, we return None since we can't tell if they share the same storage.
Taken from https://github.com/huggingface/transformers/blob/1ecf5f7c982d761b4daaa96719d162c324187c64/src/transformers/pytorch_utils.py#L278.
metaN)rr rs ri rb )r- s r rc rc s s9 }V##t}nV 4 46LV6T6TTTr! c
ddl m} | r2 \ }}t fd|D S n# t $ r Y nw xY w S # t $ rz t j z cY S # t $ r.
t j z cY cY S w xY ww xY w)z
Taken from https://github.com/huggingface/safetensors/blob/08db34094e9e59e2f9218f2df133b7b4aaff5a99/bindings/python/py_src/safetensors/torch.py#L31C1-L41C59
r re c 3 R K | ]!}t t | V "d S rh )rb rj rk s r rm z)get_torch_storage_size.. s6 WW-gfd.C.CDDWWWWWWr! )ro rf rp sumr9 untyped_storagenbytesAttributeErrorstoragesize_get_dtype_sizedtypeNotImplementedErrornelementr- rf ry rz s ` r rb rb sU
NNNNNN ( ( 0 0 X 0022HE1WWWWQVWWWWW W X
E%%''..000 E E E E>>##((**_V\-J-JJ J J J" E E E ??$$v|'D'DD D D D D D E EsB AA
AA%A> >
D :CD3C>9D=C>>Dc t j d 5| r1 ddlmc m} | }dS # t $ r Y dS w xY wdS dS )z
Checks if `torch_xla` is installed and potentially if a TPU is in the environment
Taken from https://github.com/huggingface/transformers/blob/1ecf5f7c982d761b4daaa96719d162c324187c64/src/transformers/utils/import_utils.py#L463.
ru Nr TF) importlibutil find_spectorch_xla.core.xla_modelcore xla_model
xla_device RuntimeError) check_devicexmrz s r rt rt s ~ ,,8
555555555MMOOt
uu
t 5s A
AAc F ddl m} || rt | S n# t $ r Y nw xY w | S # t $ r< | cY S # t $ r Y Y dS w xY ww xY w)z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L11.
r re ) ro rf ri r9 r data_ptrrH r r )r- rf s r rx rx s
NNNNNN ( ( 0 0 *!&)) ) *
%%''00222 >>##,,.. . . ." 111 s; #
00%A
B %%B
B
B B B B c t | }| D ]\ }}|D ]}|i }||vr|||< | |= |r d | D } | S )a6 Remove shared tensors from state_dict and update metadata accordingly (for reloading).
Warning: `state_dict` and `metadata` are mutated in-place!
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L155.
Nc > i | ]\ }}|| S r+ )
contiguous)r, kvs r r. z5_clean_state_dict_for_safetensors.. s& GGGDAqa GGGr! )_remove_duplicate_namesrL )r r r
to_removes kept_nameto_remove_group to_removes r r= r= s )44J&0&6&6&8&8 & &" ?( & &I ((&/#9%% & HGGJ4D4D4F4FGGG
r! c | rC| d d t | j z }n| }|S )z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L23.
)r viewr r r )r- stops r _end_ptrr s[ !{{2r"++-- 0M0MM Kr! rZ c g }| D ]}t | dk r| | +g }|D ]A}|| }| | t | |f B| |d \ }}} | | h |dd D ]@\ }
} }|
|k r| |h n|d | | }A|S )z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L44
r r r Nr )rR appendr r sortadd) rZ r filtered_tensorssharedareasnamer- rz last_stop last_namestartr s r _filter_shared_not_sharedr s/ v;;?? # #F + + + F FD %F LL&//++Xf-=-=tD E E E E
"'( 9i ,,,!&qrr E4 !! ''//// $((... II r! c ddl }t t }| D ]\ }}|j |j d k rct | dk rPt
| dk r=||j t | t
| f | t t |
}t || }|S )z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L69.
r Nr~ ) r: r setrL rr rx rb r listsortedvaluesr )r r: tensors_dictr r rZ s r _find_shared_tensorsr s LLL s##L "" W W1 8|u|F++ + + A!0C0CH^_`HaHaefHfHf !(KNN4J14M4MN O S STU V V V6,--//0011G'<.& s5 MMt|GFD$9$9::MMMMMMr! ) ro rf rp allr9 r rx r r r rb r s ` r r r s
NNNNNN ( ( 0 0 N 0022HE1MMMMuMMMMM M N
?? F 3 3 3 (8I8IO M M 9 ' '9( (r| )preferred_names
discard_namesr r c |g }t | }|g }t | }t }t t }|D ]}t fd|D }|st d| d t t | d } | | }
|
r"t t |
d } |r9| | }
|
r"t t |
d } t | D ]#} | | k r|| | $|S )r Nc > g | ]}t | |S r+ )r )r, r r s r
z+_remove_duplicate_names..C s, X X Xt jQUFV9W9W Xd X X Xr! zvError while trying to find names to remove to save state dict, but found no suitable name to keep for saving amongst: z. None is covering the entire storage. Refusing to save/load the model since you could be storing much more memory than needed. Please refer to https://huggingface.co/docs/safetensors/torch_shared_tensors for more information. Or open an issue.r ) r r r r r r
difference intersectionr ) r r r unique_preferred_namesunique_discard_namessharedsr r complete_names keep_name preferredr s ` r r r 0 s 1 1
}--":..GD!!I 2 2 X X X Xv X X XYY (.
4//003 #--.BCC 3 tI//2I ! 7.;;NKKI
7 "4 ?? 3 3A 6 6NN 2 2Dy )$++D111 2 r! r z torch.dtypec ddl }t |dd }t |dd }|j d|j d|j d|j d|j d|j d|j d|j d|j
d|j d|d|di }|| S ) z
Taken from https://github.com/huggingface/safetensors/blob/08db34094e9e59e2f9218f2df133b7b4aaff5a99/bindings/python/py_src/safetensors/torch.py#L344
r N
float8_e4m3fn float8_e5m2 r r ) r: rj int64float32int32bfloat16float16int16uint8int8boolfloat64)r r: _float8_e4m3fn _float8_e5m2_SIZEs r r r a s
LLL UOT::N5-66L
Q
q
Q
q
Q
Q
A
A
qa
E <r! )T)2__doc__r rO rA r? collectionsr functoolsr pathlibr typingr r r r r
r r r
r/ r r _baser r r
get_logger__file__r; r: r5 r intr r r6 r> ri rc rb rt rx r= r r r r r r r+ r! r r s ) ( # # # # # # N N N N N N N N N N N N N N N N N N N N ! ! ! ! ! ! ! ! W W W W W W W W W W
H % % LLL '+ !&4)- #P P P P#t)$P sm P
P #s(O
P tCH~&P P P P Pn '+ !&4)- #NJ NJ NJS.()NJ#t)$NJ sm NJ
NJ #s(O
NJ tCH~&NJ NJ
NJ NJ NJ NJh & F&4 I I IS.()I I #s(O I
I I I IX > eCsCx 4H.I @U UHU>SXY\^cdgildl^mYmSnpsCs=t4u U U U U"E> Ec E E E E6
( 5eCHo1E+F 4 _c S.()59#s(^W[ 0^ tCH~ 4^H[C\ aefijmfnao 8
T#~*=%>
4C>
( (D ( ( ( (. ,0)- . . .S.(). d3i(. DI& .
#tCy. . . . .b = S
r!