קgw`UddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZmZmZddlmZddlmZmZe eee efZejeZed dddd d ed e ed eee efd ede edef dZGddZ iZ!eedfe"d<ej#Z$ej#e"d<ded eddfdZ% ddeedeeefde&de efdZ'dee ej(efde e fdZ)dS) N)contextmanager) AnyCallableDictIterableIteratorListOptionalSequenceSetTupleUnion)_C_opsTensor) exposed_in)autogradutilsz torch.library) device_typesschemanamefn mutates_argsrrreturnc8fd}||S||S)aWraps a function into custom operator. Reasons why you may want to create a custom op include: - Wrapping a third-party library or custom kernel to work with PyTorch subsystems like Autograd. - Preventing torch.compile/export/FX tracing from peeking inside your function. This API is used as a decorator around a function (please see examples). The provided function must have type hints; these are needed to interface with PyTorch's various subsystems. Args: name (str): A name for the custom op that looks like "{namespace}::{name}", e.g. "mylib::my_linear". The name is used as the op's stable identifier in PyTorch subsystems (e.g. torch.export, FX graphs). To avoid name collisions, please use your project name as the namespace; e.g. all custom ops in pytorch/fbgemm use "fbgemm" as the namespace. mutates_args (Iterable[str] or "unknown"): The names of args that the function mutates. This MUST be accurate, otherwise, the behavior is undefined. If "unknown", it pessimistically assumes that all inputs to the operator are being mutated. device_types (None | str | Sequence[str]): The device type(s) the function is valid for. If no device type is provided, then the function is used as the default implementation for all device types. Examples: "cpu", "cuda". When registering a device-specific implementation for an operator that accepts no Tensors, we require the operator to have a "device: torch.device argument". schema (None | str): A schema string for the operator. If None (recommended) we'll infer a schema for the operator from its type annotations. We recommend letting us infer a schema unless you have a specific reason not to. Example: "(Tensor x, int y) -> (Tensor, Tensor)". .. note:: We recommend not passing in a ``schema`` arg and instead letting us infer it from the type annotations. It is error-prone to write your own schema. You may wish to provide your own schema if our interpretation of the type annotation is not what you want. For more info on how to write a schema string, see `here `_ Examples:: >>> import torch >>> from torch import Tensor >>> from torch.library import custom_op >>> import numpy as np >>> >>> @custom_op("mylib::numpy_sin", mutates_args=()) >>> def numpy_sin(x: Tensor) -> Tensor: >>> x_np = x.cpu().numpy() >>> y_np = np.sin(x_np) >>> return torch.from_numpy(y_np).to(device=x.device) >>> >>> x = torch.randn(3) >>> y = numpy_sin(x) >>> assert torch.allclose(y, x.sin()) >>> >>> # Example of a custom op that only works for one device type. >>> @custom_op("mylib::numpy_sin_cpu", mutates_args=(), device_types="cpu") >>> def numpy_sin_cpu(x: Tensor) -> Tensor: >>> x_np = x.numpy() >>> y_np = np.sin(x_np) >>> return torch.from_numpy(y_np) >>> >>> x = torch.randn(3) >>> y = numpy_sin_cpu(x) >>> assert torch.allclose(y, x.sin()) >>> >>> # Example of a custom op that mutates an input >>> @custom_op("mylib::numpy_sin_inplace", mutates_args={"x"}, device_types="cpu") >>> def numpy_sin_inplace(x: Tensor) -> None: >>> x_np = x.numpy() >>> np.sin(x_np, out=x_np) >>> >>> x = torch.randn(3) >>> expected = x.sin() >>> numpy_sin_inplace(x) >>> assert torch.allclose(x, expected) >>> >>> # Example of a factory function >>> @torch.library.custom_op("mylib::bar", mutates_args={}, device_types="cpu") >>> def bar(device: torch.device) -> Tensor: >>> return torch.ones(3) >>> >>> bar("cpu") c ddl} |j| }n } d\}}t ||||} {t }|jjjD]/}|j &|j j r| |j 0|t krtd d d|d|||S)Nr)r::z3Attempted to create a custom op with `mutates_args=z` and `schema=z*. The schema suggests that the op mutates z`which is different from what was provided to us in `mutates_args`. Please make these consistent.)torchlibrary infer_schemasplit CustomOpDefset _opoverload_schema arguments alias_infois_writeaddr ValueErrorregister_kernel) rr schema_str namespaceopnameresultexpectedargrrrrs U/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/_library/custom_ops.pyinnerzcustom_op..inners# >33B\3RRJJJ JJt,, 6Y B??  uuH)1; + +>-#.2I-LL***3|,,,, 5,55#)55U]555 -|,,R000 )rrrrrr4s` ``` r3 custom_opr7 sIB4 z 599r5c ,eZdZdZdededededdf dZedefd Zdefd Z e dd ed e fdZ dde deedefdZdedefdZ ddedeedefdZdddedeeddfdZd dZdefdZdZ ddeefdZdS)!r#aCustomOpDef is a wrapper around a function that turns it into a custom op. It has various methods for registering additional behavior for this custom op. You should not instantiate CustomOpDef directly; instead, use the :func:`torch.library.custom_op` API. r.rrrrNc<||_||_||_||_i|_d|_d|_d|_i|_d|_ t|j|j|_ | t|_|t|j<dSN) _namespace_namer&_init_fn _backend_fns _abstract_fn_setup_context_fn _backward_fn_torch_dispatch_fns_vmap_fnget_library_allowing_overwrite_lib_register_to_dispatcherr$_disabled_kernelOPDEFS _qualname)selfr.rrrs r3__init__zCustomOpDef.__init__s#   >@0459049; ,0 24?DJOO  $$&&&%(UU!%t~r5c$|jd|jS)Nr)r;r<rJs r3rIzCustomOpDef._qualnames/11TZ111r5cd|jdS)Nz )rIrMs r3__repr__zCustomOpDef.__repr__s1t~1111r5T device_typeenabledc#JK|rdnd}||jv}||jvrtd|||s9|rtd|nS|j|n8|std|n|j| dV|r|j|dS|j|dS#|r|j|w|j|wxYw)a Disable or re-enable an already registered kernel for this custom operator. If the kernel is already disabled/enabled, this is a no-op. Note: If a kernel is first disabled and then registered, it is disabled until enabled again. Args: device_type (str): The device type to disable/enable the kernel for. disable (bool): Whether to disable or enable the kernel. Example: >>> inp = torch.randn(1) >>> >>> # define custom op `f`. >>> @custom_op("mylib::f", mutates_args=()) >>> def f(x: Tensor) -> Tensor: >>> return torch.zeros(1) >>> >>> print(f(inp)) # tensor([0.]), default kernel >>> >>> @f.register_kernel("cpu") >>> def _(x): >>> return torch.ones(1) >>> >>> print(f(inp)) # tensor([1.]), CPU kernel >>> >>> # temporarily disable the CPU kernel >>> with f.set_kernel_enabled("cpu", enabled = False): >>> print(f(inp)) # tensor([0.]) with CPU kernel disabled enabledisablezPAttempted to %s kernel for %s but no kernel was registered for this device type.z?Attempted to disable kernel for %s but it was already disabled.z>Attempted to enable kernel for %s but it was already enabled.N)rGr>logwarningr*removediscard)rJrPrQactionoriginally_disableds r3set_kernel_enabledzCustomOpDef.set_kernel_enabledswF%3))T-BB d/ / / KKb     :" 7 U %))+6666& : T %,,[999 ; EEE# ;%))+66666%--k:::::# ;%))+6666%--k::::s +C))9D"rcfd}btjjjsDtjjj}|t d|||S||S)aRegister an implementation for a device type for this operator. Some valid device_types are: "cpu", "cuda", "xla", "mps", "ipu", "xpu". This API may be used as a decorator. Args: fn (Callable): The function to register as the implementation for the given device types. device_types (str | Sequence[str]): The device device_types to register an impl to. Examples:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> import torch >>> from torch import Tensor >>> from torch.library import custom_op >>> import numpy as np >>> >>> # Create a custom op that works on cpu >>> @custom_op("mylib::numpy_sin", mutates_args=(), device_types="cpu") >>> def numpy_sin(x: Tensor) -> Tensor: >>> x_np = x.numpy() >>> y_np = np.sin(x_np) >>> return torch.from_numpy(y_np) >>> >>> # Add implementations for the cuda device >>> @numpy_sin.register_kernel("cuda") >>> def _(x): >>> x_np = x.cpu().numpy() >>> y_np = np.sin(x_np) >>> return torch.from_numpy(y_np).to(device=x.device) >>> >>> x_cpu = torch.randn(3) >>> x_cuda = x_cpu.cuda() >>> assert torch.allclose(numpy_sin(x_cpu), x_cpu.sin()) >>> assert torch.allclose(numpy_sin(x_cuda), x_cuda.sin()) c|ttrg}nt}|D]jvr]fd}"jj|dn3jj|tjtj fd}|j<S)Ncdt||D} j |i|}|}t|ts|f}t|iD]}t |}t ||vr; j }t j|}t j d|d| ||S)NcPh|]#}t|$Sr6)iduntyped_storage).0tensors r3 zSCustomOpDef.register_kernel..inner..backend_impl..?s<$$$ &v557788$$$r5z (with implementation in a): The output of this custom operator (1) must not also be an input to this custom operator and (2) may not alias any inputs to this custom operator or other returns. The most common way to trigger this error is if we have y = custom_op(x) and y and x are the same Tensor. Please instead return a clone of the offending output tensor(s) (e.g. return x.clone()) or refactor the custom operator to not return y.) iter_tensorsr> isinstancetupler`rainspect getmodule RuntimeErrorr<r*) argskwargsstoragesr0 tuple_resultrckeyrmodulerPrJs r3 backend_implz@CustomOpDef.register_kernel..inner..backend_impl<s)$$*6tV*D*D$$$ "@!2;!?!P!P!P'- )&%885,29L&2<&D&D..F"$V%;%;%=%=">">C!&"8"8":":;;xGG%)%6{%C)0):2)>)>&2'+z %A %AF %A %A %A '" '" !"%LL----% r5CompositeExplicitAutogradc@jvr j|i|S|i|Sr:)rGr=)rkrlrPrrJs r3 wrapped_fnz>CustomOpDef.register_kernel..inner..wrapped_fnjs="d&;;;,t}d=f===!r426222r5) rfstrlistr>rEimplr<r_dispatch_key_for_devicer_disable_dynamo)rdtypesrqrtrPrrJs` @r3r4z*CustomOpDef.register_kernel..inner4s#z,'D'D#2>l++%8 <8 < d&777&&&&&&B#*  J 6Q  J(7 DD&333333'&3 2<!+..Ir5NzVFunctions without tensor inputs are required to have a `device: torch.device` argument)rhas_tensor_argr%r&get_device_arg_indexr+#_register_backend_select_dispatcher)rJrrr4device_arg_indexs`` r3r,zCustomOpDef.register_kernel sR> > > > > > @  #E,@   $- - # %9$:J:RSS ' l  4 45E F F F :LuRyyr5c||_|S)a Register a FakeTensor implementation for this custom op. This is necessary to get the operator to work efficiently with torch.compile. The Fake impl (sometimes also known as a meta kernel or abstract impl) specifies the behavior of this operator on Tensors that carry no data. Given some input Tensors with certain properties (sizes/strides/storage_offset/device), it specifies what the properties of the output Tensors are. Please see :func:`torch.library.impl_abstract` for more details. Args: fn (Callable): The function to register as the FakeTensor implementation. Examples: >>> import torch >>> import numpy as np >>> from torch import Tensor >>> >>> # Example 1: an operator without data-dependent output shape >>> @torch.library.custom_op("mylib::linear", mutates_args=()) >>> def linear(x: Tensor, weight: Tensor, bias: Tensor) -> Tensor: >>> return (x @ weight.t()) + bias >>> >>> @linear.register_fake >>> def _(x, weight, bias): >>> assert x.dim() == 2 >>> assert weight.dim() == 2 >>> assert bias.dim() == 1 >>> assert x.shape[1] == weight.shape[1] >>> assert weight.shape[0] == bias.shape[0] >>> assert x.device == weight.device >>> return x.new_empty(x.size(0), weight.size(0)) >>> >>> x = torch.randn(2, 2) >>> weight = torch.randn(2, 2) >>> bias = torch.randn(2) >>> # xdoctest: +SKIP("Requires Python <= 3.11") >>> out = torch.compile(linear, fullgraph=True)(x, weight, bias) >>> # xdoctest: +SKIP("Requires Python <= 3.11") >>> assert torch.allclose(out, torch.nn.functional.linear(x, weight, bias)) >>> >>> # Example 2: an operator with data-dependent output shape >>> @torch.library.custom_op("mylib::nonzero", mutates_args=()) >>> def nonzero(x: Tensor) -> Tensor: >>> x_np = x.cpu().numpy() >>> res = np.stack(np.nonzero(x_np), axis=1) >>> return torch.tensor(res, device=x.device) >>> >>> @nonzero.register_fake >>> def _(x): >>> # Number of nonzero-elements is data-dependent. >>> # Since we cannot peek at the data in an abstract impl, >>> # we use the ctx object to construct a new symint that >>> # represents the data-dependent size. >>> ctx = torch.library.get_ctx() >>> nnz = ctx.new_dynamic_size() >>> shape = [nnz, x.dim()] >>> result = x.new_empty(shape, dtype=torch.int64) >>> return result >>> >>> x = torch.tensor([0, 1, 2, 0, 0, 1]) >>> # xdoctest: +SKIP("Requires Python <= 3.11") >>> out = torch.compile(nonzero, fullgraph=True)(x) >>> # xdoctest: +SKIP("Requires Python <= 3.11") >>> assert torch.allclose(out, x.nonzero()) )r?)rJrs r3 register_fakezCustomOpDef.register_fakesN r5torch_dispatch_classc0fd}||S||S)aRegisters a torch_dispatch rule for the given operator and ``torch_dispatch_class``. This allows for open registration to specify the behavior between the operator and the ``torch_dispatch_class`` without needing to modify the ``torch_dispatch_class`` or the operator directly. Please see :func:`torch.library.register_torch_dispatch` for examples and more details. c|jvr'fd}jj||j<|S)Nc*j|i|Sr:)rB)rkrlrJrs r3r4zDCustomOpDef.register_torch_dispatch..register..inners+I434HI!'r5)rBrE_register_torch_dispatch_ruler<)rr4rJrs r3registerz5CustomOpDef.register_torch_dispatch..registersk#4+CCC  77J 4e>@D $%9 :Ir5r6)rJrrrs`` r3register_torch_dispatchz#CustomOpDef.register_torch_dispatchs<       :O8B<< r5) setup_contextbackwardrc|jj}tj|st d|d|d||_||_dS)adRegister a backward formula for this custom op. In order for an operator to work with autograd, you need to register a backward formula: 1. You must tell us how to compute gradients during the backward pass by providing us a "backward" function. 2. If you need any values from the forward to compute gradients, you can use `setup_context` to save values for backward. ``backward_fn`` runs during the backward pass. It accepts ``(ctx, *grads)``: - ``grads`` is one or more gradients. The number of gradients matches the number of outputs of the operator. The ``ctx`` object is `the same ctx object `_ used by :class:`torch.autograd.Function`. The semantics of ``backward_fn`` are the same as :meth:`torch.autograd.Function.backward`. ``setup_context(ctx, inputs, output)`` runs during the forward pass. Please save quantities needed for backward onto the ``ctx`` object via either :meth:`torch.autograd.function.FunctionCtx.save_for_backward` or assigning them as attributes of ``ctx``. If your custom op has kwarg-only arguments, we expect the signature of ``setup_context`` to be ``setup_context(ctx, inputs, keyword_only_inputs, output)``. Both ``setup_context_fn`` and ``backward_fn`` must be traceable. That is, they may not directly access :meth:`torch.Tensor.data_ptr` and they must not depend on or mutate global state. If you need a non-traceable backward, you can make it a separate custom_op that you call inside ``backward_fn``. Examples: >>> import torch >>> import numpy as np >>> from torch import Tensor >>> >>> @torch.library.custom_op("mylib::numpy_sin", mutates_args=()) >>> def numpy_sin(x: Tensor) -> Tensor: >>> x_np = x.cpu().numpy() >>> y_np = np.sin(x_np) >>> return torch.from_numpy(y_np).to(device=x.device) >>> >>> def setup_context(ctx, inputs, output) -> Tensor: >>> x, = inputs >>> ctx.save_for_backward(x) >>> >>> def backward(ctx, grad): >>> x, = ctx.saved_tensors >>> return grad * x.cos() >>> >>> numpy_sin.register_autograd(backward, setup_context=setup_context) >>> >>> x = torch.randn(3, requires_grad=True) >>> y = numpy_sin(x) >>> grad_x, = torch.autograd.grad(y, x, torch.ones_like(y)) >>> assert torch.allclose(grad_x, x.cos()) >>> >>> # Example with a keyword-only arg >>> @torch.library.custom_op("mylib::numpy_mul", mutates_args=()) >>> def numpy_mul(x: Tensor, *, val: float) -> Tensor: >>> x_np = x.cpu().numpy() >>> y_np = x_np * val >>> return torch.from_numpy(y_np).to(device=x.device) >>> >>> def setup_context(ctx, inputs, keyword_only_inputs, output) -> Tensor: >>> ctx.val = keyword_only_inputs["val"] >>> >>> def backward(ctx, grad): >>> return grad * ctx.val >>> >>> numpy_mul.register_autograd(backward, setup_context=setup_context) >>> >>> x = torch.randn(3, requires_grad=True) >>> y = numpy_mul(x, val=3.14) >>> grad_x, = torch.autograd.grad(y, x, torch.ones_like(y)) >>> assert torch.allclose(grad_x, torch.full_like(x, 3.14)) z=Cannot register autograd formula for non-functional operator z with schema zP. Please create a functional operator and register an autograd formula for that.N)r%r&ris_functional_schemarjrAr@)rJrrrs r3register_autogradzCustomOpDef.register_autogradswd!))&11 TTT&,TTT  %!.r5czj}jjz}tj|}t j|rtd|||tj j tj j gt j j _fd}|j|dt!jj}|j|ddjjjr&fd }|j|d ddSdS) NzUcustom_op with kwarg-only Tensor args. Please make your tensors not kwarg-only. Got: )tagscj;tjjrdSt ddjjdj|i|S)Nz&There was no fake impl registered for zM. This is necessary for torch.compile/export/fx tracing to work. Please use `z$.register_fake` to add an fake impl.)r?rcan_generate_trivial_fake_implr%rjr=__name__)rkrlrJs r3 fake_implz6CustomOpDef._register_to_dispatcher..fake_impl\s{ (78HII 4""T""#'=#9""" %4$d5f55 5r5) _stacklevelAutogradT with_keysetc.tj||D]\}}|js |jjst |t r%t jj |Tt |ttfr>|D];}t |t r$t jj |.adinplaceorview_implpsv % 0v F F J JHC>! >2! !#v..J,>>sCCCC#C%77J!$JJA)!V44J % 4 F Fq I I I9;;64+6!AADHLRs#D  DDADInplaceOrView)rEr<r&r parse_schemarhas_kwarg_only_tensorsNotImplementedErrordefineTagpt2_compliant_tagneeds_fixed_stride_order lookup_oprIr%_register_fakermake_autograd_implrw is_mutable)rJlibr- cpp_schemar autograd_implrrs` @r3rFz#CustomOpDef._register_to_dispatcherIsiZ$,. _Z00  ' 3 3 &=0:==  &*BF,KL    !?4>:: 6 6 6 6 6 4:ya@@@ 3D4DdKK  ]JDIII!)         " HH $!      '  r5r~c\fd}jj|dddS)z] Switch on the device argument to select the correct backend to dispatch to. c|j}|jvrtjd|dt j|}t tj|}jj t j |g|Ri|S)Nz' does not have a kernel registered for z&. Please use register_kernel to do so.) typer>rjr<rrxgetattr DispatchKeyr%rDispatchKeySet)rrkrldevice dispatch_keyr~rJs r3backend_selectzGCustomOpDef._register_backend_select_dispatcher..backend_selects*+0FT..."z;;&;;;6v>>L"2><@@L.4#.!,//26:@ r5 BackendSelectTrN)rErwr<)rJr~rs`` r3r}z/CustomOpDef._register_backend_select_dispatchersI       tz>?PTUUUUUr5c|j|i|Sr:)r%)rJrkrls r3__call__zCustomOpDef.__call__st0000r5funccLddlmddlmfd}||S||S)a Register a vmap implementation to support :func:`torch.vmap` for this custom op. This API may be used as a decorator. In order for an operator to work with :func:`torch.vmap`, you may need to register a vmap implementation in the following signature: ``vmap_func(info, in_dims: Tuple[Optional[int]], *args, **kwargs)``, where ``*args`` and ``**kwargs`` are the arguments and kwargs for ``op``. It specifies how do we compute the batched version of ``op`` given inputs with an additional dimension (specified by ``in_dims``). For each arg in ``args``, ``in_dims`` has a corresponding ``Optional[int]``. It is ``None`` if the arg is not a Tensor or if the arg is not being vmapped over, otherwise, it is an integer specifying what dimension of the Tensor is being vmapped over. ``info`` is a collection of additional metadata that may be helpful: ``info.batch_size`` specifies the size of the dimension being vmapped over, while ``info.randomness`` is the ``randomness`` option that was passed to :func:`torch.vmap`. The return of the function ``func`` is a tuple of ``(output, out_dims)``. Similar to ``in_dims``, ``out_dims`` should be of the same structure as ``output`` and contain one ``out_dim`` per output that specifies if the output has the vmapped dimension and what index it is in. Examples: >>> import torch >>> import numpy as np >>> from torch import Tensor >>> from typing import Tuple >>> >>> def to_numpy(tensor): >>> return tensor.cpu().numpy() >>> >>> lib = torch.library.Library("mylib", "FRAGMENT") >>> @torch.library.custom_op("mylib::numpy_cube", mutates_args=()) >>> def numpy_cube(x: Tensor) -> Tuple[Tensor, Tensor]: >>> x_np = to_numpy(x) >>> dx = torch.tensor(3 * x_np ** 2, device=x.device) >>> return torch.tensor(x_np ** 3, device=x.device), dx >>> >>> def numpy_cube_vmap(info, in_dims, x): >>> result = numpy_cube(x) >>> return result, (in_dims[0], in_dims[0]) >>> >>> numpy_cube.register_vmap(numpy_cube_vmap) >>> >>> x = torch.randn(3) >>> torch.vmap(numpy_cube)(x) >>> >>> @torch.library.custom_op("mylib::numpy_mul", mutates_args=()) >>> def numpy_mul(x: Tensor, y: Tensor) -> Tensor: >>> return torch.tensor(to_numpy(x) * to_numpy(y), device=x.device) >>> >>> @numpy_mul.register_vmap >>> def numpy_mul_vmap(info, in_dims, x, y): >>> x_bdim, y_bdim = in_dims >>> x = x.movedim(x_bdim, -1) if x_bdim is not None else x.unsqueeze(-1) >>> y = y.movedim(y_bdim, -1) if y_bdim is not None else y.unsqueeze(-1) >>> result = x * y >>> result = result.movedim(-1, 0) >>> return result, 0 >>> >>> >>> x = torch.randn(3) >>> y = torch.randn(3) >>> torch.vmap(numpy_mul)(x, y) r) custom_function_call_vmap_helper)&retrieve_current_functorch_interpretercjdu}|_|r,fd}jj|dddSdS)NcH}|jjg|Ri|Sr:)rCr%)rrkrl interpreterrrrJs r3 wrapped_funczACustomOpDef.register_vmap..register..wrapped_funcsL"H"H"J"JK;;#T]D4DGKOUr5FuncTorchBatchedTr)rCrErwr<)r need_registerrrrrJs r3rz+CustomOpDef.register_vmap..registers MT1M DM   J .@d  r5)"torch._functorch.autograd_functionrtorch._functorch.pyfunctorchr)rJrrrrs` @@r3 register_vmapzCustomOpDef.register_vmapsjR XWWWWWWWWWWW        <O8D>> !r5)Tr:)rN)r __module__ __qualname____doc__rurrKpropertyrIrOrboolr[device_types_tr r,rrrrrFintr}rrr6r5r3r#r#sE&#&S&#&8&PT&&&&(23222X22#2222C;C;cC;DC;C;C;^C;LFJvv*v080Bv vvvvpHHHHHHVCG  $' -5h-?     H-1 [/[/[/[/  ) [/  [/[/[/[/z====~VCVVVV(111 $(_"_"x _"_"_"_"_"_"r5r#ztorch.library.Library OPDEF_TO_LIBrHr.c|d|}|tvr't|t|=tj|d}|t|<|S)NrFRAGMENT)r_destroyrr Library)r.rqualnamers r3rDrDsg%%t%%H<X'')))  " -   : 6 6C L Jr5rkrlallowed_nestingc#Kfd}|D]}||Ed{V|D]}||Ed{VdS)Nc3Kt|tr|VdSdkrEt|ttfr+t t|idz Ed{VdSdSdS)Nrr)rfrrgrvre)r2rs r3checkziter_tensors..check-s c6 " " IIIIII q ZeT]%C%C #E#JJOa4GHH H H H H H H H H H! r5)values)rkrlrrr2kwargs ` r3rere*sIIIII 5::  5<<  r5opct|tr|St|tjr|j}t|t sJ|t vr t |SdSr:)rfr#r OpOverloadr<rurH)rs r3_maybe_get_opdefr9sd"k"" "do&& X b#   V||bz 4r5r:)r)*rhloggingweakref contextlibrtypingrrrrrr r r r r rrrrrtorch.utils._exposed_inrrrrur getLoggerrrUr7r#r__annotations__WeakValueDictionaryrHrDrrerrr6r5r3rs%%%%%%                           """"""""""......%Xc] 234g!! O"| $( ||| || Xc]*+ | ! | SM|||||~^ "^ "^ "^ "^ "^ "^ "^ "p46 d3//0555&Ag&A&C&C#CCC       FG   *  "38n  ?B   f       k4?C/0  k      r5