gQ ddlZddlZddlmZ ddlZn5#eef$r+ddlZeej j dkr e dYnwxYwdZ dZ dZdZd Zd Zdd Zeejjed krdZejjZeZne Ze Zeeeeeeeejjejjejjejjejjd ZdZdS)N)parsezYour currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.c tj|}ddtj|tjtjd|jz zz}||zS)av Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 ??@)tfconvert_to_tensormatherfcastsqrtdtype)xcdfs W/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/activations_tf.py_gelur"sU QA rw{{1rwrws||QW'E'E#EFFF GC s7Nc Ftj|}tjtj|j}tjd|j}ddtjtjd|z ||tj|dzzzzz}||zS)z Gaussian Error Linear Unit. This is a smoother version of the GELU. Original paper: https://arxiv.org/abs/1606.0841 Args: x: float Tensor to perform activation Returns: `x` with the GELU activation applied. Hm?rrr) r r r r pirtanhrpow)rrcoeffrs r _gelu_newr/s QA !' " "B GHag & &E rwrwsRx00Aq! 8L4LMNNN OC s7Nrctj|}|tjtj|zS)N)r r rr softplusrs rmishr As7 QA rwrw''**++ ++rctj|}tjd|j}tjd|j}d|zdtj||zd||z|zzzzzS)Nrg3E?rr)r r r rr)rcoeff1coeff2s r gelu_fastr$Gsl QA WXqw ' 'F W\17 + +F 7cBGAJ# Q2F$GHHH IIrctj|}tjd|j}|tj||zzS)NgZd;?)r r r rr sigmoid)rrs r quick_gelur'OsB QA GE17 # #E rwuqy)) ))rcHtjt|ddS)a Clip the range of possible GeLU outputs between [-10, 10]. This is especially useful for quantization purpose, as it allows mapping 2 negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602 Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 :param x: :return: i )r clip_by_valuerrs rgelu_10r+Us  E!HHc2 . ..rcztj|d|\}}|tj|zS)a Gated Linear Unit. Implementation as defined in the original paper (see https://arxiv.org/abs/1612.08083), where the input `x` is split in two halves across a dimension (`axis`), A and B, returning A * sigmoid(B). Args: `x`: float Tensor to perform activation `axis`: dimension across which `x` be split in half Returns: `x` with the GLU activation applied (with its size halved across the dimension `axis`). r)axis)r splitr r&)rr.abs rglur2cs7 8Aqt $ $ $DAq rwq!! !!rz2.4cDtj|dS)NT) approximate)keras activationsgelurs rapproximate_gelu_wrapr8us %%aT%:::r) r7r+r$gelu_newr2r r'relur&siluswishrc |tvr t|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_tf_activationrCsOF""'((h#4hhSWX^XcXcXeXeSfSfhhiiir)r,)r tensorflowr packaging.versionrtf_kerasr5ModuleNotFoundError ImportError __version__major ValueErrorrrr r$r'r+r2versionVERSIONr8r6r7r9r:r&r<rr>rCrrrOs ######  [)   LLL uU %))j &   *)    $,,, JJJ*** / / / " " " "  5 e ,,;;;   !D$HH DH      " (   #   $   "   jjjjjs/AA