NgddlmZddlZddlmZmZddlmZmZddl Z ej e Z eGddZ dS)) annotationsN) dataclassfield)AnyCallablecxeZdZUdZded<edZded<eedd Zd ed <ddZ ddZ dS)SentenceTransformerDataCollatoraCollator for a SentenceTransformers model. This encodes the text columns to {column}_input_ids and {column}_attention_mask columns. This works with the two text dataset that is used as the example in the training overview: https://www.sbert.net/docs/sentence_transformer/training_overview.html It is important that the columns are in the expected order. For example, if your dataset has columns "answer", "question" in that order, then the MultipleNegativesRankingLoss will consider "answer" as the anchor and "question" as the positive, and it will (unexpectedly) optimize for "given the answer, what is the question?". r tokenize_fnc ddgS)Nlabelscorer_/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/data_collator.pyz(SentenceTransformerDataCollator.s GWCUr)default_factory list[str]valid_label_columnsF)rinitreprzset[tuple[str]]_warned_columnsfeatureslist[dict[str, Any]]returndict[str, torch.Tensor]ct|d}i}d|vr&|d|dd|d<t||jvr|||jD]@|vr:tjfd|D|d<|nA|D] drJdtd |vr0tjfd|Dtj |<a| fd|D}| D]\}}||d |<|S) Nr dataset_namec g|] }| Srr).0row label_columns r z.-s.U.U.USs.4s2X2X2X3{3C2X2X2Xr)dtypec g|] }| Srrr%s rr"z.7s)O)O)Os#k*:)O)O)Or_)listkeysremovetuplermaybe_warn_about_column_orderrtorchtensorendswithlenintr items) selfr column_namesbatch tokenizedkeyvaluer&r!s @@r__call__z(SentenceTransformerDataCollator.__call__sHQK,,..//  \ ) )    / / /$,QK$?E. !   d&: : :  . .| < < <!4  L|++!&.U.U.U.UH.U.U.U!V!Vg##L111, ( 6 6K##$455 +F^M]I^I^H^F^:_co:o:o%*\2X2X2X2Xx2X2X2X`e`i%j%j%jk"(()O)O)O)Oh)O)O)OPPI'oo// 6 6 U05,,s,,-- 6 rr6Nonec ddddddddddd }|D]\}}||vrx|||kr_|dvrgd}n|dvrddg}n|d vrd d g}n|d vrgd }td |d||d|d|d n|jt |dS)zBWarn the user if the columns are likely not in the expected order.r) anchorpositivenegativequestionanswerqueryresponse hypothesis entailment contradiction)r@rArB)rCrDrCrD)rErFrErF)rGrHrIzColumn z is at index z?, whereas a column with this name is usually expected at index a?. Note that the column order can be important for some losses, e.g. MultipleNegativesRankingLoss will always consider the first column as the anchor and the second as the positive, regardless of the dataset column names. Consider renaming the columns to match the expected order, e.g.: dataset = dataset.select_columns()N)r4indexloggerwarningraddr-)r5r6column_name_to_expected_idxr& expected_idxproposed_fix_columnss rr.z=SentenceTransformerDataCollator.maybe_warn_about_column_order=sw ' ' #*E)J)J)L)L   %Kl**|/A/A+/N/NR^/^/^"DDD+M+M+M(( $:::,6+A(( $999,3Z+@(( $QQQ+X+X+X(PkPP,:L:L[:Y:YPPLXPP 9M PPP   |!4!455555rN)rrrr)r6rrr<) __name__ __module__ __qualname____doc____annotations__rrsetrr;r.rrrr r s  %*U;U;U%V%V%VVVVV',uSuSX'Y'Y'YOYYYY@%6%6%6%6%6%6rr ) __future__rlogging dataclassesrrtypingrrr/ getLoggerrRrLr rrrr]s""""""((((((((  8 $ $ U6U6U6U6U6U6U6 U6U6U6r