Ngw#ddlmZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZer ddlZddlmZdd ZGd deZddZGddeZeZdS)) annotationsN)Path) TYPE_CHECKINGDictListOptionalUnionDocument) BaseLoader) EntityLikerowdictreturnstrcH|d}|d}|d}|d|d|dS)zBCombine message information in a readable format ready to be used.datefromtextz on z:  )rrsenderrs i/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/telegram.pyconcatenate_rowsrs< v;D [F v;D , ,$ , ,$ , , ,,c"eZdZdZd dZd dZdS) TelegramChatFileLoaderzLoad from `Telegram chat` dump.pathUnion[str, Path]c||_dS)zInitialize with a path.N) file_path)selfrs r__init__zTelegramChatFileLoader.__init__s rrList[Document]c4t|j}t|d5}tj|}dddn #1swxYwYdd|dD}dt |i}t||gS) Load documents.utf8encodingNc3K|]:}|ddkt|dt)t|V;dS)typemessagerN) isinstancerr).0r-s r z.TelegramChatFileLoader.load..'sX  v)++ 76?C0P0P+ W % %++++  rmessagessource page_contentmetadata)rr!openjsonloadjoinrr )r"pfdrr5s rr8zTelegramChatFileLoader.load s   !f % % %  ! A               ww  Z=      c!ff%dX>>>??sAA A N)rrrr$)__name__ __module__ __qualname____doc__r#r8rrrrrsH))@@@@@@rrrUnion[str, List[str]]r$cddlm}|dgdd}t|tr|g}d|D}t |D]\}}|dz|jd <g}|D]}||j}t |D]^\}}t||jd |d  }|jd d |jd |jd<| |_|S)zIConvert a string or list of strings to a list of Documents with metadata.r)RecursiveCharacterTextSplitteri )r .!?, r*) chunk_size separators chunk_overlapc.g|]}t|S))r4r )r/pages r z text_to_docs..>s#>>>t,,,>>>rrP)rPchunkr3-rSr2) langchain_text_splittersrDr.r enumerater5 split_textr4r append) rrD text_splitter page_docsidoc doc_chunkschunksrSs r text_to_docsr_1sVGGGGGG22>>>M $v>>>>>II&&%%3 1u VJ##))#*:;;!&)) # #HAu"cl66JUV-W-WC), V(<%V%Vs|G?T%V%VCL "   c " " " "  # rcFeZdZdZ ddd ZddZddZddZddZdS) TelegramChatApiLoaderz)Load `Telegram` chat json directory dump.Ntelegram_data.json chat_entityOptional[EntityLike]api_id Optional[int]api_hash Optional[str]usernamer!rcL||_||_||_||_||_dS)aIInitialize with API parameters. Args: chat_entity: The chat entity to fetch data from. api_id: The API ID. api_hash: The API hash. username: The username. file_path: The file path to save the data to. Defaults to "telegram_data.json". N)rcrergrir!)r"rcrergrir!s rr#zTelegramChatApiLoader.__init__Vs,$'     "rrNonec 4Kddlm}g}||j|j|j4d{V}||j23d{V}|jdu}|r |jjnd}| |j |j |j |j||dc6 dddd{Vn#1d{VswxYwYt|jdd5}t#j||dd ddddS#1swxYwYdS) z8Fetch data from Telegram API and save it as a JSON file.r)TelegramClientN) sender_idrr message.idis_reply reply_to_idwzutf-8r(F) ensure_asciiindent) telethon.syncrmrirerg iter_messagesrcreply_toreply_to_msg_idrXrnrr isoformatidr6r!r7dump)r"rmdataclientr-rprqr;s rfetch_data_from_telegramz.TelegramChatApiLoader.fetch_data_from_telegramns(000000!>$-dmLL       PV!'!5!5d6F!G!G       g"+47BJTg.>>PT  %,%6 ' ' 6 6 8 8&-j$,'2     "H!G                           $.# 8 8 8 =A IdAE! < < < < = = = = = = = = = = = = = = = = = =s0B?B+AB?? C  C 'D  DDr} pd.DataFramercd fd ||d}||dd g d td <fd |d D}|S)a Create a dictionary of message threads from the given data. Args: data (pd.DataFrame): A DataFrame containing the conversation data with columns: - message.sender_id - text - date - message.id - is_reply - reply_to_id Returns: dict: A dictionary where the key is the parent message ID and the value is a list of message IDs in ascending order. parent_idint reply_datarr List[int]c||d|kd}g}|D]}||g||zz }|S)a^ Recursively find all replies to a given parent message ID. Args: parent_id (int): The parent message ID. reply_data (pd.DataFrame): A DataFrame containing reply messages. Returns: list: A list of message IDs that are replies to the parent message ID. rqro)tolist)rrdirect_replies all_repliesreply_id find_repliess rrz@TelegramChatApiLoader._get_message_threads..find_repliessj( =(AY(NOfhh  K* O OzLL:,N,NNN  rrprq)subsetc2i|]}||g|zSrr)r/rrreply_messagess r z>TelegramChatApiLoader._get_message_threads..s>     {\\)^%L%LL   rro)rrrrrr)dropnaastyper)r"r}parent_messagesmessage_threadsrrs @@r_get_message_threadsz*TelegramChatApiLoader._get_message_threadss$      0Z 001d:./66}o6NN)7}(E(L(LS(Q(Q}%     ,\:    rrDict[int, List[int]]cJd}|D]y\}}||d|dd}d|D}|d|dzz }z|S) aw Combine the message texts for each parent message ID based on the list of message threads. Args: message_threads (dict): A dictionary where the key is the parent message ID and the value is a list of message IDs in ascending order. data (pd.DataFrame): A DataFrame containing the conversation data: - message.sender_id - text - date - message.id - is_reply - reply_to_id Returns: str: A combined string of message texts sorted by date. r*ror)byrc,g|]}t|Sr)r)r/elems rrQz@TelegramChatApiLoader._combine_message_texts..sAAA4SYYAAArrJz. )itemsisin sort_valuesrr9strip)r"rr} combined_textr message_ids message_textss r_combine_message_textsz,TelegramChatApiLoader._combine_message_textss* '6&;&;&=&= = = "I{T,',,[99:''0  BA=AAAM SXXm44u< AA" B++B/2B/7B<<C)NNNNrb) rcrdrerfrgrhrirhr!r)rrk)r}rrr)rrr}rrrr=) r>r?r@rAr#rrrr8rrrraraSs33-1 $"&"&- #####0====.9999v$%$%$%$%L",",",",",",rra)rrrr)rrBrr$) __future__rrr7pathlibrtypingrrrrr langchain_core.documentsr )langchain_community.document_loaders.baser rrtelethon.hintsr rrr_raTelegramChatLoaderrrrrsK"""""" ==============------@@@@@@*))))))----@@@@@Z@@@0Du,u,u,u,u,Ju,u,u,r,r