Ng(<ddlmZddlZddlmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZGddeZGd d ZGd d eZGd deZGddZdS)) annotationsN)AnyDictListTuple TypedDictUnionDocument)Language)RecursiveCharacterTextSplitterc$eZdZdZdfd ZxZS)MarkdownTextSplitterz=Attempts to split the text along Markdown-formatted headings.kwargsrreturnNonec ||tj}tjdd|i|dS)z"Initialize a MarkdownTextSplitter. separatorsN)get_separators_for_languager MARKDOWNsuper__init__)selfrr __class__s ]/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_text_splitters/markdown.pyrzMarkdownTextSplitter.__init__sA55h6GHH 99J9&99999)rrrr)__name__ __module__ __qualname____doc__r __classcell__)rs@rrr sCGG::::::::::rrc0eZdZdZ ddd ZddZddZdS)MarkdownHeaderTextSplitterz4Splitting markdown files based on specified headers.FTheaders_to_split_onList[Tuple[str, str]]return_each_linebool strip_headerscR||_t|dd|_||_dS)a Create a new MarkdownHeaderTextSplitter. Args: headers_to_split_on: Headers we want to track return_each_line: Return each line w/ associated headers strip_headers: Strip split headers from the content of the chunk c,t|dS)Nr)len)splits rz5MarkdownHeaderTextSplitter.__init__..*s3uQx==rT)keyreverseN)r'sortedr%r)rr%r'r)s rrz#MarkdownHeaderTextSplitter.__init__s@!1$* %@%@$$ $ $  +rlinesList[LineType]rList[Document]c4g}|D]}|r8|dd|dkr |ddxxd|dzz cc<=|r|dd|dkrt|ddt|dkri|ddddddkr8|js1|ddxxd|dzz cc<|d|dd<|| d|DS) zCombine lines with common metadata into chunks Args: lines: Line of text / associated header metadata metadatacontentz  r#cHg|]}t|d|d Sr9r8 page_contentr8r .0chunks r zHMarkdownHeaderTextSplitter.aggregate_lines_to_chunks..Ts?    % "2U:=N O O O   r)r,r-r)append)rr3aggregated_chunkslines raggregate_lines_to_chunksz4MarkdownHeaderTextSplitter.aggregate_lines_to_chunks/sn -/ / /D! /%b)*5j9III ""%i000FT)_4LL0000! /%b)*5j9III)"-j9::SjAQ=R=RRR%b))4::4@@DQG3NN*O""%i000FT)_4LL000484D!"%j11"((....  *    rtextstrc|d}g}g}i}g}i}d}d} |D]} | } dttj| } |sM| dr| ddkrd}d} n3| drd}d} n| | rd}d} |r|| |j D]\} } | | rpt| t| ks| t| dkr6| | d }|r_|d d |krM| }|d |vr| |d |r|d d |kM|| | t| d d}|||d|| <|rQ|d|| d| |js|| nm| r|| nS|rQ|d|| d| | }|r+|d||d|js||Sd|DS)zASplit markdown file Args: text: Markdown filer:Fz```Tz~~~ Nr;r7levelname)rNrOdatarP)r9r8cHg|]}t|d|d Sr=r r@s rrCz9MarkdownHeaderTextSplitter.split_text..s?eI&6zARSSSr)r-stripjoinfilterrI isprintable startswithcountrDr%r,popcopyclearr)r'rG)rrHr3lines_with_metadatacurrent_contentcurrent_metadata header_stackinitial_metadata in_code_block opening_fencerF stripped_lineseprOcurrent_header_level popped_headerheaders r split_textz%MarkdownHeaderTextSplitter.split_textYs  4  .0%'+-*, +-  U 7U 7D JJLLMGGF3?M$J$JKKM ' ++E22*}7J7J57Q7QUV7V7V$(M$)MM"--e44*$(M$)M ++M::'$)M$&M &&}555"5< ,< , T ++C00/ &&#c((22mCHH6MQT6T6T'/2yy~~,) L ,R 0 9=Q Q Q-9,<,<,>,>M -V48HHH 0 4 4]65J K K K) L ,R 0 9=Q Q Q&:$($1#c((**$=$C$C$E$E.. %++F33317(.'0+22+/99_+E+E,<,A,A,C,C (--///->'..}===E ,#**=9999$,'..'+yy'A'A(8(=(=(?(? $))+++/4466      & & IIo66DTUU    $ 112EFF F0 rN)FT)r%r&r'r(r)r()r3r4rr5rHrIrr5)rrr r!rrGrgrrrr$r$sj>> "'" +++++.( ( ( ( Txxxxxxrr$c(eZdZUdZded<ded<dS)LineTypezLine type as typed dict.zDict[str, str]r8rIr9Nrrr r!__annotations__rrrrjrjs+""LLLLLrrjc2eZdZUdZded<ded<ded<dS) HeaderTypezHeader type as typed dict.intrNrIrOrPNrkrrrrnrns1$$JJJ III IIIIIrrncleZdZdZdddddddZ d&d'dZd(dZd)dZd*dZd+d Z d,d#Z d,d$Z d,d%Z d S)-&ExperimentalMarkdownSyntaxTextSplittera An experimental text splitter for handling Markdown syntax. This splitter aims to retain the exact whitespace of the original text while extracting structured metadata, such as headers. It is a re-implementation of the MarkdownHeaderTextSplitter with notable changes to the approach and additional features. Key Features: - Retains the original whitespace and formatting of the Markdown text. - Extracts headers, code blocks, and horizontal rules as metadata. - Splits out code blocks and includes the language in the "Code" metadata key. - Splits text on horizontal rules (`---`) as well. - Defaults to sensible splitting behavior, which can be overridden using the `headers_to_split_on` parameter. Parameters: ---------- headers_to_split_on : List[Tuple[str, str]], optional Headers to split on, defaulting to common Markdown headers if not specified. return_each_line : bool, optional When set to True, returns each line as a separate chunk. Default is False. Usage example: -------------- >>> headers_to_split_on = [ >>> ("#", "Header 1"), >>> ("##", "Header 2"), >>> ] >>> splitter = ExperimentalMarkdownSyntaxTextSplitter( >>> headers_to_split_on=headers_to_split_on >>> ) >>> chunks = splitter.split(text) >>> for chunk in chunks: >>> print(chunk) This class is currently experimental and subject to change based on feedback and further development. zHeader 1zHeader 2zHeader 3zHeader 4zHeader 5zHeader 6)r;z##z###z####z#####z######NFTr%"Union[List[Tuple[str, str]], None]r'r(r)cg|_td|_g|_||_|rt ||_n |j|_||_dS)NrKr?) chunksr current_chunkcurrent_header_stackr)dictsplittable_headersDEFAULT_HEADER_KEYSr'r2s rrz/ExperimentalMarkdownSyntaxTextSplitter.__init__sc ') %2666;=!*  ?&*+>&?&?D # #&*&>D # 0rrHrIrr5cz|d}|rp|d}||}||}||}|r~||js|jxj|z c_t| d}| d}| ||n|rk|| |||j_| d|jj d<|n,|r|n|jxj|z c_|p||jrd|jDS|jS)NT)keependsrrLCodecg|]J}|jD].}||t||j/KS)r>)r? splitlinesisspacer r8)rArBrFs rrCzEExperimentalMarkdownSyntaxTextSplitter.split_text..Isu!.99;; !%  dU^DDDr)rrX _match_header _match_code _match_horz_complete_chunk_docr)rvr?r,group_resolve_header_stack_resolve_code_chunkr8r'ru) rrH raw_linesraw_line header_match code_match horz_match header_depth header_texts rrgz1ExperimentalMarkdownSyntaxTextSplitter.split_text&sOOTO22  < }}Q''H--h77L))(33J))(33J <((***)@&33x?33 #<#5#5a#8#899 *0033 **<EEEE <((***262J2Ji33"/7A6F6Fq6I6I"+F3((**** <((****"//8;//3 <6   """   ![  {rrrorrct|jD]4\}\}}||kr&||f|j|<|jd|dz|_dS5|j||fdS)NrL) enumeraterwrD)rrridepth_s rrz>>LtrcTfddD}td|DdS)Nc:g|]}tj|SrrrrArulerFs rrCzFExperimentalMarkdownSyntaxTextSplitter._match_code..ws%OOOD28D$''OOOr)z^```(.*)z^~~~(.*)c3K|]}||V dSNrrArs r zEExperimentalMarkdownSyntaxTextSplitter._match_code..x'99u59U999999rnextrrFmatchess ` rrz2ExperimentalMarkdownSyntaxTextSplitter._match_codevs<OOOO4NOOO999994@@@rcTfddD}td|DdS)Nc:g|]}tj|Srrrs rrCzFExperimentalMarkdownSyntaxTextSplitter._match_horz..{s2   %)BHT4    r)z ^\*\*\*+\nz^---+\nz^___+\nc3K|]}||V dSrrrs rrzEExperimentalMarkdownSyntaxTextSplitter._match_horz..~rrrrs ` rrz2ExperimentalMarkdownSyntaxTextSplitter._match_horzzsL    -T   999994@@@r)NFT)r%rrr'r(r)r(rh)rrorrIrr)rrIrrrrI)rr)rFrIrr) rrr r!rzrrgrrrrrrrrrrqrqs&&R CG!&" 11111"))))VFFFF 7 7 7 7AAAAAAAAAArrq) __future__rrtypingrrrrrr langchain_core.documentsr langchain_text_splitters.baser "langchain_text_splitters.characterr rr$rjrnrqrrrrs|"""""" ;;;;;;;;;;;;;;;;------222222MMMMMM:::::9:::||||||||~y[A[A[A[A[A[A[A[A[A[Ar