Ng!dZddlmZddlmZmZmZddlmZm Z m Z m Z m Z m Z mZddlmZddlmZddddddddddZddZGdde ZdS)zdImplementation of chunking by title. Main entry point is the `@add_chunking_strategy()` decorator. ) annotations)IterableIteratorOptional)CHUNK_MULTI_PAGE_DEFAULTBoundaryPredicateChunkingOptionsPreChunkCombiner PreChunkeris_on_next_pageis_title)Element) lazypropertyNcombine_text_under_n_charsinclude_orig_elementsmax_charactersmultipage_sectionsnew_after_n_charsoverlap overlap_allelementsIterable[Element]r Optional[int]rOptional[bool]rrrrrreturn list[Element]c dt|||||||}t||S)a Uses title elements to identify sections within the document for chunking. Splits off into a new CompositeElement when a title is detected or if metadata changes, which happens when page numbers or sections change. Cuts off sections once they have exceeded a character length of max_characters. Parameters ---------- elements A list of unstructured elements. Usually the output of a partition function. combine_text_under_n_chars Combines elements (for example a series of titles) until a section reaches a length of n characters. Defaults to `max_characters` which combines chunks whenever space allows. Specifying 0 for this argument suppresses combining of small chunks. Note this value is "capped" at the `new_after_n_chars` value since a value higher than that would not change this parameter's effect. include_orig_elements When `True` (default), add elements from pre-chunk to the `.metadata.orig_elements` field of the chunk(s) formed from that pre-chunk. Among other things, this allows access to original-element metadata that cannot be consolidated and is dropped in the course of chunking. max_characters Chunks elements text and text_as_html (if present) into chunks of length n characters (hard max) multipage_sections If True, sections can span multiple pages. Defaults to True. new_after_n_chars Cuts off new sections once they reach a length of n characters (soft max). Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). overlap Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an oversized element is divided into multiple chunks by text-splitting. overlap_all Default: `False`. When `True`, apply overlap between "normal" chunks formed from whole elements and not subject to text-splitting. Use this with caution as it entails a certain level of "pollution" of otherwise clean semantic chunk boundaries. r)_ByTitleChunkingOptionsnew_chunk_by_title) rrrrrrrroptss W/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/chunking/title.pychunk_by_titler$sFh # & &#=3%-+ '  D 8T * **r"rcttj|||}d|DS)z-Implementation of actual "by-title" chunking.)r"c@g|]}|D]}|S) iter_chunks).0 pre_chunkchunks r# z#_chunk_by_title.._s1 S S Si9;P;P;R;R S S%E S S S Sr%)r r iter_pre_chunksiter_combined_pre_chunks)rr" pre_chunkss r#r!r!WsP""8T22   T S: S S SSr%cleZdZdZed dZed dZed dZdfd ZxZ S)ra9Adds the by-title-specific chunking options to the base case. `by_title`-specific options: combine_text_under_n_chars A remedy to over-chunking caused by elements mis-identified as Title elements. Every Title element would start a new chunk and this setting mitigates that, at the expense of sometimes violating legitimate semantic boundaries. multipage_sections Indicates that page-boundaries should not be respected while chunking, i.e. elements appearing on two different pages can appear in the same chunk. rtuple[BoundaryPredicate, ...]c>dfd }t|S)a(The semantic-boundary detectors to be applied to break pre-chunks. For the `by_title` strategy these are sections indicated by a title (section-heading), an explicit section metadata item (only present for certain document types), and optionally page boundaries. rIterator[BoundaryPredicate]c3PKtVjstVdSdS)N)r rr )selfsr#iter_boundary_predicateszM_ByTitleChunkingOptions.boundary_predicates..iter_boundary_predicatesys=NNN* (%''''''' ( (r%)rr4)tuple)r6r7s` r#boundary_predicatesz+_ByTitleChunkingOptions.boundary_predicatesps; ( ( ( ( ( ( --//000r%intcL|jd}||jn|S)aCombine consecutive text pre-chunks if former is smaller than this and both will fit. - Does not combine table chunks with text chunks even if they would both fit in the chunking window. - Does not combine text chunks if together they would exceed the chunking window. - Defaults to `max_characters` when not specified. - Is reduced to `new_after_n_chars` when it exceeds that value. r)_kwargsgethard_maxr6 arg_values r#rz2_ByTitleChunkingOptions.combine_text_under_n_charss+L$$%ABB ) 1t}}y@r%boolcf|jd}|tnt|S)z0When False, break pre-chunks on page-boundaries.r)r<r=rrAr?s r#rz*_ByTitleChunkingOptions.multipage_sectionss1L$$%9:: +4+<''$y//Qr%Nonect|jdkrtd|j|j|jkrtd|jd|jdS)z2Raise ValueError if request option-set is invalid.rz8'combine_text_under_n_chars' argument must be >= 0, got zR'combine_text_under_n_chars' argument must not exceed `max_characters` value, got z > N)super _validater ValueErrorr>)r6 __class__s r#rFz!_ByTitleChunkingOptions._validates   *Q . .:7::   *T] : :S#>SSCG=SS  ; :r%)rr2)rr:)rrA)rrC) __name__ __module__ __qualname____doc__rr9rrrF __classcell__)rHs@r#rrbs   1 1 1\ 1 A A A\ ARRR\R r%)rrrrrrrrrrrrrrrrrr)rrr"rrr)rL __future__rtypingrrrunstructured.chunking.baserrr r r r r unstructured.documents.elementsrunstructured.utilsrr$r!rr(r%r#rSsM #"""""//////////433333++++++ 15,0$()-'+!"&=+=+=+=+=+=+@TTTTGGGGGoGGGGGr%