§ íNg™ãó|—dZddlmZddlmZmZddlmZmZddl m Z ddddddœdd„Z dd„Z Gd„de¦«Z dS)a“Implementation of baseline chunking. This is the "plain-vanilla" chunking strategy. All the fundamental chunking behaviors are present in this strategy and also in all other strategies. Those are: - Maximally fill each chunk with sequential elements. - Isolate oversized elements and divide (only) those chunks by text-splitting. - Overlap when requested. "Fancier" strategies add higher-level semantic-unit boundaries to be respected. For example, in the by-title strategy, section boundaries are respected, meaning a chunk never contains text from two different sections. When a new section is detected the current chunk is closed and a new one started. é)Ú annotations)ÚIterableÚOptional)ÚChunkingOptionsÚ PreChunker)ÚElementN©Úinclude_orig_elementsÚmax_charactersÚnew_after_n_charsÚoverlapÚ overlap_allÚelementsúIterable[Element]r úOptional[bool]r ú Optional[int]r r rÚreturnú list[Element]có`—t |||||¬¦«}t||¦«S)avCombine sequential `elements` into chunks, respecting specified text-length limits. Produces a sequence of `CompositeElement`, `Table`, and `TableChunk` elements (chunks). Parameters ---------- elements A list of unstructured elements. Usually the output of a partition function. include_orig_elements When `True` (default), add elements from pre-chunk to the `.metadata.orig_elements` field of the chunk(s) formed from that pre-chunk. Among other things, this allows access to original-element metadata that cannot be consolidated and is dropped in the course of chunking. max_characters Hard maximum chunk length. No chunk will exceed this length. A single element that exceeds this length will be divided into two or more chunks using text-splitting. new_after_n_chars A chunk that of this length or greater is not extended to include the next element, even if that element would fit without exceeding `max_characters`. A "soft max" length that can be used in conjunction with `max_characters` to limit most chunks to a preferred length while still allowing larger elements to be included in a single chunk without resorting to text-splitting. Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). overlap Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an oversized element is divided into multiple chunks by text-splitting. overlap_all Default: `False`. When `True`, apply overlap between "normal" chunks formed from whole elements and not subject to text-splitting. Use this with caution as it produces a certain level of "pollution" of otherwise clean semantic chunk boundaries. r )Ú_BasicChunkingOptionsÚnewÚ_chunk_elements)rr r r r rÚoptss úW/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/chunking/basic.pyÚchunk_elementsrs@€õX !× $Ò $Ø3Ø%Ø+ØØð %ñ ô €Dõ ˜8 TÑ *Ô *Ð*órrcó@—d„tj||¦«D¦«S)z(Implementation of actual basic chunking.có@—g|]}| ¦«D]}|‘ŒŒS©)Ú iter_chunks)Ú.0Ú pre_chunkÚchunks rú z#_chunk_elements..SsK€ð ð ð à Ø×*Ò*Ñ,Ô,ð ð ð ð ð ð ð ð r)rÚiter_pre_chunks)rrs rrrOs0€ð ð å#Ô3°H¸dÑCÔCð ñ ô ðrcó—eZdZdZdS)rzOptions for `basic` chunking.N)Ú__name__Ú __module__Ú __qualname__Ú__doc__rrrrrZs€€€€€Ø'Ð'Ð'Ð'r)rrr rr rr rr rrrrr)rrrrrr)r*Ú __future__rÚtypingrrÚunstructured.chunking.baserrÚunstructured.documents.elementsrrrrrrrúr/sÝðð ð ð#Ð"Ð"Ð"Ð"Ð"à%Ð%Ð%Ð%Ð%Ð%Ð%Ð%àBÐBÐBÐBÐBÐBÐBÐBØ3Ð3Ð3Ð3Ð3Ð3ð -1Ø$(Ø'+Ø!Ø"&ð4+ð4+ð4+ð4+ð4+ð4+ðnðððð(ð(ð(ð(ð(˜Oñ(ô(ð(ð(ð(r