§ ¦ìNgéMãó—UdZddlmZddlZddlZddlmZmZddlm Z ddl mZmZm Z mZmZmZmZmZddlmZddlmZdd lmZdd lmZeZded<eje¦«Zd/d„Z d0d„Z!Gd„dee¦«Z"eddd¬¦«Gd„de"¦«¦«Z# d1d2d(„Z$eddd¬¦«Gd)„d*e"¦«¦«Z%eddd¬¦«Gd+„d,e"¦«¦«Z&eddd¬¦«Gd-„d.e"¦«¦«Z'dS)3z,Loader that uses unstructured to load files.é)ÚannotationsN)ÚABCÚabstractmethod)ÚPath)ÚIOÚAnyÚCallableÚIteratorÚListÚOptionalÚSequenceÚUnion)Ú deprecated)ÚDocument)Ú TypeAlias)Ú BaseLoaderrÚElementÚmin_versionÚstrÚreturnÚboolcó—ddlm}td„| d¦«D¦«¦«}| d¦«d}td„| d¦«D¦«¦«}||kS)zjCheck if the installed `Unstructured` version exceeds the minimum version for the feature in question.r)Ú__version__có,—g|]}t|¦«‘ŒS©©Úint©Ú.0Úxs úm/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/unstructured.pyú z6satisfies_min_unstructured_version..s€ÐFÐFÐF¨!s 1™vœvÐFÐFÐFóú.ú-có,—g|]}t|¦«‘ŒSrrrs r!r"z6satisfies_min_unstructured_version..!s€Ð:Ð:Ð:AˆQ‰ŒÐ:Ð:Ð:r#)Úunstructured.__version__rÚtupleÚsplit)rÚ__unstructured_version__Úmin_version_tupleÚ_unstructured_versionÚunstructured_version_tuples r!Ú"satisfies_min_unstructured_versionr.s›€ðQÐPÐPÐPÐPÐPåÐFÐF¨{×/@Ò/@ÀÑ/EÔ/EÐFÑFÔFÑGÔGÐð5×:Ò:¸3Ñ?Ô?ÀÔBÐÝ!&Ø:Ð:Ð.×4Ò4°SÑ9Ô9Ð:Ñ:Ô:ñ"ô"Ðð&Ð):Ò:Ð:r#Úmin_unstructured_versionÚNonecóJ—t|¦«std|›d¦«‚dS)zWRaise an error if the `Unstructured` version does not exceed the specified minimum.zunstructured>=z is required in this loader.N)r.Ú ValueError©r/s r!Úvalidate_unstructured_versionr4's@€õ.Ð.FÑGÔGð ÝØSÐ5ÐSÐSÐSñ ô ð ð ð r#cóh—eZdZdZ ddd „Zedd „¦«Zedd„¦«Zdd„Zdd„Z dd„Z dS)ÚUnstructuredBaseLoaderz%Base Loader that uses `Unstructured`.ÚsingleNÚmoderÚpost_processorsú$Optional[List[Callable[[str], str]]]Úunstructured_kwargsrcó2— ddl}n#t$rtd¦«‚wxYwhd£}||vrtd|›d|›d¦«‚td¦«sd |vr| d ¦«| ||¦«||_||_|pg|_dS) úInitialize with file path.rNzQunstructured package not found, please install it with `pip install unstructured`>Úpagedr7ÚelementszGot z# for `mode`, but should be one of `ú`z0.5.4Ústrategy) ÚunstructuredÚImportErrorr2r.ÚpopÚ5_check_if_both_mode_and_chunking_strategy_are_by_pager8r;r9)Úselfr8r9r;rBÚ_valid_modess r!Ú__init__zUnstructuredBaseLoader.__init__3sü€ð ØÐÐÐÐøÝð ð ð Ýð-ñôð ð øøøð7Ð6Ð6ˆØ|Ð#Ð#ÝØOtÐOÐOÀÐOÐOÐOñôð õ2°'Ñ:Ô:ð 4ØÐ0Ð0Ð0Ø#×'Ò'¨ Ñ3Ô3Ð3à×BÒBØÐ%ñ ô ð ðˆŒ Ø#6ˆÔ Ø.Ð4°"ˆÔÐÐs‚‡!rú List[Element]có—dS)z Get elements.Nr©rFs r!Ú _get_elementsz$UnstructuredBaseLoader._get_elementsVó€€€r#údict[str, Any]có—dS)z$Get file_path metadata if available.NrrKs r!Ú _get_metadataz$UnstructuredBaseLoader._get_metadataZrMr#r?cóN—|D]!}|jD]}| |¦«ŒŒ"|S©zÛApply post processing functions to extracted unstructured elements. Post processing functions are str -> str callables passed in using the post_processors kwarg when the loader is instantiated. ©r9Úapply©rFr?ÚelementÚpost_processors r!Ú_post_process_elementsz-UnstructuredBaseLoader._post_process_elements^óD€ð ð .ð .ˆGØ"&Ô"6ð .ð .Ø— ’ ˜nÑ-Ô-Ð-Ð-ð .àˆr#úIterator[Document]c#óVK—| ¦«}| |¦«|jdkrâ|D]Ý}| ¦«}t |d¦«r,| |j ¦«¦«t |d¦«r |j|d<| ¦« d¦«r*| ¦« d¦«|d<tt|¦«|¬¦«V—ŒÞdS|jdkrt d¦«i}i}|D]Â}| ¦«}t |d¦«r,| |j ¦«¦«| dd ¦«}||vrt|¦«d z||<|||<Œ‡||xxt|¦«d zz cc<|| |¦«ŒÃ| ¦«D]!}t||||¬¦«V—Œ"dS|jdkrH| ¦«}d d„|D¦«¦«}t||¬¦«V—dSt!d |j›d¦«‚)z Load file.r?ÚmetadataÚcategoryÚ element_id)Úpage_contentr\r>z®`mode='paged'` is deprecated in favor of the 'by_page' chunking strategy. Learn more about chunking here: https://docs.unstructured.io/open-source/core-functionality/chunkingÚpage_numberéz r7có,—g|]}t|¦«‘ŒSr©r)rÚels r!r"z4UnstructuredBaseLoader.lazy_load..—s€Ð;Ð;Ð;¨B¥ B¡¤Ð;Ð;Ð;r#zmode of z not supported.N)rLrXr8rPÚhasattrÚupdater\Úto_dictr]ÚgetrrÚloggerÚwarningÚkeysÚjoinr2) rFr?rVr\Ú text_dictÚ meta_dictr`ÚkeyÚtexts r!Ú lazy_loadz UnstructuredBaseLoader.lazy_loadisøèè€à×%Ò%Ñ'Ô'ˆØ×#Ò# HÑ-Ô-Ð-ØŒ9˜ Ò"Ð"Ø#ð Mð MØ×-Ò-Ñ/Ô/õ˜7 JÑ/Ô/ð@Ø—O’O GÔ$4×$<Ò$<Ñ$>Ô$>Ñ?Ô?Ð?Ý˜7 JÑ/Ô/ð<Ø+2Ô+;H˜ZÑ(Ø—?’?Ñ$Ô$×(Ò(¨Ñ6Ô6ðQØ-4¯_ª_Ñ->Ô->×-BÒ-BÀ<Ñ-PÔ-PH˜\Ñ*ÝC°©L¬LÀ8ÐLÑLÔLÐLÐLÐLÐLð Mð MðŒY˜'Ò !Ñ !ÝNŠNðXñ ô ð ð )+ˆIØ35ˆIà#ð <ð <Ø×-Ò-Ñ/Ô/Ý˜7 JÑ/Ô/ð@Ø—O’O GÔ$4×$<Ò$<Ñ$>Ô$>Ñ?Ô?Ð?Ø&Ÿlšl¨=¸!Ñ<Ô<ð iÐ/Ð/å-0°©\¬\¸FÑ-BI˜kÑ*Ø-5I˜kÑ*Ð*ð˜kÐ*Ð*Ô*c°'©l¬l¸VÑ.CÑCÐ*Ð*Ñ*Ø˜kÔ*×1Ò1°(Ñ;Ô;Ð;Ð;ð!—~’~Ñ'Ô'ð Uð UÝ¨I°c¬NÀYÈsÄ^ÐTÑTÔTÐTÐTÐTÐTð Uð Uà ŒY˜(Ò "Ð "Ø×)Ò)Ñ+Ô+ˆHØ—;’;Ð;Ð;°(Ð;Ñ;Ô;Ñ<Ô<ˆDÝ¨°xÐ@Ñ@Ô@Ð@Ð@Ð@Ð@Ð@åÐB¨¬ ÐBÐBÐBÑCÔCÐCr#r0cóf—|dkr(| d¦«dkrtd¦«‚dSdS)Nr>Úchunking_strategyÚby_pageziOnly one of `chunking_strategy='by_page'` or `mode='paged'` may be set. `chunking_strategy` is preferred.)rhr2)rFr8r;s r!rEzLUnstructuredBaseLoader._check_if_both_mode_and_chunking_strategy_are_by_pageœsK€ð GŠOˆOØ#×'Ò'Ð(;Ñ<Ô<À ÒIÐIåð:ñôð ð ˆOØIÐIr#)r7N)r8rr9r:r;r©rrI©rrN©r?rIrrI)rrZ)r8rr;rNrr0)Ú__name__Ú __module__Ú__qualname__Ú__doc__rHrrLrPrXrqrErr#r!r6r60s½€€€€€Ø/Ð/ðØ@Dð!5ð!5ð!5ð!5ð!5ðFðððñ„^ððð3ð3ð3ñ„^ð3ð ð ð ð ð1Dð1Dð1Dð1Dðf ð ð ð ð ð r#r6z0.2.8z1.0z)langchain_unstructured.UnstructuredLoader)ÚsinceÚremovalÚalternative_importcó:‡—eZdZdZddœdˆfd „Zdd „Zdd„ZˆxZS)ÚUnstructuredFileLoaderaôLoad files using `Unstructured`. The file loader uses the unstructured partition function and will automatically detect the file type. You can run the loader in different modes: "single", "elements", and "paged". The default "single" mode will return a single langchain Document object. If you use "elements" mode, the unstructured library will split the document into elements such as Title and NarrativeText and return those as individual langchain Document objects. In addition to these post-processing modes (which are specific to the LangChain Loaders), Unstructured has its own "chunking" parameters for post-processing elements into more useful chunks for uses cases such as Retrieval Augmented Generation (RAG). You can pass in additional unstructured kwargs to configure different unstructured settings. Examples -------- from langchain_community.document_loaders import UnstructuredFileLoader loader = UnstructuredFileLoader( "example.pdf", mode="elements", strategy="fast", ) docs = loader.load() References ---------- https://docs.unstructured.io/open-source/core-functionality/partitioning https://docs.unstructured.io/open-source/core-functionality/chunking r7©r8Ú file_pathú'Union[str, List[str], Path, List[Path]]r8rr;rcóL•—||_t¦«jdd|i|¤ŽdS©r=r8Nr)r‚ÚsuperrH)rFr‚r8r;Ú __class__s €r!rHzUnstructuredFileLoader.__init__Ës5ø€ð#ˆŒà‰ŒÔÐ:Ð:˜dÐ:Ð&9Ð:Ð:Ð:Ð:Ð:r#rrIcóx—ddlm}t|jt¦«rTg}|jD]H}t|t ¦«rt |¦«}| |dd|i|j¤Ž¦«ŒI|St|jt ¦«rt |j¦«|_|dd|ji|j¤ŽS)Nr©Ú partitionÚfilenamer) Úunstructured.partition.autorŠÚ isinstancer‚ÚlistrrÚextendr;)rFrŠr?Úfiles r!rLz$UnstructuredFileLoader._get_elements×sÕ€Ø9Ð9Ð9Ð9Ð9Ð9åd”n¥dÑ+Ô+ð RØ&(ˆHØœð Vð VÝ˜d¥DÑ)Ô)ð%Ý˜t™9œ9DØ—’ Ð TÐ T°4Ð T¸4Ô;SÐ TÐ TÑUÔUÐUÐUØˆOå˜$œ.$Ñ/Ô/ð 5Ý!$ T¤^Ñ!4Ô!4”Ø9ÐQÐQ d¤nÐQ¸Ô8PÐQÐQÐQr#rNcó—d|jiS©NÚsource©r‚rKs r!rPz$UnstructuredFileLoader._get_metadataæó€Ø˜$œ.Ð)Ð)r#)r‚rƒr8rr;rrurv)rxryrzr{rHrLrPÚ __classcell__©r‡s@r!r€r€©sƒø€€€€€ððð@ð ;ð ;ð ;ð ;ð ;ð ;ð ;ð ;ð Rð Rð Rð Rð*ð*ð*ð*ð*ð*ð*ð*r#r€ú1https://api.unstructuredapp.io/general/v0/generalÚr‚ú-Union[str, List[str], Path, List[Path], None]rú+Union[IO[bytes], Sequence[IO[bytes]], None]Úapi_urlÚapi_keyr;rrIcó —t|t¦«x}rd„|D¦«}t|t¦«s|r1ddlm}|d||||dœ|¤Ž}g}|D]} | | ¦«Œ|Sddlm} | d|t|¦«nd|||dœ|¤ŽS) z8Retrieve a list of elements from the `Unstructured API`.có,—g|]}t|¦«‘ŒSrrc)rÚpaths r!r"z)get_elements_from_api..ós€Ð5Ð5Ð5 4•S˜‘Y”YÐ5Ð5Ð5r#r)Úpartition_multiple_via_api)Ú filenamesÚfilesrrœ)Úpartition_via_apiN)r‹rrrœr)rrŽr Úunstructured.partition.apir¡rr¤r)r‚rrœrr;Úis_listr¡Ú _doc_elementsr?Ú _elementsr¤s r!Úget_elements_from_apir©ês€õ˜YÑ-Ô-Ð-€wð6Ø5Ð5¨9Ð5Ñ5Ô5ˆ Ý$Ñ!Ô!ð Wð ØIÐIÐIÐIÐIÐIà2Ð2ð ØØØØð ð ð "ð ð ˆ ðˆØ&ð 'ð 'ˆIØOŠO˜IÑ&Ô&Ð&Ð&Øˆà@Ð@Ð@Ð@Ð@Ð@à Ð ð Ø'0Ð'<•S˜‘^”^^À$ØØØð ð ð "ð ð ð r#cóF‡—eZdZdZddddœdˆfd„Zdd„Zdd„Zdd„ZˆxZS)ÚUnstructuredAPIFileLoadera•Load files using `Unstructured` API. By default, the loader makes a call to the hosted Unstructured API. If you are running the unstructured API locally, you can change the API rule by passing in the url parameter when you initialize the loader. The hosted Unstructured API requires an API key. See the links below to learn more about our API offerings and get an API key. You can run the loader in different modes: "single", "elements", and "paged". The default "single" mode will return a single langchain Document object. If you use "elements" mode, the unstructured library will split the document into elements such as Title and NarrativeText and return those as individual langchain Document objects. In addition to these post-processing modes (which are specific to the LangChain Loaders), Unstructured has its own "chunking" parameters for post-processing elements into more useful chunks for uses cases such as Retrieval Augmented Generation (RAG). You can pass in additional unstructured kwargs to configure different unstructured settings. Examples ```python from langchain_community.document_loaders import UnstructuredAPIFileLoader loader = UnstructuredAPIFileLoader( "example.pdf", mode="elements", strategy="fast", api_key="MY_API_KEY", ) docs = loader.load() References ---------- https://docs.unstructured.io/api-reference/api-services/sdk https://docs.unstructured.io/api-reference/api-services/overview https://docs.unstructured.io/open-source/core-functionality/partitioning https://docs.unstructured.io/open-source/core-functionality/chunking r7r˜r™©r8Úurlrr‚úUnion[str, List[str]]r8rrrr;rcó°•—td¬¦«||_||_tjd¦«p||_t ¦«jdd|i|¤ŽdS)r=z0.10.15r3ÚUNSTRUCTURED_API_KEYr8Nr)r4r‚rÚosÚgetenvrr†rH)rFr‚r8rrr;r‡s €r!rHz"UnstructuredAPIFileLoader.__init__7sdø€õ &¸yÐIÑIÔIÐIà"ˆŒØˆŒÝ”yÐ!7Ñ8Ô8ÐC¸GˆŒà‰ŒÔÐ:Ð:˜dÐ:Ð&9Ð:Ð:Ð:Ð:Ð:r#rrNcó—d|jiSr’r”rKs r!rPz'UnstructuredAPIFileLoader._get_metadataIr•r#rIcóJ—td|j|j|jdœ|j¤ŽS)N)r‚rrœr)r©r‚rrr;rKs r!rLz'UnstructuredAPIFileLoader._get_elementsLs:€Ý$ð Ø”nØ”LØ”Hð ð ðÔ&ð ð ð r#r?cóN—|D]!}|jD]}| |¦«ŒŒ"|SrRrSrUs r!rXz0UnstructuredAPIFileLoader._post_process_elementsTrYr#) r‚r®r8rrrrrr;rrvrurw) rxryrzr{rHrPrLrXr–r—s@r!r«r«s™ø€€€€€ð!ð!ðNØFØð ;ð;ð;ð;ð;ð;ð;ð;ð$*ð*ð*ð*ð ð ð ð ð ð ð ð ð ð ð ð r#r«cóB‡—eZdZdZddœdˆfd „Zdd „Zdd„Zdd„ZˆxZS)ÚUnstructuredFileIOLoaderaFLoad file-like objects opened in read mode using `Unstructured`. The file loader uses the unstructured partition function and will automatically detect the file type. You can run the loader in different modes: "single", "elements", and "paged". The default "single" mode will return a single langchain Document object. If you use "elements" mode, the unstructured library will split the document into elements such as Title and NarrativeText and return those as individual langchain Document objects. In addition to these post-processing modes (which are specific to the LangChain Loaders), Unstructured has its own "chunking" parameters for post-processing elements into more useful chunks for uses cases such as Retrieval Augmented Generation (RAG). You can pass in additional unstructured kwargs to configure different unstructured settings. Examples -------- from langchain_community.document_loaders import UnstructuredFileIOLoader with open("example.pdf", "rb") as f: loader = UnstructuredFileIOLoader( f, mode="elements", strategy="fast", ) docs = loader.load() References ---------- https://docs.unstructured.io/open-source/core-functionality/partitioning https://docs.unstructured.io/open-source/core-functionality/chunking r7rrú IO[bytes]r8rr;rcóL•—||_t¦«jdd|i|¤ŽdSr…)rr†rH)rFrr8r;r‡s €r!rHz!UnstructuredFileIOLoader.__init__„s5ø€ðˆŒ Ø‰ŒÔÐ:Ð:˜dÐ:Ð&9Ð:Ð:Ð:Ð:Ð:r#rrIcó6—ddlm}|dd|ji|j¤ŽS)Nrr‰rr)rŒrŠrr;)rFrŠs r!rLz&UnstructuredFileIOLoader._get_elementss4€Ø9Ð9Ð9Ð9Ð9Ð9àˆyÐDÐD˜dœiÐD¨4Ô+CÐDÐDÐDr#rNcó—iS©NrrKs r!rPz&UnstructuredFileIOLoader._get_metadata”ó€Øˆ r#r?cóN—|D]!}|jD]}| |¦«ŒŒ"|SrRrSrUs r!rXz/UnstructuredFileIOLoader._post_process_elements—rYr#)rr¸r8rr;rrurvrw© rxryrzr{rHrLrPrXr–r—s@r!r·r·`s—ø€€€€€ðððDð ;ð ;ð ;ð ;ð ;ð ;ð ;ð ;ðEðEðEðEð ðððð ð ð ð ð ð ð ð r#r·cóF‡—eZdZdZddddœdˆfd„Zdd„Zdd„Zdd„ZˆxZS)ÚUnstructuredAPIFileIOLoaderaíSend file-like objects with `unstructured-client` sdk to the Unstructured API. By default, the loader makes a call to the hosted Unstructured API. If you are running the unstructured API locally, you can change the API rule by passing in the url parameter when you initialize the loader. The hosted Unstructured API requires an API key. See the links below to learn more about our API offerings and get an API key. You can run the loader in different modes: "single", "elements", and "paged". The default "single" mode will return a single langchain Document object. If you use "elements" mode, the unstructured library will split the document into elements such as Title and NarrativeText and return those as individual langchain Document objects. In addition to these post-processing modes (which are specific to the LangChain Loaders), Unstructured has its own "chunking" parameters for post-processing elements into more useful chunks for uses cases such as Retrieval Augmented Generation (RAG). You can pass in additional unstructured kwargs to configure different unstructured settings. Examples -------- from langchain_community.document_loaders import UnstructuredAPIFileLoader with open("example.pdf", "rb") as f: loader = UnstructuredAPIFileIOLoader( f, mode="elements", strategy="fast", api_key="MY_API_KEY", ) docs = loader.load() References ---------- https://docs.unstructured.io/api-reference/api-services/sdk https://docs.unstructured.io/api-reference/api-services/overview https://docs.unstructured.io/open-source/core-functionality/partitioning https://docs.unstructured.io/open-source/core-functionality/chunking r7r˜r™r¬rú%Union[IO[bytes], Sequence[IO[bytes]]]r8rrrr;rcóú•—t|t¦«rtd¬¦«td¬¦«||_||_tjd¦«p||_t¦«j dd|i|¤ŽdS)r=z0.6.3r3z0.6.2r°r8Nr) rr r4rrr±r²rr†rH)rFrr8rrr;r‡s €r!rHz$UnstructuredAPIFileIOLoader.__init__Ísˆø€õdHÑ%Ô%ð LÝ)À7ÐKÑKÔKÐKÝ%¸wÐGÑGÔGÐGàˆŒ ØˆŒÝ”yÐ!7Ñ8Ô8ÐC¸GˆŒà‰ŒÔÐ:Ð:˜dÐ:Ð&9Ð:Ð:Ð:Ð:Ð:r#rrIcóÎ—|j d¦«r=td|j|j d¦«|j|jdœ|j¤ŽStd¦«‚)NÚmetadata_filename)rr‚rrœzLIf partitioning a file via api, metadata_filename must be specified as well.r)r;rhr©rrDrrr2rKs r!rLz)UnstructuredAPIFileIOLoader._get_elementsâsƒ€ØÔ#×'Ò'Ð(;Ñ<Ô<ð Ý(ðØ”YØÔ2×6Ò6Ð7JÑKÔKØœØœð ðð Ô*ððð õð@ñôð r#rNcó—iSr¼rrKs r!rPz)UnstructuredAPIFileIOLoader._get_metadatañr½r#r?cóN—|D]!}|jD]}| |¦«ŒŒ"|SrRrSrUs r!rXz2UnstructuredAPIFileIOLoader._post_process_elementsôrYr#) rrÂr8rrrrrr;rrurvrwr¿r—s@r!rÁrÁ£s™ø€€€€€ð"ð"ðPØFØð ;ð;ð;ð;ð;ð;ð;ð;ð* ð ð ð ððððð ð ð ð ð ð ð ð r#rÁ)rrrr)r/rrr0)NNr˜r™)r‚ršrr›rœrrrr;rrrI)(r{Ú __future__rÚloggingr±ÚabcrrÚpathlibrÚtypingrrr r rrr rÚlangchain_core._api.deprecationrÚlangchain_core.documentsrÚtyping_extensionsrÚ)langchain_community.document_loaders.baserrÚ__annotations__Ú getLoggerÚ__file__rir.r4r6r€r©r«r·rÁrr#r!úrÔsùðØ2Ð2Ð2à"Ð"Ð"Ð"Ð"Ð"à€€€Ø € € € Ø#Ð#Ð#Ð#Ð#Ð#Ð#Ð#ØÐÐÐÐÐØOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOÐOà6Ð6Ð6Ð6Ð6Ð6Ø-Ð-Ð-Ð-Ð-Ð-Ø'Ð'Ð'Ð'Ð'Ð'à@Ð@Ð@Ð@Ð@Ð@à€ÐÐÐÑà ˆÔ ˜8Ñ $Ô $€ð;ð;ð;ð;ð" ð ð ð ðvðvðvðvðv˜Z¨ñvôvðvðr€Ø ØØBðñôð 9*ð9*ð9*ð9*ð9*Ð3ñ9*ô9*ñôð 9*ðz@DØ8<ØFØð ! ð! ð! ð! ð! ðH€Ø ØØBðñôð JðJðJðJðJÐ 6ñJôJñôð JðZ€Ø ØØBðñôð ;ð;ð;ð;ð;Ð5ñ;ô;ñôð ;ð|€Ø ØØBðñôð UðUðUðUðUÐ"8ñUôUñôð UðUðUr#