NgX ddlmZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZddlmZdd lmZe eefZed ZGd d eZdS) ) annotationsN)Path)IteratorLiteralOptionalSequenceUnion)Document)BaseBlobParser) BlobLoaderFileSystemBlobLoader) GenericLoader) get_parserdefaultc ZeZdZdZ d#d$fd Zd%d Zeddddddddd&d"ZxZS)'ConcurrentLoaderz%Load and pars Documents concurrently. blob_loaderr blob_parserr num_workersintreturnNonecZt||||_dS)N)super__init__r)selfrrr __class__s k/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/concurrent.pyrzConcurrentLoader.__init__s- k222&Iterator[Document]c#PKtjj5fdjD}tj|D]}|Ed{V ddddS#1swxYwYdS)z.Load documents lazily with concurrent parsing.) max_workerscPh|]"}jj|#S)submitr lazy_parse).0blobexecutorrs r z-ConcurrentLoader.lazy_load..)s= 0 ;TBBr N) concurrentfuturesThreadPoolExecutorrr yield_blobs as_completedresult)rr-futurer*s` @r lazy_loadzConcurrentLoader.lazy_load"s  2 2(3   +  ,88::G%,99'BB + +!==??******** + + + + + + + + + + + + + + + + + + +sA#BB"Bz**/[!.]*r%NFr)globexcludesuffixes show_progressparserr parser_kwargspath _PathLiker4strr5 Sequence[str]r6Optional[Sequence[str]]r7boolr8Union[DEFAULT, BaseBlobParser]r9Optional[dict]ct|||||} t|tr;|dkr%|jtjkr|jdi|pi} nt|} n|} || | |S)aCreate a concurrent generic document loader using a filesystem blob loader. Args: path: The path to the directory to load documents from. glob: The glob pattern to use to find documents. suffixes: The suffixes to use to filter documents. If None, all files matching the glob will be loaded. exclude: A list of patterns to exclude from the loader. show_progress: Whether to show a progress bar or not (requires tqdm). Proxies to the file system loader. parser: A blob parser which knows how to parse blobs into documents num_workers: Max number of concurrent workers to use. parser_kwargs: Keyword arguments to pass to the parser. )r4r5r6r7r)rr%)r isinstancer<rr) clsr:r4r5r6r7r8rr9rrs rfrom_filesystemz ConcurrentLoader.from_filesystem0s6+ '     fc " " !""s~9Q'Q'Q,cnEE 0CEE (00  Ks; EEEEr )r)rr rr rrrr)rr!)r:r;r4r<r5r=r6r>r7r?r8r@rrr9rArr) __name__ __module__ __qualname____doc__rr3 classmethodrE __classcell__)rs@rrrs//  ''''''' + + + + !#,0#1:(,)F)F)F)F)F[)F)F)F)F)Fr r) __future__rconcurrent.futuresr,pathlibrtypingrrrrr langchain_core.documentsr )langchain_community.document_loaders.baser 1langchain_community.document_loaders.blob_loadersr r ,langchain_community.document_loaders.genericr5langchain_community.document_loaders.parsers.registryrr<r;DEFAULTrr%r rrVs2""""""??????????????------DDDDDDGFFFFFLLLLLL #t)  ) DFDFDFDFDF}DFDFDFDFDFr