NgddlmZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl mZddlmZmZddlmZmZddlmZerddlmZe eefZe d ZGd d eZd S) ) annotations)Path) TYPE_CHECKINGAnyIteratorListLiteralOptionalSequenceUnion)Document)BaseBlobParser BaseLoader) BlobLoaderFileSystemBlobLoader) get_parser) TextSplitterdefaultcneZdZdZd'dZd(d Z d)d*dZeddd ddd dd+d#Ze d,d&Z d S)- GenericLoaderaGeneric Document Loader. A generic document loader that allows combining an arbitrary blob loader with a blob parser. Examples: Parse a specific PDF file: .. code-block:: python from langchain_community.document_loaders import GenericLoader from langchain_community.document_loaders.parsers.pdf import PyPDFParser # Recursively load all text files in a directory. loader = GenericLoader.from_filesystem( "my_lovely_pdf.pdf", parser=PyPDFParser() ) .. code-block:: python from langchain_community.document_loaders import GenericLoader from langchain_community.document_loaders.blob_loaders import FileSystemBlobLoader loader = GenericLoader.from_filesystem( path="path/to/directory", glob="**/[!.]*", suffixes=[".pdf"], show_progress=True, ) docs = loader.lazy_load() next(docs) Example instantiations to change which files are loaded: .. code-block:: python # Recursively load all text files in a directory. loader = GenericLoader.from_filesystem("/path/to/dir", glob="**/*.txt") # Recursively load all non-hidden files in a directory. loader = GenericLoader.from_filesystem("/path/to/dir", glob="**/[!.]*") # Load all files in a directory without recursion. loader = GenericLoader.from_filesystem("/path/to/dir", glob="*") Example instantiations to change which parser is used: .. code-block:: python from langchain_community.document_loaders.parsers.pdf import PyPDFParser # Recursively load all text files in a directory. loader = GenericLoader.from_filesystem( "/path/to/dir", glob="**/*.pdf", parser=PyPDFParser() ) blob_loaderr blob_parserrreturnNonec"||_||_dS)zA generic document loader. Args: blob_loader: A blob loader which knows how to yield blobs blob_parser: A blob parser which knows how to parse blobs into documents N)rr)selfrrs h/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/generic.py__init__zGenericLoader.__init__as'&Iterator[Document]c#K|jD]"}|j|Ed{V#dS)z>Load documents lazily. Use this when working at a large scale.N)r yield_blobsr lazy_parse)rblobs r lazy_loadzGenericLoader.lazy_loados\$0022 9 9D'22488 8 8 8 8 8 8 8 8 9 9rN text_splitterOptional[TextSplitter]List[Document]c td)z1Load all documents and split them into sentences.zLoading and splitting is not yet implemented for generic loaders. When they will be implemented they will be added via the initializer. This method should not be used going forward.NotImplementedError)rr&s rload_and_splitzGenericLoader.load_and_splitvs" <   rz**/[!.]*Fr)globexcludesuffixes show_progressparser parser_kwargspath _PathLiker.strr/ Sequence[str]r0Optional[Sequence[str]]r1boolr2Union[DEFAULT, BaseBlobParser]r3Optional[dict]ct|||||}t|trF|dkr0 |jdi|pi} n1#t$rt|} YnwxYwt|} n|} ||| S)aCreate a generic document loader using a filesystem blob loader. Args: path: The path to the directory to load documents from OR the path to a single file to load. If this is a file, glob, exclude, suffixes will be ignored. glob: The glob pattern to use to find documents. suffixes: The suffixes to use to filter documents. If None, all files matching the glob will be loaded. exclude: A list of patterns to exclude from the loader. show_progress: Whether to show a progress bar or not (requires tqdm). Proxies to the file system loader. parser: A blob parser which knows how to parse blobs into documents, will instantiate a default parser if not provided. The default can be overridden by either passing a parser or setting the class attribute `blob_parser` (the latter should be used with inheritance). parser_kwargs: Keyword arguments to pass to the parser. Returns: A generic document loader. )r.r/r0r1rr-)r isinstancer6rr+) clsr4r.r/r0r1r2r3rrs rfrom_filesystemzGenericLoader.from_filesystemsD+ '     fc " " !""5"0#."I"IM4GR"I"IKK*555",V"4"4KKK5)00  Ks; ,,,sAAAkwargsrc t)zBOverride this method to associate a default parser with the class.r*)r@s rrzGenericLoader.get_parsers"###r)rrrrrr)rr )N)r&r'rr()r4r5r.r6r/r7r0r8r1r9r2r:r3r;rr)r@rrr) __name__ __module__ __qualname____doc__rr%r, classmethodr? staticmethodrr-rrrr s>>@ ' ' ' '99997;      !#,0#1:(,4-4-4-4-4-[4-l$$$\$$$rrN) __future__rpathlibrtypingrrrrr r r r langchain_core.documentsr )langchain_community.document_loaders.baserr1langchain_community.document_loaders.blob_loadersrr5langchain_community.document_loaders.parsers.registryrlangchain_text_splittersrr6r5DEFAULTrr-rrrQsf""""""                    .-----PPPPPPPPMLLLLL6555555 #t)  ) Z$Z$Z$Z$Z$JZ$Z$Z$Z$Z$r