NgfjddlZddlmZmZmZmZmZddlmZddl m Z ddl m Z Gdde Z dS)N)AnyListOptionalSequenceTuple)Document) TextSplitter)MultiVectorRetrievercleZdZUdZeed< dZeeed< dZee e ed< dde e dee e d e d ee e e ee e fffd Z dde e dee e d e d ed df d Z dde e dee e d e d ed df dZdS)ParentDocumentRetrieveraRetrieve small chunks then retrieve their parent documents. When splitting documents for retrieval, there are often conflicting desires: 1. You may want to have small documents, so that their embeddings can most accurately reflect their meaning. If too long, then the embeddings can lose meaning. 2. You want to have long enough documents that the context of each chunk is retained. The ParentDocumentRetriever strikes that balance by splitting and storing small chunks of data. During retrieval, it first fetches the small chunks but then looks up the parent ids for those chunks and returns those larger documents. Note that "parent document" refers to the document that a small chunk originated from. This can either be the whole raw document OR a larger chunk. Examples: .. code-block:: python from langchain_chroma import Chroma from langchain_community.embeddings import OpenAIEmbeddings from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain.storage import InMemoryStore # This text splitter is used to create the parent documents parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, add_start_index=True) # This text splitter is used to create the child documents # It should create documents smaller than the parent child_splitter = RecursiveCharacterTextSplitter(chunk_size=400, add_start_index=True) # The vectorstore to use to index the child chunks vectorstore = Chroma(embedding_function=OpenAIEmbeddings()) # The storage layer for the parent documents store = InMemoryStore() # Initialize the retriever retriever = ParentDocumentRetriever( vectorstore=vectorstore, docstore=store, child_splitter=child_splitter, parent_splitter=parent_splitter, ) child_splitterNparent_splitterchild_metadata_fieldsT documentsidsadd_to_docstorereturnc0 |j|j|}|d|D}|stdn1t|t|krtd|}g}g}t |D]\}}||} |j|g} |j| D] fd|jD _| D] | j|j<| | | | |f||fS)NcNg|]"}ttj#S)struuiduuid4).0_s j/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain/retrievers/parent_document_retriever.py zBParentDocumentRetriever._split_docs_for_adding..Qs&<<zBParentDocumentRetriever._split_docs_for_adding..es/%%%014=+%%%r) rsplit_documents ValueErrorlen enumerater rr id_keyextendappend) selfrrrdoc_idsdocs full_docsidoc_idsub_docsr"s @r_split_docs_for_addingz.ParentDocumentRetriever._split_docs_for_addingHs|   +,<%d3i % % tH~tE#x-$899 : %%%%T$( $ **>*d3i * *  *  ****:$( $ 1 1> 1d3i  1 1  1  1 1 1 1 1 1rr )rtypingrrrrrlangchain_core.documentsrlangchain_text_splittersr langchain.retrieversr r rrrrHs 77777777777777------111111555555I1I1I1I1I12I1I1I1I1I1r