Ngj#zddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZGddeZdS) N)PathLike)Path)AnyCallableDictIteratorOptionalUnion)Document) BaseLoaderc&eZdZdZ ddeeefdedeedeed ee e e ge fd ed efd Z d e e fdZdeded e e fdZded efdZde eefded e eeffdZded dfdZded dfdZdS) JSONLoadera Load a `JSON` file using a `jq` schema. Setup: .. code-block:: bash pip install -U jq Instantiate: .. code-block:: python from langchain_community.document_loaders import JSONLoader import json from pathlib import Path file_path='./sample_quiz.json' data = json.loads(Path(file_path).read_text()) loader = JSONLoader( file_path=file_path, jq_schema='.quiz', text_content=False) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quizg .json', 'seq_num': 1} Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} NFT file_path jq_schema content_keyis_content_key_jq_parsable metadata_func text_content json_linesc  ddl}||_n#t$rtdwxYwt||_|||_||_||_||_ ||_ ||_ dS)a~Initialize the JSONLoader. Args: file_path (Union[str, PathLike]): The path to the JSON or JSON Lines file. jq_schema (str): The jq schema to use to extract the data or text from the JSON. content_key (str): The key to use to extract the content from the JSON if the jq_schema results to a list of objects (dict). If is_content_key_jq_parsable is True, this has to be a jq compatible schema. If is_content_key_jq_parsable is False, this should be a simple string key. is_content_key_jq_parsable (bool): A flag to determine if content_key is parsable by jq or not. If True, content_key is treated as a jq schema and compiled accordingly. If False or if content_key is None, content_key is used as a simple string. Default is False. metadata_func (Callable[Dict, Dict]): A function that takes in the JSON object extracted by the jq_schema and the default metadata and returns a dict of the updated metadata. text_content (bool): Boolean flag to indicate whether the content is in string format, default to True. json_lines (bool): Boolean flag to indicate whether the input is in JSON Lines format. rNz=jq package not found, please install it with `pip install jq`) jq ImportErrorrresolvercompile _jq_schema_is_content_key_jq_parsable _content_key_metadata_func _text_content _json_lines) selfrrrrrrrrs l/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/json_loader.py__init__zJSONLoader.__init__TsD  IIIDGG   O   i0022**Y//+E('+)%s (returnc#zKd}|jrt|jd5}|D]:}|}|r"|||D] }|V|dz } ; ddddS#1swxYwYdS||jd|D] }|V|dz } dS)z-Load and return documents from the JSON file.rzutf-8)encodingN)r ropenstrip_parse read_text)r!indexflinedocs r" lazy_loadzJSONLoader.lazy_loads;   $$g$66 '!''D::<#;#;W#;#M#MuUU      s>A33A7:A7contentr,c#K|jtj|}|j|||j||t||dzD]X\}}| |}| |t|j |}t||VYdS)z#Convert given content to documents.Nr')sample)r3sourceseq_num) page_contentmetadata)rinputjsonloadsr_validate_content_keyr_validate_metadata_func enumerate _get_text _get_metadatastrrr )r!r1r,datair3textr7s r"r*zJSONLoader._parses$$TZ%8%899   (  & &t , , ,   *  ( ( . . ."433 A AIAv>>>00D))c$.&9&91*Hx@@@ @ @ @ @  A Ar3c|j\|jrG|j|j}||}n||j}n|}|jr5t|ts tdt|dt|tr|St|tr|rtj |ndS|t|ndS)zConvert sample to string formatNz%Expected page_content is string, got z instead. Set `text_content=False` if the desired input for `page_content` is not a string)rrrrr8firstr isinstancer@ ValueErrortypedictr9dumps)r!r3compiled_content_keyr1s r"r>zJSONLoader._get_texts   (/ 4'+wt7H'I'I$.44V<<BBDD !23G   ?j#&>&> ?4W 444  % % ?N  & & ?*194:g&&&r 9#*#63w<<rDadditional_fieldsc @|j|||S|S)z Return a metadata dictionary base on the existence of metadata_func :param sample: single data payload :param additional_fields: key-word arguments to be added as metadata values :return: )r)r!r3rNs r"r?zJSONLoader._get_metadatas*   *&&v/@AA A$ $rDrAc|}t|ts tdt |d|js2||jtd|jd|jr\|j |j | td|jddSdS)zCheck if a content key is validztExpected the jq schema to result in a list of objects (dict), so sample must be a dict but got ``Nz_Expected the jq schema to result in a list of objects (dict) with the key `z ` which should be parsable by jq) rGrHrKrIrJrgetrrrr8rC)r!rAr3s r"r;z JSONLoader._validate_content_keys"&$'' G7;F||GGG  0  4,--58#'#4888   ,  12288@@EEGGOW#'#4WWW    OOrDc|}|jK||i}t|ts"t dt |ddSdS)z*Check if the metadata_func output is validNzMExpected the metadata_func to return a dict but got `rQ)rGrrHrKrIrJ)r!rAr3sample_metadatas r"r<z"JSONLoader._validate_metadata_funcs   *"11&"==Oot44  3//333 + *  rD)NFNTF)__name__ __module__ __qualname____doc__r r@rr boolrrr#rr r0intr*rr>r?r;r<rDr"rr sFFX&*5:@D! 1&1&h'1&1&c] 1& %-TN 1&  $t); <= 1&1&1&1&1&1&f8H- AcA#A(82DAAAA&??????4 %38n %;> % c3h % % % %#$6 C D      rDr)r9osrpathlibrtypingrrrrr r langchain_core.documentsr )langchain_community.document_loaders.baser rr[rDr"ras AAAAAAAAAAAAAAAA------@@@@@@llllllllllrD