NgddlZddlmZmZmZmZmZmZddlm Z ddl m Z m Z ej eZGdde ZdS)N)AnyCallableDictIteratorListOptional)Document) BaseModelmodel_validatorceZdZUdZdZeed<dZeed<dZe ed<dZ e e e gefed <eed <ed ed e defdZdeefdZdeefdZdS)TensorflowDatasetsavAccess to the TensorFlow Datasets. The Current implementation can work only with datasets that fit in a memory. `TensorFlow Datasets` is a collection of datasets ready to use, with TensorFlow or other Python ML frameworks, such as Jax. All datasets are exposed as `tf.data.Datasets`. To get started see the Guide: https://www.tensorflow.org/datasets/overview and the list of datasets: https://www.tensorflow.org/datasets/catalog/ overview#all_datasets You have to provide the sample_to_document_function: a function that a sample from the dataset-specific format to the Document. Attributes: dataset_name: the name of the dataset to load split_name: the name of the split to load. Defaults to "train". load_max_docs: a limit to the number of loaded documents. Defaults to 100. sample_to_document_function: a function that converts a dataset sample to a Document Example: .. code-block:: python from langchain_community.utilities import TensorflowDatasets def mlqaen_example_to_document(example: dict) -> Document: return Document( page_content=decode_to_str(example["context"]), metadata={ "id": decode_to_str(example["id"]), "title": decode_to_str(example["title"]), "question": decode_to_str(example["question"]), "answer": decode_to_str(example["answers"]["text"][0]), }, ) tsds_client = TensorflowDatasets( dataset_name="mlqa/en", split_name="train", load_max_docs=MAX_DOCS, sample_to_document_function=mlqaen_example_to_document, ) dataset_nametrain split_named load_max_docsNsample_to_document_functiondatasetbefore)modevaluesreturnc  ddl}n#t$rtdwxYw ddl}n#t$rtdwxYw|dtd||d|d |d <|S) z7Validate that the python package exists in environment.rNz\Could not import tensorflow python package. Please install it with `pip install tensorflow`.znCould not import tensorflow_datasets python package. Please install it with `pip install tensorflow-datasets`.rzmsample_to_document_function is None. Please provide a function that converts a dataset sample to a Document.rr)splitr) tensorflow ImportErrortensorflow_datasets ValueErrorload)clsrrrs m/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/utilities/tensorflow_datasets.pyvalidate_environmentz'TensorflowDatasets.validate_environment?s         C     & & & & &   L   / 0 8   044 > "&*>5  y s!*AcZfdjjDS)zYDownload a selected dataset lazily. Returns: an iterator of Documents. c3PK|] }j |V!dS)N)r).0sselfs r" z/TensorflowDatasets.lazy_load..csF  /;  , ,Q / /;;;;  )rtakerr(s`r" lazy_loadzTensorflowDatasets.lazy_load]sA     \&&t'9::    r*cDt|S)zMDownload a selected dataset. Returns: a list of Documents. )listr-r,s r"r zTensorflowDatasets.loadis DNN$$%%%r*)__name__ __module__ __qualname____doc__rstr__annotations__rrintrrrrr rr classmethodr#rr-rr r*r"r r s,,\L#JM3HL(D683C*D!ELLL LLL_(###$3[$#8  8H-     &d8n&&&&&&r*r )loggingtypingrrrrrrlangchain_core.documentsr pydanticr r getLoggerr0loggerr r8r*r"r?s@@@@@@@@@@@@@@@@------////////  8 $ $e&e&e&e&e&e&e&e&e&e&r*