NgddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ej eZGdde ZdS)N)Path)DictIteratorUnion)Document) BaseLoaderc ~eZdZdZ d deeefdeedfdeedfdeddf d Zde e fd Z dS) BSHTMLLoaderaS __ModuleName__ document loader integration Setup: Install ``langchain-community`` and ``bs4``. .. code-block:: bash pip install -U langchain-community bs4 Instantiate: .. code-block:: python from langchain_community.document_loaders import BSHTMLLoader loader = BSHTMLLoader( file_path="./example_data/fake-content.html", ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Test Title My First Heading My first paragraph. {'source': './example_data/fake-content.html', 'title': 'Test Title'} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Test Title My First Heading My first paragraph. {'source': './example_data/fake-content.html', 'title': 'Test Title'} N file_path open_encoding bs_kwargsget_text_separatorreturnc ddl}n#t$rtdwxYw||_||_|2tjdstdddi}||_||_dS)ainitialize with path, and optionally, file encoding to use, and any kwargs to pass to the BeautifulSoup object. Args: file_path: The path to the file to load. open_encoding: The encoding to use when opening the file. bs_kwargs: Any kwargs to pass to the BeautifulSoup object. get_text_separator: The separator to use when calling get_text on the soup. rNzUbeautifulsoup4 package not found, please install it with `pip install beautifulsoup4`lxmlzBy default BSHTMLLoader uses the 'lxml' package. Please either install it with `pip install -U lxml` or pass in init arg `bs_kwargs={'features': '...'}` to overwrite the default BeautifulSoup kwargs.features) bs4 ImportErrorr r importlibutil find_specrr)selfr r rrrs h/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/html_bs.py__init__zBSHTMLLoader.__init__Ss   JJJJ   /   #*  >++F33 !, $V,I""4s!c#hKddlm}t|jd|j5}||fi|j}dddn #1swxYwY||j}|jrt|jj }nd}t|j|d}t||VdS) z)Load HTML document into document objects.r) BeautifulSoupr)encodingNr )sourcetitle) page_contentmetadata) rropenr r rget_textrr!strstringr)rrfsouptextr!r#s r lazy_loadzBSHTMLLoader.lazy_loadys%%%%%% $.#0B C C C 6q =55dn55D 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6}}T455 :  )**EEE$.))1 1 D8<<<<<<<r;s((((((((((------@@@@@@  8 $ $~=~=~=~=~=:~=~=~=~=~=r2