Ng~ddlZddlZddlmZmZmZmZddlmZddl m Z ddl m Z ej eZGdde ZdS)N) AsyncIteratorIteratorListOptional)Document) BaseLoader)get_user_agentceZdZdZddddeededeefdZd ed efd Z d e e fd Z d e e fd ZdS)AsyncChromiumLoaderzJScrape HTML pages from URLs using a headless instance of the Chromium.TN)headless user_agenturlsr r c||_||_|p t|_ ddl}dS#t $rt dwxYw)a^Initialize the loader with a list of URL paths. Args: urls: A list of URLs to scrape content from. headless: Whether to run browser in headless mode. user_agent: The user agent to use for the browser Raises: ImportError: If the required 'playwright' package is not installed. rNz`playwright is required for AsyncChromiumLoader. Please install it with `pip install playwright`.)rr r r playwright ImportError)selfrr r rs i/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/chromium.py__init__zAsyncChromiumLoader.__init__sn"   $8(8(8          C  s +AurlreturncRKddlm}tdd}|4d{V}|j|jd{V} ||jd{V}| |d{V| d{V}tdn#t$r}d |}Yd}~nd}~wwxYw| d{Vdddd{Vn#1d{VswxYwY|S) a Asynchronously scrape the content of a given URL using Playwright's async API. Args: url (str): The URL to scrape. Returns: str: The scraped HTML content or an error message if an exception occurs. r)async_playwrightzStarting scraping...N)r )r zContent scrapedzError: ) playwright.async_apirloggerinfochromiumlaunchr new_pager gotocontent Exceptionclose)rrrresultspbrowserpagees rascrape_playwrightz&AsyncChromiumLoader.ascrape_playwright.s :99999 *+++##%% " " " " " " "J--t}-EEEEEEEEG ($---IIIIIIIIiinn$$$$$$$ $ ...... -.... ( ( ('A-- (--// ! ! ! ! ! ! ! " " " " " " " " " " " " " " " " " " " " " " " " " " "s<'DA0C D C'C"D"C''D D #D c#K|jD]@}tj||}d|i}t ||VAdS)a- Lazily load text content from the provided URLs. This method yields Documents one at a time as they're scraped, instead of waiting to scrape all URLs before returning. Yields: Document: The scraped content encapsulated within a Document object. source page_contentmetadataN)rasynciorunr)r)rr html_contentr.s r lazy_loadzAsyncChromiumLoader.lazy_loadIsi9 I IC";t'>'>s'C'CDDL #H xHHH H H H H I IcKfdjD}tj|d{V}tj|D]\}}d|i}t ||WVdS)a Asynchronously load text content from the provided URLs. This method leverages asyncio to initiate the scraping of all provided URLs simultaneously. It improves performance by utilizing concurrent asynchronous requests. Each Document is yielded as soon as its content is available, encapsulating the scraped content. Yields: Document: A Document object containing the scraped content, along with its source URL as metadata. c:g|]}|S)r)).0rrs r z2AsyncChromiumLoader.alazy_load..fs'CCC#((--CCCr3Nr+r,)rr/gatherzipr)rtasksr$rr!r.s` r alazy_loadzAsyncChromiumLoader.alazy_loadYsDCCCCCC....... 733 D DLC #H(CCC C C C C C D Dr3)__name__ __module__ __qualname____doc__rstrboolrrr)rrr2rr<r6r3rr r s**$( 3i  SM :CC6I8H-IIII D-"9DDDDDDr3r )r/loggingtypingrrrrlangchain_core.documentsr)langchain_community.document_loaders.baser$langchain_community.utils.user_agentr getLoggerr=rr r6r3rrIs::::::::::::------@@@@@@??????  8 $ $]D]D]D]D]D*]D]D]D]D]Dr3