NgL! dZddlZddlmZmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZerddlmZddlmZdd lmZdd lmZmZmZejeZGd d eZGd deZGddeZdS)zQLoader that uses Playwright to load a page, then uses unstructured to parse html.N)ABCabstractmethod) TYPE_CHECKING AsyncIteratorDictIteratorListOptional)Document) BaseLoader)Browser)Page)Response)r rrc beZdZdZedddddddefd Zedd dd dd defd ZdS)PlaywrightEvaluatorzAbstract base class for all evaluators. Each evaluator should take a page, a browser instance, and a response object, process the page as necessary, and return the resulting text. pagerbrowserr responserreturncdS)aSynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nselfrrrs o/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/url_playwright.pyevaluatezPlaywrightEvaluator.evaluates   AsyncPage AsyncBrowser AsyncResponsec KdS)aAsynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nrrs revaluate_asyncz"PlaywrightEvaluator.evaluate_async*s  rN)__name__ __module__ __qualname____doc__rstrrr!rrrrrs   V  i  :  RU    ^      *8  DS      ^    rrcheZdZdZddeeefdZddddd d d efd Zdd ddd dd efdZ dS)UnstructuredHtmlEvaluatorz@Evaluate the page HTML content using the `unstructured` library.Nremove_selectorscZ ddl}n#t$rtdwxYw||_dS)z%Initialize UnstructuredHtmlEvaluator.rNzQunstructured package not found, please install it with `pip install unstructured`) unstructured ImportErrorr))rr)r+s r__init__z"UnstructuredHtmlEvaluator.__init__>sX         -   !1!rrrr rrrcNddlm}|jpgD]W}||}|D]+}|r|d,X|}||}dd|DS)z3Synchronously process the HTML content of the page.rpartition_htmlelement => element.remove()text c,g|]}t|Srr&.0els r z6UnstructuredHtmlEvaluator.evaluate..V777CGG777r unstructured.partition.htmlr1r)locatorall is_visiblercontentjoin rrrrr1selectorelementselement page_sources rrz"UnstructuredHtmlEvaluator.evaluateJs>>>>>>-3 D DH||H--1133H# D D%%''D$$%BCCC Dllnn !>{333{{77h777888rrrrcKddlm}|jpgD]i}||d{V}|D]7}|d{Vr|dd{V8j|d{V}||}dd|DS)z4Asynchronously process the HTML content of the page.rr0Nr2r3r5c,g|]}t|Srr7r8s rr;z.fr<rr=rDs rr!z(UnstructuredHtmlEvaluator.evaluate_asyncXs ?>>>>>-3 J JH!\\(337799999999H# J J ++--------J!**+HIIIIIIIII J!LLNN****** !>{333{{77h777888rN) r"r#r$r%r r r&r-rr!rrrr(r(;sJJ 1 1$s))< 1 1 1 1 9V 9i 9: 9RU 9 9 9 999*89DS9 999999rr(ceZdZdZ ddeedededeeedeed ee eeff d Z d e e fd Z d ee fd Zd ee fdZdS)PlaywrightURLLoaderaLoad `HTML` pages with `Playwright` and parse with `Unstructured`. This is useful for loading pages that require javascript to render. Attributes: urls (List[str]): List of URLs to load. continue_on_failure (bool): If True, continue loading other URLs on failure. headless (bool): If True, the browser will run in headless mode. proxy (Optional[Dict[str, str]]): If set, the browser will access URLs through the specified proxy. Example: .. code-block:: python from langchain_community.document_loaders import PlaywrightURLLoader urls = ["https://api.ipify.org/?format=json",] proxy={ "server": "https://xx.xx.xx:15818", # https://: "username": "username", "password": "password" } loader = PlaywrightURLLoader(urls, proxy=proxy) data = loader.load() TNurlscontinue_on_failureheadlessr) evaluatorproxyc ddl}n#t$rtdwxYw||_||_||_||_|r|rt d|pt||_dS)z%Load a list of URLs using Playwright.rNzMplaywright package not found, please install it with `pip install playwright`z:`remove_selectors` and `evaluator` cannot be both not None) playwrightr,rNrOrPrR ValueErrorr(rQ)rrNrOrPr)rQrRrTs rr-zPlaywrightURLLoader.__init__s         +    #6       L  #Q&?@P&Q&Qr.rc #HKddlm}|5}|j|j|j}|jD]} |}||}|td||j |||}d|i}t||Vt#t$r4} |jr!td|d | n| Yd} ~ d} ~ wwxYw|ddddS#1swxYwYdS) zLoad the specified URLs using Playwright and create Document instances. Returns: A list of Document instances with loaded content. r)sync_playwrightrPrRN"page.goto() returned None for url source page_contentmetadataError fetching or processing , exception: )playwright.sync_apirWchromiumlaunchrPrRrNnew_pagegotorUrQrr ExceptionrOloggererrorclose) rrWprurlrrr4r]es r lazy_loadzPlaywrightURLLoader.lazy_loads 877777 _   !j''dj'QQGy   "++--D#yy~~H'()Sc)S)STTT>224(KKD (#H"xHHHHHHH    /  QCQQaQQ    MMOOO'                  s<0DA0B54D5 C3?*C.)D.C33DDDcNKd|2d{VS)Load the specified URLs with Playwright and create Documents asynchronously. Use this function when in a jupyter notebook environment. Returns: A list of Document instances with loaded content. c"Kg|3d{V}| 6SrKr)r9docs rr;z-PlaywrightURLLoader.aload..s.77777777c7777sN) alazy_load)rs raloadzPlaywrightURLLoader.aloads887T__%6%67777777777rc Kddlm}|4d{V}|j|j|jd{V}|jD]} |d{V}||d{V}|td||j |||d{V}d|i}t||WV#t$r4} |jr!td|d | n| Yd} ~ d} ~ wwxYw|d{Vdddd{VdS#1d{VswxYwYdS) rnr)async_playwrightNrXrYrZr[r^r_)playwright.async_apirtrarbrPrRrNrcrdrUrQr!r rerOrfrgrh) rrtrirrjrrr4r]rks rrqzPlaywrightURLLoader.alazy_loadsx :99999##%% " " " " " " "J--t}DJ-WWWWWWWWGy   !(!1!1!3!3333333D%)YYs^^333333H'()Sc)S)STTT!%!>!>tWh!W!WWWWWWWD (#H"xHHHHHHHH    /  QCQQaQQ   --// ! ! ! ! ! ! !' " " " " " " " " " " " " " " " " " " " " " " " " " " " " " "s<6EBCE D*D E DE E E )TTNNN)r"r#r$r%r r&boolr rrr-rr rlrrrrqrrrrMrMis :%)0437*.RR3iR"R R #49- R /0 RS#X'RRRR>8H-:8T(^8888"-"9""""""rrM)r%loggingabcrrtypingrrrrr r langchain_core.documentsr )langchain_community.document_loaders.baser rur rrrrrr` getLoggerr"rfrr(rMrrrr}sWW########OOOOOOOOOOOOOOOO------@@@@@@<<<<<<<666666>>>>>>;;;;;;;;;;  8 $ $# # # # # ## # # L+9+9+9+9+9 3+9+9+9\|"|"|"|"|"*|"|"|"|"|"r