{h!dZddlZddlmZmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZerddlmZddlmZdd lmZdd lmZmZmZej0eZGd d eZGd deZGddeZy)zQLoader that uses Playwright to load a page, then uses unstructured to parse html.N)ABCabstractmethod) TYPE_CHECKING AsyncIteratorDictIteratorListOptional)Document) BaseLoader)Browser)Page)Response)r rrc TeZdZdZedddddddefd Zedd dd dd defd Zy)PlaywrightEvaluatorzAbstract base class for all evaluators. Each evaluator should take a page, a browser instance, and a response object, process the page as necessary, and return the resulting text. pagerbrowserr responserreturncy)aSynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nselfrrrs m/var/www/html/engine/venv/lib/python3.12/site-packages/langchain_community/document_loaders/url_playwright.pyevaluatezPlaywrightEvaluator.evaluates AsyncPage AsyncBrowser AsyncResponsec Kyw)aAsynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nrrs revaluate_asyncz"PlaywrightEvaluator.evaluate_async*s  sN)__name__ __module__ __qualname____doc__rstrrr!rrrrrsl   V  i  :  RU        *8  DS      rrcZeZdZdZddeeefdZddddd d d efd Zdd ddd dd efdZ y)UnstructuredHtmlEvaluatorz@Evaluate the page HTML content using the `unstructured` library.Nremove_selectorscL ddl}||_y#t$r tdwxYw)z%Initialize UnstructuredHtmlEvaluator.rNzQunstructured package not found, please install it with `pip install unstructured`) unstructured ImportErrorr))rr)r+s r__init__z"UnstructuredHtmlEvaluator.__init__>s7  !1  -  s#rrrr rrrcTddlm}|jxsgD]J}|j|j }|D]$}|j s|j d&L|j}||}dj|D cgc] } t| c} Scc} w)z3Synchronously process the HTML content of the page.rpartition_htmlelement => element.remove()text unstructured.partition.htmlr0r)locatorall is_visiblercontentjoinr& rrrrr0selectorelementselement page_sourceels rrz"UnstructuredHtmlEvaluator.evaluateJs>--3 DH||H-113H# D%%'$$%BC D D lln !{3{{h7CG7887s B%rrrcKddlm}|jxsgD]b}|j|j d{}|D]4}|j d{s|j dd{6d|jd{}||}dj|D cgc] } t| c} S77m7U7;cc} ww)z4Asynchronously process the HTML content of the page.rr/Nr1r2r4r5r<s rr!z(UnstructuredHtmlEvaluator.evaluate_asyncXs ?--3 JH!\\(37799H# J ++---!**+HIII J J !LLN* !{3{{h7CG788:-I*7s]: "username": "username", "password": "password" } loader = PlaywrightURLLoader(urls, proxy=proxy) data = loader.load() Nurlscontinue_on_failureheadlessr) evaluatorproxyc ddl}||_||_||_||_|r |r t d|xs t||_y#t$r tdwxYw)z%Load a list of URLs using Playwright.rNzMplaywright package not found, please install it with `pip install playwright`z:`remove_selectors` and `evaluator` cannot be both not None) playwrightr,rErFrGrI ValueErrorr(rH)rrErFrGr)rHrIrKs rr-zPlaywrightURLLoader.__init__sv   #6     L  #Q&?@P&Q# +  s AArc #ZKddlm}|5}|jj|j|j }|j D]u} |j}|j|}|td||jd|jj|||}d|i}t||w|j%dddy#t$r4} |jrt j#d |d | n| Yd} ~ d} ~ wwxYw#1swYyxYww) zLoad the specified URLs using Playwright and create Document instances. Returns: A list of Document instances with loaded content. r)sync_playwrightrGrIN"page.goto() returned None for url loadsource page_contentmetadataError fetching or processing , exception: )playwright.sync_apirNchromiumlaunchrGrIrEnew_pagegotorLwait_for_load_staterHrr ExceptionrFloggererrorclose) rrNprurlrrr3rUes r lazy_loadzPlaywrightURLLoader.lazy_loads# 8   !jj''djj'QGyy  "++-D#yy~H'(+McU)STT,,V4>>224(KD (#H"xHH & MMO+  ! // ;C5 aSQ     sHD+ADA2CD D+ D(*DDDDD($D+c`K|j2cgc3d{}| 76c}Scc}ww)Load the specified URLs with Playwright and create Documents asynchronously. Use this function when in a jupyter notebook environment. Returns: A list of Document instances with loaded content. N) alazy_load)rdocs raloadzPlaywrightURLLoader.aloads*&*__%677c7777s$.)%# %)%).c Kddlm}|4d{}|jj|j|j d{}|j D]} |jd{}|j|d{}|td||jdd{|jj|||d{}d|i}t|||j%d{dddd{y7 7777q7N#t$r5} |jrt j#d |d | n| Yd} ~ d} ~ wwxYw7c7U#1d{7swYyxYww) rgr)async_playwrightNrOrPrQrRrSrVrW)playwright.async_apirlrYrZrGrIrEr[r\rLr]rHr!r r^rFr_r`ra) rrlrbrrcrrr3rUrds rrhzPlaywrightURLLoader.alazy_loadsn :#% " "JJ--t}}DJJ-WWGyy  !(!1!1!33D%)YYs^3H'(+McU)STT226:::!%!>!>tWh!WWD (#H"xHH &--/ ! !+ " " "W43;W! // ;C5 aSQ    "+ " " " "sFD F5E2 D#E2"D-5D%6D-D'(D-7D)8$D-D+D-5E2 E. E2 FE0F#E2%D-'D-)D-+D-- E+6*E& E2&E++E20F2F8E; 9FF)TTNNN)r"r#r$r%r r&boolr rrr-rr rerjrrhrrrrDrDis:%)0437*.R3iR"R R #49- R /0 RS#X'R>8H->8T(^8"-"9"rrD)r%loggingabcrrtypingrrrrr r langchain_core.documentsr )langchain_community.document_loaders.baser rmr rrrrrrX getLoggerr"r_rr(rDrrrrusiW#OO-@<6>;;   8 $# ## L+9 3+9\@"*@"r