NgP0bddlZddlmZmZmZddlmZddlmZddl m Z GddeZ dS)N)IteratorLiteralOptional) BaseLoader)Document) get_from_envc eZdZdZdedefdZdedefdZdddddd ed eed eed e d deef dZ de e fdZ dS)FireCrawlLoadera! FireCrawlLoader document loader integration Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``. .. code-block:: bash pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key" Instantiate: .. code-block:: python from langchain_community.document_loaders import FireCrawlLoader loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} paramsreturncd}gd}|D]}||rd}n|rtjdtd|vr|ddur |d|d<|d=d|vr|ddur |d|d<|d=d |vr|d dur |d |d <|d =d |vr|d dur |d |d <|d =d |vr*:&V##*%---3J-?F>*:&&&0012d::39:Q3RF/023*f4456$>>39:U3VF/067&&f]3T::.2.P.P}-//F?+=) cd}dg}d|vrd|dvr|dddks$|dddks|dddkrd}d |dvr?|dd r|dd |d <n|dd d |d <d |dvr|dd r|dd |d <d|dvr|ddr|dd|d <|d=gd}|D]}||rd}n|r\tjdtd|vr"|ddur|d|d=d|vr"|ddur|d|d=d|vr"|ddur|d|d=d|vr"|ddur|d|d=d|vr"|ddur|d|d=d|vr"|ddur|d|d=d|vr"|ddur|d|d=d|vr|ddur |d|d<|d=d|vr|ddur |d|d <|d=d!|vr||d!<|S)"NFmarkdownextractorOptionsmodezllm-extractionzllm-extraction-from-raw-htmlzllm-extraction-from-markdownTextractionPromptpromptz-Extract page information based on the schema.extractionSchemaschema userPrompt) includeMarkdown includeHtmlincludeRawHtmlincludeExtract includeLinks screenshotfullPageScreenshotonlyIncludeTags removeTagsrr0r1htmlr2rawHtmlr3extractr4linksr5r6zscreenshot@fullPager7 includeTagsr8 excludeTagsformats)rrrrremoveappend)r r r!r? scrape_keysr#s r$rz-FireCrawlLoader.legacy_scrape_options_adapterss",  ' ' 2333-.v6:JJJ01&956601&9566*.&)V4F-GGG!"456HI/56H/I 20F8,,066H/I/M/M 2 O00F8, *V4F-GGG!"456HI/56H/I 20F8,$v.@'AAA!"45lCX/56H/I,/WF8,12      Czz# %)"  0 ) MT"   !F**+,55NN:...,-&&-(D00NN6***=)6))*+t33NN9---+,6))*+t33NN9---+,''.)T11NN7+++>*v%%,'4//NN<000<(#v--./477NN#8999/0 F**+,44,23D,EF=),-v%%,'4//,2<,@F=)<( F " " 'F9  r&Ncrawl)api_keyapi_urlr*r urlrDrEr*)rCscrapemapc ddlm}n#t$rtdwxYw|dvrtd|d|std|pt dd }||| |_||_||_|pi|_d S) aRInitialize with API key and url. Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py r) FirecrawlAppzD`firecrawl` package not found, please run `pip install firecrawl-py`)rCrGsearchrHInvalid mode 'z/'. Allowed: 'crawl', 'scrape', 'search', 'map'.zUrl must be providedrDFIRECRAWL_API_KEY)rDrEN) firecrawlrJ ImportError ValueErrorrrFr*r )r rFrDrEr*r rJs r$__init__zFireCrawlLoader.__init__s4  . . . . . . .   V   ; ; ;VVVV  5344 4I\)5HII%gwGGG l s #c#nK|jdkr;|j|j||jg}n|jdkrf|jst d|j|j||j}| dg}nz|jdkr=|jst d|j |j|j}n2|jdkrt d t d |jd |D]~}|jdkr|}i}nV| d p*| d p| dd}| di}|skt||VdS)NrG)r rCzURL is required for crawl modedatarHzURL is required for map moderKz?Search mode is not supported in this version, please downgrade.rLz%'. Allowed: 'crawl', 'scrape', 'map'.r(r9r:metadata) page_contentrU) r*rN scrape_urlrFrr rP crawl_urlr%rmap_urlr)r firecrawl_docscrawl_responsedocrVrUs r$ lazy_loadzFireCrawlLoader.lazy_load s 9 ))HT%G%G %T%T*NN Y' ! !8 C !ABBB!^55!D!DT[!Q!Q6N,//;;NN Y%  8 A !?@@@!^33DHT[3QQNN Y( " "Q QQQQ "  CyE!!" GGJ''T3776??TcggiQS>T>T77:r22 )!      r&)__name__ __module__ __qualname____doc__rr%rstrrrrQrrr]r&r$r r s77r.T.d....`jDjTjjjj`"&!%29!%,#,#,# ,## ,# # ,# ./ ,#,#,#,#,#\(8H-((((((r&r ) rtypingrrrlangchain_core.document_loadersrlangchain_core.documentsrlangchain_core.utilsrr rcr&r$rhs..........666666------------llllljlllllr&