Ng3( d dl Z d dlZd dlmZmZmZ d dlmZ d dlm Z d dl
m Z m Z d dl
mZ d dlmZ d dlmZ d d lmZmZ d d
lmZ d d lmZ d d lmZm Z d d
l m Z m Z d dl m!Z! d dl"m#Z# d dl$m%Z% d dl&m'Z' e j( e) Z* G d de Z+ edgd Z, edgd Z- G d deee. Z/ G d de Z0dS ) N)AnyListOptional)LLMChain)ConditionalPromptSelector)#AsyncCallbackManagerForRetrieverRun CallbackManagerForRetrieverRun)Document)BaseLLM)BaseOutputParser)BasePromptTemplatePromptTemplate)
BaseRetriever) VectorStore) RecursiveCharacterTextSplitter TextSplitter) BaseModelField)AsyncHtmlLoader)Html2TextTransformer)LlamaCpp)GoogleSearchAPIWrapperc D e Zd ZU dZ edd Zee ed< dS )
SearchQueriesz/Search queries to research for the user's goal..z+List of search queries to look up on Google descriptionqueriesN) __name__
__module__ __qualname____doc__r r r str__annotations__ g/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/retrievers/web_research.pyr r sM 99 F GT#Y r% r questiona! <>
You are an assistant tasked with improving Google search results.
<>
[INST] Generate THREE Google search queries that are similar to this question. The output should be a numbered list of questions and each should have a question mark at the end:
{question} [/INST])input_variablestemplatezYou are an assistant tasked with improving Google search results. Generate THREE Google search queries that are similar to this question. The output should be a numbered list of questions and each should have a question mark at the end: {question}c . e Zd ZdZdedee fdZdS )QuestionListOutputParserz/Output parser for a list of numbered questions.textreturnc 0 t j d| }|S )Nz\d+\..*?(?:\n|$))refindall)selfr, liness r& parsez QuestionListOutputParser.parse8 s
.55 r% N)r r r r! r" r r3 r$ r% r& r+ r+ 5 s@ 99# $s) r% r+ c e Zd ZU dZ edd Zeed< eed< edd Z e
ed< ed d
Z e ed < e e
d d
d Zeed< eed Zee ed< edd Zeed< dZeed< deddf fd Zedd e
d d ddfded ede
d ee d e de
dededd fd Z d edefd!Z d&d ed e dee fd"Z d ed#e!dee" fd$Z#d ed#e$dee" fd%Z% xZ&S )'WebResearchRetrieverz `Google Search API` retriever..z"Vector store for storing web pagesr vectorstore llm_chainzGoogle Search API Wrappersearch z!Number of pages per Google searchnum_search_resultsi 2 )
chunk_size
chunk_overlapz1Text splitter for splitting web pages into chunks
text_splitterzList of processed URLs)default_factoryr url_databaseFz_Whether to use the http_proxy/https_proxy env variables or check .netrc for proxy configuration trust_envallow_dangerous_requestskwargsr- Nc | dd }|st d t j di | dS )zInitialize the retriever.rB Fa WebResearchRetriever crawls URLs surfaced through the provided search engine. It is possible that some of those URLs will end up pointing to machines residing on an internal network, leadingto an SSRF (Server-Side Request Forgery) attack. To protect yourself against that risk, you can run the requests through a proxy and prevent the crawler from accidentally crawling internal resources.If've taken the necessary precautions, you can set `allow_dangerous_requests` to `True`.Nr$ )get
ValueErrorsuper__init__)r1 rC rB __class__s r& rH z WebResearchRetriever.__init___ s\ #)::.H%#P#P ' 8
""6"""""r% llmpromptc |s4t t d t fg } | | }t ||t }
| ||
||||| S )a Initialize from llm using default template.
Args:
vectorstore: Vector store for storing web pages
llm: llm for search question generation
search: GoogleSearchAPIWrapper
prompt: prompt to generating search questions
num_search_results: Number of pages per Google search
text_splitter: Text splitter for splitting web pages into chunks
trust_env: Whether to use the http_proxy/https_proxy env variables
or check .netrc for proxy configuration
allow_dangerous_requests: A flag to force users to acknowledge
the risks of SSRF attacks when using this retriever
Returns:
WebResearchRetriever
c , t | t S N)
isinstancer )rK s r& z/WebResearchRetriever.from_llm.. s C!:!: r% )default_prompt conditionals)rK rL
output_parser)r6 r7 r8 r: r> rA rB )r DEFAULT_SEARCH_PROMPTDEFAULT_LLAMA_SEARCH_PROMPT
get_promptr r+ ) clsr6 rK r8 rL r: r> rA rB QUESTION_PROMPT_SELECTORr7 s r& from_llmz WebResearchRetriever.from_llmq s @ >'@ 4::>#&& ' !#2#JE{{}} r% c f | | }| j || }|S )z3Returns num_search_results pages per Google search.)rd r8 results)r1 r[ r: query_cleanresults r& search_toolz WebResearchRetriever.search_tool s2 --e44 $$[2DEE
r% run_managerc d t d | d|i }t d| |d }t d| t d g }|D ]}| || j }t d t d| |D ]3}| dd r| |d 4t | }t | | j
} t d
| | rt | d | j }
t } t d
|
} t | | } | j | } | j | | j
| t d g } |D ]/}| | j | 0d | D }
t |
}|S )zSearch Google for documents related to the query input.
Args:
query: user query
Returns:
Relevant documents from all various urls.
z*Generating questions for Google Search ...r' z#Questions for Google Search (raw): r, z Questions for Google Search: z Searching for relevant urls...zSearch results: linkNzNew URLs to load: T)ignore_load_errorsrA zIndexing new urls...z*Grabbing most relevant splits from urls...c i | ]=}|j t t |j f|>S r$ ) page_contenttuplesortedmetadataitems).0docs r&
z@WebResearchRetriever._get_relevant_documents.. sM !
!
!
MPS
uVCL,>,>,@,@%A%A B B CS!
!
!
r% )loggerinfor7 ri r: rE appendsetlist
differencer@ r rA r loadtransform_documentsr> split_documentsr6
add_documentsextendsimilarity_searchvalues)r1 r[ rj rh questions urls_to_looksearch_resultsresurlsnew_urlsloader html2textdocsunique_documents_dictunique_documentss r& _get_relevant_documentsz,WebResearchRetriever._get_relevant_documents s @AAA U 344 B&BBCCC6N ?I??@@@ 4555 5 5E ! - -eT5L M MN KK8 9 9 9 KK;>;; < < <%
5
57764(( 5 ''F 444
5
< (9::;; 333444
/$TT^ F -..I KK. / / /;;==D 55d;;< >?? r% c K t rO )NotImplementedError)r1 r[ rj s r& _aget_relevant_documentsz-WebResearchRetriever._aget_relevant_documents s "!r% )r9 )'r r r r! r r6 r r# r r8 r r: intr r> r r{ r@ r r" rA boolrB r rH classmethodr r r
rZ rd dictri r r
r r r
__classcell__)rI s @r& r5 r5 = s (( %u = K %*U3r sb & & & & & & & & & & % % % % % % F F F F F F . - - - - - 2 2 2 2 2 2 : : : : : : E E E E E E E E 3 3 3 3 3 3 3 3 3 3 3 3 Q Q Q Q Q Q Q Q % % % % % % % % @ @ @ @ @ @ J J J J J J - - - - - - @ @ @ @ @ @ 8 $ $ I -n LL ' L6 /S : N" N" N" N" N"= N" N" N" N" N"r%