NgddlZddlmZmZmZmZmZddlmZddl m Z ddl m Z ej eZGdde ZdS)N)AnyIteratorListOptionalSequence)Document) BaseLoader) NewsURLLoaderc eZdZdZ ddeeedeededed ed df d Z d e e fd Z e d eefd Zd ee fdZdS) RSSFeedLoaderaQLoad news articles from `RSS` feeds using `Unstructured`. Args: urls: URLs for RSS feeds to load. Each articles in the feed is loaded into its own document. opml: OPML file to load feed urls from. Only one of urls or opml should be provided. The value can be a URL string, or OPML markup contents as byte or string. continue_on_failure: If True, continue loading documents even if loading fails for a particular URL. show_progress_bar: If True, use tqdm to show a loading progress bar. Requires tqdm to be installed, ``pip install tqdm``. **newsloader_kwargs: Any additional named arguments to pass to NewsURLLoader. Example: .. code-block:: python from langchain_community.document_loaders import RSSFeedLoader loader = RSSFeedLoader( urls=["", ""], ) docs = loader.load() The loader uses feedparser to parse RSS feeds. The feedparser library is not installed by default so you should install it if using this loader: https://pythonhosted.org/feedparser/ If you use OPML, you should also install listparser: https://pythonhosted.org/listparser/ Finally, newspaper is used to process each article: https://newspaper.readthedocs.io/en/latest/ NTFurlsopmlcontinue_on_failureshow_progress_barnewsloader_kwargsreturnc ~|du|dukrtd||_||_||_||_||_dS)zInitialize with urls or OPML.Nz;Provide either the urls or the opml argument, but not both.) ValueErrorr rrrr)selfr rrrrs d/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/rss.py__init__zRSSFeedLoader.__init__/sa DL DL  M   #6 !2!2c|}|jr5 ddlm}n"#t$r}td|d}~wwxYw||}t |S)Nr)tqdmzPackage tqdm must be installed if show_progress_bar=True. Please install with 'pip install tqdm' or set show_progress_bar=False.) lazy_loadrr ImportErrorlist)riterres rloadzRSSFeedLoader.loadDs~~  !  %%%%%%%   !/   4::DDzzs$ A>Ac|jr|jS ddl}n"#t$r}td|d}~wwxYw||j}d|jDS)NrzPackage listparser must be installed if the opml arg is used. Please install with 'pip install listparser' or use the urls arg instead.cg|] }|j S)url).0feeds r z+RSSFeedLoader._get_urls.._s///T///r)r listparserrparserfeeds)rr(rrsss r _get_urlszRSSFeedLoader._get_urlsRs 9 9          $    ty))//SY////s 4/4c#K ddl}n#t$rtdwxYw|jD]} ||}t |ddrt d|d|jn@#t$r3}|jr%t d|d|Yd}~{|d}~wwxYw |j D]C}td d|j gi|j}|d}||jd<|VD#t$r9}|jr+t d |j d|Yd}~|d}~wwxYwdS) NrzMfeedparser package not found, please install it with `pip install feedparser`bozoFzError fetching z , exception: r r&zError processing entry r#) feedparserrr,r)getattrrbozo_exception Exceptionrloggererrorentriesr linkrr metadata)rr/r$r&rentryloaderarticles rrzRSSFeedLoader.lazy_loadas         +   >  C !'',,4//$Q#QQDrHs::::::::::::::------@@@@@@CCCCCC  8 $ $yyyyyJyyyyyr