Ng($ddlZddlZddlmZddlmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZGd d eZed krCed d dZeZedeedejdSdS)N)Path)AnyListOptionalTuple)unquote)Document)DirectoryLoader) PyPDFLoader) WebBaseLoaderc\eZdZdZ ddedededeeeefd eed ed effd Z ddZ d e e fdZ ded efdZded e e fdZded e efdZde ed dfdZd e e fdZded e efdZded dfdZded efdZded efdZxZS)BlackboardLoadera'Load a `Blackboard` course. This loader is not compatible with all Blackboard courses. It is only compatible with courses that use the new Blackboard interface. To use this loader, you must have the BbRouter cookie. You can get this cookie by logging into the course and then copying the value of the BbRouter cookie from the browser's developer tools. Example: .. code-block:: python from langchain_community.document_loaders import BlackboardLoader loader = BlackboardLoader( blackboard_course_url="https://blackboard.example.com/webapps/blackboard/execute/announcement?method=search&context=course_entry&course_id=_123456_1", bbrouter="expires:12345...", ) documents = loader.load() TNFblackboard_course_urlbbrouterload_all_recursively basic_authcookiescontinue_on_failure show_progressct||| |dd|_n#t$rt dwxYw| ||j_|i}|d|i|jj|||_ | dS)aYInitialize with blackboard course url. The BbRouter cookie is required for most blackboard courses. Args: blackboard_course_url: Blackboard course url. bbrouter: BbRouter cookie. load_all_recursively: If True, load all documents recursively. basic_auth: Basic auth credentials. cookies: Cookies. continue_on_failure: whether to continue loading the sitemap if an error occurs loading a url, emitting a warning instead of raising an exception. Setting this to True makes the loader more robust, but also may result in missing data. Default: False show_progress: whether to show a progress bar while loading. Default: True Raises: ValueError: If blackboard course url is invalid. ) web_pathsrrz/webapps/blackboardrzpInvalid blackboard course url. Please provide a url that starts with https:///webapps/blackboardNBbRouter) super__init__splitbase_url IndexErrorsessionauthupdaterr check_bs4) selfrrrrrrr __class__s k/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/blackboard.pyrzBlackboardLoader.__init__$s: , 3'     1778MNNqQDMM   >    ! *DL  ?G H-... ##G,,,$8! s AA"returncJ ddl}dS#t$rtdwxYw)z|Check if BeautifulSoup4 is installed. Raises: ImportError: If BeautifulSoup4 is not installed. rNzeBeautifulSoup4 is required for BlackboardLoader. Please install it with `pip install beautifulsoup4`.)bs4 ImportError)r"r's r$r!zBlackboardLoader.check_bs4YsG   JJJJJ   G  s"cp|jr|}|||_||}g}|D]}|j|z}t d|||}tj t5| | |dddn #1swxYwY|St d|j |}|||_| |S)zZLoad data into Document objects. Returns: List of Documents. zFetching documents from N)rscrape_get_folder_path folder_path _get_pathsrprint_scrape contextlibsuppress ValueErrorextend_get_documentsweb_path)r" soup_inforelative_paths documentspathurls r$loadzBlackboardLoader.loadgs}  $ 2 I#44Y??D !__Y77NI& E Emd*666777 LL-- (44EE$$T%8%8%C%CDDDEEEEEEEEEEEEEEE  >  455 5((-- , ,D88F##DDOOC$8$8%%d+++rRr9c2|j|j|zd}||j}t t |j|z d5}||j ddddS#1swxYwYdS)zXDownload a file from an url. Args: path: Path to the file. T)allow_redirectswbN) rrbrparse_filenamer:openrr,writecontent)r"r9responsefilenamefs r$rlzBlackboardLoader.downloads<##DMD$8$#OO&&x|44 $t'((83T : : &a GGH$ % % % & & & & & & & & & & & & & & & & & &s$B  BBr:crt|x}r|jdkr|jS||S)zParse the filename from an url. Args: url: Url to parse the filename from. Returns: The filename. .pdf)rsuffixname_parse_filename_from_url)r"r:url_paths r$rwzBlackboardLoader.parse_filenames?S !H 6x&'@'@= 0055 5rRc6tjd|}|r|d}ntd|d|vrtd||dddz}t |}|dd}|S) zParse the filename from an url. Args: url: Url to parse the filename from. Returns: The filename. Raises: ValueError: If the filename could not be parsed. zfilename%2A%3DUTF-8%27%27(.+)zCould not parse filename from rzIncorrect file type: rz%20rA)researchgroupr2rrrN)r"r:filename_matchesr|s r$rz)BlackboardLoader._parse_filename_from_url s9%EsKK  E'--a00HHCcCCDD D  ! !?X??@@ @>>&))!,v58$$##E3//rR)TNNFT)r%N)__name__ __module__ __qualname____doc__rOboolrrdictrr!rr r;rr+r4rTrUrVr-rlrwr __classcell__)r#s@r$rrs%2&*04"&$)"33"33# 3 U38_- 3 $ 3"3333333j    2d8n22220 S S    < 3 4>    ST#Y8 &c &t & & & &h" s tCy     &S &T & & & & 6# 6# 6 6 6 6CCrRr__main__zhttps:///webapps/blackboard/content/listContent.jsp?course_id=__1&content_id=__1&mode=resetzT)rzLoaded z pages of PDFs from )r0rpathlibrtypingrrrr urllib.parserlangchain_core.documentsr .langchain_community.document_loaders.directoryr (langchain_community.document_loaders.pdfr -langchain_community.document_loaders.web_baser rrrpr;r8r.lenr5rRr$rsW ------------ ------JJJJJJ@@@@@@GGGGGGTTTTT}TTTn z   C &! F I E ICC NN I I I IJJJJJrR