Ng< ddlZddlZddlZddlmZmZmZmZmZm Z ddl m Z ddl m Z ejeZerddlmZGdde ZdS)N) TYPE_CHECKINGAnyIteratorListOptionalTuple)Document) BaseLoader) SparkSessionc eZdZdZ ddeddeeded efd Zd e e e ffd Z d e e fd Zd ee fdZdS)PySparkDataFrameLoaderzLoad `PySpark` DataFrames.Ntext皙? spark_sessionr dfpage_content_columnfraction_of_memoryc ddlm}m}n#t$rtdwxYw|r|n|j|_t||stdt|||_ ||_ ||_ | \|_|_|j jt$|_|j j|_dS)agInitialize with a Spark DataFrame object. Args: spark_session: The SparkSession object. df: The Spark DataFrame object. page_content_column: The name of the column containing the page content. Defaults to "text". fraction_of_memory: The fraction of memory to use. Defaults to 0.1. r) DataFramer zFpyspark is not installed. Please install it with `pip install pyspark`z3Expected data_frame to be a PySpark DataFrame, got N) pyspark.sqlrr ImportErrorbuilder getOrCreatespark isinstance ValueErrortyperrr get_num_rowsnum_rows max_num_rowsrddmaplistrdd_dfcolumns column_names)selfrrrrrr s r/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/pyspark_dataframe.py__init__zPySparkDataFrameLoader.__init__s    ; ; ; ; ; ; ; ; ;   ?  + RMM 0D0P0P0R0R "i(( Pd2hhPP #6 "4+/+<+<+>+>( t(gkood++  GOs %returnc ddl}n"#t$r}td|d}~wwxYw|jdd}t j|}|}|j}t||z |j z}t||j |fS)z4Gets the number of "feasible" rows for the DataFramerNzBpsutil not installed. Please install it with `pip install psutil`.) psutilrrlimitcollectsys getsizeofvirtual_memory availableintrmincount)r'r-erowestimated_row_sizemem_infoavailable_memoryr s r(rz#PySparkDataFrameLoader.get_num_rows:s  MMMM   T  gmmA&&((+ ]3//((**#-  2 2d6M M  <11<??s &!&c# KjD]efdttD}|j}|jt ||VfdS)z#A lazy loader for document content.c8i|]}j||S)r&).0ir8r's r( z4PySparkDataFrameLoader.lazy_load..Ns'NNN)!,c!fNNN) page_contentmetadataN)r$toLocalIteratorrangelenrpopr )r'rDrr8s` @r( lazy_loadz PySparkDataFrameLoader.lazy_loadKs;..00 A ACNNNNNeCHHooNNNHD45D LL1 2 2 2x@@@ @ @ @ @  A ArBc6|j|jkr=td|jd|jd|}ttj ||jS)zLoad from the dataframe.z The number of DataFrame rows is zQ, but we will only include the amount of rows that can reasonably fit in memory: .) rr6r loggerwarningrrIr# itertoolsislice)r'lazy_load_iterators r(loadzPySparkDataFrameLoader.loadSs 7==??T. . . NNO47==??OO>BmOOO    "^^--I$%7GGHHHrB)NNrr)__name__ __module__ __qualname____doc__rrstrfloatr)rr4rrr rIrrQr>rBr(r r s$$37 #)$' %,%,/%, SM%,! %, " %,%,%,%,N@eCHo@@@@"A8H-AAAA Id8n I I I I I IrBr )rNloggingr0typingrrrrrrlangchain_core.documentsr )langchain_community.document_loaders.baser getLogger__file__rLrr r r>rBr(r^s FFFFFFFFFFFFFFFF------@@@@@@  8 $ $)((((((LILILILILIZLILILILILIrB