Ng``ddlmZddlmZmZmZmZmZer ddlm Z m Z m Z GddZ dS)) annotations) TYPE_CHECKINGAnyIterableListOptional) DataFrameRow SparkSessionceZdZdZ d-d.dZe d/d0dZd1dZd1dZd2dZ d/d3dZ d2dZ d4d"Z d5d&Z d6d7d*Zd/d3d+Zd6d7d,ZdS)8SparkSQLz;SparkSQL is a utility class for interacting with Spark SQL.N spark_sessionOptional[SparkSession]catalog Optional[str]schema ignore_tablesOptional[List[str]]include_tablessample_rows_in_table_infointcX ddlm}n#t$rtdwxYw|r|n|j|_||jj|||jj|t| |_ |rt|n t|_ |j r$|j |j z }|rtd|d|rt|n t|_|jr$|j|j z }|rtd|d|} | rt| n|j |_t#|t$st'd||_dS) aInitialize a SparkSQL object. Args: spark_session: A SparkSession object. If not provided, one will be created. catalog: The catalog to use. If not provided, the default catalog will be used. schema: The schema to use. If not provided, the default schema will be used. ignore_tables: A list of tables to ignore. If not provided, all tables will be used. include_tables: A list of tables to include. If not provided, all tables will be used. sample_rows_in_table_info: The number of rows to include in the table info. Defaults to 3. rr Fpyspark is not installed. Please install it with `pip install pyspark`Nzinclude_tables  not found in databasezignore_tables z,sample_rows_in_table_info must be an integer) pyspark.sqlr ImportErrorbuilder getOrCreate_sparkrsetCurrentCatalogsetCurrentDatabaseset_get_all_table_names _all_tables_include_tables ValueError_ignore_tablesget_usable_table_names_usable_tables isinstancer TypeError_sample_rows_in_table_info) selfrrrrrrr missing_tables usable_tabless c/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/utilities/spark_sql.py__init__zSparkSQL.__init__ s2  0 0 0 0 0 0 0   X   + RMM 0D0P0P0R0R    K  1 1' : : :   K  2 26 : : :t88::;;6DOs>222#%%   !1D4DDN  LnLLL5BLc-000suu   !043CCN  K^KKK3355 4AWc-000tGW3S99 LJKK K*C''' # database_uristr engine_argsOptional[dict]kwargsrreturnc  ddlm}n#t$rtdwxYw|j|}||fi|S)zzCreating a remote Spark Session via Spark connect. For example: SparkSQL.from_uri("sc://localhost:15002") rrr)rr rrremoter )clsr5r7r9r sparks r2from_urizSparkSQL.from_uriKs  0 0 0 0 0 0 0   X   $++L99EEGGs5##F###r4 Iterable[str]cV|jr|jSt|j|jz S)zGet names of tables available.)r'sortedr&r))r/s r2r*zSparkSQL.get_usable_table_names\s/   (' 'd&)<<===c|jdd}t t d|S)Nz SHOW TABLES tableNamec|jSN)rE)rows r2z/SparkSQL._get_all_table_names..esCMrC)r!sqlselectcollectlistmap)r/rowss r2r%zSparkSQL._get_all_table_namescsK{}--44[AAIIKKC11488999rCtablec|jd|dj}|d}|d|dzS)NzSHOW CREATE TABLE rUSING;)r!rJrLcreatetab_stmtfind)r/rP statementusing_clause_indexs r2_get_create_table_stmtzSparkSQL._get_create_table_stmtgs` KOO888 9 9 A A C CA F U '^^G44,,,-33rC table_namesc|}|9t||}|rtd|d|}g}|D]Y}||}|jr&|dz }|d||dz }|dz }||Zd|}|S)Nz table_names rz /* z*/z ) r*r$ differencer(rXr._get_sample_spark_rowsappendjoin)r/rYall_table_namesr0tables table_name table_info final_strs r2get_table_infozSparkSQL.get_table_infoos5577  " --88IIN X !V!V!V!VWWW)O) & &J44Z@@J. #h& N4#>#>z#J#JNNNN d" MM* % % % %KK'' rCczd|d|j}|j|}dt t d|jj} ||}dd|D}n#t$rd}YnwxYw|jd|d |d|S) NzSELECT * FROM z LIMIT  c|jSrG)name)fs r2rIz1SparkSQL._get_sample_spark_rows..s16rCr[c8g|]}d|S)rg)r_).0rHs r2 z3SparkSQL._get_sample_spark_rows..s"(O(O(OC3(O(O(OrCz rows from z table: ) r.r!rJr_rMrNrfields_get_dataframe_results Exception)r/rPquerydf columns_str sample_rowssample_rows_strs r2r]zSparkSQL._get_sample_spark_rowssPPPt/NPP [__U # #iiS)9)929;K%L%L M MNN  !55b99K"ii(O(O;(O(O(OPPOO ! ! ! OOO !. ! !5 ! ! ! ! ! ! s$4B B('B(rHr tuplecttt|SrG)rwrNr6asDictvalues)r/rHs r2_convert_row_as_tuplezSparkSQL._convert_row_as_tuples.Scjjll113344555rCrsr rMcjtt|j|SrG)rMrNr{rL)r/rss r2rpzSparkSQL._get_dataframe_resultss%C2BJJLLAABBBrCallcommandfetchc|j|}|dkr|d}t||S)None)r!rJlimitr6rp)r/r~rrss r2runz SparkSQL.runsI [__W % % E>>!B4..r22333rCch ||S#t$r} d|cYd}~Sd}~wwxYw)afGet information about specified tables. Follows best practices as specified in: Rajkumar et al, 2022 (https://arxiv.org/abs/2204.00498) If `sample_rows_in_table_info`, the specified number of sample rows will be appended to each table description. This can increase performance as demonstrated in the paper. Error: N)rer()r/rYes r2get_table_info_no_throwz SparkSQL.get_table_info_no_throwsX !&&{33 3 ! ! ! * Q==  !s 1,11cj |||S#t$r} d|cYd}~Sd}~wwxYw)a*Execute a SQL command and return a string representing the results. If the statement returns rows, a string of the results is returned. If the statement returns no rows, an empty string is returned. If the statement throws an error, the error message is returned. rN)rrq)r/r~rrs r2 run_no_throwzSparkSQL.run_no_throwsX !88GU++ + ! ! ! * Q==  !s 2-22)NNNNNr) rrrrrrrrrrrrrG)r5r6r7r8r9rr:r )r:r@)rPr6r:r6)rYrr:r6)rHr r:rw)rsr r:rM)r})r~r6rr6r:r6)__name__ __module__ __qualname____doc__r3 classmethodr?r*r%rXrer]r{rprrrrCr2r r sUEE15!% $-1.2)*=D=D=D=D=D~>B$$$$[$ >>>>::::4444$    "6666CCCC44444 !!!!! ! ! ! ! ! ! !rCr N) __future__rtypingrrrrrrr r r r rrCr2rs""""""??????????????98888888888q!q!q!q!q!q!q!q!q!q!rC