Ng{ddlZddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z m Z mZmZmZmZddlmZmZmZ ddlZddlZn#e$rdZdZYnwxYwd dZGdd eZGd d ejZdS) N)datetime timedelta)partial)AnyDictIOListOptionalTupleUnion) file_lock get_cache_dir PathHandlerreturnc|d|j|_t|tjr9t j|d} | ||||dS#tj j $r/}td|t|jd||d}~wwxYw)Nrutf-8ConfigzError in file upload - : )seek_closeclose isinstanceioStringIOBytesIOgetvalueencodeupload_fileobjbotocore exceptions ClientErrorOSErrortype__name__)selfclientbuckets3_pathtransfer_configes L/var/www/html/ai-engine/env/lib/python3.11/site-packages/iopath/common/s3.pys3_close_and_uploadr-sIIaLLLDJ$ $$;z$--//0099::W   "        *WWWNNtAww7GNN1NNOOUVVWs6BC%*CCcxeZdZdZdZdZdZ d0deedeed ee fd Z d e efd Z d ed e eeffdZdefdZdefdZdeded efdZ d1dedededed ef dZdeeeeefdedededef dZ d2ded ed!ed"eed#eed$eed%eeded eeeeeffd&Z d1d'edededed ef d(Zded ee fd)Zdeded efd*Zdeded efd+Zdeded efd,Zdeded e efd-Z deded dfd.Z!deded dfd/Z"dS)3 S3PathHandleru Support for Amazon Simple Storage Service (S3) PathHanlder methods, at a glance: File --torch.load-> In --open(..., 'w')-> Amazon <- _exists,_isfile,_isdir,_ls,_rm ... System <-torch.save-- Mem. <-open(..., 'r')-- S3 <----------------_copy_from_local----------------- ----------------_get_local_path -----------------> Mem usage, for processing N bytes: open(..., mode) mode=='w': 2N, due to fully buffering user input, *and doing naive conversion from StringIO -> BytesIO*, before writing to S3 ^ Potential for optimization. mode=='wb': N, due to fully buffering user input, before writing to S3. mode=='r': N, due to fully buffering file in memory mode=='rb': N, due to fully buffering file in memory _copy_from_local: ≈0. boto3 streams from file system directly to s3 _get_local_path: ≈0. boto3 streams from s3 directly from s3 to file system NOTE: S3 doesn't have a notion of directories. This pathhandler simulates directories via uploading objects with a name ending in a slash, on calls to mkdir(). ls() calls return objects as if they were in a directory structure, via boto3's options. Fzs3://s3_cacheNsaml cache_dirprofiletransfer_config_kwargscP||_||_ddlm}|di|r|ni|_dS)a Args: cache_dir (str): Local filesystem directory to use for caching. If None, uses default from `file_io.get_cache_dir()`. transfer_config_kwargs (dict): Settings for boto3.s3.transfer.TransferConfig. Used to specify settings for multipart transfers. See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html for details. r)TransferConfigN)r2r3boto3.s3.transferr6r*)r&r2r3r4r6s r,__init__zS3PathHandler.__init__[sX # 444444 .~  )?G%%R  rc|jgS)zg Returns: List[str]: the list of URI prefixes this PathHandler can support ) S3_PREFIXr&s r,_get_supported_prefixesz%S3PathHandler._get_supported_prefixesus r:uric||jdd}|d}d|dd}||fS)z Parses a "s3://bucket/path" URI into `bucket` and `path` strings. Args: uri (str): A s3:// URI. Returns: bucket (str): the s3 bucket. path (str): the path on the s3 system. /rN)replacer<splitjoin)r&r?splitsr(paths r, _parse_urizS3PathHandler._parse_uri|sST^R0066s;;xxqrr ##t|r:r(ctjt}t|ds t j|j}|d|_np#tj j $rY}| d| dtd|t|jd||d}~wwxYw|jS)Nr') profile_names3z See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for method of using environment variable to point to aws credentials, and the order in which boto will search for said credentials. a&Boto3 searches via the order below. If on FAIR Cluster, method 4 may be most convenient.The order in which Boto3 searches for credentials is:1) [UNUSED] Passing credentials as parameters in the boto.client() method2) [UNUSED] Passing credentials as parameters when creating a Session object3) Environment variables AWS_ACCESS_KEY_ID - The access key for your AWS account. AWS_SECRET_ACCESS_KEY - The secret key for your AWS account. AWS_SESSION_TOKEN - The session key for your AWS account. This is only needed when you are using temporary credentials. 4) Shared credential file (~/.aws/credentials) default: ~/.aws/credentials changed via: AWS_SHARED_CREDENTIALS_FILE *for FAIR cluster usage: `export AWS_SHARED_CREDENTIALS_FILE=~/.fairusers_aws/credentials`5) AWS config file (~/.aws/config) default: ~/.aws/config changed via: AWS_CONFIG_FILE6) Assume Role provider7) Boto2 config file (/etc/boto.cfg and ~/.boto)8) Instance metadata service on an Amazon EC2 instance that has an IAM role configured.z%Error in making s3 client for bucket r)logging getLoggerr%hasattrboto3Sessionr3r'r r!NoCredentialsErrorerrorr#r$)r&r(loggersessionr+s r, _get_clientzS3PathHandler._get_clients"8,,tX&&$ # -T\BBB%nnT22 &9    N  n,/F/Aww'//+,//9 B{s4A C 4ACC rHc||\}}tjt |j|j|S)a Helper that returns a local cache path for a given uri. Args: path (str): A URI supported by this PathHandler. Returns: local_cache_path (str): a file path which exists on the local file system, in a cache directory. )rIosrHrFrr2CACHE_SUBDIR_NAME)r&rHr( file_paths r,_local_cache_pathzS3PathHandler._local_cache_pathsF!OOD11 w|| $. ) )4+A9   r:kwargsc tjt}|||drt d||sJ||}t|5tj |r| |}||d}tjtj |}||z tjdkr7|d|||cdddS|d||d z}tj |rtj|||\} } || } | | | ||j }t7j|| tj|n:#t:$rYn.wxYw# tj|w#t:$rYwwxYwxYw|d |||cdddS#1swxYwYdS) aZ Get a filepath which is compatible with native Python I/O such as `open` and `os.path`. If URI points to a remote resource, this function may download and cache the resource to local disk. In this case, the cache stays on filesystem (under `file_io.get_cache_dir()`) and will be used by a different run. Therefore this function is meant to be used with read-only resources. Args: path (str): A URI supported by this PathHandler Returns: local_path (str): a file path which exists on the local file system rBz@S3PathHandler does not currently support downloading directoriesN LastModifiedr)minuteszURL {} was already cached in {}zCaching {} ...z.tmprzURL {} cached in {})rMrNr% _check_kwargsendswithNotImplementedError_isfiler[r rXrHexists _head_objectdtr fromtimestampgetmtime astimezonerinfoformatisfileunlinkrIrV download_filer*shutilmove Exception) r&rHr\rT local_pathresponse remote_dtlocal_dttmpr(r)r's r,_get_local_pathzS3PathHandler._get_local_paths&"8,, 6""" ==   %R ||D!!!!!++D11 z " "1 1 w~~j)) * ,,T22' ( 8I!{88((44   jll!9, Q0G0G0GGG =DDT:VV *+1 1 1 1 1 1 1 1 . KK(//55 6 6 6v%Cw~~c""  #"ood33OFG%%f--F !//GS1E0 C,,,IcNNNN DIcNNNN D KK-44T:FF G G Gc1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 stCJ6.B J6<3I0IJ6 IJ6IJ6I<I,+I<, I9 6I<8I9 9I<<-J66J:=J:rrdst_path overwritec |||ds|drtd|sX||rCt jt }|d|dS| |\}}| |} | ||||j dS#tjj$rY} t jt }|dt!| Yd} ~ dSd} ~ wwxYw) a Copies a local file to the specified URI. If the URI is another local path, this should be functionally identical to copy. Args: local_path (str): a file path which exists on the local file system dst_path (str): A URI supported by this PathHandler overwrite (bool): Bool flag for forcing overwrite of existing URI Returns: status (bool): True on success rBz>S3PathHandler does not currently support uploading directories*Error: Destination path {} already exists.FrTzError in file upload - {}N)r`rarb_existsrMrNr%rSrkrIrV upload_filer*r r!r"str) r&rrrxryr\rTr(r)r'r+s r,_copy_from_localzS3PathHandler._copy_from_localsh 6"""   s # # x'8'8'='= %P  T\\(33 &x00F LLELLXVV W W W5//(33!!&))    z674CW  X X X4".   &x00F LL4;;CFFCC D D D55555 sC77E$ AEE$bufferr'r)r*c|j|_tt||||}t j|||_dS)N)r'r(r)r*)rrrr-types MethodType)r&rr'r(r)r*fns r,_decorate_buf_with_s3_methodsz+S3PathHandler._decorate_buf_with_s3_methods>sL   +    "'F33 r:rmode bufferingencodingerrorsnewlineread_chunk_sizec "||||\} } || } d|vr|tj} | | | | |jnF#tjj $r/} td| t| j d| | d} ~ wwxYw| dnt| | | |} | | | d|_d}d |vrd }tj| d |ddd S| Sd|vrKd |vrtj} ntj} || | | | |j| Std|)a Open a stream to a URI, similar to the built-in `open`. Args: path (str): A URI supported by this PathHandler mode (str): Specifies the mode in which the file is opened. It defaults to 'r'. buffering (int): An optional integer used to set the buffering policy. Pass 0 to switch buffering off and an integer >= 1 to indicate the size in bytes of a fixed-size chunk buffer. When no buffering argument is given, the default buffering policy depends on the underlying I/O implementation. Returns: file: a file-like object. rNrz%Error in making s3 client for bucekt rrBucketKey ContentLengthbrTF) write_throughrrrline_bufferingwzUnsupported open mode )r`rIrVrrdownload_fileobjr*r r!r"r#r$r%r S3ChunkReadIO get_objectlength TextIOWrapperrr)r&rHrrrrrrr\r(r)r'rr+s r,_openzS3PathHandler._open`s8 6"""//$//!!&)) $;;&  ++8L, *6!3377+33/033 A'vvwPP ++6w+GGXDKH$"'"&% #(     D[[d{{  . .1E   M9499:: :sA<<B?*B::B?src_pathc |||sX||rCtjt}|d|dS||\}}||\}} ||ks Jd||} | ||d|| |j dS#tj j $rY} tjt}|dt| Yd} ~ dSd} ~ wwxYw) aV Copies a source path to a destination path. Args: src_path (str): A URI supported by this PathHandler dst_path (str): A URI supported by this PathHandler overwrite (bool): Bool flag for forcing overwrite of existing file Returns: status (bool): True on success r{Fz*For now, can only _copy() within a bucket.rrTzError in file copy - {}N)r`r|rMrNr%rSrkrIrVcopyr*r r!r"r~) r&rrxryr\rT src_bucket src_s3_path dst_bucket dst_s3_pathr'r+s r,_copyzS3PathHandler._copysp 6""" T\\(33 &x00F LLELLXVV W W W5"&//(";"; K"&//(";"; KZ''')U'''!!*--  KK(&+    4".   &x00F LL299#a&&AA B B B55555 s!C))E=AEEc X||\}}||} |||}|S#tjj$rM}|jdddkr+td|dt|j d||Yd}~dSd}~wwxYw)NrErrorMessagez Bad RequestzError in checking s3 path  - r) rIrV head_objectr r!r"rsr#r$r%)r&rHr(r)r'rsr+s r,rezS3PathHandler._head_objects//$//!!&)) ))W)EEHO".   z'"9->>UUUd1gg>NUURSUU44444  sAB)AB$$B)c Z||||duS)z Checks if there is a resource at the given URI. Args: path (str): A URI supported by this PathHandler Returns: bool: true if the path exists N)r`rer&rHr\s r,r|zS3PathHandler._existss1 6"""  &&d22r:c t|||d o |j|fi|S)z Checks if the resource at the given URI is a file. Args: path (str): A URI supported by this PathHandler Returns: bool: true if the path is a file rBr`rar|rs r,rczS3PathHandler._isfile sH 6"""==%%%F,$,t*F*Fv*F*FFr:c r|||do |j|fi|S)z Checks if the resource at the given URI is a directory. Args: path (str): A URI supported by this PathHandler Returns: bool: true if the path is a directory rBrrs r,_isdirzS3PathHandler._isdirsE 6"""}}S!!Bldl4&B&B6&B&BBr:c ||||\}}||} |d}|||d}d|D}d|D} || zS#t jj$r0} td|dt| j d| | d } ~ wwxYw) z List the contents of the directory at the provided URI. Args: path (str): A URI supported by this PathHandler Returns: List[str]: list of contents in given path list_objects_v2rB)rPrefix DelimitercPg|]#}|dgD] }|d $S)Contentsrget.0pageobjs r, z%S3PathHandler._ls..;sK#R8P8P14E r:cPg|]#}|dgD] }|d $S)CommonPrefixesrrrs r,rz%S3PathHandler._ls..>sR88$4b99H r:zError in ls path rrN) r`rIrV get_paginatorpaginater r!r"r#r$r%) r&rHr\r(r)r' paginatorpages obj_results dir_resultsr+s r,_lszS3PathHandler._ls&s# 6"""//$//!!&)) ,,->??I&&'E ',K!K , ,".   HDHHa1AHHQHH  sA BC"+C  Cc |||ds J|||\}}||} |||dS#t jj$r0}td|dt|j d||d}~wwxYw)a Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory. Similar to the native `os.makedirs`. Args: path (str): A URI supported by this PathHandler rBrzError in mkdirs path rrN) r`rarIrV put_objectr r!r"r#r$r%r&rHr\r(r)r'r+s r,_mkdirszS3PathHandler._mkdirsIs 6"""}}S!!''4'''//$//!!&))    V  9 9 9 9 9".   LLLT!WW5ELLLL  s#A<<C+B;;Cc H||||\}}||} |||dS#tjj$r0}td|dt|j d||d}~wwxYw)z Remove the file (not directory) at the provided URI. Args: path (str): A URI supported by this PathHandler rzError in rm path rrN) r`rIrV delete_objectr r!r"r#r$r%rs r,_rmzS3PathHandler._rm_s 6"""//$//!!&))   G < < < < <".   HDHHa1AHHQHH  sAB!1+BB!)Nr1N)F)rrNNNN)#r% __module__ __qualname____doc___strict_kwargs_checkr<rYr r~rr9r r>r rIrVr[rrwboolrr rbytesrintrrrer|rcrrrrr7r:r,r/r/6s<!I" $(!'26   C= # !)     4 c    ceCHo"(#((((V     &ICI3I3IIII\AF###),#9=#QT# ####L 4bgr%y() 4 4  4  4 4 4 4 4J#' $!%)-e;e;e;e; e;3-e; e;#e;"#e;e; r#w5 ! "e;e;e;e;P?D&&&'*&7;&OR& &&&&T  $     3C 33 34 3 3 3 3 GC G3 G4 G G G G C3 C# C$ C C C C!!s!tCy!!!!FC34,str:r/c @eZdZdZedfdedededeefdZe defd Z de fd Z de fd Z de fd Zdefd ZddededefdZdefdZd deedefdZdeeefdeefdZd!dZd"dedefdZd"dedefdZdedefdZdeddfdZdS)#ri Nr(key chunk_sizetimeoutc$||_||_||_||nd|_||_d|_tdd|_tj |_ | ||d|_ dS)Nrrr)r'r(r total_secondsrroffsetrangebuffered_windowrrrrr)r&r'r(rrrs r,r9zS3ChunkReadIO.__init__ws  292Ew,,...4 $ $Q{{jll ''v3'??P r:rc|jSN)rHr=s r,namezS3ChunkReadIO.names yr:cdS)z Return a bool indicating whether object supports random access. If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek(). Tr7r=s r,seekablezS3ChunkReadIO.seekables tr:cdS)z Return a bool indicating whether object was opened for reading. If False, read() will raise OSError. Tr7r=s r,readablezS3ChunkReadIO.readables tr:cdS)z Return a bool indicating whether object was opened for writing. If False, write() and truncate() will raise OSError. Fr7r=s r,writablezS3ChunkReadIO.writables ur:ctr)AttributeErrorr=s r,filenozS3ChunkReadIO.filenosr:rrwhencec|dkr|dksJ||_n<|dkr!||jzdksJ|xj|z c_n|dkr|j|z|_|jS)a3 Change stream position. Change the stream position to byte offset offset. Argument offset is interpreted relative to the position indicated by whence. Values for whence are ints: * 0 -- start of stream (the default); offset should be zero or positive * 1 -- current stream position; offset may be negative * 2 -- end of stream; offset is usually negative Some operating systems / file systems could provide additional values. Return an int indicating the new absolute position. rrC)rr)r&rrs r,rzS3ChunkReadIO.seeksu Q;;Q;;;; DKK q[[DK'1,,,, KK6 !KKK q[[+.DK{r:c|jS)z5Return an int indicating the current stream position.)rr=s r,tellzS3ChunkReadIO.tells {r:sizec td)z Truncate file to size bytes. Size defaults to the current IO position as reported by tell(). Return the new size. zcan't truncate readonly streamr#r&rs r,truncatezS3ChunkReadIO.truncates6777r:rc td)zK Write bytes b to in-memory buffer, return number written. zcan't write to readonly streamr)r&rs r,writezS3ChunkReadIO.writes6777r:cdS)z noop Nr7r=s r,rzS3ChunkReadIO.closes  r:rc,||Sr)readrs r,read1zS3ChunkReadIO.read1syyr:c t||dkr|j|jz }t||j|jz }t}|j|jvr[|j|jjz }||j|t||zt|jz }t||kr,|xjt|z c_t|S|t|z |j kr|xjt|z c_| t|jt|j|zt|z dz |j}|xjt|z c_||zS| |jt|z||jd|t|z z }t||ksJ|xjt|z c_t|S)z Read and return up to size bytes. If the argument is omitted, None, or negative, data is read and returned until EOF is reached. An empty bytes object is returned if the stream is already at EOF. NrrC)rrmin bytearrayrstartr getbufferlenrr _read_from_s3r_read_chunk_to_buffer)r&rret buffer_offsetoutputs r,rzS3ChunkReadIO.reads <4!88;,D4t{233kk ;$. . . K$*>*DDM 4;((**MD$8#d>R:S:S T TT C s88t   KK3s88 #KK::  #c((?T_ , , KK3s88 #KK''dk3t{T'9CHH'Dq'H$+#V#VWWF KK3v;; &KK<  ""4;S#9::: t{$$&&q4#c((?':;;3xx4 s3xx Szzr:download_rangecP|j|j|jd|jd|j}|d}|j||jt}| |j D]}||z }| |S)Nzbytes=-)rrRangeBody)r) r'rr(rrstoprset_socket_timeoutr iter_chunksrr)r&rrstreaming_bodyrchunks r,rzS3ChunkReadIO._read_from_s3sk$$;G>/GG.2EGG%   V < #  - -dl ; ; ;kk#//4?/KK  E 5LCC r: start_offsetct|t||jz|j}||}|jd|j|||_dS)z download a chuck size of data start from start_offset into current buffer, then update self.buffered_window for booking which part of data is currently buffered rN) rrrrrrrrr)r&rrrs r,rz#S3ChunkReadIO._read_chunk_to_buffer*sz  #lT_rs  ((((((((>>>>>>>>>>>>>>>>>>GGGGGGGGGG LLLOOOO EHHHWWWW0yyyyyKyyyxE.E.E.E.E.B%E.E.E.E.E.sA AA