B ffHddlmZGddZGddeZdS)datetimeceZdZdZdZedZdZdZdZ dZ dZ d Z d Z d Zd Zed ZedZdS)InventoryReporta A utility class for fetching and processing inventory reports from GCS. The 'InventoryReport' class provides logic to support logic to fetch inventory reports, and process their content to obtain a final snapshot of objects in the latest inventory reports. High-Level Functionality: ------------------------ 1. Fetching Inventory Reports: - The class offers methods to fetch inventory report configurations and metadata from GCS. - It validates the inventory report information provided by the user. - Inventory report configurations include options for parsing CSV format and specifying the bucket and destination path. 2. Parsing and Processing Inventory Report Content: - The class processes the raw content of inventory reports to extract object details such as name, size, etc. - It supports listing objects using a snapshot option or filtering based on a user-defined prefix. - The class handles CSV parsing, removes header (if specified), and fetches required object metadata. 3. Constructing the Final Snapshot: - If the user wishes to use the snapshot to do listing directly, the snapshot will contain the relevant object details and subdirectory prefixes, filtered by the prefix. - If the user wishes to use the snapshot as a starting point for async listing, the snapshot will only contain a list of object names, filtered by the prefix. Note: ----- - The class should only be internally used in the 'GCSFileSystem' as an optional configuration during listing. Example Usage: -------------- # Should already be instanted in 'core.py' gcs_file_system = GCSFileSystem(...) # User defines inventory report information inventory_report_info = { "use_snapshot_listing": True, "location": "us-east1", "id": "inventory_report_id" } # User defines a prefix for filtering objects prefix = "prefix/" # Fetch the snapshot based on inventory reports items, prefixes = await InventoryReport.fetch_snapshot( gcs_file_system, inventory_report_info, prefix) z)https://storageinsights.googleapis.com/v1c0K|||d}|d}|d}||||d{V}|||}|||d{V} || } |j} ||| | d{V} ||| ||| } | | || }|S) a Main entry point of the 'InventoryReport' class. Fetches the latest snapshot of objects based on inventory report configuration. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). inventory_report_info (dict): A client-configured dictionary containing inventory report information. prefix (str): Listing prefix specified by the client. Returns: tuple: A tuple containing two lists: the 'items' list representing object details for the snapshot, and the 'prefixes' list containing subdirectory prefixes. Note: when 'use_snapshot_listing' in 'inventory_report_info' is set to False, the 'prefixes' list will be empty, and the 'items' list will contain only the object names. use_snapshot_listinglocationid)gcs_file_systemr r N)raw_inventory_report_configr)r inventory_report_config"unsorted_inventory_report_metadata)r inventory_report_metadatabucket)r inventory_report_contentr rr)objectsprefixr) _validate_inventory_report_infoget"_fetch_raw_inventory_report_config"_parse_raw_inventory_report_config _fetch_inventory_report_metadata_sort_inventory_report_metadatar"_download_inventory_report_content_parse_inventory_report_content_construct_final_snapshot)clsr inventory_report_inforrinventory_report_locationinventory_report_idr r rrrrrsnapshots e/home/panchajanya/Development/work/volatility3/lib/python3.11/site-packages/gcsfs/inventory_report.pyfetch_snapshotzInventoryReport.fetch_snapshotBs. ++,ABBB 5889OPP$9$=$=j$I$I!377==-0,R,R+."-S- - ' ' ' ' ' ' ##&"H"H(C!5#I# #  473W3W+$;4X4 4 . . . . . . * %($G$G/Q%H% % ! )/),)O)O+&?*P* * $ $ $ $ $ $  55+%=$;!5 6  00FAU1   cxd|vrtdd|vrtdd|vrtddS)a Validates the inventory report information dictionary that user passes in. Parameters: inventory_report_info (dict): A dictionary containing the inventory report information with the following keys: - "use_snapshot_listing" (bool): A flag indicating whether to use snapshot listing in the inventory report. - "location" (str): The location of the inventory report in GCS. - "id" (str): The ID of the inventory report in GCS. Raises: ValueError: If any required key (use_snapshot_listing, location, id) is missing from the inventory_report_info dictionary. rz'Use snapshot listing is not configured.r z,Inventory report location is not configured.r z&Inventory report id is not configured.N) ValueError)rs r#rz/InventoryReport._validate_inventory_report_infos]" ")> > >FGG G 2 2 2KLL L , , ,EFF F - ,r%cK|j}d}|tj|||} |d|dd{V}|S#t $r}t d|dd}~wwxYw)a Fetches the raw inventory report configuration from GCS based on the specified location and ID. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). location (str): The location of the inventory report in GCS. id (str): The ID of the inventory report in GCS. Returns: dict: A dictionary containing the raw inventory report configuration retrieved from GCS. Raises: Exception: If there is an error while fetching the inventory report configuration. z,{}/projects/{}/locations/{}/reportConfigs/{}GETT)json_outNz9Error encountered when fetching inventory report config: .)projectformatrBASE_URL_call Exceptionr')r r r r,urlr es r#rz2InventoryReport._fetch_raw_inventory_report_configs&")<jj17HbII 0?0E0EsT1F11++++++ '/ .   PAPPP  sA A1A,,A1c |d}t|d}t|d}|d}|d}|d}|d}|d} |d } tj} | |ks| |krt d | d |d |d | d} |r4 | dn#t $rt dwxYwt||| | | } | S)a Parses the raw inventory report configuration and validates its properties. Parameters: raw_inventory_report_config (dict): A dictionary containing the raw inventory report configuration retrieved from GCS. use_snapshot_listing (bool): A flag indicating whether to use snapshot listing in the inventory report. Returns: InventoryReportConfig: An instance of the InventoryReportConfig class representing the parsed inventory report configuration. Raises: ValueError: If the current date is outside the start and end range specified in the inventory report config. ValueError: If the "name" field is not present in the metadata fields of the report config. ValueError: If "size" field is not present in the metadata fields and use_snapshot_listing is True. frequencyOptions startDateendDateobjectMetadataReportOptionsstorageDestinationOptions csvOptionsrdestinationPathmetadataFieldsz Current date z& is outside the range z and z* specified by the inventory report config.namesizezIf you want to use the snapshot for listing, the object size metadata has to be included in the inventory report. csv_optionsrdestination_pathmetadata_fields obj_name_idx)rr_convert_obj_to_daternowr'indexInventoryReportConfig)r rfrequency_options start_dateend_dateobject_metadata_report_optionsstorage_destination_optionsr?rr@rAtodayrBr s r#rz2InventoryReport._parse_raw_inventory_report_configs28;;EE.cKg}|j}|j}|dd||dd{V}||dg|dd}|b|dd||d|d{V}||dg|dd}|bt |d krt d |S) a Fetches all inventory report metadata from GCS based on the specified inventory report config. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). inventory_report_config (InventoryReportConfig): An instance of the InventoryReportConfig class representing the inventory report configuration. Returns: list: A list containing dictionaries representing the metadata of objects from the inventory reports. Raises: ValueError: If the fetched inventory reports are empty. r)zb/{}/oT)rr*Nitems nextPageToken)rr* pageTokenrzbNo inventory reports to fetch. Check if your inventory report is set up correctly.)rr@r/extendrlenr')r r rrr@pagenext_page_tokens r#rz0InventoryReport._fetch_inventory_report_metadatasd,%'!)/2C%** 8V,)>???((?D99)(..') /D & , ,TXXgr-B-B C C C"hh==O) ( ) )Q . .<  )(r%c(t|ddS)a Sorts the inventory report metadata based on the 'timeCreated' field in reverse chronological order. Parameters: unsorted_inventory_report_metadata (list): A list of dictionaries representing the metadata of objects from the inventory reports. Returns: list: A sorted list of dictionaries representing the inventory report metadata, sorted in reverse chronological order based on 'timeCreated'. c\t|dS)N timeCreated)r_convert_str_to_datetimer)irs r#zAInventoryReport._sort_inventory_report_metadata..is%?CC}%%r%T)keyreverse)sortedrs r#rz/InventoryReport._sort_inventory_report_metadataYs- .     r%c K|d}t|d}g}|D]}t|d}||kr_|dd||ddd{V\}} | } || |S) ax Downloads the most recent inventory report content from GCS based on the inventory report metadata. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). inventory_report_metadata (list): A list of dictionaries representing the metadata of objects from the inventory reports. bucket (str): The name of the GCS bucket containing the inventory reports. Returns: list: A list containing the content of the most recent inventory report as strings. rrWr)z b/{}/o/{}r<media)altN)rrXrdater/decodeappend) r rrmost_recent_inventory_reportmost_recent_datermetadatainventory_report_date_headerencoded_contentdecoded_contents r#rz2InventoryReport._download_inventory_report_contentos((A'C$*CC ( , ,] ; ;  $&& $& 2 A AH$3$L$L'%%dff "%(8882A1F1F; V0D0D'2G22,,,,,,( #2"8"8":":(//@@@''r%c \|j}|dd}|dd}|dd}g} |D]]} | |} |r | dd} | D]7} t| ||||| } | | 8^| S) a Parses the raw inventory report content and extracts object details. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). inventory_report_content (list): A list of strings containing the raw content of the inventory report. inventory_report_config (InventoryReportConfig): An instance of the InventoryReportConfig class representing the inventory report configuration. use_snapshot_listing (bool): A flag indicating whether to use snapshot listing in the inventory report. bucket (str): The name of the GCS bucket containing the inventory reports. Returns: list: A list of dictionaries representing object details parsed from the inventory report content. recordSeparator delimiter,headerRequiredFN)inventory_report_linerr r rnr)r?rsplitr_parse_inventory_report_linerc)r rr rrr?record_separatorrnheader_requiredrcontentlineslineobjs r#rz/InventoryReport._parse_inventory_report_contents8.9 &??+z@InventoryReport._parse_inventory_report_line..s."Ur%r<)rBrA_process_objectzipstriprs) rrrr r rnrrBrArzs r#rtz,InventoryReport._parse_inventory_report_lines</; 1A 4 ' '!11&)')>)D)D)F)F)L)LY)W)W'' CC06688>>yII,WXC r%cd|durfd|DgfSg}t}|D]}|d}|s-|td}|d}|dkr||u|d|} ddkrdndz| z| dkrdndz} || |t|fS)aA Constructs the final snapshot based on the retrieved objects and prefix. Parameters: objects (list): A list of dictionaries representing object details from the inventory report. prefix (str): A prefix used to filter objects in the snapshot based on their names. use_snapshot_listing (bool): A flag indicating whether to use snapshot listing in the inventory report. Returns: tuple: A tuple containing two lists: the 'items' list representing object details for the snapshot, and the 'prefixes' list containing subdirectory prefixes. If 'use_snapshot_listing' is set to False, 'prefix' will also be empty, and 'items' will contains the object names in the snapshot. NFcdg|],}|d*|-S)r<)r startswith)r~rzrs r# z=InventoryReport._construct_final_snapshot..)s6QQQCcggfoo.H.H.P.PQCQQQr%r</) setrrrRfindrcrstripaddlist) rrrrNprefixesrzobj_nameobject_name_no_prefixfirst_delimiter_idxdir obj_prefixs ` r#rz)InventoryReport._construct_final_snapshotsU& >F 5 ( (QQQQ7QQQSUU U EuuH ) )776??  **622)1V(?%'<&@&@&E&E#'",,LL%%%,,@-@,@AMM#&&#r\\rrs4!BYYrrC1  Z((((d8nn$$r%cT|d}|d}|d}t|||S)a: Converts a dictionary representing a date object to a datetime object. Parameters: obj (dict): A dictionary representing a date object with keys "day", "month", and "year". Returns: datetime: A datetime object representing the converted date. daymonthyearr)rzrrrs r#rCz$InventoryReport._convert_obj_to_dateRs1%jG 6{eS)))r%cRtj|ddS)a8 Converts an ISO-formatted date string to a datetime object. Parameters: date_string (str): An ISO-formatted date string with or without timezone information (Z). Returns: datetime: A datetime object representing the converted date and time. Zz+00:00)r fromisoformatreplace)strs r#rXz(InventoryReport._convert_str_to_datetimecs#%ckk#x&@&@AAAr%N)__name__ __module__ __qualname____doc__r. classmethodr$rrrrrrrrtr staticmethodrCrXr}r%r#rrs88v;HLL[L\GGG0@N'N'N'`>)>)>)@   ,2(2(2(h888t111f@%@%@%D**\*  B B\ B B Br%rceZdZdZdZdS)rFa Represents the configuration for fetching inventory reports. Attributes: csv_options (dict): A dictionary containing options for parsing CSV format in the inventory reports. bucket (str): The name of the GCS bucket from which to fetch the inventory reports. destination_path (str): The path within the GCS bucket where the inventory reports are stored. metadata_fields (list): A list of strings representing metadata fields to be extracted from the inventory reports. obj_name_idx (int): The index of the "name" field in the 'metadata_fields' list, used to identify object names. cL||_||_||_||_||_dS)Nr>)selfr?rr@rArBs r#__init__zInventoryReportConfig.__init__s1' 0.(r%N)rrrrrr}r%r#rFrFrs- )))))r%rFN)rrobjectrFr}r%r#rsk Bk Bk Bk Bk Bk Bk Bk B\)))))F)))))r%