'wff:ddlmZGddZGddeZy)datetimeczeZdZdZdZedZdZdZdZ dZ dZ d Z d Z d Zd Zed ZedZy)InventoryReporta A utility class for fetching and processing inventory reports from GCS. The 'InventoryReport' class provides logic to support logic to fetch inventory reports, and process their content to obtain a final snapshot of objects in the latest inventory reports. High-Level Functionality: ------------------------ 1. Fetching Inventory Reports: - The class offers methods to fetch inventory report configurations and metadata from GCS. - It validates the inventory report information provided by the user. - Inventory report configurations include options for parsing CSV format and specifying the bucket and destination path. 2. Parsing and Processing Inventory Report Content: - The class processes the raw content of inventory reports to extract object details such as name, size, etc. - It supports listing objects using a snapshot option or filtering based on a user-defined prefix. - The class handles CSV parsing, removes header (if specified), and fetches required object metadata. 3. Constructing the Final Snapshot: - If the user wishes to use the snapshot to do listing directly, the snapshot will contain the relevant object details and subdirectory prefixes, filtered by the prefix. - If the user wishes to use the snapshot as a starting point for async listing, the snapshot will only contain a list of object names, filtered by the prefix. Note: ----- - The class should only be internally used in the 'GCSFileSystem' as an optional configuration during listing. Example Usage: -------------- # Should already be instanted in 'core.py' gcs_file_system = GCSFileSystem(...) # User defines inventory report information inventory_report_info = { "use_snapshot_listing": True, "location": "us-east1", "id": "inventory_report_id" } # User defines a prefix for filtering objects prefix = "prefix/" # Fetch the snapshot based on inventory reports items, prefixes = await InventoryReport.fetch_snapshot( gcs_file_system, inventory_report_info, prefix) z)https://storageinsights.googleapis.com/v1cK|j||jd}|jd}|jd}|j|||d{}|j||}|j ||d{} |j | } |j } |j|| | d{} |j|| ||| } |j| || }|S77l74w) a Main entry point of the 'InventoryReport' class. Fetches the latest snapshot of objects based on inventory report configuration. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). inventory_report_info (dict): A client-configured dictionary containing inventory report information. prefix (str): Listing prefix specified by the client. Returns: tuple: A tuple containing two lists: the 'items' list representing object details for the snapshot, and the 'prefixes' list containing subdirectory prefixes. Note: when 'use_snapshot_listing' in 'inventory_report_info' is set to False, the 'prefixes' list will be empty, and the 'items' list will contain only the object names. use_snapshot_listinglocationid)gcs_file_systemr r N)raw_inventory_report_configr)r inventory_report_config"unsorted_inventory_report_metadata)r inventory_report_metadatabucket)r inventory_report_contentr rr)objectsprefixr) _validate_inventory_report_infoget"_fetch_raw_inventory_report_config"_parse_raw_inventory_report_config _fetch_inventory_report_metadata_sort_inventory_report_metadatar"_download_inventory_report_content_parse_inventory_report_content_construct_final_snapshot)clsr inventory_report_inforrinventory_report_locationinventory_report_idr r rrrrrsnapshots j/home/panchajanya/Development/work/volatility3/venv/lib/python3.12/site-packages/gcsfs/inventory_report.pyfetch_snapshotzInventoryReport.fetch_snapshotBsc. ++,AB 5889OP$9$=$=j$I!377=-0,R,R+."-S- ' ##&"H"H(C!5#I#  473W3W+$;4X4 . * %($G$G/Q%H% ! )//),)O)O+&?*P* $  55+%=$;!5 6 00FAU1  ]' . $ s6AC=C7-C= C9 9C=C;1C=9C=;C=c^d|vr tdd|vr tdd|vr tdy)a Validates the inventory report information dictionary that user passes in. Parameters: inventory_report_info (dict): A dictionary containing the inventory report information with the following keys: - "use_snapshot_listing" (bool): A flag indicating whether to use snapshot listing in the inventory report. - "location" (str): The location of the inventory report in GCS. - "id" (str): The ID of the inventory report in GCS. Raises: ValueError: If any required key (use_snapshot_listing, location, id) is missing from the inventory_report_info dictionary. rz'Use snapshot listing is not configured.r z,Inventory report location is not configured.r z&Inventory report id is not configured.N) ValueError)rs r#rz/InventoryReport._validate_inventory_report_infosH" ")> >FG G 2 2KL L , ,EF F -cK|j}d}|jtj|||} |j d|dd{}|S7#t $r}t d|dd}~wwxYww)a Fetches the raw inventory report configuration from GCS based on the specified location and ID. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). location (str): The location of the inventory report in GCS. id (str): The ID of the inventory report in GCS. Returns: dict: A dictionary containing the raw inventory report configuration retrieved from GCS. Raises: Exception: If there is an error while fetching the inventory report configuration. z,{}/projects/{}/locations/{}/reportConfigs/{}GETT)json_outNz9Error encountered when fetching inventory report config: .)projectformatrBASE_URL_call Exceptionr&)r r r r,urlr es r#rz2InventoryReport._fetch_raw_inventory_report_configs&"))<jj117HbI 0?0E0EsT1F1+ '/ .+ KA3aP  s:1A4A A AA4A A1A,,A11A4c |jd}tj|jd}tj|jd}|jd}|jd}|jd}|jd}|jd} |jd } tj} | |ks| |kDrt d | d |d |d | j d} |r | j dt||| | | } | S#t $r t dwxYw)a Parses the raw inventory report configuration and validates its properties. Parameters: raw_inventory_report_config (dict): A dictionary containing the raw inventory report configuration retrieved from GCS. use_snapshot_listing (bool): A flag indicating whether to use snapshot listing in the inventory report. Returns: InventoryReportConfig: An instance of the InventoryReportConfig class representing the parsed inventory report configuration. Raises: ValueError: If the current date is outside the start and end range specified in the inventory report config. ValueError: If the "name" field is not present in the metadata fields of the report config. ValueError: If "size" field is not present in the metadata fields and use_snapshot_listing is True. frequencyOptions startDateendDateobjectMetadataReportOptionsstorageDestinationOptions csvOptionsrdestinationPathmetadataFieldsz Current date z& is outside the range z and z* specified by the inventory report config.namesizezIf you want to use the snapshot for listing, the object size metadata has to be included in the inventory report. csv_optionsrdestination_pathmetadata_fields obj_name_idx)rr_convert_obj_to_daternowr&indexInventoryReportConfig)r rfrequency_options start_dateend_dateobject_metadata_report_optionsstorage_destination_optionsr?rr@rAtodayrBr s r#rz2InventoryReport._parse_raw_inventory_report_configs28;;?((?D9)(..') /D & , ,TXXgr-B C"hh=O) ( )Q .<  )(C s(4C$C AC$ C" 9C$C$"C$c t|ddS)a Sorts the inventory report metadata based on the 'timeCreated' field in reverse chronological order. Parameters: unsorted_inventory_report_metadata (list): A list of dictionaries representing the metadata of objects from the inventory reports. Returns: list: A sorted list of dictionaries representing the inventory report metadata, sorted in reverse chronological order based on 'timeCreated'. cJtj|jdS)N timeCreated)r_convert_str_to_datetimer)irs r#zAInventoryReport._sort_inventory_report_metadata..is?CC}%r'T)keyreverse)sortedrs r#rz/InventoryReport._sort_inventory_report_metadataYs .   r'c K|d}tj|jdj}g}|D]}tj|dj}||k(s/|j dd||jddd{\}} | j } |j | |S7,w) ax Downloads the most recent inventory report content from GCS based on the inventory report metadata. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). inventory_report_metadata (list): A list of dictionaries representing the metadata of objects from the inventory reports. bucket (str): The name of the GCS bucket containing the inventory reports. Returns: list: A list containing the content of the most recent inventory report as strings. rrWr)z b/{}/o/{}r<media)altN)rrXrdater/decodeappend) r rrmost_recent_inventory_reportmost_recent_datermetadatainventory_report_date_headerencoded_contentdecoded_contents r#rz2InventoryReport._download_inventory_report_contentos((A'C$*CC ( , ,] ; $& $& 2 AH$3$L$L'%df "%(882A1F1F; V0D'2G2,( #2"8"8":(//@! A$(',sA*C-(CC-Cc 2|j}|jdd}|jdd}|jdd}g} |D]M} | j|} |r| dd} | D].} tj | ||||| } | j | 0O| S) a Parses the raw inventory report content and extracts object details. Parameters: gcs_file_system (GCSFileSystem): An instance of the 'GCSFileSystem' class (see 'core.py'). inventory_report_content (list): A list of strings containing the raw content of the inventory report. inventory_report_config (InventoryReportConfig): An instance of the InventoryReportConfig class representing the inventory report configuration. use_snapshot_listing (bool): A flag indicating whether to use snapshot listing in the inventory report. bucket (str): The name of the GCS bucket containing the inventory reports. Returns: list: A list of dictionaries representing object details parsed from the inventory report content. recordSeparator delimiter,headerRequiredFN)inventory_report_linerr r rnr)r?rsplitr_parse_inventory_report_linerc)r rr rrr?record_separatorrnheader_requiredrcontentlineslineobjs r#rz/InventoryReport._parse_inventory_report_contents8.99 &??+)D)D)F)L)LY)W'"UJ C 0668>>yI,WXC s B c|d}|dur3|Dcgc]%}|jdj|s$|'c}gfSg}t}|D]}|jd}|j|s&|t|d}|j d}|dk(r|j |\|d|} |j d|dk(rdndz| z| dk(rdndz} |j| |t|fScc}w)aA Constructs the final snapshot based on the retrieved objects and prefix. Parameters: objects (list): A list of dictionaries representing object details from the inventory report. prefix (str): A prefix used to filter objects in the snapshot based on their names. use_snapshot_listing (bool): A flag indicating whether to use snapshot listing in the inventory report. Returns: tuple: A tuple containing two lists: the 'items' list representing object details for the snapshot, and the 'prefixes' list containing subdirectory prefixes. If 'use_snapshot_listing' is set to False, 'prefix' will also be empty, and 'items' will contains the object names in the snapshot. NFr</) r startswithsetrRfindrcrstripaddlist) rrrrzrNprefixesobj_nameobject_name_no_prefixfirst_delimiter_idxdir obj_prefixs r#rz)InventoryReport._construct_final_snapshots1& >F 5 (#*QCcggfo.H.H.PCQSUU U EuH )776?  **62)1V(?%'<&@&@&E#'",LL%,,@-@AMM#&#r\rs4!BYrC1  Z(; )>d8n$$ORs %C3C3c:|d}|d}|d}t|||S)a: Converts a dictionary representing a date object to a datetime object. Parameters: obj (dict): A dictionary representing a date object with keys "day", "month", and "year". Returns: datetime: A datetime object representing the converted date. daymonthyearr)rzrrrs r#rCz$InventoryReport._convert_obj_to_dateRs.%jG 6{eS))r'cLtj|jddS)a8 Converts an ISO-formatted date string to a datetime object. Parameters: date_string (str): An ISO-formatted date string with or without timezone information (Z). Returns: datetime: A datetime object representing the converted date and time. Zz+00:00)r fromisoformatreplace)strs r#rXz(InventoryReport._convert_str_to_datetimecs %%ckk#x&@AAr'N)__name__ __module__ __qualname____doc__r. classmethodr$rrrrrrrrtr staticmethodrCrXr'r#rrs8v;HLL\G0@N'`>)@ ,2(h8t1f@%D**  B Br'rceZdZdZdZy)rFa Represents the configuration for fetching inventory reports. Attributes: csv_options (dict): A dictionary containing options for parsing CSV format in the inventory reports. bucket (str): The name of the GCS bucket from which to fetch the inventory reports. destination_path (str): The path within the GCS bucket where the inventory reports are stored. metadata_fields (list): A list of strings representing metadata fields to be extracted from the inventory reports. obj_name_idx (int): The index of the "name" field in the 'metadata_fields' list, used to identify object names. cJ||_||_||_||_||_y)Nr>)selfr?rr@rArBs r#__init__zInventoryReportConfig.__init__s+' 0.(r'N)rrrrrrr'r#rFrFrs  )r'rFN)rrobjectrFrr'r#rs#k Bk B\)F)r'