Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22266

Adapt pipe_analysis to run on post-processing parquet tables in Gen2

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      24
    • Epic Link:
    • Sprint:
      DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug), DRP F20-5 (Oct)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As a step towards converting the pipe_analysis scripts to Gen3, adapt the scripts in pipe_analysis to work with the parquet tables produced in the pipe_tasks postprocessing.py step. The reading of afwTables will be deprecated eventually, but we still leave the ability to do so for compatibility with older datasets for which the postprocessing.py step was not run (so no parquet catalog tables exist). As such, the first step here is to convert all functionality to be run with pandas DataFrames (so, if reading in afwTable catalogs, convert them immediately to pandas DataFrames). Then add a function to read in the *_obj parquet tables and add a config parameter to control which tables are read in (default it to read the parquet catalogs).

        Attachments

          Issue Links

            Activity

            No work has yet been logged on this issue.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Tim Morton [X] (Inactive)
              Watchers:
              Jim Bosch, John Parejko, Lauren MacArthur, Nate Lust, Sophie Reed, Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.