Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22266

Adapt pipe_analysis to run on post-processing parquet tables in Gen2

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      24
    • Epic Link:
    • Sprint:
      DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug), DRP F20-5 (Oct)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As a step towards converting the pipe_analysis scripts to Gen3, adapt the scripts in pipe_analysis to work with the parquet tables produced in the pipe_tasks postprocessing.py step. The reading of afwTables will be deprecated eventually, but we still leave the ability to do so for compatibility with older datasets for which the postprocessing.py step was not run (so no parquet catalog tables exist). As such, the first step here is to convert all functionality to be run with pandas DataFrames (so, if reading in afwTable catalogs, convert them immediately to pandas DataFrames). Then add a function to read in the *_obj parquet tables and add a config parameter to control which tables are read in (default it to read the parquet catalogs).

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27931 Port DM-27800 code to use pandas

          • Won't Fix
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27027 Enable persistence of "source" parquet tables in obs_subaru

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-28168 Look into pipe_analysis script failures in RC2 w_2020_50 run

          • Done

            Activity

            Ascending order - Click to sort in descending order
            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Epic Link DM-21254 [ 414685 ]
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-21254 [ 414685 ] DM-22586 [ 427653 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov) [ 972 ] DRP F19-6 (Nov), DRP S20-2 (Jan) [ 972, 998 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov), DRP S20-2 (Jan) [ 972, 998 ] DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb) [ 972, 998, 1000 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-22586 [ 427653 ] DM-23737 [ 431393 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb) [ 972, 998, 1000 ] DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr) [ 972, 998, 1000, 1014 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr) [ 972, 998, 1000, 1014 ] DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May) [ 972, 998, 1000, 1014, 1021 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-23737 [ 431393 ] DM-25268 [ 435627 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-25268 [ 435627 ] DM-25272 [ 435632 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May) [ 972, 998, 1000, 1014, 1021 ] DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June) [ 972, 998, 1000, 1014, 1021, 1030 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June) [ 972, 998, 1000, 1014, 1021, 1030 ] DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July) [ 972, 998, 1000, 1014, 1021, 1030, 1036 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July) [ 972, 998, 1000, 1014, 1021, 1030, 1036 ] DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug) [ 972, 998, 1000, 1014, 1021, 1030, 1036, 1042 ]
            lauren Lauren MacArthur made changes -
            Summary Adapt coaddAnalysis to run on deepCoadd_obj parquet table in Gen2 Adapt pipe_analysis to run on post-processing parquet tables in Gen2
            lauren Lauren MacArthur made changes -
            Description As a step towards converting the {{pipe_analysis}} scripts to Gen3, adapt the coaddAnalysis.py script to be run reading the {{deepCoadd_obj}} parquet table that gets produced by {{pipe_tasks}} {{WriteObjectTableTask}}. This parquet table is the filter and datasets {{forced_src}}, {{meas}}, {{ref}} -merged catalogs at the patch level. As a step towards converting the {{pipe_analysis}} scripts to Gen3, adapt the scripts in {{pipe_analysis}} to work with the parquet tables produced in the {{pipe_tasks}} {{postprocessing.py}} step. The reading of {{afwTables}} will be deprecated eventually, but we still leave the ability to do so for compatibility with older datasets for which the {{postprocessing.py}} step was not run (so no parquet catalog tables exist). As such, the first step here is to convert all functionality to be run with pandas DataFrames (so, if reading in {{afwTable}} catalogs, convert them immediately to pandas DataFrames). Then add a function to read in the {{*_obj}} parquet tables and add a config parameter to control which tables are read in (default it to read the parquet catalogs).
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-27027 [ DM-27027 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-25272 [ 435632 ] DM-26790 [ 439740 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug) [ 972, 998, 1000, 1014, 1021, 1030, 1036, 1042 ] DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug), DRP F20-5 (Oct) [ 972, 998, 1000, 1014, 1021, 1030, 1036, 1042, 1054 ]
            lauren Lauren MacArthur made changes -
            Reviewers Tim Morton [ tmorton ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            fred3m Fred Moolekamp made changes -
            Link This issue blocks DM-27931 [ DM-27931 ]
            yusra Yusra AlSayyad made changes -
            Story Points 24
            Urgent? off
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-28168 [ DM-28168 ]

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Tim Morton [X] (Inactive)
              Watchers:
              Jim Bosch, John Parejko, Lauren MacArthur, Nate Lust, Sophie Reed, Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.