Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18735

Convert dax_ppdb to use Pandas data frames (rather than afw::table) as an interface

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

          Activity

          swinbank John Swinbank created issue -
          swinbank John Swinbank made changes -
          Field Original Value New Value
          Epic Link DM-16720 [ 235353 ]
          swinbank John Swinbank made changes -
          Rank Ranked higher
          swinbank John Swinbank made changes -
          Link This issue blocks DM-18736 [ DM-18736 ]
          Hide
          salnikov Andy Salnikov added a comment -

          Are we positive that we'll never need to support afw::table in the future? If there is a chance that we'll need both pandas and afw::table (or pandas and something else) then it may be a good idea to generalize ppdb to support more than one in-memory data format.

          Show
          salnikov Andy Salnikov added a comment - Are we positive that we'll never need to support afw::table in the future? If there is a chance that we'll need both pandas and afw::table (or pandas and something else) then it may be a good idea to generalize ppdb to support more than one in-memory data format.
          ebellm Eric Bellm made changes -
          Remote Link This issue links to "Page (Confluence)" [ 20170 ]
          Hide
          ebellm Eric Bellm added a comment -

          John Swinbank and I suggested that Chris Morrison evaluate how much work he expected it would take to support both interfaces.

          Show
          ebellm Eric Bellm added a comment - John Swinbank and I suggested that Chris Morrison evaluate how much work he expected it would take to support both interfaces.
          cmorrison Chris Morrison made changes -
          Link This issue is triggering DM-19388 [ DM-19388 ]
          cmorrison Chris Morrison made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          cmorrison Chris Morrison made changes -
          Reviewers Andy Salnikov [ salnikov ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          cmorrison Chris Morrison added a comment -

          Ticket didn't end up completely converting but created a separate code path. Code supports both datatypes but uses different SQL writing/reading methods to retrieve the data.

          Show
          cmorrison Chris Morrison added a comment - Ticket didn't end up completely converting but created a separate code path. Code supports both datatypes but uses different SQL writing/reading methods to retrieve the data.
          Hide
          cmorrison Chris Morrison added a comment - - edited

          Jenkins passes: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29698/pipeline/46/

          Thinking about it, I will also fire off an ap_verify run to make sure that the Ppdb's current usage is not broken by these changes.

          Update: ap_verify/pipe seem to be running without error across 2 ccds and several visits.

          Show
          cmorrison Chris Morrison added a comment - - edited Jenkins passes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29698/pipeline/46/ Thinking about it, I will also fire off an ap_verify run to make sure that the Ppdb's current usage is not broken by these changes. Update: ap_verify/pipe seem to be running without error across 2 ccds and several visits.
          Hide
          salnikov Andy Salnikov added a comment -

          Looks OK, see comments on PR.

          Show
          salnikov Andy Salnikov added a comment - Looks OK, see comments on PR.
          salnikov Andy Salnikov made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          Show
          cmorrison Chris Morrison added a comment - Jenkins after review:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29699/pipeline
          cmorrison Chris Morrison made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Chris Morrison, Eric Bellm, John Swinbank
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel