Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18735

Convert dax_ppdb to use Pandas data frames (rather than afw::table) as an interface

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

          Activity

          Show
          cmorrison Chris Morrison added a comment - Jenkins after review:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29699/pipeline
          Hide
          salnikov Andy Salnikov added a comment -

          Looks OK, see comments on PR.

          Show
          salnikov Andy Salnikov added a comment - Looks OK, see comments on PR.
          Hide
          cmorrison Chris Morrison added a comment - - edited

          Jenkins passes: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29698/pipeline/46/

          Thinking about it, I will also fire off an ap_verify run to make sure that the Ppdb's current usage is not broken by these changes.

          Update: ap_verify/pipe seem to be running without error across 2 ccds and several visits.

          Show
          cmorrison Chris Morrison added a comment - - edited Jenkins passes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29698/pipeline/46/ Thinking about it, I will also fire off an ap_verify run to make sure that the Ppdb's current usage is not broken by these changes. Update: ap_verify/pipe seem to be running without error across 2 ccds and several visits.
          Hide
          cmorrison Chris Morrison added a comment -

          Ticket didn't end up completely converting but created a separate code path. Code supports both datatypes but uses different SQL writing/reading methods to retrieve the data.

          Show
          cmorrison Chris Morrison added a comment - Ticket didn't end up completely converting but created a separate code path. Code supports both datatypes but uses different SQL writing/reading methods to retrieve the data.
          Hide
          ebellm Eric Bellm added a comment -

          John Swinbank and I suggested that Chris Morrison evaluate how much work he expected it would take to support both interfaces.

          Show
          ebellm Eric Bellm added a comment - John Swinbank and I suggested that Chris Morrison evaluate how much work he expected it would take to support both interfaces.
          Hide
          salnikov Andy Salnikov added a comment -

          Are we positive that we'll never need to support afw::table in the future? If there is a chance that we'll need both pandas and afw::table (or pandas and something else) then it may be a good idea to generalize ppdb to support more than one in-memory data format.

          Show
          salnikov Andy Salnikov added a comment - Are we positive that we'll never need to support afw::table in the future? If there is a chance that we'll need both pandas and afw::table (or pandas and something else) then it may be a good idea to generalize ppdb to support more than one in-memory data format.

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Chris Morrison, Eric Bellm, John Swinbank
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel