Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18736

Convert ap_association to use Pandas data frames (rather than afw::table) as an interface

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

          Activity

          swinbank John Swinbank created issue -
          swinbank John Swinbank made changes -
          Field Original Value New Value
          Epic Link DM-16720 [ 235353 ]
          swinbank John Swinbank made changes -
          Link This issue is blocked by DM-18735 [ DM-18735 ]
          swinbank John Swinbank made changes -
          Rank Ranked higher
          ebellm Eric Bellm made changes -
          Remote Link This issue links to "Page (Confluence)" [ 20167 ]
          swinbank John Swinbank made changes -
          Sprint AP S19-5 [ 833 ] AP S19-6 [ 834 ]
          cmorrison Chris Morrison made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          cmorrison Chris Morrison added a comment -

          Made an initial pass though a Pandas conversion of AssociationTask. Ended up simplifying the selection of DiaObjects in update to an inefficient boolean mask on the data id as the indexing refused to work.

          You can find outputs for the DiaObjects and DiaSources in /project/morriscb/src/ap_verify_hits2015/test_pandas/association.db

          Show
          cmorrison Chris Morrison added a comment - Made an initial pass though a Pandas conversion of AssociationTask. Ended up simplifying the selection of DiaObjects in update to an inefficient boolean mask on the data id as the indexing refused to work. You can find outputs for the DiaObjects and DiaSources in /project/morriscb/src/ap_verify_hits2015/test_pandas/association.db
          cmorrison Chris Morrison made changes -
          Reviewers Eric Bellm [ ebellm ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          cmorrison Chris Morrison added a comment -

          I made a small change to the AssociationTask usage of pandas to make use of the DataFrame object's built in indexing to allow for easy and fast look up of diaObjects and diaSources by their diaObjectId.

          Show
          cmorrison Chris Morrison added a comment - I made a small change to the AssociationTask usage of pandas to make use of the DataFrame object's built in indexing to allow for easy and fast look up of diaObjects and diaSources by their diaObjectId.
          Hide
          cmorrison Chris Morrison added a comment -

          Looked into the data output from the current master to that of this ticket. Found a bug where if a column was not specified when inserting a new DiaObject dataframe in the Ppdb any columns not declared would default to a value of 0 instead of NaN/Null. Fixed in a new commit.

          Show
          cmorrison Chris Morrison added a comment - Looked into the data output from the current master to that of this ticket. Found a bug where if a column was not specified when inserting a new DiaObject dataframe in the Ppdb any columns not declared would default to a value of 0 instead of NaN/Null. Fixed in a new commit.
          Hide
          cmorrison Chris Morrison added a comment -

          After running ap_verify_hits2015, all visits, ccds 24 and 25, I have confirmed that the output values are identical between the pandas and previous afw version.

          Show
          cmorrison Chris Morrison added a comment - After running ap_verify_hits2015, all visits, ccds 24 and 25, I have confirmed that the output values are identical between the pandas and previous afw version.
          Hide
          ebellm Eric Bellm added a comment -

          Nice work!

          Show
          ebellm Eric Bellm added a comment - Nice work!
          ebellm Eric Bellm made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          Show
          cmorrison Chris Morrison added a comment - Knew I forgot something. Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29957/pipeline
          cmorrison Chris Morrison made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          cmorrison Chris Morrison made changes -
          Link This issue relates to DM-20189 [ DM-20189 ]

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Eric Bellm
              Watchers:
              Chris Morrison, Eric Bellm, John Swinbank
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel