Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18736

Convert ap_association to use Pandas data frames (rather than afw::table) as an interface

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

          Activity

          Show
          cmorrison Chris Morrison added a comment - Knew I forgot something. Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29957/pipeline
          Hide
          ebellm Eric Bellm added a comment -

          Nice work!

          Show
          ebellm Eric Bellm added a comment - Nice work!
          Hide
          cmorrison Chris Morrison added a comment -

          After running ap_verify_hits2015, all visits, ccds 24 and 25, I have confirmed that the output values are identical between the pandas and previous afw version.

          Show
          cmorrison Chris Morrison added a comment - After running ap_verify_hits2015, all visits, ccds 24 and 25, I have confirmed that the output values are identical between the pandas and previous afw version.
          Hide
          cmorrison Chris Morrison added a comment -

          Looked into the data output from the current master to that of this ticket. Found a bug where if a column was not specified when inserting a new DiaObject dataframe in the Ppdb any columns not declared would default to a value of 0 instead of NaN/Null. Fixed in a new commit.

          Show
          cmorrison Chris Morrison added a comment - Looked into the data output from the current master to that of this ticket. Found a bug where if a column was not specified when inserting a new DiaObject dataframe in the Ppdb any columns not declared would default to a value of 0 instead of NaN/Null. Fixed in a new commit.
          Hide
          cmorrison Chris Morrison added a comment -

          I made a small change to the AssociationTask usage of pandas to make use of the DataFrame object's built in indexing to allow for easy and fast look up of diaObjects and diaSources by their diaObjectId.

          Show
          cmorrison Chris Morrison added a comment - I made a small change to the AssociationTask usage of pandas to make use of the DataFrame object's built in indexing to allow for easy and fast look up of diaObjects and diaSources by their diaObjectId.
          Hide
          cmorrison Chris Morrison added a comment -

          Made an initial pass though a Pandas conversion of AssociationTask. Ended up simplifying the selection of DiaObjects in update to an inefficient boolean mask on the data id as the indexing refused to work.

          You can find outputs for the DiaObjects and DiaSources in /project/morriscb/src/ap_verify_hits2015/test_pandas/association.db

          Show
          cmorrison Chris Morrison added a comment - Made an initial pass though a Pandas conversion of AssociationTask. Ended up simplifying the selection of DiaObjects in update to an inefficient boolean mask on the data id as the indexing refused to work. You can find outputs for the DiaObjects and DiaSources in /project/morriscb/src/ap_verify_hits2015/test_pandas/association.db

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Eric Bellm
              Watchers:
              Chris Morrison, Eric Bellm, John Swinbank
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel