Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18736

Convert ap_association to use Pandas data frames (rather than afw::table) as an interface

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

        is blocked by

        Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18735 Convert dax_ppdb to use Pandas data frames (rather than afw::table) as an interface

        • Done
        relates to

        Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-20189 Investigate change in processing time when using pandas in ap_association

        • Done
        mentioned in

        Page Loading...

          Activity

          Ascending order - Click to sort in descending order
          swinbank John Swinbank created issue -
          swinbank John Swinbank made changes -
          Field Original Value New Value
          Epic Link DM-16720 [ 235353 ]
          swinbank John Swinbank made changes -
          Link This issue is blocked by DM-18735 [ DM-18735 ]
          swinbank John Swinbank made changes -
          Rank Ranked higher
          ebellm Eric Bellm made changes -
          Remote Link This issue links to "Page (Confluence)" [ 20167 ]
          swinbank John Swinbank made changes -
          Sprint AP S19-5 [ 833 ] AP S19-6 [ 834 ]
          cmorrison Chris Morrison made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          Permalink
          cmorrison Chris Morrison added a comment -

          Made an initial pass though a Pandas conversion of AssociationTask. Ended up simplifying the selection of DiaObjects in update to an inefficient boolean mask on the data id as the indexing refused to work.

          You can find outputs for the DiaObjects and DiaSources in /project/morriscb/src/ap_verify_hits2015/test_pandas/association.db

          Show
          cmorrison Chris Morrison added a comment - Made an initial pass though a Pandas conversion of AssociationTask. Ended up simplifying the selection of DiaObjects in update to an inefficient boolean mask on the data id as the indexing refused to work. You can find outputs for the DiaObjects and DiaSources in /project/morriscb/src/ap_verify_hits2015/test_pandas/association.db
          cmorrison Chris Morrison made changes -
          Reviewers Eric Bellm [ ebellm ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          Permalink
          cmorrison Chris Morrison added a comment -

          I made a small change to the AssociationTask usage of pandas to make use of the DataFrame object's built in indexing to allow for easy and fast look up of diaObjects and diaSources by their diaObjectId.

          Show
          cmorrison Chris Morrison added a comment - I made a small change to the AssociationTask usage of pandas to make use of the DataFrame object's built in indexing to allow for easy and fast look up of diaObjects and diaSources by their diaObjectId.
          Hide
          Permalink
          cmorrison Chris Morrison added a comment -

          Looked into the data output from the current master to that of this ticket. Found a bug where if a column was not specified when inserting a new DiaObject dataframe in the Ppdb any columns not declared would default to a value of 0 instead of NaN/Null. Fixed in a new commit.

          Show
          cmorrison Chris Morrison added a comment - Looked into the data output from the current master to that of this ticket. Found a bug where if a column was not specified when inserting a new DiaObject dataframe in the Ppdb any columns not declared would default to a value of 0 instead of NaN/Null. Fixed in a new commit.
          Hide
          Permalink
          cmorrison Chris Morrison added a comment -

          After running ap_verify_hits2015, all visits, ccds 24 and 25, I have confirmed that the output values are identical between the pandas and previous afw version.

          Show
          cmorrison Chris Morrison added a comment - After running ap_verify_hits2015, all visits, ccds 24 and 25, I have confirmed that the output values are identical between the pandas and previous afw version.
          Hide
          Permalink
          ebellm Eric Bellm added a comment -

          Nice work!

          Show
          ebellm Eric Bellm added a comment - Nice work!
          ebellm Eric Bellm made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          Hide
          Permalink
          cmorrison Chris Morrison added a comment -

          Knew I forgot something. Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29957/pipeline

          Show
          cmorrison Chris Morrison added a comment - Knew I forgot something. Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29957/pipeline
          cmorrison Chris Morrison made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          cmorrison Chris Morrison made changes -
          Link This issue relates to DM-20189 [ DM-20189 ]

            People

            Assignee:
            cmorrison Chris Morrison
            Reporter:
            swinbank John Swinbank
            Reviewers:
            Eric Bellm
            Watchers:
            Chris Morrison, Eric Bellm, John Swinbank
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: