Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18000

Implement PipelineTask and yaml file to convert DiaSources for SDM system inside ap_association

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_association
    • Labels:
      None
    • Story Points:
      12
    • Epic Link:
    • Sprint:
      AP F19-5 (October), AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February), AP S20-4 (March), AP F20-2 (July), AP F20-3 (August), AP F20-4 (September), AP F20-5 (October), AP F20-6 (November), AP S21-3 (February)
    • Team:
      Alert Production

      Description

      That is, replacing the current DPDDifier system.

      This will have to include some sort of interface with the Apdb. At the point you're ready to start working on this, we should either have talked to Andy about whether he will support that, or we should make another ticket to capture you putting in some appropriate interface.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21267 Enforce consistency between the expected filter columns in ap_association and dax_ppdb.

          • To Do
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23030 Modify Photometry SDM Functor to use stored calibration value.

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23031 Create dipole separation functor

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23103 Update functor unittests to no longer rely on test_multilevel_parq.csv.gz

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16234 Move qa_explorer/postprocess.py and writeObjectTable to pipe_tasks

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17999 Demonstrate proof of concept use of SDM system

          • Done
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23032 CreateSDM functor for bitpacking mutiple flag columns

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-28940 Create functor for pix^2->arcsec^2 unit conversion

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-28942 Create dipMeanFlux/dipFluxDiff functor.

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Before starting work to completely switch to using the SDM-ifier for diaPipe, we new to create/modify a few functors listed below. 

            List of functors that are needed for DiaSources to replicate the current apPipe processing:

            • Nanojansky (Need to modify to use stored calibration column)
            • NanojanskyErr (Need to modify to use stored calibration column)
            • DipoleSeparation
            • FlagBitPacker

            Others are values copied from the exposure which can just be appended to the pandas DataFrame within AssociationTask or diaPipe. This sub tickets will block this ticket until completed.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Before starting work to completely switch to using the SDM-ifier for diaPipe, we new to create/modify a few functors listed below.  List of functors that are needed for DiaSources to replicate the current apPipe processing: Nanojansky (Need to modify to use stored calibration column) NanojanskyErr (Need to modify to use stored calibration column) DipoleSeparation FlagBitPacker Others are values copied from the exposure which can just be appended to the pandas DataFrame within AssociationTask or diaPipe. This sub tickets will block this ticket until completed.
            Hide
            Permalink
            swinbank John Swinbank added a comment -

            Definition of done for this ticket (per discussion of 2020-06-30): there should be a YAML file (ap.yaml, or whatever) which defines interfaces between the AP pipeline and the database. At this point, MapApData is gone.

            Show
            swinbank John Swinbank added a comment - Definition of done for this ticket (per discussion of 2020-06-30): there should be a YAML file (ap.yaml, or whatever) which defines interfaces between the AP pipeline and the database. At this point, MapApData is gone.
            Hide
            Permalink
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited

            Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33648/pipeline

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33648/pipeline
            Hide
            Permalink
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited

            Instead of what John Swinbank has above, I'm going to modify the goal of this ticket create the pipeline task and the yaml file. A few functors need to be created to replicate the mapApData functionality. The functors needed are listed below.

            • SNR
            • dipMeanFlux
            • dipFluxDiff
            • bitpacking
            • arcsec^2 functor similar to the currently pix->arcsec functor
            • Possibly others I've forgotten.
            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited Instead of what John Swinbank has above, I'm going to modify the goal of this ticket create the pipeline task and the yaml file. A few functors need to be created to replicate the mapApData functionality. The functors needed are listed below. SNR dipMeanFlux dipFluxDiff bitpacking arcsec^2 functor similar to the currently pix->arcsec functor Possibly others I've forgotten.
            Hide
            Permalink
            tmorton Tim Morton [X] (Inactive) added a comment -

            Looks good to me; just left some minor comments on the PR.

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - Looks good to me; just left some minor comments on the PR.
            Hide
            Permalink
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            DM-24638 added Gen3 functionality to the TransformCatalog tasks so I had to do some editing of the inheritance for the the TransformDiaSourceCatalogTask in the ticket. Was a quick fix and Jenkins is now passing.

            Let me know Tim Morton [X] if you'd like to take another quick look and if not I'll merge after the full Jenkins run completes.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - DM-24638 added Gen3 functionality to the TransformCatalog tasks so I had to do some editing of the inheritance for the the TransformDiaSourceCatalogTask in the ticket. Was a quick fix and Jenkins is now passing. Let me know Tim Morton [X] if you'd like to take another quick look and if not I'll merge after the full Jenkins run completes.
            Hide
            Permalink
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Found out that Index and IDColumn were not properly copying diaSourceId into the parquet table. Have switched the functor to Column.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Found out that Index and IDColumn were not properly copying diaSourceId into the parquet table. Have switched the functor to Column.
            Hide
            Permalink
            tmorton Tim Morton [X] (Inactive) added a comment -

            OK, yeah- I figured the reason it wasn't inheriting before was because DM-24638 wasn't merged yet, but I'm glad it's all coherent now.  

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - OK, yeah- I figured the reason it wasn't inheriting before was because DM-24638 wasn't merged yet, but I'm glad it's all coherent now.  

              People

              Assignee:
              cmorrison Chris Morrison [X] (Inactive)
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Tim Morton [X] (Inactive)
              Watchers:
              Chris Morrison [X] (Inactive), Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.