Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17029

Update LoadReferenceObjectsTask to output fluxes in nanojansky

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP S19-1, AP S19-2, AP S19-3, AP S19-4, AP S19-5
    • Team:
      Alert Production

      Description

      PhotoCalib is now defined in terms of nanojansky, but our reference catalogs are saved in Jy, and remain in Jy when the user loads them. We need to get our refcat loaders to produce nJy to keep our calibrations self-consistent.

      It may be enough to just multiply fluxes and flux errors by 1e9 after the data is read from disk, but Simon Krughoff may have other suggestions.

      We will have to add a few work-arounds for this to PhotoCal and other tasks that use the Calib object, as it will remained defined in Jy. If only we could do DM-10153 simultaneously (but I think that would be maddening).

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-10156 Replace all uses of Calib with PhotoCalib

          • Done
          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-549 Use nanojansky for calibrated fluxes in DM code and intermediate data products

          • Implemented

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-575 Convert old refcats to nJy for RFC-549

          • Implemented
          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. DM-10153 Replace Calib with finished PhotoCalib

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16650 Use nJy in PhotoCalib as the unit for calibrated fluxes

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-18303 Add explicit support for dataset name to refcat API

          • Invalid

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-851 Remove support for version 0 reference catalogs

          • Adopted

          Bug - A problem which impairs or prevents the functions of the product. DM-18302 IngestReferenceRunner should return exitStatus

          • Won't Fix

            Activity

            Ascending order - Click to sort in descending order
            8 older comments
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            I've incorporated your comments and added extra version checking and better version output for the convert refcat script.

            Post cleanup jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29520/pipeline

            I think I'm going to hold off on merging this until DM-10156 is ready to merge: I think those two should go in as part of the same weekly.

            Show
            Parejkoj John Parejko added a comment - I've incorporated your comments and added extra version checking and better version output for the convert refcat script. Post cleanup jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29520/pipeline I think I'm going to hold off on merging this until DM-10156 is ready to merge: I think those two should go in as part of the same weekly.
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            Sounds good.

            Show
            krughoff Simon Krughoff added a comment - Sounds good.
            Hide
            Permalink
            jbosch Jim Bosch added a comment - - edited

            I've put a commit that adds the version to the reference catalog headers as well on u/jbosch/DM-17029 of meas_algorithms (you can also just cherry-pick 2157407684968b41fdf9615260b980b5754c2f9d).  All meas_algorithms tests pass (the version1 test data repo was updated on that commit), but I have not tested other packages; in general you'll need to re-run the upgrade script or re-ingest version 1 catalogs to get those to work.

            If you opt not to include that on this ticket, please let me know and I'll put it on a new one.  I would like to get it in before we start upgrading on-disk catalogs in earnest, though.

            Show
            jbosch Jim Bosch added a comment - - edited I've put a commit that adds the version to the reference catalog headers as well on u/jbosch/ DM-17029 of meas_algorithms (you can also just cherry-pick 2157407684968b41fdf9615260b980b5754c2f9d).  All meas_algorithms tests pass (the version1 test data repo was updated on that commit), but I have not tested other packages; in general you'll need to re-run the upgrade script or re-ingest version 1 catalogs to get those to work. If you opt not to include that on this ticket, please let me know and I'll put it on a new one.  I would like to get it in before we start upgrading on-disk catalogs in earnest, though.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Post Jim Bosch gen3 additions and rebasing Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29603/pipeline

            Show
            Parejkoj John Parejko added a comment - Post Jim Bosch gen3 additions and rebasing Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29603/pipeline
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Thank you for the review Simon Krughoff, and for your gen3 butler updates, Jim Bosch.

            Merged and done (and now trying to merge DM-10156 on the same day!).

            Show
            Parejkoj John Parejko added a comment - Thank you for the review Simon Krughoff , and for your gen3 butler updates, Jim Bosch . Merged and done (and now trying to merge DM-10156 on the same day!).

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Simon Krughoff
              Watchers:
              Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Paul Price, Russell Owen, Simon Krughoff, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.