Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38546

Implement new CalibrateImageTask

    XMLWordPrintable

Details

    • 24
    • AP F23-1 (June), AP F23-2 (July)
    • Alert Production
    • No

    Description

      Assist erykoff with refactoring the existing image characterization and calibration tasks into a single unified CalibrateImageTask, following the design laid out in DM-37584.

      Some relevant design notes from jbosch in this google doc:
      https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit

      Further design notes in the February 2023 update branch of DMTN-172

      Attachments

        Issue Links

          Activity

            No builds found.
            sullivan Ian Sullivan created issue -
            sullivan Ian Sullivan made changes -
            Field Original Value New Value
            Link This issue relates to DM-37584 [ DM-37584 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked higher
            Parejkoj John Parejko made changes -
            Description Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified task, following the design laid out in DM-37584. Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.
            Parejkoj John Parejko made changes -
            Watchers Eli Rykoff, Ian Sullivan [ Eli Rykoff, Ian Sullivan ] Eli Rykoff, Ian Sullivan, Jim Bosch [ Eli Rykoff, Ian Sullivan, Jim Bosch ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-35940 [ DM-35940 ]
            Parejkoj John Parejko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-38632 [ DM-38632 ]
            sullivan Ian Sullivan made changes -
            Story Points 8 12
            sullivan Ian Sullivan made changes -
            Sprint AP S23-5 (April) [ 1227 ] AP S23-6 (May) [ 1234 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked lower
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-32701 [ DM-32701 ]
            Parejkoj John Parejko added a comment - - edited

            Packages modified on this ticket:

            Parejkoj John Parejko added a comment - - edited Packages modified on this ticket: pipe_tasks (where the new task is): https://github.com/lsst/pipe_tasks/pull/802 meas_base (improvements to TestDataset): https://github.com/lsst/meas_base/pull/247 meas_algorithms (new refcat mocker and source selector bugfix): https://github.com/lsst/meas_algorithms/pull/338 meas_astrom (better error handling of small datasets and disabling source selectors): https://github.com/lsst/meas_astrom/pull/181 afw (tweaks to exposure apCorrMap interface): https://github.com/lsst/afw/pull/693
            sullivan Ian Sullivan made changes -
            Story Points 12 16
            sullivan Ian Sullivan made changes -
            Story Points 16 18
            sullivan Ian Sullivan made changes -
            Sprint AP S23-6 (May) [ 1234 ] AP F23-1 (June) [ 1239 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked lower
            Parejkoj John Parejko made changes -
            Summary Refactor characterizeImageTask and calibrateTask Implement new CalibrateImageTask
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-39796 [ DM-39796 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-5309 [ DM-5309 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-39842 [ DM-39842 ]

            This pipeline task config worked to let me run ap_verify DC2 data through the new task. I added it to an ApVerify.yaml file from a previous run.

            tasks:
              calibrateImage:
                class: lsst.pipe.tasks.calibrateImage.CalibrateImageTask
                config:
                  connections.astrometry_ref_cat: "cal_ref_cat_2_2"
                  connections.photometry_ref_cat: "cal_ref_cat_2_2"
                  astrometry_ref_loader.anyFilterMapsToThis: None
                  python: >
                    config.astrometry_ref_loader.filterMap = {band: 'lsst_%s_smeared' % (band) for band in 'ugrizy'};
                    config.photometry_ref_loader.filterMap = {band: 'lsst_%s_smeared' % (band) for band in 'ugrizy'};
                  psf_detection.background.approxOrderX: 1
                  psf_detection.tempLocalBackground.approxOrderX: 1
                  psf_detection.tempWideBackground.approxOrderX: 1
                  psf_repair.cosmicray.background.approxOrderX: 1
                  psf_detection.isotropicGrow: True
            

            Parejkoj John Parejko added a comment - This pipeline task config worked to let me run ap_verify DC2 data through the new task. I added it to an ApVerify.yaml file from a previous run. tasks: calibrateImage: class: lsst.pipe.tasks.calibrateImage.CalibrateImageTask config: connections.astrometry_ref_cat: "cal_ref_cat_2_2" connections.photometry_ref_cat: "cal_ref_cat_2_2" astrometry_ref_loader.anyFilterMapsToThis: None python: > config.astrometry_ref_loader.filterMap = {band: 'lsst_%s_smeared' % (band) for band in 'ugrizy'}; config.photometry_ref_loader.filterMap = {band: 'lsst_%s_smeared' % (band) for band in 'ugrizy'}; psf_detection.background.approxOrderX: 1 psf_detection.tempLocalBackground.approxOrderX: 1 psf_detection.tempWideBackground.approxOrderX: 1 psf_repair.cosmicray.background.approxOrderX: 1 psf_detection.isotropicGrow: True
            Parejkoj John Parejko added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/39082/pipeline

            jbosch: Thanks for volunteering to review while erykoff is on vacation. No rush on the review: it's ~1300 lines total. The changes outside of the new task and its tests should all be pretty straightforward: they were things that helped me write the new task and test it, or were small fixes I did along the way. The commits should all be fully atomic, so you can review them independently if that helps with context.

            I left quite a few TODOs in the new task and its tests: I hope that we can answer most of them during the review, and whatever is left we can file tickets for (some I'll have a ticket for once I get the config defaults RFC finalized).

            Parejkoj John Parejko added a comment - jbosch : Thanks for volunteering to review while erykoff is on vacation. No rush on the review: it's ~1300 lines total. The changes outside of the new task and its tests should all be pretty straightforward: they were things that helped me write the new task and test it, or were small fixes I did along the way. The commits should all be fully atomic, so you can review them independently if that helps with context. I left quite a few TODOs in the new task and its tests: I hope that we can answer most of them during the review, and whatever is left we can file tickets for (some I'll have a ticket for once I get the config defaults RFC finalized).
            Parejkoj John Parejko made changes -
            Reviewers Jim Bosch [ jbosch ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Description Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584. Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.


            https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit
            Parejkoj John Parejko made changes -
            Description Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.


            https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit
            Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.

            Some relevant design notes from [~jbosch] in this google docs:
            https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit
            Parejkoj John Parejko made changes -
            Description Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.

            Some relevant design notes from [~jbosch] in this google docs:
            https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit
            Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.

            Some relevant design notes from [~jbosch] in this google doc:
            https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit
            sullivan Ian Sullivan made changes -
            Story Points 18 24
            Parejkoj John Parejko made changes -
            Link This issue blocks DM-39858 [ DM-39858 ]
            sullivan Ian Sullivan made changes -
            Link This issue blocks DM-34975 [ DM-34975 ]
            yusra Yusra AlSayyad made changes -
            Epic Link PREOPS-1497 [ 2232737 ] PREOPS-1498 [ 2232747 ]
            Parejkoj John Parejko made changes -
            Remote Link This issue links to "Page (Confluence)" [ 37852 ]
            sullivan Ian Sullivan made changes -
            Sprint AP F23-1 (June) [ 1239 ] AP F23-1 (June), AP F23-2 (July) [ 1239, 1248 ]
            jbosch Jim Bosch added a comment -

            Most of my PR comments are about existing TODO comments, and I haven't tried at all to enforce that TODOs have ticket references.  Some changes are definitely needed before this becomes the default (in many cases due to TODOs already present in the code), but I have no problem with merging this pretty much as-is and addressing however much of that as you'd like later.

            Certainly a lot of those TODOs can only be addressed by actually experimenting with this task on real data (often significant amounts of real data), and that'll be much easier after this has been merged to main.

             

            jbosch Jim Bosch added a comment - Most of my PR comments are about existing TODO comments, and I haven't tried at all to enforce that TODOs have ticket references.  Some changes are definitely needed before this becomes the default (in many cases due to TODOs already present in the code), but I have no problem with merging this pretty much as-is and addressing however much of that as you'd like later. Certainly a lot of those TODOs can only be addressed by actually experimenting with this task on real data (often significant amounts of real data), and that'll be much easier after this has been merged to main .  
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Parejkoj John Parejko made changes -
            Description Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.

            Some relevant design notes from [~jbosch] in this google doc:
            https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit
            Assist [~erykoff] with refactoring the existing image characterization and calibration tasks into a single unified {{CalibrateImageTask}}, following the design laid out in DM-37584.

            Some relevant design notes from [~jbosch] in this google doc:
            https://docs.google.com/document/d/1V0DYlKbj7BVIguDI_Jlmh6zI5Q1DeW8nZXPKLO7SY8w/edit

            Further design notes in the [February 2023 update branch of DMTN-172|https://dmtn-172.lsst.io/v/DM-37584/index.html]
            Parejkoj John Parejko made changes -
            Link This issue blocks DM-40061 [ DM-40061 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to RFC-857 [ RFC-857 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-40064 [ DM-40064 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-39203 [ DM-39203 ]
            mfisherlevine Merlin Fisher-Levine made changes -
            Link This issue blocks SITCOM-642 [ SITCOM-642 ]
            skleinman Scot Kleinman made changes -
            Remote Link This issue links to "Page (Confluence)" [ 38545 ]
            skleinman Scot Kleinman made changes -
            Remote Link This issue links to "Page (Confluence)" [ 38548 ]
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to RFC-918 [ RFC-918 ]
            Parejkoj John Parejko made changes -
            Labels CalibrateImageTask
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-41600 [ DM-41600 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-41640 [ DM-41640 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-41899 [ DM-41899 ]

            People

              Parejkoj John Parejko
              sullivan Ian Sullivan
              Jim Bosch
              Eli Rykoff, Ian Sullivan, Jim Bosch, John Parejko, Merlin Fisher-Levine
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.