Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15751

Configure ability to build coadds with either Jointcal or meas_mosaic

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      John, I added you as a watcher in case I was duplicating effort.

      Plan is to make lightweight task that can "applyJointcalResults" to calexps (based on https://github.com/lsst/jointcal/blob/master/python/lsst/jointcal/jointcalCoadd.py)  and make choice of meas_mosaic or jointcal retargetable.  Need to add the ability to do apply the photometric calibration in addition to the WCS (like https://github.com/lsst/meas_mosaic/blob/master/python/lsst/meas/mosaic/updateExposure.py#L47) 

       

      Outstanding question: do we want to be able to mix and match? Photo from one and WCS from another?

       

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16397 Do release 17.0 of science pipelines

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15254 Option to normalize warps by Jointcal's PhotoCalib

          • Done
          is triggering

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-545 Deprecate ScaleZeroPointTask

          • Adopted
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16082 nopytest_test_coadds.py throws warnings, and should be fixed

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16596 Quality assessment of coadds created using jointcal calibrations

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18492 Update forcedPhotCcd to use PhotoCalib.calibrateImage

          • Invalid

          Bug - A problem which impairs or prevents the functions of the product. DM-10595 Add test for jointcalCoadd.py

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            23 older comments
            Hide
            Permalink
            Parejkoj John Parejko added a comment - - edited

            New Jenkins run, now that I think I've fixed ci_hsc: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29050/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited New Jenkins run, now that I think I've fixed ci_hsc: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29050/pipeline
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Yusra AlSayyad: Jenkins passed through ci_hsc, so it's time for review!

            Show
            Parejkoj John Parejko added a comment - Yusra AlSayyad : Jenkins passed through ci_hsc, so it's time for review!
            Hide
            Permalink
            yusra Yusra AlSayyad added a comment -

            Couple notes:
            (1) CoaddDriver.py takes longer (working on quantifying how much to taking out effects of GPFS performance.)
            (2) QA: https://lsst-web.ncsa.illinois.edu/~yusra/RC_QA/w_2018_44_jointcal vs. https://lsst-web.ncsa.illinois.edu/~yusra/RC_QA/w_2018_44

            • Overall, QA plots look better for deep areas and fainter stars.
            • Lauren MacArthur noticed that the brightest stars with calib_psf_used have cmodel fluxes > psf fluxes (=> are classifed as galaxies) in jointcal and not meas mosaic. Spatially this appears in the areas with 1-2 visits and ghosting around very bright stars. I'm not sure how jointcal and meas_mosaic would differ if they use the same input catalogs. By eye the coadds look identical and differences in clipping very minimal in these problematic areas.
            • Recommendation at this point would be to decouple this ticket from the decision of using jointcal for obs_subaru add useMeasMosaic=True to obs_subaru and a ticket to remove it when we're ready. We can take the decision making process to #subaru-hsc.

            Finally, obs_lsstCam (which copied all the configs over from obs_subaru initially)

            Show
            yusra Yusra AlSayyad added a comment - Couple notes: (1) CoaddDriver.py takes longer (working on quantifying how much to taking out effects of GPFS performance.) (2) QA: https://lsst-web.ncsa.illinois.edu/~yusra/RC_QA/w_2018_44_jointcal vs. https://lsst-web.ncsa.illinois.edu/~yusra/RC_QA/w_2018_44 Overall, QA plots look better for deep areas and fainter stars. Lauren MacArthur noticed that the brightest stars with calib_psf_used have cmodel fluxes > psf fluxes (=> are classifed as galaxies) in jointcal and not meas mosaic. Spatially this appears in the areas with 1-2 visits and ghosting around very bright stars. I'm not sure how jointcal and meas_mosaic would differ if they use the same input catalogs. By eye the coadds look identical and differences in clipping very minimal in these problematic areas. Recommendation at this point would be to decouple this ticket from the decision of using jointcal for obs_subaru add useMeasMosaic=True to obs_subaru and a ticket to remove it when we're ready. We can take the decision making process to #subaru-hsc. Finally, obs_lsstCam (which copied all the configs over from obs_subaru initially)
            Hide
            Permalink
            Parejkoj John Parejko added a comment - - edited

            Thanks for the review.

            Changes in what happens around bright vs. faint stars might be due to differences in source selection: I don't know how meas_mosaic does it's source selection, but jointcal uses the AstrometrySourceSelector which selects every "good" "non-blended" source with s/n>10, whether star or galaxy. You can play with configuring that differently, if you think incorrect sources are being included near bright stars.

            New post-review Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29063/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited Thanks for the review. Changes in what happens around bright vs. faint stars might be due to differences in source selection: I don't know how meas_mosaic does it's source selection, but jointcal uses the AstrometrySourceSelector which selects every "good" "non-blended" source with s/n>10 , whether star or galaxy. You can play with configuring that differently, if you think incorrect sources are being included near bright stars. New post-review Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29063/pipeline
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Thank you for the review, and for looking at my subsequent updates.

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Thank you for the review, and for looking at my subsequent updates. Merged and done.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              yusra Yusra AlSayyad
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.