Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16596

Quality assessment of coadds created using jointcal calibrations

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: jointcal, meas_mosaic
    • Labels:
      None
    • Story Points:
      16
    • Epic Link:
    • Team:
      Data Release Production

      Description

      This issue is to account for and detail investigations into the quality of the coadds that were built using the calibrations from jointcal. They will be assessed independently and in comparison to the associated coadds that were built using meas_mosaic calibrations.

        Attachments

          Issue Links

            Activity

            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Epic Link DM-14402 [ 79665 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-15751 [ DM-15751 ]
            Hide
            Parejkoj John Parejko added a comment -

            Will the acceptance criteria be defined prior to the investigation? Having a well defined pass/fail metric made DM-11783 much more tractable.

            Show
            Parejkoj John Parejko added a comment - Will the acceptance criteria be defined prior to the investigation? Having a well defined pass/fail metric made DM-11783 much more tractable.
            Hide
            lauren Lauren MacArthur added a comment -

            I mean this just to post my general observations.  I imagined the pass/fail definition and determination will be made elsewhere and by others.  This issue is just to track relevant observations made along the way to the formal call.

            Show
            lauren Lauren MacArthur added a comment - I mean this just to post my general observations.  I imagined the pass/fail definition and determination will be made elsewhere and by others.  This issue is just to track relevant observations made along the way to the formal call.
            swinbank John Swinbank made changes -
            Link This issue blocks DM-15846 [ DM-15846 ]
            Hide
            swinbank John Swinbank added a comment -

            Per discussion offline:

            We do not think it practical to boil this ticket down to a short list of numerical targets.

            Lauren MacArthur will spend some time exploring Jointcal outputs, and will post observations here. She will collaborate with Yusra AlSayyad and other members of the DRP team to decide whether those observations relate to issues that need resolving in the short term (before making Jointcal the default for processing with obs_subaru), and, if so, turn them into tickets which “block” this one.

            Show
            swinbank John Swinbank added a comment - Per discussion offline: We do not think it practical to boil this ticket down to a short list of numerical targets. Lauren MacArthur will spend some time exploring Jointcal outputs, and will post observations here. She will collaborate with Yusra AlSayyad and other members of the DRP team to decide whether those observations relate to issues that need resolving in the short term (before making Jointcal the default for processing with obs_subaru), and, if so, turn them into tickets which “block” this one.
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lauren Lauren MacArthur made changes -
            Hide
            lauren Lauren MacArthur added a comment -

            Some comments on the plots attached which are based on a jointcal run by Yusra AlSayyad using w_2018_44 on tract 9697 of the RC2 dataset (DM-) in comparison with the meas_mosaic built coadds with the same weekly (DM-16099). The plots attached here were all created running the pipe_analysis scripts coaddAnalysis.py, colorAnalysis.py, and compareCoaddAnalysis.py. The full suites can be perused here for the jointcal run and here for the meas_mosaic and direct comparison between the two runs. (But note we make not guarantees on the lifetime of these directories!)

            Overall, the color-color locus looks quite good (i.e. both runs compare well):
            vs.

            However, as noted in DM-15751, the brightest stars with calib_psf_used have cmodel fluxes > psf fluxes (=> are classifed as galaxies) in jointcal and not meas_mosaic. This was first noted in this plot:
            vs.
            This is also seen in the CModel - PSF flux plots, where the star/galaxy separation is currently defined. There seems to be a dip towards bright mags in the jointcal run that does not occur with meas_mosaic:
            vs.

            And the following plot directly compares the CModel fluxes of a matched (based on RA/Dec) catalog between the two runs (jointcal vs. meas_mosaic):

            Spatially this appears in the areas with 1-2 visits and ghosting around very bright stars. This run comparison plot (sky distribution of the previous plot) shows the concentration of larger discrepancies (red-ish color) around bright (often masked, but not always, e.g. patch 7,7) star areas:

            The input counts and locations of bright stars can be seen in the following:

            Yusra AlSayyad also notes: By eye the coadds look identical and differences in clipping very minimal in these problematic areas.

            Show
            lauren Lauren MacArthur added a comment - Some comments on the plots attached which are based on a  jointcal run by Yusra AlSayyad using w_2018_44 on tract 9697 of the RC2 dataset (DM-) in comparison with the meas_mosaic built coadds with the same weekly ( DM-16099 ). The plots attached here were all created running the pipe_analysis scripts coaddAnalysis.py, colorAnalysis.py, and compareCoaddAnalysis.py. The full suites can be perused here for the jointcal run and here for the meas_mosaic and direct comparison between the two runs . (But note we make not guarantees on the lifetime of these directories!) Overall, the color-color locus looks quite good (i.e. both runs compare well): vs. However, as noted in DM-15751 , the brightest stars with calib_psf_used have cmodel fluxes > psf fluxes (=> are classifed as galaxies) in jointcal and not meas_mosaic . This was first noted in this plot: vs. This is also seen in the CModel - PSF flux plots, where the star/galaxy separation is currently defined. There seems to be a dip towards bright mags in the jointcal run that does not occur with meas_mosaic : vs. And the following plot directly compares the CModel fluxes of a matched (based on RA/Dec) catalog between the two runs ( jointcal vs. meas_mosaic ): Spatially this appears in the areas with 1-2 visits and ghosting around very bright stars. This run comparison plot (sky distribution of the previous plot) shows the concentration of larger discrepancies (red-ish color) around bright (often masked, but not always, e.g. patch 7,7) star areas: The input counts and locations of bright stars can be seen in the following: Yusra AlSayyad also notes: By eye the coadds look identical and differences in clipping very minimal in these problematic areas.
            Hide
            Parejkoj John Parejko added a comment -

            How would one modify the SourceSelector to reject stars in ghosted areas? What properties do those areas have that you could select on? Is this just around saturated stars, or is any "bright enough" star measurably worse in jointcal?

            Show
            Parejkoj John Parejko added a comment - How would one modify the SourceSelector to reject stars in ghosted areas? What properties do those areas have that you could select on? Is this just around saturated stars, or is any "bright enough" star measurably worse in jointcal?
            Hide
            jbosch Jim Bosch added a comment -

            Is the meas_mosaic "control sample" here coming from the old approach that used the fcr outputs directly, or the PhotoCalib scaling path with meas_mosaic's PhotoCalibs outputs?

            (I'm trying to understand whether we're actually comparing jointcal calibration vs. meas_mosaic calibration, PhotoCalib scaling vs. fcr scaling, or both)

            Show
            jbosch Jim Bosch added a comment - Is the meas_mosaic "control sample" here coming from the old approach that used the fcr outputs directly, or the PhotoCalib scaling path with meas_mosaic's PhotoCalibs outputs? (I'm trying to understand whether we're actually comparing jointcal calibration vs. meas_mosaic calibration, PhotoCalib scaling vs. fcr scaling, or both)
            Hide
            lauren Lauren MacArthur added a comment -

            The above observations (among others) inspired Yusra AlSayyad to request DM-16598 to see if the difference in variance handling/scaling was causing any of these discrepancies.  As noted there, Good news is it resolves the magnitude-dependent stellar size issue. Bad news is that there's now an offset. As seen here (note missing concentration of red points at bright mags, i.e., the "good" but the overall offset between the measured and reference magnitudes...the "bad"!)

            The full suite of plots from that run can be found (again with no guarantees on the lifetime of this link) here.

            Such offsets are also seen in the other bands with MagPsf - reference mean and std of:

            Filter mean (mmag) std (mmag)
            HSC-G -59.45 89.50
            HSC-R 20.25 88.11
            HSC-I 98.41 66.53
            HSC-Z 119.36 91.56
            HSC-Y 65.19 125.38

            This clearly needs resolving. My only speculation at this point is in recognizing that, currently, jointcal does not apply the color terms between the reference catalog (Pan-STARRS1) and HSC filters. However, this theory does not explain how the offset was not seen without the variance scaling omission of DM-16598 (i.e. there would have to be some kind of interplay between these two issues that is not obvious to me).

            Show
            lauren Lauren MacArthur added a comment - The above observations (among others) inspired Yusra AlSayyad  to request DM-16598 to see if the difference in variance handling/scaling was causing any of these discrepancies.  As noted there, Good news is it resolves the magnitude-dependent stellar size issue. Bad news is that there's now an offset . As seen here (note missing concentration of red points at bright mags, i.e., the "good" but the overall offset between the measured and reference magnitudes...the "bad"!) The full suite of plots from that run can be found (again with no guarantees on the lifetime of this link) here . Such offsets are also seen in the other bands with MagPsf - reference mean and std of: Filter mean (mmag) std (mmag) HSC-G -59.45 89.50 HSC-R 20.25 88.11 HSC-I 98.41 66.53 HSC-Z 119.36 91.56 HSC-Y 65.19 125.38 This clearly needs resolving. My only speculation at this point is in recognizing that, currently, jointcal does not apply the color terms between the reference catalog (Pan-STARRS1) and HSC filters. However, this theory does not explain how the offset was not seen without the variance scaling omission of DM-16598 (i.e. there would have to be some kind of interplay between these two issues that is not obvious to me).
            Hide
            lauren Lauren MacArthur added a comment -

            Jim Bosch, the meas_mosaic run is that of Hsin-Fang Chiang's w_2018_44 processing of the RC2 dataset (DM-16099). I'm not sure if/when we switched how the meas_mosaic outputs are applied...I'll try to track that down.

            Show
            lauren Lauren MacArthur added a comment - Jim Bosch , the meas_mosaic run is that of Hsin-Fang Chiang 's w_2018_44 processing of the RC2 dataset ( DM-16099 ). I'm not sure if/when we switched how the meas_mosaic outputs are applied...I'll try to track that down.
            Hide
            jbosch Jim Bosch added a comment -

            If this ticket reflects the state of things before we changed how PhotoCalib scales images, there's no need - in that case I already understand what's going on here, and it's the post-DM-16598 behavior that I'm actually interested in.

            Show
            jbosch Jim Bosch added a comment - If this ticket reflects the state of things before we changed how PhotoCalib scales images, there's no need - in that case I already understand what's going on here, and it's the post- DM-16598 behavior that I'm actually interested in.
            Hide
            lauren Lauren MacArthur added a comment -

            It looks to me that we are still using the original meas_mosaic method of updating the exposures:
            https://github.com/lsst/obs_subaru/blob/master/config/hsc/coaddBase.py
            https://github.com/lsst/obs_subaru/blob/master/config/hsc/compareWarpAssembleCoadd.py
            Where useMeasMosaic doc = "Use meas_mosaic's applyMosaicResultsExposure() to do the photometric calibration/wcs update (deprecated)."
            https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/coaddBase.py#L73

            Show
            lauren Lauren MacArthur added a comment - It looks to me that we are still using the original meas_mosaic method of updating the exposures: https://github.com/lsst/obs_subaru/blob/master/config/hsc/coaddBase.py https://github.com/lsst/obs_subaru/blob/master/config/hsc/compareWarpAssembleCoadd.py Where useMeasMosaic doc = "Use meas_mosaic's applyMosaicResultsExposure() to do the photometric calibration/wcs update (deprecated)." https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/coaddBase.py#L73
            yusra Yusra AlSayyad made changes -
            Epic Link DM-14402 [ 79665 ] DM-16680 [ 235240 ]
            Hide
            Parejkoj John Parejko added a comment -

            The mosaic PhotoCalib is very slow, which is why I didn't use it to make the warps. See my comment and Jim Bosch's followup here: https://jira.lsstcorp.org/browse/DM-15751?focusedCommentId=178255&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-178255

            Show
            Parejkoj John Parejko added a comment - The mosaic PhotoCalib is very slow, which is why I didn't use it to make the warps. See my comment and Jim Bosch 's followup here: https://jira.lsstcorp.org/browse/DM-15751?focusedCommentId=178255&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-178255
            Hide
            Parejkoj John Parejko added a comment -

            Regarding color terms in jointcal, I've got two tickets related to that: DM-12793 and DM-13054

            Show
            Parejkoj John Parejko added a comment - Regarding color terms in jointcal, I've got two tickets related to that: DM-12793 and DM-13054
            Hide
            lauren Lauren MacArthur added a comment - - edited

            John Parejko, I'm not sure this is simply a matter of source selection (I assume you mean what sources are selected and used for the fit in jointcal vs. meas_mosaic). What is being observed is a marked difference between the CModel and PSF fluxes measured on the jointcal coadds (not present in the meas_mosaic coadds). This is notable at the bright end because these two flux measurements have been observed to agree quite well for unresolved sources (PSFs). What the root cause of this is may be more complex than boiling down to just source selection (not that I don't think that could play a role, but we would probably just want to look directly at the differences in the source selection between meas_mosaic and jointcal first to see if there really is a significant/systematic difference in the sources that go into their respective fits before tinkering with source selectors?).

            Show
            lauren Lauren MacArthur added a comment - - edited John Parejko , I'm not sure this is simply a matter of source selection (I assume you mean what sources are selected and used for the fit in jointcal vs. meas_mosaic ). What is being observed is a marked difference between the CModel and PSF fluxes measured on the jointcal coadds (not present in the meas_mosaic coadds). This is notable at the bright end because these two flux measurements have been observed to agree quite well for unresolved sources (PSFs). What the root cause of this is may be more complex than boiling down to just source selection (not that I don't think that could play a role, but we would probably just want to look directly at the differences in the source selection between meas_mosaic and jointcal first to see if there really is a significant/systematic difference in the sources that go into their respective fits before tinkering with source selectors?).
            Hide
            lauren Lauren MacArthur added a comment -

            Yeah, as for the colorterm theory, my suggestion for a quick-and-(really)dirty attempt to see what impact (and its magnitude) it could have is to do a meas_mosaic run but NOT using them (this may, however, only reveal unrelated cans-of-worms...)

            Show
            lauren Lauren MacArthur added a comment - Yeah, as for the colorterm theory, my suggestion for a quick-and-(really)dirty attempt to see what impact (and its magnitude) it could have is to do a meas_mosaic run but NOT using them (this may, however, only reveal unrelated cans-of-worms...)
            Hide
            jbosch Jim Bosch added a comment -

            The root cause of this is the error propagation logic prior to DM-16598; that totally blows up the variance plane around bright objects, and hence changes (dramatically and incorrectly) the relative importance of the likelihood and prior in the CModel fit.

            Show
            jbosch Jim Bosch added a comment - The root cause of this is the error propagation logic prior to DM-16598 ; that totally blows up the variance plane around bright objects, and hence changes (dramatically and incorrectly) the relative importance of the likelihood and prior in the CModel fit.
            Hide
            lauren Lauren MacArthur added a comment -

            Ok, so we really are now down to "just" the ref vs. meas offset issue?

            Show
            lauren Lauren MacArthur added a comment - Ok, so we really are now down to "just" the ref vs. meas offset issue?
            Hide
            jbosch Jim Bosch added a comment -

            Ok, so we really are now down to "just" the ref vs. meas offset issue?

            I think so.

            Show
            jbosch Jim Bosch added a comment - Ok, so we really are now down to "just" the ref vs. meas offset issue? I think so.
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-16598 [ DM-16598 ]
            Hide
            lauren Lauren MacArthur added a comment - - edited

            NOTE: The following comments apply to the w_2018_48 RC2 reprocessing which includes coadds made using both the meas_mosaic and jointcal fits.

            I have just confirmed that the colorterm issue is our main contributor to the offsets noted above. The way things are currently running are actually inconsistent. The meas_mosaic and jointcal fits use the same Single Frame Processing run as input. The sfm's apply the colorterm corrections to the reference catalog magnitudes to do the initial photometric calibration. meas_mosaic also applies the colorterms when using the reference stars in its fit. jointcal does not. This puts jointcal in the position of "erasing" those corrections from the initial sfm photoCal and also, effectively, is using a mix-match of effective filters for the fitting (the reference stars are still on their own system whereas the newly added "internal" reference stars are on the observed system).

            My confirmation comes from remaking my plots but not applying the colorterm to the reference stars for the plots comparing ref to src matches (they look up the colorterms essentially how sfm would, so were being applied as they were unaware that jointcal was not applying them). This brings the offset down from 88[-59] mmag to 25[-4.5] mmag in HSC-I[HSC-G].

            When directly comparing the meas_mosaic and jointcal cModel mags (based on a matched catalog), the offset between them is also of order 88[-64] mmag, so the magnitude of the effect is in the right ballpark. The remaining 25[-4.5] mmag offset between the reference catalog could very well result from the relative weight of the effective mix-matching between filter systems in the jointcal fit.

            Taking into account the offset, there is still evidence for increased discrepancies around bright objects. This is perhaps the remaining effect that will be resolved with DM-16598. Yusra AlSayyad, let me know if you want me to point my scripts at any particular run that incorporates that change.

            Show
            lauren Lauren MacArthur added a comment - - edited NOTE: The following comments apply to the w_2018_48 RC2 reprocessing which includes coadds made using both the meas_mosaic and jointcal fits. I have just confirmed that the colorterm issue is our main contributor to the offsets noted above. The way things are currently running are actually inconsistent. The meas_mosaic and jointcal fits use the same Single Frame Processing run as input. The sfm's apply the colorterm corrections to the reference catalog magnitudes to do the initial photometric calibration. meas_mosaic also applies the colorterms when using the reference stars in its fit. jointcal does not. This puts jointcal in the position of "erasing" those corrections from the initial sfm photoCal and also, effectively, is using a mix-match of effective filters for the fitting (the reference stars are still on their own system whereas the newly added "internal" reference stars are on the observed system). My confirmation comes from remaking my plots but not applying the colorterm to the reference stars for the plots comparing ref to src matches (they look up the colorterms essentially how sfm would, so were being applied as they were unaware that jointcal was not applying them). This brings the offset down from 88 [ -59 ] mmag to 25 [ -4.5 ] mmag in HSC-I [ HSC-G ] . When directly comparing the meas_mosaic and jointcal cModel mags (based on a matched catalog), the offset between them is also of order 88 [ -64 ] mmag, so the magnitude of the effect is in the right ballpark. The remaining 25 [ -4.5 ] mmag offset between the reference catalog could very well result from the relative weight of the effective mix-matching between filter systems in the jointcal fit. Taking into account the offset, there is still evidence for increased discrepancies around bright objects. This is perhaps the remaining effect that will be resolved with DM-16598 . Yusra AlSayyad , let me know if you want me to point my scripts at any particular run that incorporates that change.
            Hide
            lauren Lauren MacArthur added a comment -

            My bottom line for now: the colorterms must be applied to the reference catalog mags in jointcal. I also believe the work for this has been done (but is perhaps a bit rotted...heh, heh) by Dominique Boutigny on DM-13054 in these commits: https://github.com/lsst/jointcal/commit/e1d0292c71275315a6b17b67ecaf47acd2ffbee8
            https://github.com/lsst/jointcal/commit/05f60a9a2c9b86a33487cccddd9b0640ca90faf1
            Given the significant magnitude of the effect, I believe this should block any further QA efforts (but I would, of course, be happy to pick them up along with the functionality being implemented). Hopefully the recent decision to Removing from current sprint per discussion of 2018-12-19. can be amended?

            Show
            lauren Lauren MacArthur added a comment - My bottom line for now: the colorterms must be applied to the reference catalog mags in jointcal . I also believe the work for this has been done (but is perhaps a bit rotted...heh, heh) by Dominique Boutigny on DM-13054 in these commits: https://github.com/lsst/jointcal/commit/e1d0292c71275315a6b17b67ecaf47acd2ffbee8 https://github.com/lsst/jointcal/commit/05f60a9a2c9b86a33487cccddd9b0640ca90faf1 Given the significant magnitude of the effect, I believe this should block any further QA efforts (but I would, of course, be happy to pick them up along with the functionality being implemented). Hopefully the recent decision to Removing from current sprint per discussion of 2018-12-19. can be amended?
            Hide
            lauren Lauren MacArthur added a comment -

            Let me know if you think the work to date on this ticket covers its scope and can be closed out.  If so, I would expect we would add further similar tickets following decisions and outcomes based on my recommendations here.

            Show
            lauren Lauren MacArthur added a comment - Let me know if you think the work to date on this ticket covers its scope and can be closed out.  If so, I would expect we would add further similar tickets following decisions and outcomes based on my recommendations here.
            lauren Lauren MacArthur made changes -
            Reviewers John Swinbank, Yusra AlSayyad [ swinbank, yusra ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            yusra Yusra AlSayyad added a comment -

            The post DM-16598 rerun is in: https://lsst-web.ncsa.illinois.edu/~yusra/RC_QA/w_2018_48_jointcal_noScaleVar_redo/ --> /datasets/hsc/repo/rerun/private/yusra/RC2/w_2018_48_jointcal_noScaleVar_redo/

            Yes, we can reassess after colorterms have been applied. On DM-16598, I just have to update pipe_tasks, run jenkins and that ticket should be good to merge.

            Show
            yusra Yusra AlSayyad added a comment - The post DM-16598 rerun is in: https://lsst-web.ncsa.illinois.edu/~yusra/RC_QA/w_2018_48_jointcal_noScaleVar_redo/ --> /datasets/hsc/repo/rerun/private/yusra/RC2/w_2018_48_jointcal_noScaleVar_redo/ Yes, we can reassess after colorterms have been applied. On DM-16598 , I just have to update pipe_tasks, run jenkins and that ticket should be good to merge.
            yusra Yusra AlSayyad made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            yusra Yusra AlSayyad made changes -
            Story Points 16
            yusra Yusra AlSayyad made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                John Swinbank, Yusra AlSayyad
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Lauren MacArthur, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel