# Investigate color-dependent offsets from ref cat in jointcal vs. meas_mosaic

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
12
• Sprint:
DRP S19-3, DRP S19-4, DRP S19-5
• Team:
Data Release Production

#### Description

Now that jointcal applies the colorterms, we can look at much smaller differences in how meas_mosaic an jointcal match the to reference catalog. Some observations. There is a shift that jumps from

and the 3 mmag shift in the w principle component from jointcal to meas_mosaic  new_jointcal_gri  which is probably a manifestation of the offsets above (and just another way of looking at it)

I don't know if these are significant or not. Will those with more expertise comment on whether this is a problem or not.

NOTE: these comparisons were done with meas_mosaic before DM-13054. If https://community.lsst.org/t/incorrect-reference-magnitude-errors-from-photocaltask/3581 affects meas_mosaic too, I propose we wait for Hsin-Fang Chiang's w_2019_06 run next week before we spend time thinking about this.

#### Attachments

1. psfMagRefOffset_stars_only_internal.png
17 kB
2. psfMagRefOffset.png
17 kB
3. split_tract_AF1_design.png
42 kB
4. split_tract_AM1.png
44 kB
5. split_tract_AM1 (2).png
32 kB
6. split_tract_PA1.png
39 kB
7. split_tract_reviewAM1.png
55 kB
8. split_tract_reviewPA1.png
51 kB
9. wOffset_stars_only.png
11 kB
10. wOffset.png
9 kB
11. wStd_fit_std.png
11 kB
12. wStd.png
9 kB

#### Activity

Hide
Eli Rykoff added a comment -

We had a discussion at group meeting last week about the philosophy of individually coded source selectors vs catch-all selectors like ScienceSourceSelector (which can be configured). I actually don't know where we concluded (Yusra AlSayyad or Jim Bosch can chime in), other than to agree that the present situation is muddled and confusing to developers and users alike.

Show
Eli Rykoff added a comment - We had a discussion at group meeting last week about the philosophy of individually coded source selectors vs catch-all selectors like ScienceSourceSelector (which can be configured). I actually don't know where we concluded ( Yusra AlSayyad or Jim Bosch can chime in), other than to agree that the present situation is muddled and confusing to developers and users alike.
Hide
John Parejko added a comment -

I've played with the two selectors a bit, and one thing that came up is that ScienceSourceSelector does not check the footprints when testing for blended sources, while AstrometrySourceSelector does (doing the check is quite slow). Jim Bosch's implication in DM-7100 is that a footprint check is unnecessary if the deblender has been run, but I don't know how to determine that from a given catalog.

Show
John Parejko added a comment - I've played with the two selectors a bit, and one thing that came up is that ScienceSourceSelector does not check the footprints when testing for blended sources, while AstrometrySourceSelector does (doing the check is quite slow). Jim Bosch 's implication in DM-7100 is that a footprint check is unnecessary if the deblender has been run, but I don't know how to determine that from a given catalog.
Hide
Eli Rykoff added a comment - - edited

Well, if we need that behavior in the general selector, it can be added as an option to ScienceSourceSelector! (Only slightly joking).

But I believe that deblending is run by default when calexp s are calibrate d: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/calibrate.py#L132

Though I don't know how to programatically check for that from an arbitrary catalog.

Show
Eli Rykoff added a comment - - edited Well, if we need that behavior in the general selector, it can be added as an option to ScienceSourceSelector ! (Only slightly joking). But I believe that deblending is run by default when calexp s are calibrate d: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/calibrate.py#L132 Though I don't know how to programatically check for that from an arbitrary catalog.
Hide
Jim Bosch added a comment -

We had a discussion at group meeting last week about the philosophy of individually coded source selectors vs catch-all selectors like ScienceSourceSelector (which can be configured). I actually don't know where we concluded (Yusra AlSayyad or Jim Bosch can chime in), other than to agree that the present situation is muddled and confusing to developers and users alike.

After that meeting, Nate Lust and I had a brief chat that lead to a pretty simple idea for how to improve matters by making selectors composible.  Since then I've been a little torn about whether to make a push for that, since it'd still be a disruption, but the fact that this question has come up again so soon suggests that I should.  I'll put together an RFC tonight or tomorrow.  In the meantime, don't block anything here waiting for that.

But I believe that deblending is run by default when calexp s are calibrate d: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/calibrate.py#L132

Though I don't know how to programatically check for that from an arbitrary catalog.

If you haven't run the deblender, you won't get any sources with parent != 0.  I would also be quite comfortable declaring that the deblender must be run in CalibrateTask, and checking this in CalibrateConfig.validate.

Show
Jim Bosch added a comment - We had a discussion at group meeting last week about the philosophy of individually coded source selectors vs catch-all selectors like ScienceSourceSelector (which can be configured). I actually don't know where we concluded ( Yusra AlSayyad or Jim Bosch can chime in), other than to agree that the present situation is muddled and confusing to developers and users alike. After that meeting, Nate Lust and I had a brief chat that lead to a pretty simple idea for how to improve matters by making selectors composible.  Since then I've been a little torn about whether to make a push for that, since it'd still be a disruption, but the fact that this question has come up again so soon suggests that I should.  I'll put together an RFC tonight or tomorrow.  In the meantime, don't block anything here waiting for that. But I believe that deblending is run by default when calexp s are calibrate d: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/calibrate.py#L132 Though I don't know how to programatically check for that from an arbitrary catalog. If you haven't run the deblender, you won't get any sources with parent != 0 .  I would also be quite comfortable declaring that the deblender must be run in CalibrateTask, and checking this in CalibrateConfig.validate.
Hide

Going to merge this in time for w_2019_18 (Hsin-Fang Chiang's next RC2 processing).
https://ci.lsst.codes/job/stack-os-matrix/29788/display/redirect

Thank you John Parejko for looking it over again and pushing the commit to get the jointcal unit tests to pass. When I rebased, github ended up thinking it was a single author commit instead of a double author commit, and the record of exchange disappeared...

Show
Yusra AlSayyad added a comment - Going to merge this in time for w_2019_18 ( Hsin-Fang Chiang 's next RC2 processing). https://ci.lsst.codes/job/stack-os-matrix/29788/display/redirect Thank you John Parejko for looking it over again and pushing the commit to get the jointcal unit tests to pass. When I rebased, github ended up thinking it was a single author commit instead of a double author commit, and the record of exchange disappeared...

#### People

Assignee:
Reporter:
Reviewers:
John Parejko
Watchers:
Eli Rykoff, Hsin-Fang Chiang, James Chiang, Jim Bosch, John Parejko, John Swinbank, Lauren MacArthur, Yusra AlSayyad