# jointcal constrained photometry model is not better than processCcd photometry on DC2 run1.2

XMLWordPrintable

#### Details

• Type: Bug
• Status: Won't Fix
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:

#### Description

From a set of 7 visits from DESC DC2 Run1.2p I am taking 1 visit as an arbitrary reference and for each matched sources in the other visits I compute the difference of magnitude using 1- the processCcd calibration and 2- the jointcal calibration from the default constrained model.

The dispersion (19.9 mmag) is identical in both cases which looks very surprising. The size of the dispersion also looks very large.

In order to reproduce the problem the data are available on lsst-dev:/home/boutigny/2018-10-11 and the notebook to reproduce the plot is on: https://github.com/boutigny/LSST_notebooks/blob/master/DC2/multi_visit_Joincal.ipynb

I am not excluding an error on my side or a misusage of jointcal

#### Activity

No builds found.
Dominique Boutigny created issue -
Field Original Value New Value
Risk Score 0
 Description From a set of 7 visits from DESC DC2 Run1.2p I am taking 1 visit as an arbitrary reference and for each matched sources in the other visits I compute the difference of magnitude using 1- the processCcd calibration and 2- the jointcal calibration from the default constrained model. The dispersion (19.9 mmag) is identical in both cases which looks very surprising.   In order to reproduce the problem the data are available on lsst-dev:/home/boutigny/2018-10-11 and the notebook to reproduce the plot is on: [https://github.com/boutigny/LSST_notebooks/blob/master/DC2/multi_visit_Joincal.ipynb] I am not excluding an error on my side or a misusage of jointcal From a set of 7 visits from DESC DC2 Run1.2p I am taking 1 visit as an arbitrary reference and for each matched sources in the other visits I compute the difference of magnitude using 1- the processCcd calibration and 2- the jointcal calibration from the default constrained model. The dispersion (19.9 mmag) is identical in both cases which looks very surprising. The size of the dispersion also looks very large.  In order to reproduce the problem the data are available on lsst-dev:/home/boutigny/2018-10-11 and the notebook to reproduce the plot is on: [https://github.com/boutigny/LSST_notebooks/blob/master/DC2/multi_visit_Joincal.ipynb] I am not excluding an error on my side or a misusage of jointcal
Hide
John Parejko added a comment -

I gave up on running validate_drp on your output, because the dataIds did not have the int detector field, just the string detectorName. If you could help fix that, I can do some more data exploration (validate_drp computes a number of different metrics).

Show
John Parejko added a comment - I gave up on running validate_drp on your output, because the dataIds did not have the int detector field, just the string detectorName . If you could help fix that, I can do some more data exploration (validate_drp computes a number of different metrics).
 Summary jointcal constrained photometry model is not better than processCcd photometry jointcal constrained photometry model is not better than processCcd photometry on DC2 run1.2
 Epic Link DM-14447 [ 80385 ]
 Team Alert Production [ 10300 ]
Hide
Jim Bosch added a comment -

Dominique Boutigny, we discussed this a bit today and that raised some questions about whether known issues with DC2 reference catalogs might be in play here:

• I know there are some problems with galaxies and extinction; are those the kind that would make the reference catalog fluxes inconsistent with the images, or just the kind that would make both unphysical but self-consistent?
• I'd like to find a way to investigate the possibility of both the single-epoch and jointcal fits locking on to the reference catalog as the source of truth and disregarding (in the case of jointcal) the extra information from multiple observations, which could happen if there are many more sources with reference catalog matches and/or lower-than-expected uncertainties in the reference catalog.  That would indicate that the only "problem" here is an unrealistically small dispersion for the single-epoch photometry, and that might be what we're seeing: 19mm is definite much larger than what we want to see for LSST, but I think it's pretty similar to what we get with meas_mosaic and the current state of calibrations for HSC (though there we know we are limited by chromatic effects in the filters that may not be in play here, and I'm not at all certain that's an apples-to-apples comparison in terms of SNR cuts).  We could investigate this by re-running with a SourceSelector for both single-epoch and jointcal that filters the reference catalog down to more realistic magnitude limits and densities.
• Is the timeline for a new run that will fix some of those issues soon enough that we should just wait for that?

All that said, I'm not thrilled about the prospect of trying to debug this by just turning knobs and hoping the results improve; it'd be much more productive to make targeted diagnostic plots (RMS relative to reference catalog, filter galaxies in and out of various plots, etc.) to actually test some of these hypotheses.  We're talking about how to get better at doing that, too (the notebook you attached is definitely a very helpful first step!).

Show
Jim Bosch added a comment - Dominique Boutigny , we discussed this a bit today and that raised some questions about whether known issues with DC2 reference catalogs might be in play here: I know there are some problems with galaxies and extinction; are those the kind that would make the reference catalog fluxes inconsistent with the images, or just the kind that would make both unphysical but self-consistent? I'd like to find a way to investigate the possibility of both the single-epoch and jointcal fits locking on to the reference catalog as the source of truth and disregarding (in the case of jointcal) the extra information from multiple observations, which could happen if there are many more sources with reference catalog matches and/or lower-than-expected uncertainties in the reference catalog.  That would indicate that the only "problem" here is an unrealistically small dispersion for the single-epoch photometry, and that might be what we're seeing: 19mm is definite much larger than what we want to see for LSST, but I think it's pretty similar to what we get with meas_mosaic and the current state of calibrations for HSC (though there we know we are limited by chromatic effects in the filters that may not be in play here, and I'm not at all certain that's an apples-to-apples comparison in terms of SNR cuts).  We could investigate this by re-running with a SourceSelector for both single-epoch and jointcal that filters the reference catalog down to more realistic magnitude limits and densities. Is the timeline for a new run that will fix some of those issues soon enough that we should just wait for that? All that said, I'm not thrilled about the prospect of trying to debug this by just turning knobs and hoping the results improve; it'd be much more productive to make targeted diagnostic plots (RMS relative to reference catalog, filter galaxies in and out of various plots, etc.) to actually test some of these hypotheses.  We're talking about how to get better at doing that, too (the notebook you attached is definitely a very helpful first step!).
Hide
Dominique Boutigny added a comment -

Jim Bosch I came to the same conclusion about the reference catalog and phosim DC2 data. We can't draw any conclusions if the reference catalog is polluted by sources with fluxes that are not matching the data. Today I saw that I can easily modify loadIndexedReferenceObjects.py to exclude galaxies from the reference catalog. I will rerun jointcal with this modification as soon as I have some time to do it.

The other possibility is to run jointcal on imsim data as we know that the reference catalog is correctly matching the data.

In a general way, it would be useful to have selectors for the reference sources as we have for the detected sources.

Show
Dominique Boutigny added a comment - Jim Bosch I came to the same conclusion about the reference catalog and phosim DC2 data. We can't draw any conclusions if the reference catalog is polluted by sources with fluxes that are not matching the data. Today I saw that I can easily modify loadIndexedReferenceObjects.py to exclude galaxies from the reference catalog. I will rerun jointcal with this modification as soon as I have some time to do it. The other possibility is to run jointcal on imsim data as we know that the reference catalog is correctly matching the data. In a general way, it would be useful to have selectors for the reference sources as we have for the detected sources.
 Link This issue relates to DM-13053 [ DM-13053 ]
 Epic Link DM-14447 [ 80385 ] DM-16722 [ 235355 ]
Hide
Dominique Boutigny added a comment -

Today I reran joincal on imsim data (Run1.2i) and I get a very similar result as what I obtained initially on Run1.2p and the chi2/dof is = 67 for the photometry.
I really don't know how to handle this...

Show
Dominique Boutigny added a comment - Today I reran joincal on imsim data (Run1.2i) and I get a very similar result as what I obtained initially on Run1.2p and the chi2/dof is = 67 for the photometry. I really don't know how to handle this...
Hide
John Parejko added a comment -

Lets look at this in the new year: one of the priorities for January is building some more analysis tools for jointcal's output.

Show
John Parejko added a comment - Lets look at this in the new year: one of the priorities for January is building some more analysis tools for jointcal's output.
 Epic Link DM-16722 [ 235355 ] DM-17887 [ 240317 ]
Hide
Jim Bosch added a comment -

Recording discussion from the DESC-DM-DC2 discussion of 2019-02-26:

• Dominique Boutigny has tried something he believes is equivalent to enabling the new use-only-stars-from-reference-catalog option added by Eli Rykoff on DM-17917, and saw no significant change.  He would like this to be checked more carefully.
• Someone from DM (I'm asking Yusra AlSayyad to coordinate, though it may not be her) should try running jointcal on the DC2 data on lsst-dev with the option from DM-17917 enabled, and then try to reproduce Dominique Boutigny's plots on the results of that processing.

Thinking about this a bit further now, it's not totally clear to me that the DM-17917 code can be used with jointcal to select only stars from the reference catalog without limiting jointcal to using only reference catalog objects entirely.  Eli Rykoff or John Parejko, can you comment on that?

Show
Jim Bosch added a comment - Recording discussion from the DESC-DM-DC2 discussion of 2019-02-26: Dominique Boutigny has tried something he believes is equivalent to enabling the new use-only-stars-from-reference-catalog option added by Eli Rykoff on DM-17917 , and saw no significant change.  He would like this to be checked more carefully. Someone from DM (I'm asking Yusra AlSayyad to coordinate, though it may not be her) should try running jointcal on the DC2 data on lsst-dev with the option from DM-17917 enabled, and then try to reproduce Dominique Boutigny 's plots on the results of that processing. Thinking about this a bit further now, it's not totally clear to me that the DM-17917 code can be used with jointcal to select only stars from the reference catalog without limiting jointcal to using only reference catalog objects entirely.  Eli Rykoff or John Parejko , can you comment on that?
Hide
John Parejko added a comment -

I think we might have to implement DM-13053, before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?

Show
John Parejko added a comment - I think we might have to implement DM-13053 , before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?
Hide
Jim Bosch added a comment -

I think we might have to implement DM-13053, before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?

Yes, please.  (I'm just taking your word that it's what's needed to try what I've said want to try, but I'm happy to do that unless you think I need to understand the details).

Show
Jim Bosch added a comment - I think we might have to implement DM-13053 , before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket? Yes, please.  (I'm just taking your word that it's what's needed to try what I've said want to try, but I'm happy to do that unless you think I need to understand the details).
Hide
Eli Rykoff added a comment -

So the fix in DM-17917 applies to astrometryTask in processCcd only. What John said that DM-13053 needs to be done for selecting stars from the reference catalog (or doing any refcat selections) in jointcal.

Show
Eli Rykoff added a comment - So the fix in DM-17917 applies to astrometryTask in processCcd only. What John said that DM-13053 needs to be done for selecting stars from the reference catalog (or doing any refcat selections) in jointcal .
Hide
Robert Lupton added a comment -

This just came up on the DC2 phonecon.  I am not all that surprised that jointcal doesn't do better than processCcd given a noise-free reference catalogue to 23rd magnitude – all that jointcal can do is bootstrap a deeper catalogue, and there's not much need under the circumstances.   If I'm misinformed about the reference catalogue that we're using please forgive this note.

I am still very concerned about a 20mmag scatter, though.  Is this still present after all the work discussed on this ticket?

Show
Robert Lupton added a comment - This just came up on the DC2 phonecon.  I am not all that surprised that jointcal doesn't do better than processCcd given a noise-free reference catalogue to 23rd magnitude – all that jointcal can do is bootstrap a deeper catalogue, and there's not much need under the circumstances.   If I'm misinformed about the reference catalogue that we're using please forgive this note. I am still very concerned about a 20mmag scatter, though.  Is this still present after all the work discussed on this ticket?
 Attachment run2.1i_jointcal_delta_mags_w_2019_09.png [ 37228 ]
Hide
James Chiang added a comment -

> I am still very concerned about a 20mmag scatter, though. Is this still present after all the work discussed on this ticket?

Using 7 overlapping visits of Run2.1i data, jointcal in w_2019_09, and a stars-only reference catalog, I get the same ~20mmag scatter:

Show
James Chiang added a comment - > I am still very concerned about a 20mmag scatter, though. Is this still present after all the work discussed on this ticket?   Using 7 overlapping visits of Run2.1i data, jointcal in w_2019_09, and a stars-only reference catalog, I get the same ~20mmag scatter:
Hide
Eli Rykoff added a comment -

So processCcd (photoCal precisely) can only do one zeropoint per ccd. In general, jointcal should be able to do higher order fits in the constrained (vs simple) models, though I'm not sure if James Chiang's recent run uses the higher order stuff (I know he was building up to getting that working).

Show
Eli Rykoff added a comment - So processCcd ( photoCal precisely) can only do one zeropoint per ccd. In general, jointcal should be able to do higher order fits in the constrained (vs simple) models, though I'm not sure if James Chiang 's recent run uses the higher order stuff (I know he was building up to getting that working).
Hide
James Chiang added a comment -

Here's what I used:

 config.photometryRejectBadFluxes=False   config.astrometryModel="constrained" config.astrometryChipOrder=1 config.astrometryVisitOrder=3   config.photometryModel="simpleMagnitude" 

I guess I need to use "constrainedMagnitude" for the photometryModel and dial up photometryVisitOrder for which the default is 7?

Show
James Chiang added a comment - Here's what I used: config.photometryRejectBadFluxes=False   config.astrometryModel= "constrained" config.astrometryChipOrder= 1 config.astrometryVisitOrder= 3   config.photometryModel= "simpleMagnitude" I guess I need to use "constrainedMagnitude" for the photometryModel and dial up photometryVisitOrder for which the default is 7?
Hide
John Parejko added a comment -

Try running jointcal with the defaults: that will use the constrained model for both astrometry and photometry, with a 5th and 7th order polynomial, respectively. What version are you running?

The "simple" models are just extensions of the processCcd model (one model per ccd/visit: 2d polynomial for astrometry, one calibration factor for photometry). They are primarily useful for debugging or fitting non-mosaic camera data.

Show
John Parejko added a comment - Try running jointcal with the defaults: that will use the constrained model for both astrometry and photometry, with a 5th and 7th order polynomial, respectively. What version are you running? The "simple" models are just extensions of the processCcd model (one model per ccd/visit: 2d polynomial for astrometry, one calibration factor for photometry). They are primarily useful for debugging or fitting non-mosaic camera data.
Hide
Dominique Boutigny added a comment -

We should also keep in mind that the astrometry fit is not giving very good results either. Running jointcal with default configuration I get an astrometric dispersion = 11.4 mas (mag<21) to be compared to 7.9 mas with the processCcd astrometry (which is also not particularly good given an almost perfect reference catalog).
I think that we cannot exclude a problem in the reference catalog itself. I am excluding a problem in the simulation as I got similar results with phosim.

Show
Dominique Boutigny added a comment - We should also keep in mind that the astrometry fit is not giving very good results either. Running jointcal with default configuration I get an astrometric dispersion = 11.4 mas (mag<21) to be compared to 7.9 mas with the processCcd astrometry (which is also not particularly good given an almost perfect reference catalog). I think that we cannot exclude a problem in the reference catalog itself. I am excluding a problem in the simulation as I got similar results with phosim.
Hide
John Parejko added a comment - - edited

Based on the discussion on slack, it appears to me that the reason jointcal doesn't do better than processCcd on this imsim data, is that there is nothing for it to fit beyond processCcd's model. Quoting myself from slack:

If there are no throughput variations due to flat fielding, optics or atmosphere transparency, then there's really no way for jointcal (single value per sensor+focal plane polynomial per visit) to do better than processCcd (single value per sensor/visit).

Show
John Parejko added a comment - - edited Based on the discussion on slack , it appears to me that the reason jointcal doesn't do better than processCcd on this imsim data, is that there is nothing for it to fit beyond processCcd's model. Quoting myself from slack: If there are no throughput variations due to flat fielding, optics or atmosphere transparency, then there's really no way for jointcal (single value per sensor+focal plane polynomial per visit) to do better than processCcd (single value per sensor/visit).
 Epic Link DM-17887 [ 240317 ] DM-19979 [ 307530 ]
Hide
John Parejko added a comment -

Can we close this as Invalid or Won't Fix, due to DC2 not having any photometric distortions for jointcal to model, and the known issues with the refcat?

Show
John Parejko added a comment - Can we close this as Invalid or Won't Fix, due to DC2 not having any photometric distortions for jointcal to model, and the known issues with the refcat?
 Epic Link DM-19979 [ 307530 ] DM-21441 [ 423048 ]
 Epic Link DM-21441 [ 423048 ] DM-22484 [ 427311 ]
 Epic Link DM-22484 [ 427311 ] DM-24339 [ 433026 ]
Hide
John Swinbank added a comment -

Yes.

Show
John Swinbank added a comment - Yes.
 Resolution Done [ 10000 ] Status To Do [ 10001 ] Won't Fix [ 10405 ]

#### People

Assignee:
Unassigned
Reporter:
Dominique Boutigny
Watchers:
Dominique Boutigny, Eli Rykoff, James Chiang, Jim Bosch, John Parejko, John Swinbank, Robert Lupton