Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16226

jointcal constrained photometry model is not better than processCcd photometry on DC2 run1.2

    XMLWordPrintable

Details

    • Bug
    • Status: Won't Fix
    • Resolution: Done
    • None
    • jointcal
    • None

    Description

      From a set of 7 visits from DESC DC2 Run1.2p I am taking 1 visit as an arbitrary reference and for each matched sources in the other visits I compute the difference of magnitude using 1- the `processCcd` calibration and 2- the `jointcal` calibration from the default constrained model.

      The dispersion (19.9 mmag) is identical in both cases which looks very surprising. The size of the dispersion also looks very large. 

      In order to reproduce the problem the data are available on `lsst-dev:/home/boutigny/2018-10-11` and the notebook to reproduce the plot is on: `https://github.com/boutigny/LSST_notebooks/blob/master/DC2/multi_visit_Joincal.ipynb`

      I am not excluding an error on my side or a misusage of `jointcal`

      Attachments

        Issue Links

          Activity

            I gave up on running validate_drp on your output, because the dataIds did not have the int detector field, just the string detectorName. If you could help fix that, I can do some more data exploration (validate_drp computes a number of different metrics).

            Parejkoj John Parejko added a comment - I gave up on running validate_drp on your output, because the dataIds did not have the int detector field, just the string detectorName . If you could help fix that, I can do some more data exploration (validate_drp computes a number of different metrics).
            jbosch Jim Bosch added a comment -

            boutigny, we discussed this a bit today and that raised some questions about whether known issues with DC2 reference catalogs might be in play here:

            • I know there are some problems with galaxies and extinction; are those the kind that would make the reference catalog fluxes inconsistent with the images, or just the kind that would make both unphysical but self-consistent?
            • I'd like to find a way to investigate the possibility of both the single-epoch and jointcal fits locking on to the reference catalog as the source of truth and disregarding (in the case of jointcal) the extra information from multiple observations, which could happen if there are many more sources with reference catalog matches and/or lower-than-expected uncertainties in the reference catalog.  That would indicate that the only "problem" here is an unrealistically small dispersion for the single-epoch photometry, and that might be what we're seeing: 19mm is definite much larger than what we want to see for LSST, but I think it's pretty similar to what we get with meas_mosaic and the current state of calibrations for HSC (though there we know we are limited by chromatic effects in the filters that may not be in play here, and I'm not at all certain that's an apples-to-apples comparison in terms of SNR cuts).  We could investigate this by re-running with a SourceSelector for both single-epoch and jointcal that filters the reference catalog down to more realistic magnitude limits and densities.
            • Is the timeline for a new run that will fix some of those issues soon enough that we should just wait for that?

            All that said, I'm not thrilled about the prospect of trying to debug this by just turning knobs and hoping the results improve; it'd be much more productive to make targeted diagnostic plots (RMS relative to reference catalog, filter galaxies in and out of various plots, etc.) to actually test some of these hypotheses.  We're talking about how to get better at doing that, too (the notebook you attached is definitely a very helpful first step!).

            jbosch Jim Bosch added a comment - boutigny , we discussed this a bit today and that raised some questions about whether known issues with DC2 reference catalogs might be in play here: I know there are some problems with galaxies and extinction; are those the kind that would make the reference catalog fluxes inconsistent with the images, or just the kind that would make both unphysical but self-consistent? I'd like to find a way to investigate the possibility of both the single-epoch and jointcal fits locking on to the reference catalog as the source of truth and disregarding (in the case of jointcal) the extra information from multiple observations, which could happen if there are many more sources with reference catalog matches and/or lower-than-expected uncertainties in the reference catalog.  That would indicate that the only "problem" here is an unrealistically small dispersion for the single-epoch photometry, and that might be what we're seeing: 19mm is definite much larger than what we want to see for LSST, but I think it's pretty similar to what we get with meas_mosaic and the current state of calibrations for HSC (though there we know we are limited by chromatic effects in the filters that may not be in play here, and I'm not at all certain that's an apples-to-apples comparison in terms of SNR cuts).  We could investigate this by re-running with a SourceSelector for both single-epoch and jointcal that filters the reference catalog down to more realistic magnitude limits and densities. Is the timeline for a new run that will fix some of those issues soon enough that we should just wait for that? All that said, I'm not thrilled about the prospect of trying to debug this by just turning knobs and hoping the results improve; it'd be much more productive to make targeted diagnostic plots (RMS relative to reference catalog, filter galaxies in and out of various plots, etc.) to actually test some of these hypotheses.  We're talking about how to get better at doing that, too (the notebook you attached is definitely a very helpful first step!).

            jbosch I came to the same conclusion about the reference catalog and phosim DC2 data. We can't draw any conclusions if the reference catalog is polluted by sources with fluxes that are not matching the data. Today I saw that I can easily modify loadIndexedReferenceObjects.py to exclude galaxies from the reference catalog. I will rerun jointcal with this modification as soon as I have some time to do it.

            The other possibility is to run jointcal on imsim data as we know that the reference catalog is correctly matching the data.

            In a general way, it would be useful to have selectors for the reference sources as we have for the detected sources. 

             

            boutigny Dominique Boutigny added a comment - jbosch I came to the same conclusion about the reference catalog and phosim DC2 data. We can't draw any conclusions if the reference catalog is polluted by sources with fluxes that are not matching the data. Today I saw that I can easily modify loadIndexedReferenceObjects.py to exclude galaxies from the reference catalog. I will rerun jointcal with this modification as soon as I have some time to do it. The other possibility is to run jointcal on imsim data as we know that the reference catalog is correctly matching the data. In a general way, it would be useful to have selectors for the reference sources as we have for the detected sources.   

            Today I reran joincal on imsim data (Run1.2i) and I get a very similar result as what I obtained initially on Run1.2p and the chi2/dof is = 67 for the photometry.
            I really don't know how to handle this...

            boutigny Dominique Boutigny added a comment - Today I reran joincal on imsim data (Run1.2i) and I get a very similar result as what I obtained initially on Run1.2p and the chi2/dof is = 67 for the photometry. I really don't know how to handle this...

            Lets look at this in the new year: one of the priorities for January is building some more analysis tools for jointcal's output.

            Parejkoj John Parejko added a comment - Lets look at this in the new year: one of the priorities for January is building some more analysis tools for jointcal's output.
            jbosch Jim Bosch added a comment -

            Recording discussion from the DESC-DM-DC2 discussion of 2019-02-26:

            • boutigny has tried something he believes is equivalent to enabling the new use-only-stars-from-reference-catalog option added by erykoff on DM-17917, and saw no significant change.  He would like this to be checked more carefully.
            • Someone from DM (I'm asking yusra to coordinate, though it may not be her) should try running jointcal on the DC2 data on lsst-dev with the option from DM-17917 enabled, and then try to reproduce boutigny's plots on the results of that processing.

            Thinking about this a bit further now, it's not totally clear to me that the DM-17917 code can be used with jointcal to select only stars from the reference catalog without limiting jointcal to using only reference catalog objects entirely.  erykoff or Parejkoj, can you comment on that?

            jbosch Jim Bosch added a comment - Recording discussion from the DESC-DM-DC2 discussion of 2019-02-26: boutigny has tried something he believes is equivalent to enabling the new use-only-stars-from-reference-catalog option added by erykoff on DM-17917 , and saw no significant change.  He would like this to be checked more carefully. Someone from DM (I'm asking yusra to coordinate, though it may not be her) should try running jointcal on the DC2 data on lsst-dev with the option from DM-17917 enabled, and then try to reproduce boutigny 's plots on the results of that processing. Thinking about this a bit further now, it's not totally clear to me that the DM-17917 code can be used with jointcal to select only stars from the reference catalog without limiting jointcal to using only reference catalog objects entirely.  erykoff or Parejkoj , can you comment on that?
            Parejkoj John Parejko added a comment -

            I think we might have to implement DM-13053, before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?

            Parejkoj John Parejko added a comment - I think we might have to implement DM-13053 , before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?
            jbosch Jim Bosch added a comment -

            I think we might have to implement DM-13053, before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?

            Yes, please.  (I'm just taking your word that it's what's needed to try what I've said want to try, but I'm happy to do that unless you think I need to understand the details).

             

            jbosch Jim Bosch added a comment - I think we might have to implement DM-13053 , before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket? Yes, please.  (I'm just taking your word that it's what's needed to try what I've said want to try, but I'm happy to do that unless you think I need to understand the details).  
            erykoff Eli Rykoff added a comment -

            So the fix in DM-17917 applies to astrometryTask in processCcd only. What John said that DM-13053 needs to be done for selecting stars from the reference catalog (or doing any refcat selections) in jointcal.

            erykoff Eli Rykoff added a comment - So the fix in DM-17917 applies to astrometryTask in processCcd only. What John said that DM-13053 needs to be done for selecting stars from the reference catalog (or doing any refcat selections) in jointcal .

            This just came up on the DC2 phonecon.  I am not all that surprised that jointcal doesn't do better than processCcd given a noise-free reference catalogue to 23rd magnitude – all that jointcal can do is bootstrap a deeper catalogue, and there's not much need under the circumstances.   If I'm misinformed about the reference catalogue that we're using please forgive this note.

            I am still very concerned about a 20mmag scatter, though.  Is this still present after all the work discussed on this ticket?

            rhl Robert Lupton added a comment - This just came up on the DC2 phonecon.  I am not all that surprised that jointcal doesn't do better than processCcd given a noise-free reference catalogue to 23rd magnitude – all that jointcal can do is bootstrap a deeper catalogue, and there's not much need under the circumstances.   If I'm misinformed about the reference catalogue that we're using please forgive this note. I am still very concerned about a 20mmag scatter, though.  Is this still present after all the work discussed on this ticket?
            jchiang James Chiang added a comment -

            > I am still very concerned about a 20mmag scatter, though. Is this still present after all the work discussed on this ticket?

             

            Using 7 overlapping visits of Run2.1i data, jointcal in w_2019_09, and a stars-only reference catalog, I get the same ~20mmag scatter:

            jchiang James Chiang added a comment - > I am still very concerned about a 20mmag scatter, though. Is this still present after all the work discussed on this ticket?   Using 7 overlapping visits of Run2.1i data, jointcal in w_2019_09, and a stars-only reference catalog, I get the same ~20mmag scatter:
            erykoff Eli Rykoff added a comment -

            So processCcd (photoCal precisely) can only do one zeropoint per ccd. In general, jointcal should be able to do higher order fits in the constrained (vs simple) models, though I'm not sure if jchiang's recent run uses the higher order stuff (I know he was building up to getting that working).

            erykoff Eli Rykoff added a comment - So processCcd ( photoCal precisely) can only do one zeropoint per ccd. In general, jointcal should be able to do higher order fits in the constrained (vs simple) models, though I'm not sure if jchiang 's recent run uses the higher order stuff (I know he was building up to getting that working).
            jchiang James Chiang added a comment -

            Here's what I used:

            config.photometryRejectBadFluxes=False
             
            config.astrometryModel="constrained"
            config.astrometryChipOrder=1
            config.astrometryVisitOrder=3
             
            config.photometryModel="simpleMagnitude" 
            

            I guess I need to use "constrainedMagnitude" for the photometryModel and dial up photometryVisitOrder for which the default is 7?

            jchiang James Chiang added a comment - Here's what I used: config.photometryRejectBadFluxes=False   config.astrometryModel= "constrained" config.astrometryChipOrder= 1 config.astrometryVisitOrder= 3   config.photometryModel= "simpleMagnitude" I guess I need to use "constrainedMagnitude" for the photometryModel and dial up photometryVisitOrder for which the default is 7?
            Parejkoj John Parejko added a comment -

            Try running jointcal with the defaults: that will use the constrained model for both astrometry and photometry, with a 5th and 7th order polynomial, respectively. What version are you running?

            The "simple" models are just extensions of the processCcd model (one model per ccd/visit: 2d polynomial for astrometry, one calibration factor for photometry). They are primarily useful for debugging or fitting non-mosaic camera data.

            Parejkoj John Parejko added a comment - Try running jointcal with the defaults: that will use the constrained model for both astrometry and photometry, with a 5th and 7th order polynomial, respectively. What version are you running? The "simple" models are just extensions of the processCcd model (one model per ccd/visit: 2d polynomial for astrometry, one calibration factor for photometry). They are primarily useful for debugging or fitting non-mosaic camera data.

            We should also keep in mind that the astrometry fit is not giving very good results either. Running `jointcal` with default configuration I get an astrometric dispersion = 11.4 mas (mag<21) to be compared to 7.9 mas with the processCcd astrometry (which is also not particularly good given an almost perfect reference catalog).
            I think that we cannot exclude a problem in the reference catalog itself. I am excluding a problem in the simulation as I got similar results with phosim.  

            boutigny Dominique Boutigny added a comment - We should also keep in mind that the astrometry fit is not giving very good results either. Running `jointcal` with default configuration I get an astrometric dispersion = 11.4 mas (mag<21) to be compared to 7.9 mas with the processCcd astrometry (which is also not particularly good given an almost perfect reference catalog). I think that we cannot exclude a problem in the reference catalog itself. I am excluding a problem in the simulation as I got similar results with phosim.  
            Parejkoj John Parejko added a comment - - edited

            Based on the discussion on slack, it appears to me that the reason jointcal doesn't do better than processCcd on this imsim data, is that there is nothing for it to fit beyond processCcd's model. Quoting myself from slack:

            If there are no throughput variations due to flat fielding, optics or atmosphere transparency, then there's really no way for jointcal (single value per sensor+focal plane polynomial per visit) to do better than processCcd (single value per sensor/visit).

            Parejkoj John Parejko added a comment - - edited Based on the discussion on slack , it appears to me that the reason jointcal doesn't do better than processCcd on this imsim data, is that there is nothing for it to fit beyond processCcd's model. Quoting myself from slack: If there are no throughput variations due to flat fielding, optics or atmosphere transparency, then there's really no way for jointcal (single value per sensor+focal plane polynomial per visit) to do better than processCcd (single value per sensor/visit).
            Parejkoj John Parejko added a comment -

            Can we close this as Invalid or Won't Fix, due to DC2 not having any photometric distortions for jointcal to model, and the known issues with the refcat?

            Parejkoj John Parejko added a comment - Can we close this as Invalid or Won't Fix, due to DC2 not having any photometric distortions for jointcal to model, and the known issues with the refcat?

            Yes.

            swinbank John Swinbank added a comment - Yes.

            People

              Unassigned Unassigned
              boutigny Dominique Boutigny
              Dominique Boutigny, Eli Rykoff, James Chiang, Jim Bosch, John Parejko, John Swinbank, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.