Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18016

Test fgcmcal reference stars on S18a

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Sprint:
      DRP S19-3, DRP S19-4, DRP S19-5, DRP S19-6a, DRP S19-6b
    • Team:
      Data Release Production

      Description

      When testing adding reference stars to fgcmcal, use the HSC S18a dataset to check that things are working and investigate questions of color terms / signal-to-noise / etc from the PS1 reference catalog.  Furthermore, this exercises the cross-calibration of the HSC r/r2 and i/i2 filters, and further checks for bimodality issues in the solution as first uncovered in DM-15895.

       

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            Using the reference stars has been extremely instructive on figuring out why my previous fits (without reference stars) have not been as successful as I had hoped. There are two issues going on.
            First: the observation strategy of HSC is not optimized for self-calibration. Generally, on a single night a given region of sky will be observed in a single filter. This means that while my code assumes that repeated observations of a star are sampling different atmospheres, this is not typically the case. Therefore, I believe the fitter is unable to get past the noise of all the same-night stars to the useful stars that are observed across multiple nights. I have some plans to try to rectify this.
            Second: the FGCM instrument model assumes that in between mirror washes (more common on Blanco than Subaru) and/or coatings that the transmission declines at a constant rate with no band dependence. While this seems to be the case for Blanco during the DES survey, this is definitely not the case for Subaru/HSC (see below). So I have to modify how I fit the instrumental degradation to take this into account.
            The following plots show the comparison between the individual observations in s18a and the reference stars (for bright reference stars). All individual observations have been adjusted to "top-of-the-atmosphere" magnitudes given the airmass and the standard atmosphere. Only the simplest non-photometric cut has been applied at this point, but the global trends are evident.

            First thing to note is the vertical red dashed line. This is the date of the mirror recoating (https://www.naoj.org/Observing/Telescope/Parameters/Reflectivity/) on October 20, 2017. (I hadn't previously it was recoated in the span of the S18a observations.)
            Second is that the degradation slope is not the same in the g, r, and i bands.
            Third is that there was something that happened between ~800-1000 days on the plot, or roughly May-October 2016, that accelerated the degradation. This is most prominent in the g band, but squinting I see the same pattern in the r and i bands as well.
            Perhaps coincidentally, this was the time of a new eruption of Kilauea.

            Show
            erykoff Eli Rykoff added a comment - Using the reference stars has been extremely instructive on figuring out why my previous fits (without reference stars) have not been as successful as I had hoped. There are two issues going on. First: the observation strategy of HSC is not optimized for self-calibration. Generally, on a single night a given region of sky will be observed in a single filter. This means that while my code assumes that repeated observations of a star are sampling different atmospheres, this is not typically the case. Therefore, I believe the fitter is unable to get past the noise of all the same-night stars to the useful stars that are observed across multiple nights. I have some plans to try to rectify this. Second: the FGCM instrument model assumes that in between mirror washes (more common on Blanco than Subaru) and/or coatings that the transmission declines at a constant rate with no band dependence. While this seems to be the case for Blanco during the DES survey, this is definitely not the case for Subaru/HSC (see below). So I have to modify how I fit the instrumental degradation to take this into account. The following plots show the comparison between the individual observations in s18a and the reference stars (for bright reference stars). All individual observations have been adjusted to "top-of-the-atmosphere" magnitudes given the airmass and the standard atmosphere. Only the simplest non-photometric cut has been applied at this point, but the global trends are evident. First thing to note is the vertical red dashed line. This is the date of the mirror recoating ( https://www.naoj.org/Observing/Telescope/Parameters/Reflectivity/ ) on October 20, 2017. (I hadn't previously it was recoated in the span of the S18a observations.) Second is that the degradation slope is not the same in the g, r, and i bands. Third is that there was something that happened between ~800-1000 days on the plot, or roughly May-October 2016, that accelerated the degradation. This is most prominent in the g band, but squinting I see the same pattern in the r and i bands as well. Perhaps coincidentally, this was the time of a new eruption of Kilauea.
            Hide
            erykoff Eli Rykoff added a comment - - edited

            Much progress has been made!

            Key improvements include: better understanding of the instrument (as noted above); fix the derivative math, correctly this time (DM-18014); and fix a typo in the derivative computation.

            The fit is now much better behaved, and this makes a big difference in all metrics: repeatability; chromatic consistence; uniformity (in r-band compared to Gaia as a baseline).

            First, an update to the instrumental plots above. With the latest fixes, I have better estimates of these "raw" plots, and this shows the form of the mirror much clearer:

            Vertical lines show my "mirror intervals", where I have determined (by eye) that the rate of decline changed; the right-most line is the date of the mirror recoating.

            I have done 3 separate runs for comparing repeatability stats and to compare to Gaia DR2 G-band (against HSC r-band, with stars with 0.5<g-i<1.5 and a random-forest generated color correction). These 3 runs are with no reference stars (noref); PS1 stars with s/n > 10 (ref10), and PS1 stars with s/n > 50 (ref50). PS1 stars were used with the default color corrections. In all cases HSC-R was corrected to HSC-R2 and HSC-I was corrected to HSC-I2 using the FGCM-derived transformations from the transmission curve information.

            There are a few stats to compare.

            First: internal repeatability of bright stars (s/n>100), after accounting for photometric error. All quantities are mmag rms for g/r/i/z/y.

            • ref50: 5.4/5.1/5.0/4.2/4.5
            • ref10: 7.4/6.6/5.8/5.3/6.9
            • noref: 4.5/4.3/3.5/3.0/3.4
              For this test, ref50 is clearly better than ref10 (the lower s/n stars are very noisy in the refcat and show some systematics as well). Interesting noref is better still, but from plots below it might be "overfitting" locally because the global uniformity isn't as good.

            Second: Consistency with the reference stars. All quantities are mmag rms:

            • ref50: 21.5/21.0/20.2/17.5/21.7
            • ref10: 38.3/33.1/27.9/26.0/37.2
              (I don't have a measurement for the noref case because of the way the code is structured; I will return to this at a later date).
              Here it's clear that the ref50 is much better, well beyond what the additional photometric errors one would expect from going s/n 50 to 10.

            Third: Consistency with Gaia G-band. This is the pixel-to-pixel rms over the full footprint, measured in nside=128 pixels. All quantities are mmag. Also, plots!

            • ref50: 1.9
            • ref10: 2.2
            • noref: 3.0
            • noref, before latest fixes: 5.4

            The fixes have made a huge improvement in the uniformity even without any reference stars. I'm very pleased that I can tie these disparate fields together so well! But there's definitely information to be gained from the reference stars especially at large scales.
            I'm going to try a run with s/n>100 to see what happens.

            Show
            erykoff Eli Rykoff added a comment - - edited Much progress has been made! Key improvements include: better understanding of the instrument (as noted above); fix the derivative math, correctly this time ( DM-18014 ); and fix a typo in the derivative computation. The fit is now much better behaved, and this makes a big difference in all metrics: repeatability; chromatic consistence; uniformity (in r-band compared to Gaia as a baseline). First, an update to the instrumental plots above. With the latest fixes, I have better estimates of these "raw" plots, and this shows the form of the mirror much clearer: Vertical lines show my "mirror intervals", where I have determined (by eye) that the rate of decline changed; the right-most line is the date of the mirror recoating. I have done 3 separate runs for comparing repeatability stats and to compare to Gaia DR2 G-band (against HSC r-band, with stars with 0.5<g-i<1.5 and a random-forest generated color correction). These 3 runs are with no reference stars ( noref ); PS1 stars with s/n > 10 ( ref10 ), and PS1 stars with s/n > 50 ( ref50 ). PS1 stars were used with the default color corrections. In all cases HSC-R was corrected to HSC-R2 and HSC-I was corrected to HSC-I2 using the FGCM-derived transformations from the transmission curve information. There are a few stats to compare. First: internal repeatability of bright stars (s/n>100), after accounting for photometric error. All quantities are mmag rms for g/r/i/z/y . ref50 : 5.4/5.1/5.0/4.2/4.5 ref10 : 7.4/6.6/5.8/5.3/6.9 noref : 4.5/4.3/3.5/3.0/3.4 For this test, ref50 is clearly better than ref10 (the lower s/n stars are very noisy in the refcat and show some systematics as well). Interesting noref is better still, but from plots below it might be "overfitting" locally because the global uniformity isn't as good. Second: Consistency with the reference stars. All quantities are mmag rms: ref50 : 21.5/21.0/20.2/17.5/21.7 ref10 : 38.3/33.1/27.9/26.0/37.2 (I don't have a measurement for the noref case because of the way the code is structured; I will return to this at a later date). Here it's clear that the ref50 is much better, well beyond what the additional photometric errors one would expect from going s/n 50 to 10. Third: Consistency with Gaia G-band. This is the pixel-to-pixel rms over the full footprint, measured in nside=128 pixels. All quantities are mmag. Also, plots! ref50 : 1.9 ref10 : 2.2 noref : 3.0 noref , before latest fixes: 5.4 The fixes have made a huge improvement in the uniformity even without any reference stars. I'm very pleased that I can tie these disparate fields together so well! But there's definitely information to be gained from the reference stars especially at large scales. I'm going to try a run with s/n>100 to see what happens.
            Hide
            erykoff Eli Rykoff added a comment -

            I have done the ref100 run and I find:
            Internal reproducibility: 4.8/4.5/4.3/3.4/3.9 mmag rms
            Consistency with reference stars: 18.6/17.3/16.6/14.2/20.5 mmag rms
            Uniformity in r-band with Gaia: 2.1 mmag rms

            I think this looks the best to me, in that the internal reproducibility is significantly improved, and the uniformity is also very good. So I'm going to recommend these settings as the defaults.

            Show
            erykoff Eli Rykoff added a comment - I have done the ref100 run and I find: Internal reproducibility: 4.8/4.5/4.3/3.4/3.9 mmag rms Consistency with reference stars: 18.6/17.3/16.6/14.2/20.5 mmag rms Uniformity in r-band with Gaia: 2.1 mmag rms I think this looks the best to me, in that the internal reproducibility is significantly improved, and the uniformity is also very good. So I'm going to recommend these settings as the defaults.
            Hide
            erykoff Eli Rykoff added a comment -

            I've now added some slides summarizing the most relevant plots: hsc_s18a_FGCM.pdf

            Show
            erykoff Eli Rykoff added a comment - I've now added some slides summarizing the most relevant plots: hsc_s18a_FGCM.pdf

              People

              • Assignee:
                erykoff Eli Rykoff
                Reporter:
                erykoff Eli Rykoff
                Watchers:
                Eli Rykoff
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel