Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27361

Insert flat_models Fake Sources Provided by the LSST:UK LSB WG Into Visit-Level Imaging

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 9
    • Data Release Production
    • No

    Description

      This ticket injects the fake sources specified in the flat_models.fits catalogue provided by the LSST:UK LSB WG into visit-level imaging, propagating fake sources through to the final deepCoadd_calexp stage. In summary, we've injected 144 fake sources into all 149 visits x 103 CCDs covering the COSMOS tract 9813. The following data processing steps have taken place:

      1. Generate tract, patch, visit and CCD lists
      2. Run jointCal.py to generate updated calibs
      3. Run processCcdWithFakes.py to inject fakes into calexps
      4. Run skyCorrection.py to generate full focal plane sky maps
      5. Run coaddDriver.py to mosaic visit-level calexps into coadds
      6. Run multiBandDriver.py to generate final output catalogues

      The plots attached to this ticket show example outputs for data within tract 9813, either patch 3,5 (for deepCoadd level imaging) or visit 1166 CCD 12 (for visit-level imaging). Additionally, the original flat_models.fits input catalogue and a plot showing the distrubution of input Sersic values within this table is also attached to this ticket. The full output processing can be found on the RSP here: /home/lskelvin/fakes_lsstuk/output/flat_models_sfm. Comments below provide more detail on the analysis of these data.

      Attachments

        1. fakes_inputValues.png
          fakes_inputValues.png
          114 kB
        2. flat_models.fits
          31 kB
        3. t9813p35i-deepCoadd_calexp_diff.png
          t9813p35i-deepCoadd_calexp_diff.png
          1.13 MB
        4. t9813p35i-deepCoadd_calexp.png
          t9813p35i-deepCoadd_calexp.png
          982 kB
        5. t9813p73i-deepCoadd_calexp_diff.png
          t9813p73i-deepCoadd_calexp_diff.png
          1.11 MB
        6. t9813p73i-deepCoadd_calexp.png
          t9813p73i-deepCoadd_calexp.png
          942 kB
        7. t9813v1166c012-calexp_detmasks.png
          t9813v1166c012-calexp_detmasks.png
          1.67 MB
        8. t9813v1166c012-calexp_diffs.png
          t9813v1166c012-calexp_diffs.png
          84 kB
        9. t9813v1166c012-calexp_masks.png
          t9813v1166c012-calexp_masks.png
          1.42 MB
        10. t9813v1166c012-calexp_stripzooms.png
          t9813v1166c012-calexp_stripzooms.png
          1.15 MB
        11. t9813v1166c012-calexps.png
          t9813v1166c012-calexps.png
          1.63 MB
        12. t9813v1166-fp.png
          t9813v1166-fp.png
          1.08 MB
        13. v1166-calexp.png
          v1166-calexp.png
          545 kB
        14. v1166-diff.png
          v1166-diff.png
          78 kB
        15. v1166-fakes_calexp.png
          v1166-fakes_calexp.png
          543 kB
        16. v1166-skyCorr.png
          v1166-skyCorr.png
          493 kB

        Issue Links

          Activity

            lskelvin Lee Kelvin added a comment - - edited

            Visit 1166 CCD 12 comparison strips


            This figure shows a zoomed-in strip from the original calexp (top panel), fakes_calexp (middle panel), and detection mask difference (lower panel). As discussed above, here we see the impact of the undetected fake source (at ~[500,1700] in pixel coordinates) on the footprints of existing true sources, artifically expanding them beyond their otherwise 'true' footprint.

            lskelvin Lee Kelvin added a comment - - edited Visit 1166 CCD 12 comparison strips This figure shows a zoomed-in strip from the original calexp (top panel), fakes_calexp (middle panel), and detection mask difference (lower panel). As discussed above, here we see the impact of the undetected fake source (at ~ [500,1700] in pixel coordinates) on the footprints of existing true sources, artifically expanding them beyond their otherwise 'true' footprint.
            lskelvin Lee Kelvin added a comment - - edited

            Tract 9813 Patch 3,5 deepCoadd calexp comparison plots



            These two figures show the deepCoadd_calexp data for tract 9813 patch 3,5. Three distinct panels are shown: the vanilla deepCoadd_calexp (repeated in each image in the left panel), the fakes_deepCoadd_calexp, and the difference image. The fake sources are clearly evident, however, what looks like background oversubtraction around the fake sources is evident as a slightly negative 'halo' (note that the colour bar goes from white for low values to black for high values, and is logarithmically scaled to emphasise this negative offset). In addition, in the difference image, all the existing 'true' sources are also still clearly evident.

            On closer inspection of these data, it was discovered that differing photometric solutions between the vanilla processing and the fakes processing was the root cause. Vanilla processing uses FGCM to provide global photometric calibrations (see Burke, Rykoff et al. 2018 for more details). Fakes processing however formerly used the older legacy jointcal to provide these solutions, and it was not possible to use FGCM. This discovery, and the desire to bring processCcdWithFakes in line with processCcd, led to ticket DM-26999 - a ticket which implements FGCM within the fake source injection routines. This update successfully resolves this issue, and will not impact future fakes processing.

            lskelvin Lee Kelvin added a comment - - edited Tract 9813 Patch 3,5 deepCoadd calexp comparison plots These two figures show the deepCoadd_calexp data for tract 9813 patch 3,5. Three distinct panels are shown: the vanilla deepCoadd_calexp (repeated in each image in the left panel), the fakes_deepCoadd_calexp, and the difference image. The fake sources are clearly evident, however, what looks like background oversubtraction around the fake sources is evident as a slightly negative 'halo' (note that the colour bar goes from white for low values to black for high values, and is logarithmically scaled to emphasise this negative offset). In addition, in the difference image, all the existing 'true' sources are also still clearly evident. On closer inspection of these data, it was discovered that differing photometric solutions between the vanilla processing and the fakes processing was the root cause. Vanilla processing uses FGCM to provide global photometric calibrations (see Burke, Rykoff et al. 2018 for more details). Fakes processing however formerly used the older legacy jointcal to provide these solutions, and it was not possible to use FGCM. This discovery, and the desire to bring processCcdWithFakes in line with processCcd, led to ticket DM-26999 - a ticket which implements FGCM within the fake source injection routines. This update successfully resolves this issue, and will not impact future fakes processing.
            lskelvin Lee Kelvin added a comment - - edited

            Tract 9813 Patch 7,3 deepCoadd calexp comparison plots



            The figures shown here are as above, but for tract 9813 patch 7,3. As highlighted above, slight negative haloes are visible around fake injected sources, and existing 'true' sources are still present in the difference image. Both of these issues have been addressed with ticket DM-26999, and therefore will be resolved for all future fakes processing.

            lskelvin Lee Kelvin added a comment - - edited Tract 9813 Patch 7,3 deepCoadd calexp comparison plots The figures shown here are as above, but for tract 9813 patch 7,3. As highlighted above, slight negative haloes are visible around fake injected sources, and existing 'true' sources are still present in the difference image. Both of these issues have been addressed with ticket DM-26999 , and therefore will be resolved for all future fakes processing.
            dtaranu Dan Taranu added a comment -

            Looks good. My only scientific comment was that the choice of fake parameters seems slightly odd, but I see that has already been addressed for the next run.

            Your home directory isn't group-readable on lsst-devl even though it is on the RSP. Rather than make it readable, though, I would suggest just moving the repo to /project.

            dtaranu Dan Taranu added a comment - Looks good. My only scientific comment was that the choice of fake parameters seems slightly odd, but I see that has already been addressed for the next run. Your home directory isn't group-readable on lsst-devl even though it is on the RSP. Rather than make it readable, though, I would suggest just moving the repo to /project.
            lskelvin Lee Kelvin added a comment -

            Thanks for the feedback on this Dan, much appreciated. I've contacted the LSST:UK LSB side to work out how they would like to access these data going forward. As you mention, the next run will use an updated set of input fake paremeters, designed to more accurately capture the full range of source profile types. Thanks again for these comments, very useful, cheers.

            lskelvin Lee Kelvin added a comment - Thanks for the feedback on this Dan, much appreciated. I've contacted the LSST:UK LSB side to work out how they would like to access these data going forward. As you mention, the next run will use an updated set of input fake paremeters, designed to more accurately capture the full range of source profile types. Thanks again for these comments, very useful, cheers.

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              Dan Taranu
              Dan Taranu, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.