Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34391

Change coadd defaults to use "finalized" psf models.

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: drp_pipe, pipe_tasks
    • Labels:
      None

      Description

      When the DRP reprocessing schedule says it is ready, we can turn on the application of the finalized psf in pipe_tasks/drp_pipe by default.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Heh...you tell me! It does seem like things are slightly worse on your branch, but this is a very small dataset and not the most "diagnostic" of plots for PSF model quality.

            Indeed, the deepCoadd_calexps differences look small to me (but I don't have a good feel for how much difference would be expected here).

            Show
            lauren Lauren MacArthur added a comment - Heh...you tell me! It does seem like things are slightly worse on your branch, but this is a very small dataset and not the most "diagnostic" of plots for PSF model quality. Indeed, the deepCoadd_calexps differences look small to me (but I don't have a good feel for how much difference would be expected here).
            Hide
            lauren Lauren MacArthur added a comment -

            Ok, for my next test, I ran a “mini RC2 DRP” step1 through step3 selecting only a subset with:

            dataQuery: "instrument='HSC' AND physical_filter IN ('HSC-G', 'HSC-R', 'HSC-I') AND visit IN (11690, 11692, 11694, 11696, 11698, 11700, 11702, 11704, 11706, 11708, 11710, 11712, 1202, 1204, 1206, 1208, 1210, 1212, 1214, 1216, 1218, 1220, 23692, 23694, 23704, 23706, 23716, 23718, 1228, 1230, 1232, 1238, 1240, 1242, 1244, 1246, 1248) AND skymap='hsc_rings_v1'"
            

            After Yusra AlSayyad found and fixed a bug in the PSF ellipticity & size residual scripts that expected all 5 bands to be present, all the coadd plots from the analysis_drp pipelines were generated. While these cannot be compared directly to the full RC2 plots, any glaring issues can be gleaned. As an example, the following compare the two for PSF size residuals:

            Full RC2 for w_202216:

            Mini RC2 with this ticket:

            I don’t see any obvious problems, so am taking this as a decent validation of the new PSF models.

            On the other hand, the stellar locus plots did look worse on the mini-RC2 run, but it may well be that fgcm - as configured for the full RC2 - won’t work properly in this context, so I’m not sure if the stellar locus being worse is unexpected (Eli Rykoff will confirm!).

            As such I am marking the PRs as approved, but am hoping to get formal approval from Yusra AlSayyad that she is on board with merging this on time for the weekly cut tonight (which is the basis for the next DC2 processing run).

            Show
            lauren Lauren MacArthur added a comment - Ok, for my next test, I ran a “mini RC2 DRP” step1 through step3 selecting only a subset with: dataQuery: "instrument='HSC' AND physical_filter IN ('HSC-G', 'HSC-R', 'HSC-I') AND visit IN (11690, 11692, 11694, 11696, 11698, 11700, 11702, 11704, 11706, 11708, 11710, 11712, 1202, 1204, 1206, 1208, 1210, 1212, 1214, 1216, 1218, 1220, 23692, 23694, 23704, 23706, 23716, 23718, 1228, 1230, 1232, 1238, 1240, 1242, 1244, 1246, 1248) AND skymap='hsc_rings_v1'" After Yusra AlSayyad found and fixed a bug in the PSF ellipticity & size residual scripts that expected all 5 bands to be present, all the coadd plots from the analysis_drp pipelines were generated. While these cannot be compared directly to the full RC2 plots, any glaring issues can be gleaned. As an example, the following compare the two for PSF size residuals: Full RC2 for w_202216 : Mini RC2 with this ticket: I don’t see any obvious problems, so am taking this as a decent validation of the new PSF models. On the other hand, the stellar locus plots did look worse on the mini-RC2 run, but it may well be that fgcm - as configured for the full RC2 - won’t work properly in this context, so I’m not sure if the stellar locus being worse is unexpected ( Eli Rykoff will confirm!). As such I am marking the PRs as approved, but am hoping to get formal approval from Yusra AlSayyad that she is on board with merging this on time for the weekly cut tonight (which is the basis for the next DC2 processing run).
            Hide
            yusra Yusra AlSayyad added a comment -

            For posterity and future inspection, the coadd-level plots for w_2022_16 vs DM-34391 were written out into:

            browse via e.g.
            ssh -L 9999:127.0.0.1:5678 lsst-devl01 and nav to http://localhost:9999/plot-navigator/dashboard_gen3

            Scatter is higher on the elliptically difference plots, but lower on the shape size plots. (it's kind of hard to tell by blinking because the y-axis limits change) but for example:

            Show
            yusra Yusra AlSayyad added a comment - For posterity and future inspection, the coadd-level plots for w_2022_16 vs DM-34391 were written out into: u/yusra/coadd/RC2/w_2022_16/ DM-34451 u/yusra/coadd/RC2/ DM-34391 -debug browse via e.g. ssh -L 9999:127.0.0.1:5678 lsst-devl01 and nav to http://localhost:9999/plot-navigator/dashboard_gen3 Scatter is higher on the elliptically difference plots, but lower on the shape size plots. (it's kind of hard to tell by blinking because the y-axis limits change) but for example:
            Hide
            yusra Yusra AlSayyad added a comment -

            Nothing catastrophic though, so probably safe to merge.  Overall I'm fine either way: merging tonight or waiting 2 weeks. 

             

            Show
            yusra Yusra AlSayyad added a comment - Nothing catastrophic though, so probably safe to merge.  Overall I'm fine either way: merging tonight or waiting 2 weeks.   
            Hide
            lauren Lauren MacArthur added a comment -

            Ok…your choice Eli!

            Show
            lauren Lauren MacArthur added a comment - Ok…your choice Eli!

              People

              Assignee:
              erykoff Eli Rykoff
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Lauren MacArthur
              Watchers:
              Eli Rykoff, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.