Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4730

Adapt qa analysis script for LSST vs. HSC single visit processing comparison

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      The qa analysis script ported from HSC and adapted to LSST on DM-4393 currently performs qa on single visit processing by comparing outputs of a single run from the different measurement algorithms and comparing those with the astrometry/photemetry reference catalog. Here we will add functionality to directly compare two different runs of the same dataset (requiring accommodations for two butlers). Since the goal is to compare outputs from runs on the LSST vs. HSC stacks, this will require a mapping of the different schemas of the persisted source catalogs of the two stacks.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Ok, great – I think you can mark this as done. Thanks!

            Show
            swinbank John Swinbank added a comment - Ok, great – I think you can mark this as done. Thanks!
            Hide
            lauren Lauren MacArthur added a comment -

            Yes, your summary seems accurate to me (and any additional views or removal of anything we end up deeming redundant/uninformative can always be accommodated). The only thing I'd add is that the LSST-only plots can also be generated for HSC-only runs.

            Show
            lauren Lauren MacArthur added a comment - Yes, your summary seems accurate to me (and any additional views or removal of anything we end up deeming redundant/uninformative can always be accommodated). The only thing I'd add is that the LSST-only plots can also be generated for HSC-only runs.
            Hide
            swinbank John Swinbank added a comment -

            Before signing off on this, I'd like to get a feeling for what exactly it is we're producing other than "lots and lots of plots". By a combination of squinting at the code and the example (and reading Lauren MacArthur's comment above), I think we've got the following:

            Match sources between LSST and HSC stack processing of the same data. For each of PSF, Gaussian, Kron and CModel fluxes (if they exist in the schema) calculate the difference in magnitude between the LSST and HSC results. Plot these differences as:

            • A histogram;
            • A scatter plot of difference against PSF magnitude;
            • A scatter plot of difference against RA;
            • A scatter plot of difference against dec;
            • A plot showing difference as a function of position in the focal plane.

            We also plot a series of figures showing the absolute value of LSST results (ie, not the difference relative to HSC):

            • The difference between each of (Gaussian, Kron and CModel magnitude) and PSF magnitude as a function of:
              • Position on CCD;
              • Distribution on focal plane;
              • PSF magnitude;
              • RA;
              • dec;
              • Position on the sky.
            • A histogram of the difference between each of (Gaussian, Kron and CModel magnitude) and PSF magnitude.

            Finally, for each source in the LSST results matched against the reference catalogue, we plot the difference between the measured and catalogue sources of each of:

            • RA;
            • dec;
            • Magnitude;
            • Position.

            We plot those differences as a function of:

            • Position on CCD;
            • Position on focal plane;
            • PSF magnitude;
            • RA;
            • dec;
            • Position on sky.

            Lauren MacArthur, does that sound plausible? Is there anything I'm missing or incorrectly claiming? If not, I'm happy to sign off on this.

            Show
            swinbank John Swinbank added a comment - Before signing off on this, I'd like to get a feeling for what exactly it is we're producing other than "lots and lots of plots". By a combination of squinting at the code and the example (and reading Lauren MacArthur 's comment above), I think we've got the following: Match sources between LSST and HSC stack processing of the same data. For each of PSF, Gaussian, Kron and CModel fluxes (if they exist in the schema) calculate the difference in magnitude between the LSST and HSC results. Plot these differences as: A histogram; A scatter plot of difference against PSF magnitude; A scatter plot of difference against RA; A scatter plot of difference against dec; A plot showing difference as a function of position in the focal plane. We also plot a series of figures showing the absolute value of LSST results (ie, not the difference relative to HSC): The difference between each of (Gaussian, Kron and CModel magnitude) and PSF magnitude as a function of: Position on CCD; Distribution on focal plane; PSF magnitude; RA; dec; Position on the sky. A histogram of the difference between each of (Gaussian, Kron and CModel magnitude) and PSF magnitude. Finally, for each source in the LSST results matched against the reference catalogue, we plot the difference between the measured and catalogue sources of each of: RA; dec; Magnitude; Position. We plot those differences as a function of: Position on CCD; Position on focal plane; PSF magnitude; RA; dec; Position on sky. Lauren MacArthur , does that sound plausible? Is there anything I'm missing or incorrectly claiming? If not, I'm happy to sign off on this.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            John, would you mind giving this a look? The scripts can now accommodate inter-run comparisons, i.e. two different runs of the same dataset. The different runs can be either two different LSST stack runs, or between an LSST and HSC stack run. I attach some sample figures here for the latter using HSC commissioning data. These figures are telling us that we are doing reasonably well as far as having matched stacks (rms = 0.009 mag between Gaussian fluxes, for example). There are some systematics that will need to be tracked down in future tickets. For example, there is a 0.0166 mag offset in the zeropoint between the two stacks (it is systematic, i.e. no trend with magnitude) and some strange patterns appear in the "sky" view (i.e. certain CCDs are much worse than others). Causes will be tracked down (likely leading to additional diagnostic plots) in future tickets.

            As far as what gets plotted, there is a list of fluxes that gets looped through. If the given flux exists in the schema, it gets plotted. Currently, the list includes: ["base_PsfFlux", "base_GaussianFlux", "ext_photometryKron_KronFlux", "modelfit_Cmodel"]

            The analysis script and alias mapper between the LSST and HSC stacks are in u/lauren/DM-4730 branches in pipe_tasks and obs_subaru, respectively, for anyone who wants to have a look, but note that there is no intention of merging anything as of yet.

            Show
            lauren Lauren MacArthur added a comment - - edited John, would you mind giving this a look? The scripts can now accommodate inter-run comparisons, i.e. two different runs of the same dataset. The different runs can be either two different LSST stack runs, or between an LSST and HSC stack run. I attach some sample figures here for the latter using HSC commissioning data. These figures are telling us that we are doing reasonably well as far as having matched stacks (rms = 0.009 mag between Gaussian fluxes, for example). There are some systematics that will need to be tracked down in future tickets. For example, there is a 0.0166 mag offset in the zeropoint between the two stacks (it is systematic, i.e. no trend with magnitude) and some strange patterns appear in the "sky" view (i.e. certain CCDs are much worse than others). Causes will be tracked down (likely leading to additional diagnostic plots) in future tickets. As far as what gets plotted, there is a list of fluxes that gets looped through. If the given flux exists in the schema, it gets plotted. Currently, the list includes: ["base_PsfFlux", "base_GaussianFlux", "ext_photometryKron_KronFlux", "modelfit_Cmodel"] The analysis script and alias mapper between the LSST and HSC stacks are in u/lauren/ DM-4730 branches in pipe_tasks and obs_subaru , respectively, for anyone who wants to have a look, but note that there is no intention of merging anything as of yet.

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                John Swinbank
                Watchers:
                John Swinbank, Jonathan Sick, Lauren MacArthur, Michael Wood-Vasey
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel