Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34625

Add sky coverage plots of visit summary quantities to analysis_tools

    XMLWordPrintable

    Details

    • Story Points:
      18
    • Epic Link:
    • Sprint:
      DRP S22B
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As part of the DP0.2 V&V efforts, plots showing various "visit summary" quantities as a function of RA/Dec and focal plane coordinates were generated (see PREOPS-849). These are (small-ish) survey-level plots with a point per detector. The following are examples for the ~300 sq deg, 6 band, 19852 visit, 2804994 detector dataset that comprises the DP0.2 release (and probably represents the upper limit to which this granularity scales):

      This ticket is to get these plots added to analysis_drp.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            These can now make use of the ccdVisitTable datasets with the quantities added to those tables on DM-34630.

            Show
            lauren Lauren MacArthur added a comment - - edited These can now make use of the ccdVisitTable datasets with the quantities added to those tables on DM-34630 .
            Hide
            sophiereed Sophie Reed added a comment -

            I added a few generic plot points in the validation team slack channel.

            I think there are two many panels on each plot, you can't read the labels or see the details without scrolling around on your screen, at which point you might as well have multiple plots. Maybe they should be divided by band?

            Show
            sophiereed Sophie Reed added a comment - I added a few generic plot points in the validation team slack channel. I think there are two many panels on each plot, you can't read the labels or see the details without scrolling around on your screen, at which point you might as well have multiple plots. Maybe they should be divided by band?
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Here are some examples based on your (Sophie Reed) suggestions on slack (namely to use the colorblind-friendly colormap and binning...I had to factor out that functionality from the skyPlot action and put it in plotUtils to allow reuse).  I'm still including all the panels as I want to ask a wider audience as to which groupings would be best and then we can lock those into pipelines (it's all configurable, so making multiple versions with specific band/parameter subsets is easy enough).

            Show
            lauren Lauren MacArthur added a comment - - edited Here are some examples based on your ( Sophie Reed ) suggestions on slack (namely to use the colorblind-friendly colormap and binning...I had to factor out that functionality from the skyPlot  action and put it in plotUtils  to allow reuse).  I'm still including all the panels as I want to ask a wider audience as to which groupings would be best and then we can lock those into pipelines (it's all configurable, so making multiple versions with specific band/parameter subsets is easy enough).
            Hide
            lauren Lauren MacArthur added a comment -

            I have tested this thoroughly on several different datasets (including HSC RC2, imsim/DC2, and LATISS collections) and altering config defaults using the attached notebook (note: there are a few things to edit if you want to run yourself, namely the details specifying the specific collection you want to look at and the output directory if you want to persist pngs of the plots). Various examples are attached.

            I am leaving adding an entry into out regularly run pipelines for another ticket as it will require a discussion on how we want these plots to be partitioned/named. As such, I also did not want to officially register any dataset names into our repos until that discussion has occurred, so I tested running the attached pipeline coveragePlots.yaml (again varying many of the config parameters) on a local ci_hsc_gen3 repo with the following command:

            $ pipetask --long-log run -d "instrument='HSC' AND skymap='discrete/ci_hsc'" -b /sdf/data/rubin/user/laurenma/LSST/ci_hsc_gen3/DATA --input HSC/runs/ci_hsc --output u/laurenma/DM-34625 -p /sdf/group/rubin/user/laurenma/tickets/DM-34625/coveragePlots.yaml#raDecCoveragePlots,focalPlaneCoveragePlots
            

            The resulting plots look like (note there are very few visits in ci_hsc_gen3!):



            Show
            lauren Lauren MacArthur added a comment - I have tested this thoroughly on several different datasets (including HSC RC2, imsim/DC2, and LATISS collections) and altering config defaults using the attached notebook (note: there are a few things to edit if you want to run yourself, namely the details specifying the specific collection you want to look at and the output directory if you want to persist pngs of the plots). Various examples are attached. I am leaving adding an entry into out regularly run pipelines for another ticket as it will require a discussion on how we want these plots to be partitioned/named. As such, I also did not want to officially register any dataset names into our repos until that discussion has occurred, so I tested running the attached pipeline coveragePlots.yaml (again varying many of the config parameters) on a local ci_hsc_gen3 repo with the following command: $ pipetask - - long - log run - d "instrument='HSC' AND skymap='discrete/ci_hsc'" - b / sdf / data / rubin / user / laurenma / LSST / ci_hsc_gen3 / DATA - - input HSC / runs / ci_hsc - - output u / laurenma / DM - 34625 - p / sdf / group / rubin / user / laurenma / tickets / DM - 34625 / coveragePlots.yaml #raDecCoveragePlots,focalPlaneCoveragePlots The resulting plots look like (note there are very few visits in ci_hsc_gen3 !):
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Would you mind giving this a look when you get a chance?

            Jenkins is: running green.

            Show
            lauren Lauren MacArthur added a comment - - edited Would you mind giving this a look when you get a chance? Jenkins is:  running green .
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks for the review, Clare.  I addressed all your comments in the PR and reran Jenkins. Merged to main and done!

            Show
            lauren Lauren MacArthur added a comment - Thanks for the review, Clare.  I addressed all your comments in the PR and reran Jenkins . Merged to main and done!

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Clare Saunders
              Watchers:
              Clare Saunders, Lauren MacArthur, Sophie Reed, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.