Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34625

Add sky coverage plots of visit summary quantities to analysis_tools

    XMLWordPrintable

Details

    • 18
    • DRP S22B
    • Data Release Production
    • No

    Description

      As part of the DP0.2 V&V efforts, plots showing various "visit summary" quantities as a function of RA/Dec and focal plane coordinates were generated (see PREOPS-849). These are (small-ish) survey-level plots with a point per detector. The following are examples for the ~300 sq deg, 6 band, 19852 visit, 2804994 detector dataset that comprises the DP0.2 release (and probably represents the upper limit to which this granularity scales):

      This ticket is to get these plots added to analysis_drp.

      Attachments

        Issue Links

          Activity

            lauren Lauren MacArthur added a comment - - edited

            These can now make use of the ccdVisitTable datasets with the quantities added to those tables on DM-34630.

            lauren Lauren MacArthur added a comment - - edited These can now make use of the ccdVisitTable datasets with the quantities added to those tables on DM-34630 .
            sophiereed Sophie Reed added a comment -

            I added a few generic plot points in the validation team slack channel.

            I think there are two many panels on each plot, you can't read the labels or see the details without scrolling around on your screen, at which point you might as well have multiple plots. Maybe they should be divided by band?

            sophiereed Sophie Reed added a comment - I added a few generic plot points in the validation team slack channel. I think there are two many panels on each plot, you can't read the labels or see the details without scrolling around on your screen, at which point you might as well have multiple plots. Maybe they should be divided by band?
            lauren Lauren MacArthur added a comment - - edited

            Here are some examples based on your (sophiereed) suggestions on slack (namely to use the colorblind-friendly colormap and binning...I had to factor out that functionality from the skyPlot action and put it in plotUtils to allow reuse).  I'm still including all the panels as I want to ask a wider audience as to which groupings would be best and then we can lock those into pipelines (it's all configurable, so making multiple versions with specific band/parameter subsets is easy enough).

            lauren Lauren MacArthur added a comment - - edited Here are some examples based on your ( sophiereed ) suggestions on slack (namely to use the colorblind-friendly colormap and binning...I had to factor out that functionality from the skyPlot  action and put it in plotUtils  to allow reuse).  I'm still including all the panels as I want to ask a wider audience as to which groupings would be best and then we can lock those into pipelines (it's all configurable, so making multiple versions with specific band/parameter subsets is easy enough).

            I have tested this thoroughly on several different datasets (including HSC RC2, imsim/DC2, and LATISS collections) and altering config defaults using the attached notebook (note: there are a few things to edit if you want to run yourself, namely the details specifying the specific collection you want to look at and the output directory if you want to persist pngs of the plots). Various examples are attached.

            I am leaving adding an entry into out regularly run pipelines for another ticket as it will require a discussion on how we want these plots to be partitioned/named. As such, I also did not want to officially register any dataset names into our repos until that discussion has occurred, so I tested running the attached pipeline coveragePlots.yaml (again varying many of the config parameters) on a local ci_hsc_gen3 repo with the following command:

            $ pipetask --long-log run -d "instrument='HSC' AND skymap='discrete/ci_hsc'" -b /sdf/data/rubin/user/laurenma/LSST/ci_hsc_gen3/DATA --input HSC/runs/ci_hsc --output u/laurenma/DM-34625 -p /sdf/group/rubin/user/laurenma/tickets/DM-34625/coveragePlots.yaml#raDecCoveragePlots,focalPlaneCoveragePlots
            

            The resulting plots look like (note there are very few visits in ci_hsc_gen3!):



            lauren Lauren MacArthur added a comment - I have tested this thoroughly on several different datasets (including HSC RC2, imsim/DC2, and LATISS collections) and altering config defaults using the attached notebook (note: there are a few things to edit if you want to run yourself, namely the details specifying the specific collection you want to look at and the output directory if you want to persist pngs of the plots). Various examples are attached. I am leaving adding an entry into out regularly run pipelines for another ticket as it will require a discussion on how we want these plots to be partitioned/named. As such, I also did not want to officially register any dataset names into our repos until that discussion has occurred, so I tested running the attached pipeline coveragePlots.yaml (again varying many of the config parameters) on a local ci_hsc_gen3 repo with the following command: $ pipetask - - long - log run - d "instrument='HSC' AND skymap='discrete/ci_hsc'" - b / sdf / data / rubin / user / laurenma / LSST / ci_hsc_gen3 / DATA - - input HSC / runs / ci_hsc - - output u / laurenma / DM - 34625 - p / sdf / group / rubin / user / laurenma / tickets / DM - 34625 / coveragePlots.yaml #raDecCoveragePlots,focalPlaneCoveragePlots The resulting plots look like (note there are very few visits in ci_hsc_gen3 !):
            lauren Lauren MacArthur added a comment - - edited

            Would you mind giving this a look when you get a chance?

            Jenkins is: running green.

            lauren Lauren MacArthur added a comment - - edited Would you mind giving this a look when you get a chance? Jenkins is:  running green .

            Thanks for the review, Clare.  I addressed all your comments in the PR and reran Jenkins. Merged to main and done!

            lauren Lauren MacArthur added a comment - Thanks for the review, Clare.  I addressed all your comments in the PR and reran Jenkins . Merged to main and done!

            People

              lauren Lauren MacArthur
              lauren Lauren MacArthur
              Clare Saunders
              Clare Saunders, Lauren MacArthur, Sophie Reed, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.