Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13687

Make exploratory plots from HiTS processing

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP S18-4, AP S18-5
    • Team:
      Alert Production

      Description

      As part of LDM-503-3, we processed 3 fields of HiTS through ap_pipe and ap_verify.  The resulting test report (DM-12534) mostly examined whether the expected products were produced.  This ticket is to do some initial scientific validation of the processed data, which is located on lsst-dev at /scratch/ebellm/ap_verify_HiTS.

      In particular, this may include:

      • plotting histograms of the number of DIASources associated with each DIAObject
      • calculating the median magnitude and RMS for each DIAObject in one band, and plotting them against each other
      • plotting lightcurves for DIAObjects which are outliers on the above plots
      • examining cutouts for a few DIAObjects with only a handful of associated DIASources

       

        Attachments

          Activity

          Hide
          mrawls Meredith Rawls added a comment -

          I've made a notebook that walks through most of the suggested steps and explains the caveats of the current processed dataset. I did not look at median magnitudes yet, and while there are light curves, they are quite preliminary since retrieving timestamps turns out to be borderline impossible with the current databases. The notebook will be a good starting point for making more illustrative exploratory plots when we reprocess the dataset.

          The notebook lives on lsst-dev in /home/mrawls/notebooks. I can provide more detailed directions of how to access and run it over ssh if that would be useful, and I can move it to a different location on lsst-dev if desired.

          Show
          mrawls Meredith Rawls added a comment - I've made a notebook that walks through most of the suggested steps and explains the caveats of the current processed dataset. I did not look at median magnitudes yet, and while there are light curves, they are quite preliminary since retrieving timestamps turns out to be borderline impossible with the current databases. The notebook will be a good starting point for making more illustrative exploratory plots when we reprocess the dataset. The notebook lives on lsst-dev in /home/mrawls/notebooks . I can provide more detailed directions of how to access and run it over ssh if that would be useful, and I can move it to a different location on lsst-dev if desired.
          Hide
          ebellm Eric Bellm added a comment -

          Before we close this out I'd like to see a little more on bullets 1 and 4–we don't actually need timestamps to calculate median and RMS values, for instance.  Given the slow speed with which we can retrieve photometry it doesn't make sense to do a large sample of lightcurves, but I'd like to see a scatter plot of at least 1k or so.

          I'd also like to see some prototype code for going from srcId to displaying a cutout of the difference/science/template images.

          Show
          ebellm Eric Bellm added a comment - Before we close this out I'd like to see a little more on bullets 1 and 4–we don't actually need timestamps to calculate median and RMS values, for instance.  Given the slow speed with which we can retrieve photometry it doesn't make sense to do a large sample of lightcurves, but I'd like to see a scatter plot of at least 1k or so. I'd also like to see some prototype code for going from srcId to displaying a cutout of the difference/science/template images.
          Hide
          mrawls Meredith Rawls added a comment -

          Fair enough. I went ahead and expanded on the spatial plots and also did bullet 2 (median magnitude plot). The latter has just over 100 DIAObjects (it is realllllly slow to do many more than that) which all have at least 10 visits each. If you want to see closer to 1000 DIAObjects, I'll need to let it churn for over an hour.

          Unfortunately we can't do anything for cutouts just yet because we don't have a way to back out the visits and ccdnums to construct a dataId. It will be straightforward to display a diffim cutout side-by-side with any other plot once we have that information.

          I put an html rendering of the notebook and its outputs (since some of the steps are quite slow to rerun yourself) in a place you should be able to see: /project/mrawls/DM-13687.html. Please take a look and see if anything else is missing.

          Show
          mrawls Meredith Rawls added a comment - Fair enough. I went ahead and expanded on the spatial plots and also did bullet 2 (median magnitude plot). The latter has just over 100 DIAObjects (it is realllllly slow to do many more than that) which all have at least 10 visits each. If you want to see closer to 1000 DIAObjects, I'll need to let it churn for over an hour. Unfortunately we can't do anything for cutouts just yet because we don't have a way to back out the visits and ccdnums to construct a dataId. It will be straightforward to display a diffim cutout side-by-side with any other plot once we have that information. I put an html rendering of the notebook and its outputs (since some of the steps are quite slow to rerun yourself) in a place you should be able to see: /project/mrawls/ DM-13687 .html . Please take a look and see if anything else is missing.
          Hide
          ebellm Eric Bellm added a comment -

          ... I just remembered/realized we're working with difference image flux, so RMS in the sense I described is not very meaningful.  Scratch that.

          Show
          ebellm Eric Bellm added a comment - ... I just remembered/realized we're working with difference image flux, so RMS in the sense I described is not very meaningful.  Scratch that.
          Hide
          mrawls Meredith Rawls added a comment -

          OK, thanks for working with me on this some more yesterday! I think it's done for now, but of course there are lots more plots we can make in the future. The updated notebook exported as html is at /project/mrawls/DM-13687.html for your viewing pleasure.

          Show
          mrawls Meredith Rawls added a comment - OK, thanks for working with me on this some more yesterday! I think it's done for now, but of course there are lots more plots we can make in the future. The updated notebook exported as html is at /project/mrawls/ DM-13687 .html for your viewing pleasure.
          Hide
          ebellm Eric Bellm added a comment -

          Looks good.

          Show
          ebellm Eric Bellm added a comment - Looks good.
          Hide
          mrawls Meredith Rawls added a comment -

          I pushed the notebook to u/mrawls/notebooks-etc on ap_pipe just so it's preserved somewhere other than lsst-dev. It's now visible here.

          Show
          mrawls Meredith Rawls added a comment - I pushed the notebook to u/mrawls/notebooks-etc on ap_pipe just so it's preserved somewhere other than lsst-dev. It's now visible here .

            People

            • Assignee:
              mrawls Meredith Rawls
              Reporter:
              ebellm Eric Bellm
              Reviewers:
              Eric Bellm
              Watchers:
              Eric Bellm, Meredith Rawls
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel