Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17259

Turn plotting scripts into reusable format

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP S19-2, AP S19-3, AP S19-4
    • Team:
      Alert Production

      Description

      In order to make my AAS 233 poster, I threw a lot of things into a giant jupyter notebook which generates pretty light curves, object/source plots and histograms, and image cutouts. This code is a mess, but the core functionality will be very helpful for future analysis of ap_pipe reruns. This ticket is to preserve that work somewhere other than in a monolithic notebook on lsst-dev (perhaps a new ap_analysis package).

        Attachments

        1. cw_processed2_cos.pdf
          196 kB
        2. cw_processed2_lcs.pdf
          68 kB
        3. cw_processed2_objHist.png
          cw_processed2_objHist.png
          28 kB
        4. cw_processed2_objSky.png
          cw_processed2_objSky.png
          310 kB

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment - - edited

            I have added two python scripts to the ap_pipe-notebooks repo. They are both customized for the ap_verify_hits2015 dataset. They each take a repository that contains a PPDB as an input argument and save figures to the working directory. The light curve script additionally requires the repository containing difference imaging template coadds to be provided as input.

            You may test them on lsst-dev by running, e.g.,

            diaObjectAnalysis.py /project/mrawls/hits2015/rerun/cw_processed2
            (should run relatively quickly and write two PNGs to disk)

            plotLightcurve.py /project/mrawls/hits2015/rerun/cw_processed2 /project/mrawls/hits2014/coadds_processed2
            (will take some time to run, as it is plotting light curves for > 100 objects, and will write two multi-page PDFs to disk)

            I attached the plots I got from running the scripts as stated above, except the PDFs have been truncated to show just the first few objects (and yes, some of them are presently bad/edge objects!).

            Show
            mrawls Meredith Rawls added a comment - - edited I have added two python scripts to the ap_pipe-notebooks repo. They are both customized for the ap_verify_hits2015 dataset. They each take a repository that contains a PPDB as an input argument and save figures to the working directory. The light curve script additionally requires the repository containing difference imaging template coadds to be provided as input. You may test them on lsst-dev by running, e.g., diaObjectAnalysis.py /project/mrawls/hits2015/rerun/cw_processed2 (should run relatively quickly and write two PNGs to disk) plotLightcurve.py /project/mrawls/hits2015/rerun/cw_processed2 /project/mrawls/hits2014/coadds_processed2 (will take some time to run, as it is plotting light curves for > 100 objects, and will write two multi-page PDFs to disk) I attached the plots I got from running the scripts as stated above, except the PDFs have been truncated to show just the first few objects (and yes, some of them are presently bad/edge objects!).
            Hide
            mrawls Meredith Rawls added a comment -

            Lauren MacArthur, would you be willing to review this? It is not particularly urgent. I thought your experience with pipe_analysis would be particularly relevant, since I will probably wind up using these scripts as a starting point for something like an ap_analysis package.

            Show
            mrawls Meredith Rawls added a comment - Lauren MacArthur , would you be willing to review this? It is not particularly urgent. I thought your experience with pipe_analysis would be particularly relevant, since I will probably wind up using these scripts as a starting point for something like an ap_analysis package.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            They are both customized for the ap_verify_hits2015 dataset.

            Does this imply that the scripts would not run on any other dataset? If so, you should update the docs that imply they would run on any ap_pipe run, e.g. in diaObjectAnalysis.py:

            "Script to make plots of DIAObjects using a Prompt Products Database
            (PPDB) resulting from a run of ap_pipe."
            

            Could you provide context/descriptions for the two individual plots posted here (as is they are not very informative to the uninitiated). Perhaps pasting/linking to your AAS poster would do the trick?

            It is somewhat cumbersome to have the two pdf files...seeing all the difference images along with the light curve would help with the interpretation of trends/anomalies seen in the light curve. Since you are only plotting a single source per page, it seems there is enough room for them to be merged (of, course, this only applies for a limited number of diffims, but that will require some accounting for in the "cos" pdf plots anyway. I don't think you intend to do that here, but do you have a strategy in mind for how to display 100s+ of difference image inputs?

            I have not inspected the code beyond a cursory glance. I am under the impression that this is not required/expected here since this code is not being moved anywhere "official" at the moment (and that a full code-review would happen if/when such a move were planned). Please let me know if I'm mistaken!

            Show
            lauren Lauren MacArthur added a comment - - edited They are both customized for the ap_verify_hits2015 dataset. Does this imply that the scripts would not run on any other dataset? If so, you should update the docs that imply they would run on any ap_pipe run, e.g. in diaObjectAnalysis.py : "Script to make plots of DIAObjects using a Prompt Products Database (PPDB) resulting from a run of ap_pipe." Could you provide context/descriptions for the two individual plots posted here (as is they are not very informative to the uninitiated). Perhaps pasting/linking to your AAS poster would do the trick? It is somewhat cumbersome to have the two pdf files...seeing all the difference images along with the light curve would help with the interpretation of trends/anomalies seen in the light curve. Since you are only plotting a single source per page, it seems there is enough room for them to be merged (of, course, this only applies for a limited number of diffims, but that will require some accounting for in the "cos" pdf plots anyway. I don't think you intend to do that here, but do you have a strategy in mind for how to display 100s+ of difference image inputs? I have not inspected the code beyond a cursory glance. I am under the impression that this is not required/expected here since this code is not being moved anywhere "official" at the moment (and that a full code-review would happen if/when such a move were planned). Please let me know if I'm mistaken!
            Hide
            mrawls Meredith Rawls added a comment -

            Thank you for taking a look at this, Lauren MacArthur. I apologize for the slow response; I was out sick for a week and then on vacation and moving, so it's been a fun few weeks. I agree you don't need to look at the code in detail since it is just going to land in my ap_pipe-notebooks repo for now.

            Good idea to provide a more complete caption for each of the plots. I think what I'll do is add the plots themselves to the repo as references/examples and also add a text file with captions, and then merge it.

            The scripts truly do work with any run of ap_pipe, but in diaObjectAnalysis.py it checks to see if "hits2015" is in the repo name, and if so, it uses a special set of axis limits to make the objSky plot. Sorry for the confusion there.

            I agree it would be nice to have all the calexp and difference image cutouts displayed together with the light curve, but many of the sources have 20+ visits, and all that whitespace goes away quickly! I don't currently have a plan for how to display zillions of visits but I am very open to suggestions. I think this is future work outside the scope of this ticket.

            Show
            mrawls Meredith Rawls added a comment - Thank you for taking a look at this, Lauren MacArthur . I apologize for the slow response; I was out sick for a week and then on vacation and moving, so it's been a fun few weeks. I agree you don't need to look at the code in detail since it is just going to land in my ap_pipe-notebooks repo for now. Good idea to provide a more complete caption for each of the plots. I think what I'll do is add the plots themselves to the repo as references/examples and also add a text file with captions, and then merge it. The scripts truly do work with any run of ap_pipe, but in diaObjectAnalysis.py it checks to see if "hits2015" is in the repo name, and if so, it uses a special set of axis limits to make the objSky plot. Sorry for the confusion there. I agree it would be nice to have all the calexp and difference image cutouts displayed together with the light curve, but many of the sources have 20+ visits, and all that whitespace goes away quickly! I don't currently have a plan for how to display zillions of visits but I am very open to suggestions. I think this is future work outside the scope of this ticket.

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              mrawls Meredith Rawls
              Reviewers:
              Lauren MacArthur
              Watchers:
              Lauren MacArthur, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.