Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14559

Make pipe_analysis compliant with LSST coding standards

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Sprint:
      DRP F19-5, DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug), DRP F20-5 (Oct), DRP S21a (Dec Jan)
    • Team:
      Data Release Production

      Description

      The pipe_analysis repository provides scripts which inspect a repository of processed data and generate static plots of the distributions of scientifically relevant quantities. These may be manually inspected to demonstrate the internal consistency and fidelity of photometric and astrometric measurements on the visit-stage and coadd-stage catalog outputs. They also can be used to compare catalog outputs from two different reruns.

      The pipe_analysis repository currently resides in the lsst-dm GitHub organization and has historically been developed largely from the point of view of performing QA on HSC-SSP processing to reveal and diagnose any issues arising in the LSST codebase.  As such, it has not been developed and code-reviewed under the same strict guidelines to which the main lsst repository code development must comply.  

      More recently, the products of this repository have proven (and been deemed) more generally useful (e.g. they are now currently run and inspected as part of the bi-weekly HSC RC2 dataset reprocessing, see https://confluence.lsstcorp.org/display/DM/Reprocessing+of+the+HSC+RC2+dataset), so there is a desire to have it moved into the main lsst repository (and, to begin, have the main functionality CI'd by running the scripts as part of ci_hsc, noting that a human eye inspecting the individual plots would still be required to check their fidelity).  Before this can happen, however, the code must be brought into compliance with the lsst coding standards (as outlined in the LSST DM Developer Guide: https://developer.lsst.io/index.html).
       
      This ticket is meant as a bucket to list all the required/desired changes to be made (input from others is strongly encouraged/desired!).  Individual tickets will be created for the individual changes (to the extent to which they can be atomized).  When we believe compliance has been achieved, an RFC to make the move from the lsst-dm to the lsst GitHub organization will be created.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            List of todo's so far:

            • wipe out accommodations for the old processing with the (now obsolete) "HSC stack"
            • unify all string formating to use .format style while minimizing verbosity (i.e. no ":s" needed in {} as string is the default, no numbers needed in {#:} variable holders if variables are listed in the order they are quoted)
            • unify return values for all functions: specifically, only return None in functions that otherwise return a "value", otherwise use a bare return
            • ensure pep8 compliance
            • add proper copyright information
            • add/update documentation for all functions
            • minimize HSC-specific code (should likely include getting scripts to run on other camera packages/datasets: e.g. obs_lsstSims [done for most scripts in DM-11452], obs_decam, obs_cfht, obs_lsstCam)
            • persist the config
            • convert the plot persistence to make use of the MatplotlibStorage implemented DM-14356
            Show
            lauren Lauren MacArthur added a comment - - edited List of todo's so far: wipe out accommodations for the old processing with the (now obsolete) "HSC stack" unify all string formating to use .format style while minimizing verbosity (i.e. no ":s" needed in {} as string is the default, no numbers needed in {#:} variable holders if variables are listed in the order they are quoted) unify return values for all functions: specifically, only return None in functions that otherwise return a "value", otherwise use a bare return ensure pep8 compliance add proper copyright information add/update documentation for all functions minimize HSC-specific code (should likely include getting scripts to run on other camera packages/datasets: e.g. obs_lsstSims [done for most scripts in DM-11452] , obs_decam , obs_cfht, obs_lsstCam) persist the config convert the plot persistence to make use of the MatplotlibStorage implemented DM-14356
            Hide
            lauren Lauren MacArthur added a comment -

            The pipe_analysis repo is in the process of being retired and replaced with analysis_drp.

            Show
            lauren Lauren MacArthur added a comment - The pipe_analysis repo is in the process of being retired and replaced with analysis_drp .

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Watchers:
              Jim Bosch, John Swinbank, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.