Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-440

Update calibrations in /datasets/hsc/repo and enable new HSC pipeline steps

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      We (Paul Price and Jim Bosch) would like to update the master calibrations used by default for HSC processing on lsst-dev, including both existing calibration frame types (flats, biases, darks, fringes) and two new calibration-like datasets:

      • Sky Frames: measurements of the persistent (average) sky background in HSC visits across different filters.  This will be used in the new visit-level background estimation step (https://community.lsst.org/t/sky-subtraction/2415).
      • Transmission Curves: measurements of the wavelength-dependent transmission for different parts of the optical system, which will be combined and added to Exposures in during ISR after the merge of DM-12375.

      Both of these new processing steps will be enabled by default after the necessary calibrations are added to lsst-dev, which means any processing of HSC data with the default configuration on other platforms will also required those new calibrations to be added.  Instructions for adding the sky frames can be found in the above community.lsst.org link; while adding transmission curves can be done (after DM-12375) with:

      setup obs_subaru
      installTransmissionCurves.py /path/to/repo

      Note that the above path is to the regular data repository root, not a calibRoot; transmission curves are not yet versioned (they will be in the Gen3 butler, if not earlier).

      Both of these steps will only be used in HSC processing for now.  They could be enabled for other cameras once the work is done to provide the necessary calibrations for those cameras.

      As per our policy on modifications to /datasets:

      • Description and reason for addition/change/deletion: updates to the existing calibration frames reflect our most up-to-date understanding of the camera's characteristics.  The new datasets enable new processing steps that improve the quality of the reductions.
      • Target top-level-directory for location of addition/change/deletion: /datasets/hsc/repo
      • Organization of data:
        • calibration and sky frames will go in a new subdirectory of /datasets/hsc/repo/calibs, and the /datasets/hsc/repo/CALIB symlink will be updated to point to the new subdirectory
        • transmission curves will go into /datasets/hsc/repo/transmission
      • Other necessary domain knowledge as identified by project members relating to the contents of the data: provided in the community.lsst.org link above

       

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Description We ([~price] and [~jbosch]) would like to update the master calibrations used by default for HSC processing on lsst-dev, including both existing calibration frame types (flats, biases, darks, fringes) and two new calibration-like datasets:
             * Sky Frames: measurements of the persistent (average) sky background in HSC visits across different filters.  This will be used in the new visit-level background estimation step ([https://community.lsst.org/t/sky-subtraction/2415)|https://community.lsst.org/t/sky-subtraction/2415),].
             * Transmission Curves: measurements of the wavelength-dependent transmission for different parts of the optical system, which will be combined and added to {{Exposures}} in during ISR after the merge of DM-12375.

            Both of these new processing steps will be enabled by default after the necessary calibrations are added to lsst-dev, which means any processing of HSC data with the default configuration on other platforms will also required those new calibrations to be added.  Instructions for adding the sky frames can be found in the above community.lsst.org link; while adding transmission curves can be done (after DM-12375) with:
            {code:java}setup obs_subaru
            installTransmissionCurves.py /path/to/repo{code}
            Note that the above path is to the regular data repository root, not a {{calibRoot}}; transmission curves are not yet versioned (they will be in the Gen3 butler, if not earlier).

            Both of these steps will only be used in HSC processing for now.  They could be enabled for other cameras once the work is done to provide the necessary calibrations for those cameras.

            As per [our policy on modifications to /datasets|https://developer.lsst.io/services/datasets.html]:
             * *Description and reason for addition/change/deletion:* updates to the existing calibration frames reflect our most up-to-date understanding of the camera's characteristics.  The new datasets enable new processing steps that improve the quality of the reductions.
             * *Target top-level-directory for location of addition/change/deletion:* /datasets/hsc/repo
             * *Organization of data:*
             ** calibration and sky frames will go in a new subdirectory of /datasets/hsc/repo/calibs, and the /datasets/hsc/repo/CALIB symlink will be updated to point to the new subdirectory
             ** transmission curves will go into /datasets/hsc/repo/transmission
             * *Other necessary domain knowledge as identified by project members relating to the contents of the data:* provided in the community.lsst.org link**

             
            We ([~price] and [~jbosch]) would like to update the master calibrations used by default for HSC processing on lsst-dev, including both existing calibration frame types (flats, biases, darks, fringes) and two new calibration-like datasets:
             * Sky Frames: measurements of the persistent (average) sky background in HSC visits across different filters.  This will be used in the new visit-level background estimation step ([https://community.lsst.org/t/sky-subtraction/2415)|https://community.lsst.org/t/sky-subtraction/2415),].
             * Transmission Curves: measurements of the wavelength-dependent transmission for different parts of the optical system, which will be combined and added to {{Exposures}} in during ISR after the merge of DM-12375.

            Both of these new processing steps will be enabled by default after the necessary calibrations are added to lsst-dev, which means any processing of HSC data with the default configuration on other platforms will also required those new calibrations to be added.  Instructions for adding the sky frames can be found in the above community.lsst.org link; while adding transmission curves can be done (after DM-12375) with:
            {code:java}setup obs_subaru
            installTransmissionCurves.py /path/to/repo{code}
            Note that the above path is to the regular data repository root, not a {{calibRoot}}; transmission curves are not yet versioned (they will be in the Gen3 butler, if not earlier).

            Both of these steps will only be used in HSC processing for now.  They could be enabled for other cameras once the work is done to provide the necessary calibrations for those cameras.

            As per [our policy on modifications to /datasets|https://developer.lsst.io/services/datasets.html]:
             * *Description and reason for addition/change/deletion:* updates to the existing calibration frames reflect our most up-to-date understanding of the camera's characteristics.  The new datasets enable new processing steps that improve the quality of the reductions.
             * *Target top-level-directory for location of addition/change/deletion:* /datasets/hsc/repo
             * *Organization of data:*
             ** calibration and sky frames will go in a new subdirectory of /datasets/hsc/repo/calibs, and the /datasets/hsc/repo/CALIB symlink will be updated to point to the new subdirectory
             ** transmission curves will go into /datasets/hsc/repo/transmission
             * *Other necessary domain knowledge as identified by project members relating to the contents of the data:* provided in the community.lsst.org link above

             
            jbosch Jim Bosch made changes -
            Link This issue is triggering DM-13387 [ DM-13387 ]
            jbosch Jim Bosch made changes -
            Link This issue is triggering DM-13388 [ DM-13388 ]
            jbosch Jim Bosch made changes -
            Link This issue is triggering DM-13389 [ DM-13389 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is triggering IHS-755 [ IHS-755 ]
            Hide
            price Paul Price added a comment -

            Adopted.

            Show
            price Paul Price added a comment - Adopted.
            price Paul Price made changes -
            Status Proposed [ 10805 ] Adopted [ 10806 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Hsin-Fang Chiang, Jim Bosch, John Swinbank, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.