Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1942

Test data development and HSC stack integration

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      Test data development and HSC stack integration
    • Story Points:
      45
    • WBS:
      02C.04.06
    • Team:
      Data Release Production
    • Cycle:
      Summer 2015

      Description

      Breakdown: price 20%; lauren 80%

      This epic is focused on general stack testing and integration on HSC data, with a goal of getting LSST-side reductions of HSC data to the same level of quality and robustness currently present on the HSC fork of the stack, which will involve a combination of backporting minor fixes from the HSC codebase and tuning parameters for LSST-side algorithms that supercede their HSC-side counterparts. This will allow science-level algorithms to be tested using HSC data, and will provide functionality important for science-grade tests on other real data (such as the ongoing CFHTLS reprocsessing at IN2P3).

      This effort will be focused on single-epoch processing (i.e. ProcessCcdTask), with coadd-level processing a stretch goal dependent somewhat on the completion of astrometric calibration work at UW.

        Attachments

          Issue Links

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          DM-3473

          Add transformed source entries to HSC mapper policy

          0.1 John Swinbank Done
           
          DM-3442

          Processing y-band HSC data fails in loading reference sources

          2 Paul Price Done
           
          DM-3404

          Port HSC updates to ingestImages.py

          2 Paul Price Done
           
          DM-3243

          Include polygon bounds in CoaddPsf logic

          1 John Swinbank Done
           
          DM-3137

          Handle bad pixels in image stacker

          1 John Swinbank Done
           
          DM-3136

          Add & use new mask plane for out-of-bounds regions

          2 John Swinbank Done
           
          DM-3108

          Use aperture flux for photometric calibration

          1 John Swinbank Done
           
          DM-3106

          Add slot for calibration flux

          2 John Swinbank Done
           
          DM-3258

          CoaddPsf.getAveragePosition() is not a valid position

          1 John Swinbank Done
           
          DM-2981

          polygon masking in CoaddPsf

          8 Russell Owen Done
           
          DM-2980

          refactor coaddition code

          5 John Swinbank Done
           
          DM-2977

          Miscellaneous CModel improvements from HSC

          2 Jim Bosch Done
           
          DM-2961

          Port the psfextractor external library from HSC to LSST

          14 Nate Lust Done
           
          DM-2923

          Port HSC-1199 to LSST (UNMASKEDNAN mask propagates to all amplifiers)

          4 Nate Lust Done
           
          DM-2436

          Cherry-pick "fix makeRGB so it can replace saturated pixels and produce an image" from HSC

          0.5 Paul Price Done
           
          DM-2380

          Retrieve HSC engineering data

          2 Paul Price Done
           
          DM-527

          make Image construction robust against integer overflow

          1 John Swinbank Done
           
          DM-2796

          obs_subaru cannot be built due to a missing hsc header file

          Lauren MacArthur Won't Fix
           
          DM-2975

          Quantify how much objects are blended

          2 Lauren MacArthur Won't Fix

            Activity

            Hide
            swinbank John Swinbank added a comment -

            The first major task in this epic will involve properly checking through the HSC codebase and deriving a list of functionality which will need to be migrated. Our hope is that Nate Lust will be able to start on that when he arrives in Princeton in early May.

            Show
            swinbank John Swinbank added a comment - The first major task in this epic will involve properly checking through the HSC codebase and deriving a list of functionality which will need to be migrated. Our hope is that Nate Lust will be able to start on that when he arrives in Princeton in early May.
            Hide
            swinbank John Swinbank added a comment -

            Note that the HSC stack uses its own process distribution middleware which provides functionality not available through LSST. At time of writing, discussions are ongoing about whether it is appropriate to port the HSC middleware to LSST as a stop-gap solution, or to wait until LSST middleware development has ramped up to the extent that it can provide all necessary functionality.

            Considerations include:

            • Porting the HSC functionality to a new (LSST-developed) middleware would presumably be more effort than continuing to run with HSC-like middleware.
            • Development of LSST middleware will be carried out at NCSA, which is currently in the process of staffing up.
            • Introducing HSC-like middleware to the LSST stack could create additional work now which will cause our schedule to slip and will ultimately just need to be replaced (and, indeed, its very existence may delay its replacement).
            • We regarding merging and re-forking HSC as a technical priority.
            • There may be some project-level pressure to process eg DECam data, requiring the functionality now available on HSC.

            Discussions are currently ongoing; we should revisit this at the May 2015 DMLT meeting.

            Show
            swinbank John Swinbank added a comment - Note that the HSC stack uses its own process distribution middleware which provides functionality not available through LSST. At time of writing, discussions are ongoing about whether it is appropriate to port the HSC middleware to LSST as a stop-gap solution, or to wait until LSST middleware development has ramped up to the extent that it can provide all necessary functionality. Considerations include: Porting the HSC functionality to a new (LSST-developed) middleware would presumably be more effort than continuing to run with HSC-like middleware. Development of LSST middleware will be carried out at NCSA, which is currently in the process of staffing up. Introducing HSC-like middleware to the LSST stack could create additional work now which will cause our schedule to slip and will ultimately just need to be replaced (and, indeed, its very existence may delay its replacement). We regarding merging and re-forking HSC as a technical priority. There may be some project-level pressure to process eg DECam data, requiring the functionality now available on HSC. Discussions are currently ongoing; we should revisit this at the May 2015 DMLT meeting.
            Hide
            swinbank John Swinbank added a comment -

            The definitive status of middleware will ultimately be a call for the Project Engineer to make, but the noises at the DMLT seemed to indicate that simply merging what's required from HSC would not be a problem in the short term.

            Increase the estimated SP total of this epic to 65 at the Summer 2015 mid-cycle reset to reflect the increased scope of work here.

            Show
            swinbank John Swinbank added a comment - The definitive status of middleware will ultimately be a call for the Project Engineer to make, but the noises at the DMLT seemed to indicate that simply merging what's required from HSC would not be a problem in the short term. Increase the estimated SP total of this epic to 65 at the Summer 2015 mid-cycle reset to reflect the increased scope of work here.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Jim Bosch, John Swinbank, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.