Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28344

Integrate WEP into DM stack: creation of the pipeline task

    XMLWordPrintable

Details

    Description

      We need to make WEP code part of the DM standard CI.

      All the changes proposed in https://tstn-026.lsst.io need to be implemented (as of 1/14/21 these are still on a branch and to be reviewed).

      This is a high priority because we need to get this up running before we tackle other image processing challenges.

      It is expected that tribeiro will lead the effort to set up the new architecture (Tiago's ticket DM-28520). Bryce, with assistance from ttsai, will lead in migrating the existing code into the new architecture.

      Completion criteria:

      The CI job takes images from butler, calculates the WF for each CCD center, and compares solutions to PhoSim truth.

      A mechanism needs to be set up for monitoring the wavefront RMS error with each code change.

      Attachments

        Issue Links

          Activity

            jbkalmbach Bryce Kalmbach added a comment - - edited

            Merged new centroid finding algorithm CentroidConvolveTemplate into ts_wep: https://github.com/lsst-ts/ts_wep/pull/58

            jbkalmbach Bryce Kalmbach added a comment - - edited Merged new centroid finding algorithm CentroidConvolveTemplate into ts_wep:  https://github.com/lsst-ts/ts_wep/pull/58

            Merged template generation from model code into ts_wep: https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.1 

            jbkalmbach Bryce Kalmbach added a comment - Merged template generation from model code into ts_wep: https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.1  

            Merged template generation from phosim code into ts_wep: https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.3

            jbkalmbach Bryce Kalmbach added a comment - Merged template generation from phosim code into ts_wep:  https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.3

            Notebook with work in progress on Gen 3 Wavefront Estimation Pipeline: https://github.com/lsst-ts/ts_analysis_notebooks/pull/4

            jbkalmbach Bryce Kalmbach added a comment - Notebook with work in progress on Gen 3 Wavefront Estimation Pipeline:  https://github.com/lsst-ts/ts_analysis_notebooks/pull/4

            As discussed with tribeiro we agree to review this ticket recognizing that it acheived the following goals: 

            • Creation of the pipeline task
            • Set up the Continuous integration.

            However, the integration of the this pipeline task into the DM stack CI will be tracked in another ticket.

            sthomas Sandrine Thomas added a comment - As discussed with tribeiro we agree to review this ticket recognizing that it acheived the following goals:  Creation of the pipeline task Set up the Continuous integration. However, the integration of the this pipeline task into the DM stack CI will be tracked in another ticket.

            tribeiro, Have you had time to review this ticket? Thanks!

            sthomas Sandrine Thomas added a comment - tribeiro , Have you had time to review this ticket? Thanks!

            tribeiro, Could you review this ticket so we can close it?

            sthomas Sandrine Thomas added a comment - tribeiro , Could you review this ticket so we can close it?

            Pipeline tasks are software abstractions that represents how data is processed using the DM stack. This allows one to develop data processing steps that takes fully advantage of the Vera Rubin Science Pipelines, which includes, accessing data through the butler, parallelization, access to other pipelines tasks and much more. For example, when developing a pipeline task to process data, one can integrate steps that will also perform Instrument Signal Removal (ISR) using the standard routines.

            An initial version of the wep pipeline task was created, thus integrating the wavefront estivation pipeline with the DM stack. The pipeline was used to process phosim simulated data and also integrated with the MTAOS CSC.

            tribeiro Tiago Ribeiro added a comment - Pipeline tasks are software abstractions that represents how data is processed using the DM stack. This allows one to develop data processing steps that takes fully advantage of the Vera Rubin Science Pipelines, which includes, accessing data through the butler, parallelization, access to other pipelines tasks and much more. For example, when developing a pipeline task to process data, one can integrate steps that will also perform Instrument Signal Removal (ISR) using the standard routines. An initial version of the wep pipeline task was created, thus integrating the wavefront estivation pipeline with the DM stack. The pipeline was used to process phosim simulated data and also integrated with the MTAOS CSC.

            People

              jbkalmbach Bryce Kalmbach
              bxin Bo Xin [X] (Inactive)
              Tiago Ribeiro
              Bo Xin [X] (Inactive), Bryce Kalmbach, Sandrine Thomas, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.