Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28344

Integrate WEP into DM stack: creation of the pipeline task

    XMLWordPrintable

    Details

      Description

      We need to make WEP code part of the DM standard CI.

      All the changes proposed in https://tstn-026.lsst.io need to be implemented (as of 1/14/21 these are still on a branch and to be reviewed).

      This is a high priority because we need to get this up running before we tackle other image processing challenges.

      It is expected that Tiago Ribeiro will lead the effort to set up the new architecture (Tiago's ticket DM-28520). Bryce, with assistance from Te-Wei Tsai, will lead in migrating the existing code into the new architecture.

      Completion criteria:

      The CI job takes images from butler, calculates the WF for each CCD center, and compares solutions to PhoSim truth.

      A mechanism needs to be set up for monitoring the wavefront RMS error with each code change.

        Attachments

          Issue Links

            Activity

            Hide
            jbkalmbach Bryce Kalmbach added a comment - - edited

            Merged new centroid finding algorithm CentroidConvolveTemplate into ts_wep: https://github.com/lsst-ts/ts_wep/pull/58

            Show
            jbkalmbach Bryce Kalmbach added a comment - - edited Merged new centroid finding algorithm CentroidConvolveTemplate into ts_wep:  https://github.com/lsst-ts/ts_wep/pull/58
            Hide
            jbkalmbach Bryce Kalmbach added a comment -

            Merged template generation from model code into ts_wep: https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.1 

            Show
            jbkalmbach Bryce Kalmbach added a comment - Merged template generation from model code into ts_wep: https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.1  
            Hide
            jbkalmbach Bryce Kalmbach added a comment -

            Merged template generation from phosim code into ts_wep: https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.3

            Show
            jbkalmbach Bryce Kalmbach added a comment - Merged template generation from phosim code into ts_wep:  https://github.com/lsst-ts/ts_wep/releases/tag/v1.5.3
            Hide
            jbkalmbach Bryce Kalmbach added a comment -

            Notebook with work in progress on Gen 3 Wavefront Estimation Pipeline: https://github.com/lsst-ts/ts_analysis_notebooks/pull/4

            Show
            jbkalmbach Bryce Kalmbach added a comment - Notebook with work in progress on Gen 3 Wavefront Estimation Pipeline:  https://github.com/lsst-ts/ts_analysis_notebooks/pull/4
            Hide
            sthomas Sandrine Thomas added a comment -

            As discussed with Tiago Ribeiro we agree to review this ticket recognizing that it acheived the following goals: 

            • Creation of the pipeline task
            • Set up the Continuous integration.

            However, the integration of the this pipeline task into the DM stack CI will be tracked in another ticket.

            Show
            sthomas Sandrine Thomas added a comment - As discussed with Tiago Ribeiro we agree to review this ticket recognizing that it acheived the following goals:  Creation of the pipeline task Set up the Continuous integration. However, the integration of the this pipeline task into the DM stack CI will be tracked in another ticket.
            Hide
            sthomas Sandrine Thomas added a comment -

            Tiago Ribeiro, Have you had time to review this ticket? Thanks!

            Show
            sthomas Sandrine Thomas added a comment - Tiago Ribeiro , Have you had time to review this ticket? Thanks!
            Hide
            sthomas Sandrine Thomas added a comment -

            Tiago Ribeiro, Could you review this ticket so we can close it?

            Show
            sthomas Sandrine Thomas added a comment - Tiago Ribeiro , Could you review this ticket so we can close it?
            Hide
            tribeiro Tiago Ribeiro added a comment -

            Pipeline tasks are software abstractions that represents how data is processed using the DM stack. This allows one to develop data processing steps that takes fully advantage of the Vera Rubin Science Pipelines, which includes, accessing data through the butler, parallelization, access to other pipelines tasks and much more. For example, when developing a pipeline task to process data, one can integrate steps that will also perform Instrument Signal Removal (ISR) using the standard routines.

            An initial version of the wep pipeline task was created, thus integrating the wavefront estivation pipeline with the DM stack. The pipeline was used to process phosim simulated data and also integrated with the MTAOS CSC.

            Show
            tribeiro Tiago Ribeiro added a comment - Pipeline tasks are software abstractions that represents how data is processed using the DM stack. This allows one to develop data processing steps that takes fully advantage of the Vera Rubin Science Pipelines, which includes, accessing data through the butler, parallelization, access to other pipelines tasks and much more. For example, when developing a pipeline task to process data, one can integrate steps that will also perform Instrument Signal Removal (ISR) using the standard routines. An initial version of the wep pipeline task was created, thus integrating the wavefront estivation pipeline with the DM stack. The pipeline was used to process phosim simulated data and also integrated with the MTAOS CSC.

              People

              Assignee:
              jbkalmbach Bryce Kalmbach
              Reporter:
              bxin Bo Xin [X] (Inactive)
              Reviewers:
              Tiago Ribeiro
              Watchers:
              Bo Xin [X] (Inactive), Bryce Kalmbach, Sandrine Thomas, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.