Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10849

Incorporate Synpipe into the stack

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      DRP F17-1, DRP F17-2, DRP F17-3, DRP F17-4, DRP F17-5, DRP F17-6
    • Team:
      Data Release Production

      Description

      Define and implement how we'll be making Synpipe available within the LSST stack. That may include treating it as a regular third party package, or tracking it via an LSST fork, or adopting the package entirely.

      Although Nate Lust will lead this work, Jim Bosch will coordinate with the upstream maintainer (Song Huang) to figure out how we can best cooperate with him.

      This ticket is to make SynPipe able to be installed through the LSST distribution system. Success will be marked when the repository is is put in place such that the fake object insertion tasks are usable be used in the course of LSST processing. Full functionality of all scripts in the repository is not necessary, as that work will be completed on other tickets.

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment -

            Here is a note on the timeline of this (and related tickets)

            While verifying that synpipe was successfully ported, such that it could reproduce plots found within the synpipe paper, it was necessary to reprocess a tract of data. During the course of processing I encountered a number of gotchas. Some of these might be known to people who regularly process data, but are not widely known outside that group.

            These gotchas include the stack not clobbering data that exists within the butler hierarchy if it already exists. There are many different config parameters scattered about the code that control this clobbering, and I kept encountering yet another that needed to be set. Additionally I found a bug within pipe_drivers where clobber options for detections should exist but currently do not. Some of these issues arose because I was branching off a parent rerun where all the processing was done within one output rerun, so the processing was finding, and using, results that were not expected. After setting what seemed to be all the necessary clobber options, there were still errors which cropped up in the processing.

            Upon consultation with Jim Bosch the processing was started directly from raw data outside the existing rerun structures. As of today this processing is still not going through successfully. The slurm workers for multibandDriver are crashing for unknown reasons, as there is no error output. My current hypothesis is a segfault, and I am in the process of tracking down the problem outside the slurm submission system.

            I believe the work on synpipe itself is all but complete, as I have been able to work on fragments of data that have made it through various stages of processing. However, I can not verify this until the data processing is complete.

            Show
            nlust Nate Lust added a comment - Here is a note on the timeline of this (and related tickets) While verifying that synpipe was successfully ported, such that it could reproduce plots found within the synpipe paper, it was necessary to reprocess a tract of data. During the course of processing I encountered a number of gotchas. Some of these might be known to people who regularly process data, but are not widely known outside that group. These gotchas include the stack not clobbering data that exists within the butler hierarchy if it already exists. There are many different config parameters scattered about the code that control this clobbering, and I kept encountering yet another that needed to be set. Additionally I found a bug within pipe_drivers where clobber options for detections should exist but currently do not. Some of these issues arose because I was branching off a parent rerun where all the processing was done within one output rerun, so the processing was finding, and using, results that were not expected. After setting what seemed to be all the necessary clobber options, there were still errors which cropped up in the processing. Upon consultation with Jim Bosch the processing was started directly from raw data outside the existing rerun structures. As of today this processing is still not going through successfully. The slurm workers for multibandDriver are crashing for unknown reasons, as there is no error output. My current hypothesis is a segfault, and I am in the process of tracking down the problem outside the slurm submission system. I believe the work on synpipe itself is all but complete, as I have been able to work on fragments of data that have made it through various stages of processing. However, I can not verify this until the data processing is complete.
            Hide
            nlust Nate Lust added a comment -

            Update: Upon digging into the above issue, I discovered the failure was not due to a segmentation fault, but an issue where consistent catalogs for objects could not be generated. This lead to digging into individual fake sources, and I discovered that many of the fake sources that were being injected were then being clipped out in the coaddition stage. The clipping of these objects resulted from the same fake object not being inserted in the same part of the sky for each of the frames. My investigation lead to the conclusion that fake objects were being inserted before the wcs for the frame was being redetermined. This mean that the fake source task was inserting objects according to the wcs that comes from the telescope, which can sometimes be off by a number of minutes. Moving the insertion point for fake objects spawned DM-12108. The changes have now been made and the data is reprocessing.

            One final gotcha was encountered when the processing encountered some bad amps in some of the data which caused the slurm jobs to abort. Changing the slurm command, used to invoke the processing, to ignore such errors seems to have the processing back on track.

            Show
            nlust Nate Lust added a comment - Update: Upon digging into the above issue, I discovered the failure was not due to a segmentation fault, but an issue where consistent catalogs for objects could not be generated. This lead to digging into individual fake sources, and I discovered that many of the fake sources that were being injected were then being clipped out in the coaddition stage. The clipping of these objects resulted from the same fake object not being inserted in the same part of the sky for each of the frames. My investigation lead to the conclusion that fake objects were being inserted before the wcs for the frame was being redetermined. This mean that the fake source task was inserting objects according to the wcs that comes from the telescope, which can sometimes be off by a number of minutes. Moving the insertion point for fake objects spawned DM-12108 . The changes have now been made and the data is reprocessing. One final gotcha was encountered when the processing encountered some bad amps in some of the data which caused the slurm jobs to abort. Changing the slurm command, used to invoke the processing, to ignore such errors seems to have the processing back on track.
            Hide
            rearmstr Bob Armstrong added a comment -

            Looks good, a few small edits are on the pull request.

            Show
            rearmstr Bob Armstrong added a comment - Looks good, a few small edits are on the pull request.
            Hide
            nlust Nate Lust added a comment -

            Synpipe is now included with lsst_distrib

            Show
            nlust Nate Lust added a comment - Synpipe is now included with lsst_distrib

              People

              Assignee:
              nlust Nate Lust
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Bob Armstrong
              Watchers:
              Bob Armstrong, John Swinbank, Nate Lust
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: