Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:4
-
Epic Link:
-
Sprint:DRP F17-1, DRP F17-2, DRP F17-3, DRP F17-4, DRP F17-5, DRP F17-6
-
Team:Data Release Production
Description
Define and implement how we'll be making Synpipe available within the LSST stack. That may include treating it as a regular third party package, or tracking it via an LSST fork, or adopting the package entirely.
Although Nate Lust will lead this work, Jim Bosch will coordinate with the upstream maintainer (Song Huang) to figure out how we can best cooperate with him.
This ticket is to make SynPipe able to be installed through the LSST distribution system. Success will be marked when the repository is is put in place such that the fake object insertion tasks are usable be used in the course of LSST processing. Full functionality of all scripts in the repository is not necessary, as that work will be completed on other tickets.
Attachments
Issue Links
Activity
Update: Upon digging into the above issue, I discovered the failure was not due to a segmentation fault, but an issue where consistent catalogs for objects could not be generated. This lead to digging into individual fake sources, and I discovered that many of the fake sources that were being injected were then being clipped out in the coaddition stage. The clipping of these objects resulted from the same fake object not being inserted in the same part of the sky for each of the frames. My investigation lead to the conclusion that fake objects were being inserted before the wcs for the frame was being redetermined. This mean that the fake source task was inserting objects according to the wcs that comes from the telescope, which can sometimes be off by a number of minutes. Moving the insertion point for fake objects spawned DM-12108. The changes have now been made and the data is reprocessing.
One final gotcha was encountered when the processing encountered some bad amps in some of the data which caused the slurm jobs to abort. Changing the slurm command, used to invoke the processing, to ignore such errors seems to have the processing back on track.
Looks good, a few small edits are on the pull request.
Here is a note on the timeline of this (and related tickets)
While verifying that synpipe was successfully ported, such that it could reproduce plots found within the synpipe paper, it was necessary to reprocess a tract of data. During the course of processing I encountered a number of gotchas. Some of these might be known to people who regularly process data, but are not widely known outside that group.
These gotchas include the stack not clobbering data that exists within the butler hierarchy if it already exists. There are many different config parameters scattered about the code that control this clobbering, and I kept encountering yet another that needed to be set. Additionally I found a bug within pipe_drivers where clobber options for detections should exist but currently do not. Some of these issues arose because I was branching off a parent rerun where all the processing was done within one output rerun, so the processing was finding, and using, results that were not expected. After setting what seemed to be all the necessary clobber options, there were still errors which cropped up in the processing.
Upon consultation with Jim Bosch the processing was started directly from raw data outside the existing rerun structures. As of today this processing is still not going through successfully. The slurm workers for multibandDriver are crashing for unknown reasons, as there is no error output. My current hypothesis is a segfault, and I am in the process of tracking down the problem outside the slurm submission system.
I believe the work on synpipe itself is all but complete, as I have been able to work on fragments of data that have made it through various stages of processing. However, I can not verify this until the data processing is complete.