Details
-
Type:
Improvement
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_tasks, Science Pipelines
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Team:Data Release Production
Description
During investigations in DM-32826, it became clear that the entries in sourceTable_visit are not sorted, and thus are not identical from one run to the next (with all configs the same). Update the relevant pipe_task code (postprocess.py? elsewhere?) to order by detector when creating sourceTable_visit.
(See this slack thread for details.)
Attachments
Issue Links
- relates to
-
DM-31701 Enable deterministic dataset loading order in Gen3 middleware
- Reviewed
Activity
LGTM. SinceJeffrey Carlin has a handy test case, it would be nice to check that this fixes things for him (but that need not hold up the merge if timing is an issue).
I'll check it out, but it will take a couple of days, so I think it's fine to go ahead and merge now. I'll let you know what I find.
I confirmed that running the same pipeline+configs twice in a row now produces identical FGCM (and coadd) outputs. 
DM-31701 is a long-standing ticket for doing this down in the middleware, once and for all (I assume the order of this catalog is set by the order of inputs to a PipelineTask). If you're planning to work on this today and want to tackle the general problem, feel free to ping me over in #dm-middleware-support for pointers. I shouldn't commit to doing it myself today as I'll have too many distractions, but I could help someone else along.