Fix Version/s: None
Component/s: pipe_tasks, Science Pipelines
During investigations in
DM-32826, it became clear that the entries in sourceTable_visit are not sorted, and thus are not identical from one run to the next (with all configs the same). Update the relevant pipe_task code (postprocess.py? elsewhere?) to order by detector when creating sourceTable_visit.
(See this slack thread for details.)
- relates to
DM-31701 Enable deterministic dataset loading order in Gen3 middleware
LGTM. SinceJeffrey Carlin has a handy test case, it would be nice to check that this fixes things for him (but that need not hold up the merge if timing is an issue).
I'll check it out, but it will take a couple of days, so I think it's fine to go ahead and merge now. I'll let you know what I find.
I confirmed that running the same pipeline+configs twice in a row now produces identical FGCM (and coadd) outputs.
DM-31701 is a long-standing ticket for doing this down in the middleware, once and for all (I assume the order of this catalog is set by the order of inputs to a PipelineTask). If you're planning to work on this today and want to tackle the general problem, feel free to ping me over in #dm-middleware-support for pointers. I shouldn't commit to doing it myself today as I'll have too many distractions, but I could help someone else along.