Fix Version/s: None
We have at least a couple of algorithms whose output is dependent on the order of the inputs (e.g jointcal & fgcm). While this may not be an issue for many processing contexts, it is critically important for our regular CI/QA reprocessing campaigns where even the smallest output differences must be scrutinized if not anticipated from explicit code changes. Thus, it is of great use to be able to ensure a deterministic ordering of inputs (visits in the above examples) within the context of the Gen3 middleware. It may be that imposing this ordering implies some efficiency hits, so it should be a configurable/optional setting.
|Remote Link||This issue links to "Page (Confluence)" [ 31004 ]|
|Remote Link||This issue links to "Page (Confluence)" [ 31685 ]|
|Assignee||Paul Price [ price ]|
|Team||Data Release Production [ 10301 ]||External [ 12117 ]|
|Status||To Do [ 10001 ]||In Progress [ 3 ]|
|Reviewers||Jim Bosch [ jbosch ]|
|Status||In Progress [ 3 ]||In Review [ 10004 ]|
|Status||In Review [ 10004 ]||Reviewed [ 10101 ]|
I'd prefer to hold off; I don't want to merge over Nate Lust's objections, and I suspect what you actually wanted out of this (being able to zip-iterate) is something we've concluded it cannot be assumed to provide. If you were just looking for the guaranteed determinism that was originally requested in the ticket, let's discuss - maybe we can find some way to achieve that while addressing Nate's concerns (e.g. by making the order deterministic but intentionally different for different dataset types).
OK. I have an attempt at zip iteration that I'm working on that I'll add here if it works.
|Component/s||pipe_base [ 10727 ]|
|Component/s||gen3-middleware [ 19000 ]|
Note there are a couple of TODOs pending this ticket that should be addressed either way: