Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31701

Enable deterministic dataset loading order in Gen3 middleware

    XMLWordPrintable

    Details

    • Team:
      External
    • Urgent?:
      No

      Description

      We have at least a couple of algorithms whose output is dependent on the order of the inputs (e.g jointcal & fgcm). While this may not be an issue for many processing contexts, it is critically important for our regular CI/QA reprocessing campaigns where even the smallest output differences must be scrutinized if not anticipated from explicit code changes. Thus, it is of great use to be able to ensure a deterministic ordering of inputs (visits in the above examples) within the context of the Gen3 middleware. It may be that imposing this ordering implies some efficiency hits, so it should be a configurable/optional setting.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29820 Compare Gen2 vs. Gen3 fgcm photoCalibs up to w_2022_04 HSC-RC2 run

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29821 Compare the jointcal SkyWcs's of the gen2 vs. gen3 w_2021_22 RC2 runs

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-33158 Sort sourceTable_visit by detector

          • Done
          mentioned in

          Page Loading...

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Link This issue relates to DM-29821 [ DM-29821 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-29820 [ DM-29820 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31004 ]
            yusra Yusra AlSayyad made changes -
            Labels drp-gen3 drp-parity drp-parity
            tjenness Tim Jenness made changes -
            Labels drp-parity drp-parity gen3-middleware
            lguy Leanne Guy made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31685 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-33158 [ DM-33158 ]
            price Paul Price made changes -
            Assignee Paul Price [ price ]
            price Paul Price made changes -
            Team Data Release Production [ 10301 ] External [ 12117 ]
            price Paul Price made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            price Paul Price added a comment -

            Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36302/pipeline/

            Show
            price Paul Price added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36302/pipeline/
            price Paul Price made changes -
            Reviewers Jim Bosch [ jbosch ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            Permalink
            price Paul Price added a comment -

            Jim Bosch and Nate Lust debated the usefulness of this patch on the GitHub PR. What was the final conclusion? Shall I merge?

            Show
            price Paul Price added a comment - Jim Bosch and Nate Lust debated the usefulness of this patch on the GitHub PR. What was the final conclusion? Shall I merge?
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            I'd prefer to hold off; I don't want to merge over Nate Lust's objections, and I suspect what you actually wanted out of this (being able to zip-iterate) is something we've concluded it cannot be assumed to provide. If you were just looking for the guaranteed determinism that was originally requested in the ticket, let's discuss - maybe we can find some way to achieve that while addressing Nate's concerns (e.g. by making the order deterministic but intentionally different for different dataset types).

            Show
            jbosch Jim Bosch added a comment - I'd prefer to hold off; I don't want to merge over Nate Lust 's objections, and I suspect what you actually wanted out of this (being able to zip-iterate) is something we've concluded it cannot be assumed to provide. If you were just looking for the guaranteed determinism that was originally requested in the ticket, let's discuss - maybe we can find some way to achieve that while addressing Nate's concerns (e.g. by making the order deterministic but intentionally different for different dataset types).
            Hide
            Permalink
            price Paul Price added a comment -

            OK. I have an attempt at zip iteration that I'm working on that I'll add here if it works.

            Show
            price Paul Price added a comment - OK. I have an attempt at zip iteration that I'm working on that I'll add here if it works.
            tjenness Tim Jenness made changes -
            Component/s pipe_base [ 10727 ]
            Component/s gen3-middleware [ 19000 ]
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Paul Price did your experiment go anywhere?

            Show
            tjenness Tim Jenness added a comment - Paul Price did your experiment go anywhere?
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Paul Price have you abandoned this?

            Show
            tjenness Tim Jenness added a comment - Paul Price have you abandoned this?
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Note there are a couple of TODOs pending this ticket that should be addressed either way:

            https://github.com/search?q=org%3Alsst%20DM-31701&type=code

             

            Show
            lauren Lauren MacArthur added a comment - Note there are a couple of TODOs pending this ticket that should be addressed either way: https://github.com/search?q=org%3Alsst%20DM-31701&type=code  

              People

              Assignee:
              price Paul Price
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Jim Bosch
              Watchers:
              Eli Rykoff, Jim Bosch, John Parejko, Lauren MacArthur, Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.