Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33158

Sort sourceTable_visit by detector

    XMLWordPrintable

Details

    Description

      During investigations in DM-32826, it became clear that the entries in sourceTable_visit are not sorted, and thus are not identical from one run to the next (with all configs the same). Update the relevant pipe_task code (postprocess.py? elsewhere?) to order by detector when creating sourceTable_visit.

      (See this slack thread for details.)

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            DM-31701 is a long-standing ticket for doing this down in the middleware, once and for all (I assume the order of this catalog is set by the order of inputs to a PipelineTask). If you're planning to work on this today and want to tackle the general problem, feel free to ping me over in #dm-middleware-support for pointers. I shouldn't commit to doing it myself today as I'll have too many distractions, but I could help someone else along.

            jbosch Jim Bosch added a comment - DM-31701 is a long-standing ticket for doing this down in the middleware, once and for all (I assume the order of this catalog is set by the order of inputs to a PipelineTask). If you're planning to work on this today and want to tackle the general problem, feel free to ping me over in #dm-middleware-support for pointers. I shouldn't commit to doing it myself today as I'll have too many distractions, but I could help someone else along.
            erykoff Eli Rykoff added a comment - PR is here: https://github.com/lsst/pipe_tasks/pull/620 Jenkins is here: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35657/pipeline

            LGTM.  Sincejcarlin has a handy test case, it would be nice to check that this fixes things for him (but that need not hold up the merge if timing is an issue).

            lauren Lauren MacArthur added a comment - LGTM.  Since jcarlin  has a handy test case, it would be nice to check that this fixes things for him (but that need not hold up the merge if timing is an issue).

            I'll check it out, but it will take a couple of days, so I think it's fine to go ahead and merge now. I'll let you know what I find.

            jcarlin Jeffrey Carlin added a comment - I'll check it out, but it will take a couple of days, so I think it's fine to go ahead and merge now. I'll let you know what I find.

             I confirmed that running the same pipeline+configs twice in a row now produces identical FGCM (and coadd) outputs. 

            jcarlin Jeffrey Carlin added a comment -  I confirmed that running the same pipeline+configs twice in a row now produces identical FGCM (and coadd) outputs. 

            People

              erykoff Eli Rykoff
              jcarlin Jeffrey Carlin
              Lauren MacArthur
              Eli Rykoff, Jeffrey Carlin, Jim Bosch, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.