Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18892

Investigate deletes in Gen3 ci_hsc conversion or processing

    Details

    • Story Points:
      1
    • Team:
      Data Release Production

      Description

      From Michelle Gower at https://lsstc.slack.com/archives/C2JPT1KB7/p1553801514017200:

      For each run there were a bunch of
      ```DELETE FROM "PatchSkyPixJoin" WHERE "PatchSkyPixJoin".tract = :tract_1 AND "PatchSkyPixJoin".skymap = :skymap_1 AND "PatchSkyPixJoin".patch = :patch_1```
      statements which at first glance seem to be coming from the pipeline. If they are from the pipeline (instead of the gen3.py or other bootstrap codes), then we should talk sometime about why there is a delete.

      These tables should be inserted into in gen2convert, but not deleted from.  They should not be modified at all during processing.  It's possible I'm just clearing subset of a table before inserting into it, which would mean in this case these DELETEs are no-ops because there are no matching record.  If that's the case, we might be able to ignore this.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Michelle Gower, this should have been fixed on DM-17023.  Could you please verify, and close this as Invalid if I'm right?

            Show
            jbosch Jim Bosch added a comment - Michelle Gower , this should have been fixed on DM-17023 .  Could you please verify, and close this as Invalid if I'm right?
            Hide
            mgower Michelle Gower added a comment -

            Christopher Stephens Can you check Oracle trace files from ci_hsc_gen3 runs (gen3_mgower_1) to see if there are still delete statements? Let me know if you need me to run again.

            Show
            mgower Michelle Gower added a comment - Christopher Stephens Can you check Oracle trace files from ci_hsc_gen3 runs (gen3_mgower_1) to see if there are still delete statements? Let me know if you need me to run again.
            Hide
            cs2018 Christopher Stephens added a comment -

            Michelle Gower I grep'd through all ci_hsc_gen3 runs from Oct 7. There were no DELETEs in any of the tracefiles.

            Show
            cs2018 Christopher Stephens added a comment - Michelle Gower I grep'd through all ci_hsc_gen3 runs from Oct 7. There were no DELETEs in any of the tracefiles.
            Hide
            mgower Michelle Gower added a comment -

            Looks like this was fixed when doing the work in DM-17023. Closing ticket.

            Show
            mgower Michelle Gower added a comment - Looks like this was fixed when doing the work in DM-17023 . Closing ticket.

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Christopher Stephens, Jim Bosch, Michelle Gower
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel