Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18892

Investigate deletes in Gen3 ci_hsc conversion or processing

    XMLWordPrintable

Details

    • 1
    • Data Release Production

    Description

      From mgower at https://lsstc.slack.com/archives/C2JPT1KB7/p1553801514017200:

      For each run there were a bunch of
      ```DELETE FROM "PatchSkyPixJoin" WHERE "PatchSkyPixJoin".tract = :tract_1 AND "PatchSkyPixJoin".skymap = :skymap_1 AND "PatchSkyPixJoin".patch = :patch_1```
      statements which at first glance seem to be coming from the pipeline. If they are from the pipeline (instead of the gen3.py or other bootstrap codes), then we should talk sometime about why there is a delete.

      These tables should be inserted into in gen2convert, but not deleted from.  They should not be modified at all during processing.  It's possible I'm just clearing subset of a table before inserting into it, which would mean in this case these DELETEs are no-ops because there are no matching record.  If that's the case, we might be able to ignore this.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            mgower, this should have been fixed on DM-17023.  Could you please verify, and close this as Invalid if I'm right?

            jbosch Jim Bosch added a comment - mgower , this should have been fixed on DM-17023 .  Could you please verify, and close this as Invalid if I'm right?

            cs2018 Can you check Oracle trace files from ci_hsc_gen3 runs (gen3_mgower_1) to see if there are still delete statements? Let me know if you need me to run again.

            mgower Michelle Gower added a comment - cs2018 Can you check Oracle trace files from ci_hsc_gen3 runs (gen3_mgower_1) to see if there are still delete statements? Let me know if you need me to run again.

            mgower I grep'd through all ci_hsc_gen3 runs from Oct 7. There were no DELETEs in any of the tracefiles.

            cs2018 Christopher Stephens [X] (Inactive) added a comment - mgower I grep'd through all ci_hsc_gen3 runs from Oct 7. There were no DELETEs in any of the tracefiles.

            Looks like this was fixed when doing the work in DM-17023. Closing ticket.

            mgower Michelle Gower added a comment - Looks like this was fixed when doing the work in DM-17023 . Closing ticket.

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Christopher Stephens [X] (Inactive), Jim Bosch, Michelle Gower
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.