Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18892

Investigate deletes in Gen3 ci_hsc conversion or processing

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Team:
      Data Release Production

      Description

      From Michelle Gower at https://lsstc.slack.com/archives/C2JPT1KB7/p1553801514017200:

      For each run there were a bunch of
      ```DELETE FROM "PatchSkyPixJoin" WHERE "PatchSkyPixJoin".tract = :tract_1 AND "PatchSkyPixJoin".skymap = :skymap_1 AND "PatchSkyPixJoin".patch = :patch_1```
      statements which at first glance seem to be coming from the pipeline. If they are from the pipeline (instead of the gen3.py or other bootstrap codes), then we should talk sometime about why there is a delete.

      These tables should be inserted into in gen2convert, but not deleted from.  They should not be modified at all during processing.  It's possible I'm just clearing subset of a table before inserting into it, which would mean in this case these DELETEs are no-ops because there are no matching record.  If that's the case, we might be able to ignore this.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17023 Refactor the Dimensions and query system

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Michelle Gower, this should have been fixed on DM-17023.  Could you please verify, and close this as Invalid if I'm right?

            Show
            jbosch Jim Bosch added a comment - Michelle Gower , this should have been fixed on DM-17023 .  Could you please verify, and close this as Invalid if I'm right?
            Hide
            Permalink
            mgower Michelle Gower added a comment -

            Christopher Stephens [X] Can you check Oracle trace files from ci_hsc_gen3 runs (gen3_mgower_1) to see if there are still delete statements? Let me know if you need me to run again.

            Show
            mgower Michelle Gower added a comment - Christopher Stephens [X] Can you check Oracle trace files from ci_hsc_gen3 runs (gen3_mgower_1) to see if there are still delete statements? Let me know if you need me to run again.
            Hide
            Permalink
            cs2018 Christopher Stephens [X] (Inactive) added a comment -

            Michelle Gower I grep'd through all ci_hsc_gen3 runs from Oct 7. There were no DELETEs in any of the tracefiles.

            Show
            cs2018 Christopher Stephens [X] (Inactive) added a comment - Michelle Gower I grep'd through all ci_hsc_gen3 runs from Oct 7. There were no DELETEs in any of the tracefiles.
            Hide
            Permalink
            mgower Michelle Gower added a comment -

            Looks like this was fixed when doing the work in DM-17023. Closing ticket.

            Show
            mgower Michelle Gower added a comment - Looks like this was fixed when doing the work in DM-17023 . Closing ticket.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Christopher Stephens [X] (Inactive), Jim Bosch, Michelle Gower
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.