Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30866

Add support for replacing bad DimensionRecords, use it in DefineVisits and RawIngest, and fix old LATISS visit regions

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      We bumped the LATISS visit region padding on DM-30869, but have no way to update dimension records already in the database (other than dangerous direct-SQL usage).

      On this ticket:

      • add Registry.replaceDimensionData (should be able to delegate essentially everything to Database.replace);
      • add config options to RawIngestTask and DefineVisitsTask that switch them to use this method instead of syncDimensionData;
      • run DefineVisitsTask for LATISS in /repo/main to update those regions.

      All of these new interfaces need to come with warnings about them being advanced/admin operations that should be only used to fix bad data after careful consideration - e.g. when the visit padding is increased, or we fix something in the metadata translation.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          I think this is ready for review.  I'll actually re-run DefineVisitsTask on LATISS after the review is complete.  Somebody should probably also run it on the OODS repos, though I have no idea what the current state of visits is there.

          Instead of adding Registry.replaceDimensionData, I added an update flag to syncDimensionData and a replace flag to insertDimensionData; as with those methods more generally, the sync version is more flexible and returns more information but can't be vectorized.

          PRs are:

          First Jenkins run almost succeeded (and the ci_hscs did complete) but I accidentally included ci_cpp_gen2, which seems to no longer be maintained.  So I'm running another one without ci_hsc here:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34522/pipeline

           

          Show
          jbosch Jim Bosch added a comment - I think this is ready for review.  I'll actually re-run DefineVisitsTask on LATISS after the review is complete.  Somebody should probably also run it on the OODS repos, though I have no idea what the current state of visits is there. Instead of adding Registry.replaceDimensionData , I added an update flag to syncDimensionData and a replace flag to insertDimensionData ; as with those methods more generally, the sync version is more flexible and returns more information but can't be vectorized. PRs are: https://github.com/lsst/daf_butler/pull/543 https://github.com/lsst/obs_base/pull/384 First Jenkins run almost succeeded (and the ci_hscs did complete) but I accidentally included ci_cpp_gen2, which seems to no longer be maintained.  So I'm running another one without ci_hsc here:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34522/pipeline  
          Hide
          tjenness Tim Jenness added a comment -

          Looks good. Thanks.

          Show
          tjenness Tim Jenness added a comment - Looks good. Thanks.
          Hide
          jbosch Jim Bosch added a comment -

          I've added one more commit to the daf_butler branch to add support for testing for NULL in query expressions, which was missing functionality I discovered while trying to identify the LATISS exposures that needed padding.

          Show
          jbosch Jim Bosch added a comment - I've added one more commit to the daf_butler branch to add support for testing for NULL in query expressions, which was missing functionality I discovered while trying to identify the LATISS exposures that needed padding.

            People

            Assignee:
            jbosch Jim Bosch
            Reporter:
            jbosch Jim Bosch
            Reviewers:
            Tim Jenness
            Watchers:
            Jim Bosch, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.