Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31890

Attempt to re-register the LATISS instrument with butler register-instrument fails for /repo/main

    XMLWordPrintable

    Details

    • Team:
      Data Facility
    • Urgent?:
      No

      Description

      Newer versions of the LSST Stack introduced new filters for LATISS instrument. As a result, ingest attempts for Auxtel images to the /repo/main at NCSA fail due to the Butler not recognizing them. For example:

      psycopg2.errors.ForeignKeyViolation: insert or update on table "exposure" violates foreign key constraint "fkey_exposure_physical_filter_instrument_name_instrume_ee4bb371"
      DETAIL:  Key (instrument, physical_filter)=(LATISS, FELH0600~empty) is not present in table "physical_filter".
      

      According to the discussion in this thread, the new version of the instrument (introducing the new filters) can be re-registered with butler register-instrument. However, my attempt to do that failed with the following error:

      lsst.daf.butler.registry.interfaces._database.DatabaseConflictError: Conflict in sync for table instrument on column(s) visit_max: 3050123199999 != 6050123199999, exposure_max: 3050123199999 != 6050123199999.
      

      The full traceback can be found here.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Yes, that change happened in DM-16297 I think. There would need to be a way to pass down a flag from register-instrument saying that items that are different can be overwritten.

            Show
            tjenness Tim Jenness added a comment - Yes, that change happened in DM-16297 I think. There would need to be a way to pass down a flag from register-instrument saying that items that are different can be overwritten.
            Hide
            jbosch Jim Bosch added a comment -

            That doesn't make sense to me; DM-16297 was merged long before this repo was created (which was February of this year).  My previous guess from some git blaming (not rigorously confirmed) was

            https://github.com/lsst/obs_lsst/commit/16a9178d8f4d6119ad9d21a6940fa5c86cdb16a1

            and there's more discussion on this slack thread:

            https://lsstc.slack.com/archives/C2JPT1KB7/p1631885321252900

            I can go make the code changes to add a update kwarg/option to instrument registration, but I'd really like somebody else to check whether the new or old values are actually the desired ones for LATISS.

            Show
            jbosch Jim Bosch added a comment - That doesn't make sense to me; DM-16297 was merged long before this repo was created (which was February of this year).  My previous guess from some git blaming (not rigorously confirmed) was https://github.com/lsst/obs_lsst/commit/16a9178d8f4d6119ad9d21a6940fa5c86cdb16a1 and there's more discussion on this slack thread: https://lsstc.slack.com/archives/C2JPT1KB7/p1631885321252900 I can go make the code changes to add a update kwarg/option to instrument registration, but I'd really like somebody else to check whether the new or old values are actually the desired ones for LATISS.
            Hide
            tjenness Tim Jenness added a comment -

            Jim Bosch neither of us are right. My change moved from hard-coded to using the general code. The change you mention optimized the calculation for arbitrary controllers. I think the real problem was caused by DM-30142 which added new controllers to the list and that forced the number to go up.

            This code assumes that when visit != exposure the visit we do generate is a smaller int than the one coming from exposure. I'd have to check on that since I can't remember.

            Show
            tjenness Tim Jenness added a comment - Jim Bosch neither of us are right. My change moved from hard-coded to using the general code. The change you mention optimized the calculation for arbitrary controllers. I think the real problem was caused by DM-30142 which added new controllers to the list and that forced the number to go up. This code assumes that when visit != exposure the visit we do generate is a smaller int than the one coming from exposure. I'd have to check on that since I can't remember.
            Hide
            jbosch Jim Bosch added a comment -

            Ok, "new controller" makes it sounds like the new values are indeed better, so a forced update is the way to go.

            DM-31903 has the functionality for forced updates and is now in review, so this ticket can just be re-running butler register-instrument with that branch.

            Show
            jbosch Jim Bosch added a comment - Ok, "new controller" makes it sounds like the new values are indeed better, so a forced update is the way to go. DM-31903 has the functionality for forced updates and is now in review, so this ticket can just be re-running butler register-instrument with that branch.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mkowalik Mikolaj Kowalik
              Watchers:
              Jim Bosch, Mikolaj Kowalik, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins Builds

                  No builds found.