Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21016

Handle DECam instrument signature data in gen3

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Sprint:
      AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February), AP S20-4 (March)
    • Team:
      Alert Production

      Description

      DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This will require at-minimum implementing writeCuratedCalibrations.

      I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Just discovered this ticket. Should it be part of the gen3-middleware work? I'm not sure how we can do DM-21915 without it. (Though I also don't understand why DM-21862 is unrelated.)

            Show
            krzys Krzysztof Findeisen added a comment - - edited Just discovered this ticket. Should it be part of the gen3-middleware work? I'm not sure how we can do DM-21915 without it. (Though I also don't understand why DM-21862 is unrelated.)
            Hide
            Parejkoj John Parejko added a comment -

            Yes, this is a blocker on using decam data for testing.

            I think Jim Bosch didn't remember that I'd filed this when he filed DM-21862: this ticket is one component of 21862 (which I think should be a small epic, but whatever).

            Show
            Parejkoj John Parejko added a comment - Yes, this is a blocker on using decam data for testing. I think Jim Bosch didn't remember that I'd filed this when he filed DM-21862 : this ticket is one component of 21862 (which I think should be a small epic, but whatever).
            Hide
            krzys Krzysztof Findeisen added a comment -

            AFAIK we're all free to break up these tickets if we think it makes sense.

            Show
            krzys Krzysztof Findeisen added a comment - AFAIK we're all free to break up these tickets if we think it makes sense.
            Hide
            jbosch Jim Bosch added a comment -

            .  I have been using blocker umbrella tickets instead of Epics just because Epics can have special meaning for T/CAMs and special rules about Team assignments in some contexts and I didn't want to stomp on any of that.

            Show
            jbosch Jim Bosch added a comment - .  I have been using blocker umbrella tickets instead of Epics just because Epics can have special meaning for T/CAMs and special rules about Team assignments in some contexts and I didn't want to stomp on any of that.
            Show
            Parejkoj John Parejko added a comment - - edited Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30899/pipeline
            Hide
            tjenness Tim Jenness added a comment - - edited

            I've had a look at the branches on this ticket and I can try to see if I can rescue them. It seems to me that we are using the term "instrument signature data" rather than "curated calibrations" – can someone explain whether these are different or the same? Is "instrument signature data" a superset of curated calibrations? We have started to call "curated calibrations" the standardized calibrations that live in the obs_x_data packages. Should writeCuratedCalibrations be renamed to writeInstrumentSignatureData?

            The obs_lsst branch removes gen3 from the module hierarchy. If we are going to do that then sooner rather than later is better before we start embedding the class name everywhere? Jim Bosch I assume you don't mind if we drop gen3. Also, do we want to use the Task infrastructure for running writeCuratedCalibrations or is a simple command that becomes part of the unified command line tools fine? Is any configuration needed other than specifying the instrument and (possibly) collection?

            John Swinbank From an earned value perspective, since I'm not on AP and haven't already spent a lot of time on this ticket, should I do the work on a different ticket (DM-23976) that I thought I would be working on anyhow?

            Show
            tjenness Tim Jenness added a comment - - edited I've had a look at the branches on this ticket and I can try to see if I can rescue them. It seems to me that we are using the term "instrument signature data" rather than "curated calibrations" – can someone explain whether these are different or the same? Is "instrument signature data" a superset of curated calibrations? We have started to call "curated calibrations" the standardized calibrations that live in the obs_x_data packages. Should writeCuratedCalibrations be renamed to writeInstrumentSignatureData? The obs_lsst branch removes gen3 from the module hierarchy. If we are going to do that then sooner rather than later is better before we start embedding the class name everywhere? Jim Bosch I assume you don't mind if we drop gen3. Also, do we want to use the Task infrastructure for running writeCuratedCalibrations or is a simple command that becomes part of the unified command line tools fine? Is any configuration needed other than specifying the instrument and (possibly) collection? John Swinbank From an earned value perspective, since I'm not on AP and haven't already spent a lot of time on this ticket, should I do the work on a different ticket ( DM-23976 ) that I thought I would be working on anyhow?
            Hide
            Parejkoj John Parejko added a comment - - edited

            I started the "curated calibrations" -> "instrument signature data" rename because nobody had provided a definition of "curated calibrations" and the things we referred to by that name are the things that go into instrument signature removal. `writeCuratedCalibrations` would be renamed `writeInstrumentSignatureData`, yes.

            Jim Bosch and I had agreed to remove the `gen3` hierarchy from things as soon as we could. It should also get pulled from HSC.

            We should also decide on a coherent set of names for Instrument classes: are they `BlahCameraInstrument` or just `BlahCamera`? I don't know how much it matters, but the time to standardize is now.

            Show
            Parejkoj John Parejko added a comment - - edited I started the "curated calibrations" -> "instrument signature data" rename because nobody had provided a definition of "curated calibrations" and the things we referred to by that name are the things that go into instrument signature removal. `writeCuratedCalibrations` would be renamed `writeInstrumentSignatureData`, yes. Jim Bosch and I had agreed to remove the `gen3` hierarchy from things as soon as we could. It should also get pulled from HSC. We should also decide on a coherent set of names for Instrument classes: are they `BlahCameraInstrument` or just `BlahCamera`? I don't know how much it matters, but the time to standardize is now.
            Hide
            swinbank John Swinbank added a comment -

            From a pure EV perspective, yes, if this work is being handed off from AP to Architecture, let's do that by writing a brief summary of what happened on this ticket, marking it as done, and then using another ticket for Arch activities. (It's not obvious from the description here whether that other ticket is DM-23976, but I have no horse in that race.)

            Show
            swinbank John Swinbank added a comment - From a pure EV perspective, yes, if this work is being handed off from AP to Architecture, let's do that by writing a brief summary of what happened on this ticket, marking it as done, and then using another ticket for Arch activities. (It's not obvious from the description here whether that other ticket is DM-23976 , but I have no horse in that race.)
            Hide
            tjenness Tim Jenness added a comment -

            I think I will put renaming of instrument classes (removal of gen3) onto a new ticket since it doesn't affect any of the other discussions. I absolutely agree that now is the time to change names and class paths.

            Show
            tjenness Tim Jenness added a comment - I think I will put renaming of instrument classes (removal of gen3) onto a new ticket since it doesn't affect any of the other discussions. I absolutely agree that now is the time to change names and class paths.
            Hide
            tjenness Tim Jenness added a comment -

            DM-23980 has been created for the instrument class renames.

            Show
            tjenness Tim Jenness added a comment - DM-23980 has been created for the instrument class renames.
            Hide
            Parejkoj John Parejko added a comment -

            Should we close this as either `Invalid` or `Done`, given DM-23976?

            Show
            Parejkoj John Parejko added a comment - Should we close this as either `Invalid` or `Done`, given DM-23976 ?
            Hide
            Parejkoj John Parejko added a comment -

            DM-23976 has implemented writeCuratedCalibrations for obs_decam, so defects and camera are now taken care of. DM-23980 took care of removing the `gen3` bit from the module hierarchies. The other cleanups and renames on this ticket are probably not worth trying to sort out in what would likely be an ugly rebase, and many of them depend on the not yet finished plan for how calibrations will be handled in general.

            I will close the four PRs on this ticket, and am marking the ticket as "done" because at least defects are now being managed per the ticket description.

            Show
            Parejkoj John Parejko added a comment - DM-23976 has implemented writeCuratedCalibrations for obs_decam, so defects and camera are now taken care of. DM-23980 took care of removing the `gen3` bit from the module hierarchies. The other cleanups and renames on this ticket are probably not worth trying to sort out in what would likely be an ugly rebase, and many of them depend on the not yet finished plan for how calibrations will be handled in general. I will close the four PRs on this ticket, and am marking the ticket as "done" because at least defects are now being managed per the ticket description.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: