Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21016

Handle DECam instrument signature data in gen3

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Sprint:
      AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February), AP S20-4 (March)
    • Team:
      Alert Production

      Description

      DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This will require at-minimum implementing writeCuratedCalibrations.

      I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).

        Attachments

          Issue Links

            Activity

            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Link This issue relates to DM-7286 [ DM-7286 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-7287 [ DM-7287 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-6495 [ DM-6495 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-17538 [ DM-17538 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggered by DM-20763 [ DM-20763 ]
            swinbank John Swinbank made changes -
            Labels SciencePipelines
            swinbank John Swinbank made changes -
            Epic Link DM-21442 [ 423049 ]
            swinbank John Swinbank made changes -
            Team Alert Production [ 10300 ]
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Just discovered this ticket. Should it be part of the gen3-middleware work? I'm not sure how we can do DM-21915 without it. (Though I also don't understand why DM-21862 is unrelated.)

            Show
            krzys Krzysztof Findeisen added a comment - - edited Just discovered this ticket. Should it be part of the gen3-middleware work? I'm not sure how we can do DM-21915 without it. (Though I also don't understand why DM-21862 is unrelated.)
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-21915 [ DM-21915 ]
            Hide
            Parejkoj John Parejko added a comment -

            Yes, this is a blocker on using decam data for testing.

            I think Jim Bosch didn't remember that I'd filed this when he filed DM-21862: this ticket is one component of 21862 (which I think should be a small epic, but whatever).

            Show
            Parejkoj John Parejko added a comment - Yes, this is a blocker on using decam data for testing. I think Jim Bosch didn't remember that I'd filed this when he filed DM-21862 : this ticket is one component of 21862 (which I think should be a small epic, but whatever).
            Hide
            krzys Krzysztof Findeisen added a comment -

            AFAIK we're all free to break up these tickets if we think it makes sense.

            Show
            krzys Krzysztof Findeisen added a comment - AFAIK we're all free to break up these tickets if we think it makes sense.
            Parejkoj John Parejko made changes -
            Link This issue is child task of DM-21862 [ DM-21862 ]
            Parejkoj John Parejko made changes -
            Labels gen2-deprecation-blocker gen3-middleware
            Parejkoj John Parejko made changes -
            Assignee John Parejko [ parejkoj ]
            Parejkoj John Parejko made changes -
            Watchers Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness [ Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness ] Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness [ Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness ]
            Hide
            jbosch Jim Bosch added a comment -

            .  I have been using blocker umbrella tickets instead of Epics just because Epics can have special meaning for T/CAMs and special rules about Team assignments in some contexts and I didn't want to stomp on any of that.

            Show
            jbosch Jim Bosch added a comment - .  I have been using blocker umbrella tickets instead of Epics just because Epics can have special meaning for T/CAMs and special rules about Team assignments in some contexts and I didn't want to stomp on any of that.
            ebellm Eric Bellm made changes -
            Sprint AP F19-6 (November) [ 958 ]
            Story Points 8
            Parejkoj John Parejko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-22370 [ DM-22370 ]
            Parejkoj John Parejko made changes -
            Summary Handle DECam calibrations in gen3 Handle DECam instrument signature data in gen3
            Parejkoj John Parejko made changes -
            Description DM-20763 dealt with ingesting DECam data, but punted on managing the calibration data. This is probably best done with a {{gen2convert}}-style script, and will require at-minimum implementing {{writeCuratedCalibrations}}.

            I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).
            DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This is probably best done with a {{gen2convert}}-style script, and will require at-minimum implementing {{writeCuratedCalibrations}}.

            I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).
            Parejkoj John Parejko made changes -
            Link This issue is blocked by DM-21155 [ DM-21155 ]
            Show
            Parejkoj John Parejko added a comment - - edited Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30899/pipeline
            swinbank John Swinbank made changes -
            Sprint AP F19-6 (November) [ 958 ] AP F19-6 (November), AP S20-1 (December) [ 958, 981 ]
            swinbank John Swinbank made changes -
            Epic Link DM-21442 [ 423049 ] DM-22633 [ 427742 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-22794 [ DM-22794 ]
            swinbank John Swinbank made changes -
            Description DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This is probably best done with a {{gen2convert}}-style script, and will require at-minimum implementing {{writeCuratedCalibrations}}.

            I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).
            DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This will require at-minimum implementing {{writeCuratedCalibrations}}.

            I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).
            swinbank John Swinbank made changes -
            Sprint AP F19-6 (November), AP S20-1 (December) [ 958, 981 ] AP F19-6 (November), AP S20-1 (December), AP S20-2 (January) [ 958, 981, 983 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-23230 [ DM-23230 ]
            swinbank John Swinbank made changes -
            Sprint AP F19-6 (November), AP S20-1 (December), AP S20-2 (January) [ 958, 981, 983 ] AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February) [ 958, 981, 983, 984 ]
            tjenness Tim Jenness made changes -
            Component/s obs_lsst [ 16504 ]
            Component/s obs_subaru [ 10747 ]
            Component/s daf_butler [ 11300 ]
            swinbank John Swinbank made changes -
            Sprint AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February) [ 958, 981, 983, 984 ] AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February), AP S20-4 (March) [ 958, 981, 983, 984, 985 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-23976 [ DM-23976 ]
            tjenness Tim Jenness made changes -
            Watchers Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness [ Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness ] Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness [ Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness ]
            tjenness Tim Jenness made changes -
            Watchers Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness [ Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness ] Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness [ Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness ]
            Hide
            tjenness Tim Jenness added a comment - - edited

            I've had a look at the branches on this ticket and I can try to see if I can rescue them. It seems to me that we are using the term "instrument signature data" rather than "curated calibrations" – can someone explain whether these are different or the same? Is "instrument signature data" a superset of curated calibrations? We have started to call "curated calibrations" the standardized calibrations that live in the obs_x_data packages. Should writeCuratedCalibrations be renamed to writeInstrumentSignatureData?

            The obs_lsst branch removes gen3 from the module hierarchy. If we are going to do that then sooner rather than later is better before we start embedding the class name everywhere? Jim Bosch I assume you don't mind if we drop gen3. Also, do we want to use the Task infrastructure for running writeCuratedCalibrations or is a simple command that becomes part of the unified command line tools fine? Is any configuration needed other than specifying the instrument and (possibly) collection?

            John Swinbank From an earned value perspective, since I'm not on AP and haven't already spent a lot of time on this ticket, should I do the work on a different ticket (DM-23976) that I thought I would be working on anyhow?

            Show
            tjenness Tim Jenness added a comment - - edited I've had a look at the branches on this ticket and I can try to see if I can rescue them. It seems to me that we are using the term "instrument signature data" rather than "curated calibrations" – can someone explain whether these are different or the same? Is "instrument signature data" a superset of curated calibrations? We have started to call "curated calibrations" the standardized calibrations that live in the obs_x_data packages. Should writeCuratedCalibrations be renamed to writeInstrumentSignatureData? The obs_lsst branch removes gen3 from the module hierarchy. If we are going to do that then sooner rather than later is better before we start embedding the class name everywhere? Jim Bosch I assume you don't mind if we drop gen3. Also, do we want to use the Task infrastructure for running writeCuratedCalibrations or is a simple command that becomes part of the unified command line tools fine? Is any configuration needed other than specifying the instrument and (possibly) collection? John Swinbank From an earned value perspective, since I'm not on AP and haven't already spent a lot of time on this ticket, should I do the work on a different ticket ( DM-23976 ) that I thought I would be working on anyhow?
            Hide
            Parejkoj John Parejko added a comment - - edited

            I started the "curated calibrations" -> "instrument signature data" rename because nobody had provided a definition of "curated calibrations" and the things we referred to by that name are the things that go into instrument signature removal. `writeCuratedCalibrations` would be renamed `writeInstrumentSignatureData`, yes.

            Jim Bosch and I had agreed to remove the `gen3` hierarchy from things as soon as we could. It should also get pulled from HSC.

            We should also decide on a coherent set of names for Instrument classes: are they `BlahCameraInstrument` or just `BlahCamera`? I don't know how much it matters, but the time to standardize is now.

            Show
            Parejkoj John Parejko added a comment - - edited I started the "curated calibrations" -> "instrument signature data" rename because nobody had provided a definition of "curated calibrations" and the things we referred to by that name are the things that go into instrument signature removal. `writeCuratedCalibrations` would be renamed `writeInstrumentSignatureData`, yes. Jim Bosch and I had agreed to remove the `gen3` hierarchy from things as soon as we could. It should also get pulled from HSC. We should also decide on a coherent set of names for Instrument classes: are they `BlahCameraInstrument` or just `BlahCamera`? I don't know how much it matters, but the time to standardize is now.
            Hide
            swinbank John Swinbank added a comment -

            From a pure EV perspective, yes, if this work is being handed off from AP to Architecture, let's do that by writing a brief summary of what happened on this ticket, marking it as done, and then using another ticket for Arch activities. (It's not obvious from the description here whether that other ticket is DM-23976, but I have no horse in that race.)

            Show
            swinbank John Swinbank added a comment - From a pure EV perspective, yes, if this work is being handed off from AP to Architecture, let's do that by writing a brief summary of what happened on this ticket, marking it as done, and then using another ticket for Arch activities. (It's not obvious from the description here whether that other ticket is DM-23976 , but I have no horse in that race.)
            Hide
            tjenness Tim Jenness added a comment -

            I think I will put renaming of instrument classes (removal of gen3) onto a new ticket since it doesn't affect any of the other discussions. I absolutely agree that now is the time to change names and class paths.

            Show
            tjenness Tim Jenness added a comment - I think I will put renaming of instrument classes (removal of gen3) onto a new ticket since it doesn't affect any of the other discussions. I absolutely agree that now is the time to change names and class paths.
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-23980 [ DM-23980 ]
            Hide
            tjenness Tim Jenness added a comment -

            DM-23980 has been created for the instrument class renames.

            Show
            tjenness Tim Jenness added a comment - DM-23980 has been created for the instrument class renames.
            Hide
            Parejkoj John Parejko added a comment -

            Should we close this as either `Invalid` or `Done`, given DM-23976?

            Show
            Parejkoj John Parejko added a comment - Should we close this as either `Invalid` or `Done`, given DM-23976 ?
            Hide
            Parejkoj John Parejko added a comment -

            DM-23976 has implemented writeCuratedCalibrations for obs_decam, so defects and camera are now taken care of. DM-23980 took care of removing the `gen3` bit from the module hierarchies. The other cleanups and renames on this ticket are probably not worth trying to sort out in what would likely be an ugly rebase, and many of them depend on the not yet finished plan for how calibrations will be handled in general.

            I will close the four PRs on this ticket, and am marking the ticket as "done" because at least defects are now being managed per the ticket description.

            Show
            Parejkoj John Parejko added a comment - DM-23976 has implemented writeCuratedCalibrations for obs_decam, so defects and camera are now taken care of. DM-23980 took care of removing the `gen3` bit from the module hierarchies. The other cleanups and renames on this ticket are probably not worth trying to sort out in what would likely be an ugly rebase, and many of them depend on the not yet finished plan for how calibrations will be handled in general. I will close the four PRs on this ticket, and am marking the ticket as "done" because at least defects are now being managed per the ticket description.
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Epic Link DM-22633 [ 427742 ] DM-24341 [ 433028 ]

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: