# Handle DECam instrument signature data in gen3

XMLWordPrintable

#### Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
8
• Sprint:
AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February), AP S20-4 (March)
• Team:

#### Description

DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This will require at-minimum implementing writeCuratedCalibrations.

I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).

#### Activity

John Parejko created issue -
Field Original Value New Value
Link This issue relates to DM-7286 [ DM-7286 ]
 Link This issue relates to DM-7287 [ DM-7287 ]
 Link This issue relates to DM-6495 [ DM-6495 ]
 Link This issue relates to DM-17538 [ DM-17538 ]
 Link This issue is triggered by DM-20763 [ DM-20763 ]
 Labels SciencePipelines
 Epic Link DM-21442 [ 423049 ]
 Team Alert Production [ 10300 ]
Hide
Krzysztof Findeisen added a comment - - edited

Just discovered this ticket. Should it be part of the gen3-middleware work? I'm not sure how we can do DM-21915 without it. (Though I also don't understand why DM-21862 is unrelated.)

Show
Krzysztof Findeisen added a comment - - edited Just discovered this ticket. Should it be part of the gen3-middleware work? I'm not sure how we can do DM-21915 without it. (Though I also don't understand why DM-21862 is unrelated.)
 Link This issue blocks DM-21915 [ DM-21915 ]
Hide
John Parejko added a comment -

Yes, this is a blocker on using decam data for testing.

I think Jim Bosch didn't remember that I'd filed this when he filed DM-21862: this ticket is one component of 21862 (which I think should be a small epic, but whatever).

Show
John Parejko added a comment - Yes, this is a blocker on using decam data for testing. I think Jim Bosch didn't remember that I'd filed this when he filed DM-21862 : this ticket is one component of 21862 (which I think should be a small epic, but whatever).
Hide
Krzysztof Findeisen added a comment -

AFAIK we're all free to break up these tickets if we think it makes sense.

Show
Krzysztof Findeisen added a comment - AFAIK we're all free to break up these tickets if we think it makes sense.
 Labels gen2-deprecation-blocker gen3-middleware
 Assignee John Parejko [ parejkoj ]
 Watchers Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness [ Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness ] Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness [ Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness ]
Hide
Jim Bosch added a comment -

.  I have been using blocker umbrella tickets instead of Epics just because Epics can have special meaning for T/CAMs and special rules about Team assignments in some contexts and I didn't want to stomp on any of that.

Show
Jim Bosch added a comment - .  I have been using blocker umbrella tickets instead of Epics just because Epics can have special meaning for T/CAMs and special rules about Team assignments in some contexts and I didn't want to stomp on any of that.
 Sprint AP F19-6 (November) [ 958 ] Story Points 8
 Status To Do [ 10001 ] In Progress [ 3 ]
 Link This issue relates to DM-22370 [ DM-22370 ]
 Summary Handle DECam calibrations in gen3 Handle DECam instrument signature data in gen3
 Description DM-20763 dealt with ingesting DECam data, but punted on managing the calibration data. This is probably best done with a {{gen2convert}}-style script, and will require at-minimum implementing {{writeCuratedCalibrations}}. I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific). DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This is probably best done with a {{gen2convert}}-style script, and will require at-minimum implementing {{writeCuratedCalibrations}}. I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).
 Link This issue is blocked by DM-21155 [ DM-21155 ]
Hide
John Parejko added a comment - - edited
Show
John Parejko added a comment - - edited Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30899/pipeline
 Sprint AP F19-6 (November) [ 958 ] AP F19-6 (November), AP S20-1 (December) [ 958, 981 ]
 Epic Link DM-21442 [ 423049 ] DM-22633 [ 427742 ]
 Link This issue relates to DM-22794 [ DM-22794 ]
 Description DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This is probably best done with a {{gen2convert}}-style script, and will require at-minimum implementing {{writeCuratedCalibrations}}. I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific). DM-20763 dealt with ingesting DECam data, but punted on managing the instrument signature data. This will require at-minimum implementing {{writeCuratedCalibrations}}. I've linked a variety of DECam calibration-related tickets as "related', since we can maybe close some of those as won't fix once this is done (since they might be gen2-specific).
 Sprint AP F19-6 (November), AP S20-1 (December) [ 958, 981 ] AP F19-6 (November), AP S20-1 (December), AP S20-2 (January) [ 958, 981, 983 ]
 Link This issue relates to DM-23230 [ DM-23230 ]
 Sprint AP F19-6 (November), AP S20-1 (December), AP S20-2 (January) [ 958, 981, 983 ] AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February) [ 958, 981, 983, 984 ]
 Component/s obs_lsst [ 16504 ] Component/s obs_subaru [ 10747 ] Component/s daf_butler [ 11300 ]
 Sprint AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February) [ 958, 981, 983, 984 ] AP F19-6 (November), AP S20-1 (December), AP S20-2 (January), AP S20-3 (February), AP S20-4 (March) [ 958, 981, 983, 984, 985 ]
 Link This issue relates to DM-23976 [ DM-23976 ]
 Watchers Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness [ Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness ] Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness [ Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness ]
 Watchers Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness [ Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness ] Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness [ Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness ]
Hide
Tim Jenness added a comment - - edited

I've had a look at the branches on this ticket and I can try to see if I can rescue them. It seems to me that we are using the term "instrument signature data" rather than "curated calibrations" – can someone explain whether these are different or the same? Is "instrument signature data" a superset of curated calibrations? We have started to call "curated calibrations" the standardized calibrations that live in the obs_x_data packages. Should writeCuratedCalibrations be renamed to writeInstrumentSignatureData?

The obs_lsst branch removes gen3 from the module hierarchy. If we are going to do that then sooner rather than later is better before we start embedding the class name everywhere? Jim Bosch I assume you don't mind if we drop gen3. Also, do we want to use the Task infrastructure for running writeCuratedCalibrations or is a simple command that becomes part of the unified command line tools fine? Is any configuration needed other than specifying the instrument and (possibly) collection?

John Swinbank From an earned value perspective, since I'm not on AP and haven't already spent a lot of time on this ticket, should I do the work on a different ticket (DM-23976) that I thought I would be working on anyhow?

Show
Tim Jenness added a comment - - edited I've had a look at the branches on this ticket and I can try to see if I can rescue them. It seems to me that we are using the term "instrument signature data" rather than "curated calibrations" – can someone explain whether these are different or the same? Is "instrument signature data" a superset of curated calibrations? We have started to call "curated calibrations" the standardized calibrations that live in the obs_x_data packages. Should writeCuratedCalibrations be renamed to writeInstrumentSignatureData? The obs_lsst branch removes gen3 from the module hierarchy. If we are going to do that then sooner rather than later is better before we start embedding the class name everywhere? Jim Bosch I assume you don't mind if we drop gen3. Also, do we want to use the Task infrastructure for running writeCuratedCalibrations or is a simple command that becomes part of the unified command line tools fine? Is any configuration needed other than specifying the instrument and (possibly) collection? John Swinbank From an earned value perspective, since I'm not on AP and haven't already spent a lot of time on this ticket, should I do the work on a different ticket ( DM-23976 ) that I thought I would be working on anyhow?
Hide
John Parejko added a comment - - edited

I started the "curated calibrations" -> "instrument signature data" rename because nobody had provided a definition of "curated calibrations" and the things we referred to by that name are the things that go into instrument signature removal. writeCuratedCalibrations would be renamed writeInstrumentSignatureData, yes.

Jim Bosch and I had agreed to remove the gen3 hierarchy from things as soon as we could. It should also get pulled from HSC.

We should also decide on a coherent set of names for Instrument classes: are they BlahCameraInstrument or just BlahCamera? I don't know how much it matters, but the time to standardize is now.

Show
John Parejko added a comment - - edited I started the "curated calibrations" -> "instrument signature data" rename because nobody had provided a definition of "curated calibrations" and the things we referred to by that name are the things that go into instrument signature removal. writeCuratedCalibrations would be renamed writeInstrumentSignatureData, yes. Jim Bosch and I had agreed to remove the gen3 hierarchy from things as soon as we could. It should also get pulled from HSC. We should also decide on a coherent set of names for Instrument classes: are they BlahCameraInstrument or just BlahCamera? I don't know how much it matters, but the time to standardize is now.
Hide
John Swinbank added a comment -

From a pure EV perspective, yes, if this work is being handed off from AP to Architecture, let's do that by writing a brief summary of what happened on this ticket, marking it as done, and then using another ticket for Arch activities. (It's not obvious from the description here whether that other ticket is DM-23976, but I have no horse in that race.)

Show
John Swinbank added a comment - From a pure EV perspective, yes, if this work is being handed off from AP to Architecture, let's do that by writing a brief summary of what happened on this ticket, marking it as done, and then using another ticket for Arch activities. (It's not obvious from the description here whether that other ticket is DM-23976 , but I have no horse in that race.)
Hide
Tim Jenness added a comment -

I think I will put renaming of instrument classes (removal of gen3) onto a new ticket since it doesn't affect any of the other discussions. I absolutely agree that now is the time to change names and class paths.

Show
Tim Jenness added a comment - I think I will put renaming of instrument classes (removal of gen3) onto a new ticket since it doesn't affect any of the other discussions. I absolutely agree that now is the time to change names and class paths.
 Link This issue relates to DM-23980 [ DM-23980 ]
Hide
Tim Jenness added a comment -

DM-23980 has been created for the instrument class renames.

Show
Tim Jenness added a comment - DM-23980 has been created for the instrument class renames.
Hide
John Parejko added a comment -

Should we close this as either Invalid or Done, given DM-23976?

Show
John Parejko added a comment - Should we close this as either Invalid or Done, given DM-23976 ?
Hide
John Parejko added a comment -

DM-23976 has implemented writeCuratedCalibrations for obs_decam, so defects and camera are now taken care of. DM-23980 took care of removing the gen3 bit from the module hierarchies. The other cleanups and renames on this ticket are probably not worth trying to sort out in what would likely be an ugly rebase, and many of them depend on the not yet finished plan for how calibrations will be handled in general.

I will close the four PRs on this ticket, and am marking the ticket as "done" because at least defects are now being managed per the ticket description.

Show
John Parejko added a comment - DM-23976 has implemented writeCuratedCalibrations for obs_decam, so defects and camera are now taken care of. DM-23980 took care of removing the gen3 bit from the module hierarchies. The other cleanups and renames on this ticket are probably not worth trying to sort out in what would likely be an ugly rebase, and many of them depend on the not yet finished plan for how calibrations will be handled in general. I will close the four PRs on this ticket, and am marking the ticket as "done" because at least defects are now being managed per the ticket description.
 Resolution Done [ 10000 ] Status In Progress [ 3 ] Done [ 10002 ]
 Epic Link DM-22633 [ 427742 ] DM-24341 [ 433028 ]

#### People

Assignee:
John Parejko
Reporter:
John Parejko
Watchers:
Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness