# Update obs_lsst to fix common problems.

XMLWordPrintable

#### Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Data Release Production

#### Description

All data is now taken with 64 columns of serial overscan, so we should make that the default in the camera definition.  This will avoid the need to rework the bounding boxes on read.

In addition, the serial overscan should extend to the top of the image, so the parallel overscan region can be corrected as well.

#### Activity

Hide
James Chiang added a comment - - edited

I believe the issue is not that the size of the serial overscan should be 64 pixels.  Instead, it's that the Camera code is writing out the data for each amp using a raw bounding box of size 576 x 2048 pixels for both e2V and ITL CCDs.   As a result, for e2V, the serial overscan is 54 pixels wide, while for ITL it's 64 pixels.   Here are the relevant header excerpts from the BOT data taken on 2021-12-12:

For an e2V CCD:

 XTENSION= 'IMAGE ' / marks beginning of new HDU  BITPIX = 32 / bits per data value  NAXIS = 2 / number of axis  NAXIS1 = 576 / size of the n'th axis  NAXIS2 = 2048 / size of the n'th axis  PCOUNT = 0 / Required value  GCOUNT = 1 / Required value  CHANNEL = 1  EXTNAME = 'Segment10'  CCDSUM = '1 1 '  DATASEC = '[11:522,1:2002]'  DETSEC = '[512:1,1:2002]'  DETSIZE = '[1:4096,1:4004]' 

For an ITL CCD:

 XTENSION= 'IMAGE ' / marks beginning of new HDU  BITPIX = 32 / bits per data value  NAXIS = 2 / number of axis  NAXIS1 = 576 / size of the n'th axis  NAXIS2 = 2048 / size of the n'th axis  PCOUNT = 0 / Required value  GCOUNT = 1 / Required value  CHANNEL = 1  EXTNAME = 'Segment10'  CCDSUM = '1 1 '  DATASEC = '[4:512,1:2000]'  DETSEC = '[509:1,1:2000]'  DETSIZE = '[1:4072,1:4000]' 

We should check with Tony Johnson whether these bounding boxes sizes are final or not.

Show
James Chiang added a comment - - edited I believe the issue is not that the size of the serial overscan should be 64 pixels.  Instead, it's that the Camera code is writing out the data for each amp using a raw bounding box of size 576 x 2048 pixels for both e2V and ITL CCDs.   As a result, for e2V, the serial overscan is 54 pixels wide, while for ITL it's 64 pixels.   Here are the relevant header excerpts from the BOT data taken on 2021-12-12: For an e2V CCD: XTENSION= 'IMAGE ' / marks beginning of new HDU BITPIX = 32 / bits per data value NAXIS = 2 / number of axis NAXIS1 = 576 / size of the n'th axis NAXIS2 = 2048 / size of the n'th axis PCOUNT = 0 / Required value GCOUNT = 1 / Required value CHANNEL = 1 EXTNAME = 'Segment10' CCDSUM = '1 1 ' DATASEC = '[11:522,1:2002]' DETSEC = '[512:1,1:2002]' DETSIZE = '[1:4096,1:4004]' For an ITL CCD: XTENSION= 'IMAGE ' / marks beginning of new HDU BITPIX = 32 / bits per data value NAXIS = 2 / number of axis NAXIS1 = 576 / size of the n'th axis NAXIS2 = 2048 / size of the n'th axis PCOUNT = 0 / Required value GCOUNT = 1 / Required value CHANNEL = 1 EXTNAME = 'Segment10' CCDSUM = '1 1 ' DATASEC = '[4:512,1:2000]' DETSEC = '[509:1,1:2000]' DETSIZE = '[1:4072,1:4000]' We should check with Tony Johnson  whether these bounding boxes sizes are final or not.
Hide
Robert Lupton added a comment -

Different overscans for ITL and E2V wouldn't be a problem for DM

Show
Robert Lupton added a comment - Different overscans for ITL and E2V wouldn't be a problem for DM
Hide
Tony Johnson added a comment - - edited

The overscan size is a parameter fed to the sequencer, which could in principle change at any time (even image to image, or even for different REBs within a single image). CCS will update the relevant parameters in the header if that happens.

There is a small probability overscan will change in practice, although it could do because a) it could be reduced to decrease the readout time or b) someone may want to do some special studies with increased overscan.

Note that CCS also supports reading postage stamps from science CCDs even in normal readout mode. Again I am not sure this is a feature which will ever be used in practice, but there are headers defined to allow that feature to be supported in principle.

Show
Tony Johnson added a comment - - edited The overscan size is a parameter fed to the sequencer, which could in principle change at any time (even image to image, or even for different REBs within a single image). CCS will update the relevant parameters in the header if that happens. There is a small probability overscan will change in practice, although it could do because a) it could be reduced to decrease the readout time or b) someone may want to do some special studies with increased overscan. Note that CCS also supports reading postage stamps from science CCDs even in normal readout mode. Again I am not sure this is a feature which will ever be used in practice, but there are headers defined to allow that feature to be supported in principle.
Hide
Christopher Waters added a comment -

The implementation I've added fixes the ITL overscan size by setting it to 64 pixels.  I did not extend it to cover the doubly-overscanned region, as the current ip_isr overscan code automatically does extension when doParallelOverscan is enabled.

The main goal of this fix was to remove the log messages that are sometimes confusing.  In addition, I removed another log message that would trigger for every assembly, as the header DETSEC and the values in the cameraGeom object disagree.

Finally, to ensure consistent processing of obs_lsst data, I've made the MEDIAN_PER_ROW overscan correction the default.  This is widely acknowledged as the current best, and having it not be the default has created a number of confusing results that were traced back to overscan fit mismatch.

Show
Christopher Waters added a comment - The implementation I've added fixes the ITL overscan size by setting it to 64 pixels.  I did not extend it to cover the doubly-overscanned region, as the current ip_isr overscan code automatically does extension when doParallelOverscan is enabled. The main goal of this fix was to remove the log messages that are sometimes confusing.  In addition, I removed another log message that would trigger for every assembly, as the header DETSEC and the values in the cameraGeom object disagree. Finally, to ensure consistent processing of obs_lsst data, I've made the MEDIAN_PER_ROW overscan correction the default.  This is widely acknowledged as the current best, and having it not be the default has created a number of confusing results that were traced back to overscan fit mismatch.
Hide
Christopher Waters added a comment -
Show
Christopher Waters added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/37587/pipeline
Hide
Eli Rykoff added a comment -

Small change requested on the wording of the comment/todo.

Show
Eli Rykoff added a comment - Small change requested on the wording of the comment/todo.

#### People

Assignee:
Christopher Waters
Reporter:
Christopher Waters
Reviewers:
Eli Rykoff
Watchers:
Christopher Waters, Eli Rykoff, James Chiang, Robert Lupton, Tony Johnson