Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34989

Update obs_lsst to fix common problems.

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production

      Description

      All data is now taken with 64 columns of serial overscan, so we should make that the default in the camera definition.  This will avoid the need to rework the bounding boxes on read.

      In addition, the serial overscan should extend to the top of the image, so the parallel overscan region can be corrected as well.

        Attachments

          Issue Links

            Activity

            Hide
            jchiang James Chiang added a comment - - edited

            I believe the issue is not that the size of the serial overscan should be 64 pixels.  Instead, it's that the Camera code is writing out the data for each amp using a raw bounding box of size 576 x 2048 pixels for both e2V and ITL CCDs.   As a result, for e2V, the serial overscan is 54 pixels wide, while for ITL it's 64 pixels.   Here are the relevant header excerpts from the BOT data taken on 2021-12-12:

            For an e2V CCD:

            XTENSION= 'IMAGE   '           / marks beginning of new HDU                     
            BITPIX  =                   32 / bits per data value                            
            NAXIS   =                    2 / number of axis                                 
            NAXIS1  =                  576 / size of the n'th axis                          
            NAXIS2  =                 2048 / size of the n'th axis                          
            PCOUNT  =                    0 / Required value                                 
            GCOUNT  =                    1 / Required value                                 
            CHANNEL =                    1                                                  
            EXTNAME = 'Segment10'                                                           
            CCDSUM  = '1 1     '                                                            
            DATASEC = '[11:522,1:2002]'                                                     
            DETSEC  = '[512:1,1:2002]'                                                      
            DETSIZE = '[1:4096,1:4004]'                                                     
            

            For an ITL CCD:

            XTENSION= 'IMAGE   '           / marks beginning of new HDU                     
            BITPIX  =                   32 / bits per data value                            
            NAXIS   =                    2 / number of axis                                 
            NAXIS1  =                  576 / size of the n'th axis                          
            NAXIS2  =                 2048 / size of the n'th axis                          
            PCOUNT  =                    0 / Required value                                 
            GCOUNT  =                    1 / Required value                                 
            CHANNEL =                    1                                                  
            EXTNAME = 'Segment10'                                                           
            CCDSUM  = '1 1     '                                                            
            DATASEC = '[4:512,1:2000]'                                                      
            DETSEC  = '[509:1,1:2000]'                                                      
            DETSIZE = '[1:4072,1:4000]'                                                    
            

            We should check with Tony Johnson whether these bounding boxes sizes are final or not.

            Show
            jchiang James Chiang added a comment - - edited I believe the issue is not that the size of the serial overscan should be 64 pixels.  Instead, it's that the Camera code is writing out the data for each amp using a raw bounding box of size 576 x 2048 pixels for both e2V and ITL CCDs.   As a result, for e2V, the serial overscan is 54 pixels wide, while for ITL it's 64 pixels.   Here are the relevant header excerpts from the BOT data taken on 2021-12-12: For an e2V CCD: XTENSION= 'IMAGE ' / marks beginning of new HDU BITPIX = 32 / bits per data value NAXIS = 2 / number of axis NAXIS1 = 576 / size of the n'th axis NAXIS2 = 2048 / size of the n'th axis PCOUNT = 0 / Required value GCOUNT = 1 / Required value CHANNEL = 1 EXTNAME = 'Segment10' CCDSUM = '1 1 ' DATASEC = '[11:522,1:2002]' DETSEC = '[512:1,1:2002]' DETSIZE = '[1:4096,1:4004]' For an ITL CCD: XTENSION= 'IMAGE ' / marks beginning of new HDU BITPIX = 32 / bits per data value NAXIS = 2 / number of axis NAXIS1 = 576 / size of the n'th axis NAXIS2 = 2048 / size of the n'th axis PCOUNT = 0 / Required value GCOUNT = 1 / Required value CHANNEL = 1 EXTNAME = 'Segment10' CCDSUM = '1 1 ' DATASEC = '[4:512,1:2000]' DETSEC = '[509:1,1:2000]' DETSIZE = '[1:4072,1:4000]' We should check with Tony Johnson  whether these bounding boxes sizes are final or not.
            Hide
            rhl Robert Lupton added a comment -

            Different overscans for ITL and E2V wouldn't be a problem for DM

            Show
            rhl Robert Lupton added a comment - Different overscans for ITL and E2V wouldn't be a problem for DM
            Hide
            tjohnson Tony Johnson added a comment - - edited

            The overscan size is a parameter fed to the sequencer, which could in principle change at any time (even image to image, or even for different REBs within a single image). CCS will update the relevant parameters in the header if that happens.

            There is a small probability overscan will change in practice, although it could do because a) it could be reduced to decrease the readout time or b) someone may want to do some special studies with increased overscan.

            Note that CCS also supports reading postage stamps from science CCDs even in normal readout mode. Again I am not sure this is a feature which will ever be used in practice, but there are headers defined to allow that feature to be supported in principle. 

            Show
            tjohnson Tony Johnson added a comment - - edited The overscan size is a parameter fed to the sequencer, which could in principle change at any time (even image to image, or even for different REBs within a single image). CCS will update the relevant parameters in the header if that happens. There is a small probability overscan will change in practice, although it could do because a) it could be reduced to decrease the readout time or b) someone may want to do some special studies with increased overscan. Note that CCS also supports reading postage stamps from science CCDs even in normal readout mode. Again I am not sure this is a feature which will ever be used in practice, but there are headers defined to allow that feature to be supported in principle. 
            Hide
            czw Christopher Waters added a comment -

            The implementation I've added fixes the ITL overscan size by setting it to 64 pixels.  I did not extend it to cover the doubly-overscanned region, as the current ip_isr overscan code automatically does extension when doParallelOverscan is enabled.

            The main goal of this fix was to remove the log messages that are sometimes confusing.  In addition, I removed another log message that would trigger for every assembly, as the header DETSEC and the values in the cameraGeom object disagree.

            Finally, to ensure consistent processing of obs_lsst data, I've made the MEDIAN_PER_ROW overscan correction the default.  This is widely acknowledged as the current best, and having it not be the default has created a number of confusing results that were traced back to overscan fit mismatch.

            Show
            czw Christopher Waters added a comment - The implementation I've added fixes the ITL overscan size by setting it to 64 pixels.  I did not extend it to cover the doubly-overscanned region, as the current ip_isr overscan code automatically does extension when doParallelOverscan is enabled. The main goal of this fix was to remove the log messages that are sometimes confusing.  In addition, I removed another log message that would trigger for every assembly, as the header DETSEC and the values in the cameraGeom object disagree. Finally, to ensure consistent processing of obs_lsst data, I've made the MEDIAN_PER_ROW overscan correction the default.  This is widely acknowledged as the current best, and having it not be the default has created a number of confusing results that were traced back to overscan fit mismatch.
            Show
            czw Christopher Waters added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/37587/pipeline
            Hide
            erykoff Eli Rykoff added a comment -

            Small change requested on the wording of the comment/todo.

            Show
            erykoff Eli Rykoff added a comment - Small change requested on the wording of the comment/todo.

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              czw Christopher Waters
              Reviewers:
              Eli Rykoff
              Watchers:
              Christopher Waters, Eli Rykoff, James Chiang, Robert Lupton, Tony Johnson
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.