Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19336

Columnar defect identifcation

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The system for identifying defects on DM-17490 could be more effective if it could simply flag whole (or partial) columns as bad when they contain obviously-suspicious patterns (e.g. alternating on/off pixels).

        Attachments

        1. FindDefects_may22.py
          19 kB
        2. FindDefects.py
          18 kB
        3. gap_columnar.pdf
          42 kB
        4. image-2019-05-17-12-10-19-793.png
          image-2019-05-17-12-10-19-793.png
          43 kB
        5. image-2019-05-17-12-10-44-970.png
          image-2019-05-17-12-10-44-970.png
          43 kB
        6. image-2019-05-17-12-16-18-807.png
          image-2019-05-17-12-16-18-807.png
          44 kB
        7. image-2019-05-22-16-49-07-235.png
          image-2019-05-22-16-49-07-235.png
          55 kB
        8. image-2019-05-22-16-49-55-692.png
          image-2019-05-22-16-49-55-692.png
          48 kB
        9. no_gap_columnar.pdf
          35 kB

          Issue Links

            Activity

            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            To address what is described in this ticket, I added a few lines of code at the end of the function `find_hot_and_cold_pixels` in the code described in DM-19334. Updated version of the code: FindDefects.py

            TS8 data, detector S10, amp C5: 

            After: 

             ^ ^

             

             

             

             

             

             

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited To address what is described in this ticket, I added a few lines of code at the end of the function `find_hot_and_cold_pixels` in the code described in  DM-19334 . Updated version of the code:  FindDefects.py TS8 data, detector S10, amp C5:   After:   ^ ^            
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Columns are often bad from a point up or down (depending on readout direction, i.e. which half of the chip they're on) and then good for the rest.

            I think this code should be able to work out, for a column that's got a line of pixels on the cusp of being bad but then goes good again, how big the bbox needs to be, as opposed to just masking the whole thing.

            I realise it kind of does this already, given that it takes min and max y on that column, but a single pixel near the edge would result in the whole thing being marked bad.

            Does that sound like a feasible update?

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Columns are often bad from a point up or down (depending on readout direction, i.e. which half of the chip they're on) and then good for the rest. I think this code should be able to work out, for a column that's got a line of pixels on the cusp of being bad but then goes good again, how big the bbox needs to be, as opposed to just masking the whole thing. I realise it kind of does this already, given that it takes min and max y on that column, but a single pixel near the edge would result in the whole thing being marked bad. Does that sound like a feasible update?
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            I added a few extra lines to the same of the code (function `find_hot_and_cold_pixels`) to address Merlin's comment above. Updated code: FindDefects_may22.py

             

            Example: 

            1) Masking all the pixels between min_y and max_y on a particular column, even if there are relatively large gaps of "good" pixels in between: 

             

            2) If there's a gap of "good pixels" larger than the new parameter "gap_threshold" (30 in this example), then don't mask them: 

             

             

            Full PDFs: 

             no_gap_columnar.pdf

             gap_columnar.pdf

             

             

             

             

             

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited I added a few extra lines to the same of the code (function `find_hot_and_cold_pixels`) to address Merlin's comment above. Updated code:  FindDefects_may22.py   Example:  1) Masking all the pixels between min_y and max_y on a particular column, even if there are relatively large gaps of "good" pixels in between:    2) If there's a gap of "good pixels" larger than the new parameter "gap_threshold" (30 in this example), then don't mask them:      Full PDFs:    no_gap_columnar.pdf   gap_columnar.pdf          
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment - - edited

            Haven't looked at the code, but that doesn't sound like a bad approach for now. Could you file a new ticket that says to add this as a feature to the findDefectsTask in cp_pipe please? (and post the relevant code on that ticket too)

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - - edited Haven't looked at the code, but that doesn't sound like a bad approach for now. Could you file a new ticket that says to add this as a feature to the findDefectsTask in cp_pipe please? (and post the relevant code on that ticket too)
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Looks good. Just post the code on DM-19903 and then mark as Done

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Looks good. Just post the code on DM-19903 and then mark as Done
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Code posted on DM-19903 

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Code posted on  DM-19903  

              People

              • Assignee:
                plazas Andrés Alejandro Plazas Malagón
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Merlin Fisher-Levine
                Watchers:
                Andrés Alejandro Plazas Malagón, John Swinbank, Merlin Fisher-Levine
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel