Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-37421

Cutout failures: diaSource outside bbox

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Some cutouts failed to generate because pixels are outside of the image bounding box. How can there be diaSources outside the exposure BBox? This looks like more of DM-29512, which doesn't appear to have resulted in any code changes.

      Ian's suggestion from standup was that the centroid can be off of the footprint for very bad/strangely shaped sources, so if the source is on the edge, the ra/dec will be outside the bbox. If this is why, we could add a "centroid outside footprint" flag for this situation, but I also wonder whether the flags in DM-37386 might be enough? We'll have to look at these sources manually on the difference image to see what's going on.

      Some examples:

      lsst.zooniverseCutouts ERROR: InvalidParameterError processing diaSourceId 140015933850250:
        File "src/image/Exposure.cc", line 215, in lsst::afw::image::Exposure<ImagePixelT, MaskPixelT, VariancePixelT> lsst::afw::image::Exposure<ImageT, MaskT, VarianceT>::getCutout(const lsst::geom::SpherePoint&, const Extent2I&) const [with ImageT = float; MaskT = int; VarianceT = float; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
          Point (150.4554217955, +1.4866807950) lies at pixel (-119.388, 3026.3), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176)) {0}
      lsst::pex::exceptions::InvalidParameterError: 'Point (150.4554217955, +1.4866807950) lies at pixel (-119.388, 3026.3), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176))'
       
      lsst.zooniverseCutouts ERROR: InvalidParameterError processing diaSourceId 153792041451941:
        File "src/image/Exposure.cc", line 215, in lsst::afw::image::Exposure<ImagePixelT, MaskPixelT, VariancePixelT> lsst::afw::image::Exposure<ImageT, MaskT, VarianceT>::getCutout(const lsst::geom::SpherePoint&, const Extent2I&) const [with ImageT = float; MaskT = int; VarianceT = float; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
          Point (149.6450048592, +1.7786541142) lies at pixel (1647.46, 4273.83), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176)) {0}
      lsst::pex::exceptions::InvalidParameterError: 'Point (149.6450048592, +1.7786541142) lies at pixel (1647.46, 4273.83), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176))'
       
      lsst.zooniverseCutouts ERROR: InvalidParameterError processing diaSourceId 504450351366776:
        File "src/image/Exposure.cc", line 215, in lsst::afw::image::Exposure<ImagePixelT, MaskPixelT, VariancePixelT> lsst::afw::image::Exposure<ImageT, MaskT, VarianceT>::getCutout(const lsst::geom::SpherePoint&, const Extent2I&) const [with ImageT = float; MaskT = int; VarianceT = float; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
          Point (149.3223656928, +2.6694136516) lies at pixel (2106.59, 3430.03), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176)) {0}
      lsst::pex::exceptions::InvalidParameterError: 'Point (149.3223656928, +2.6694136516) lies at pixel (2106.59, 3430.03), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176))'
       
      lsst.zooniverseCutouts ERROR: InvalidParameterError processing diaSourceId 506009424495146:
        File "src/image/Exposure.cc", line 215, in lsst::afw::image::Exposure<ImagePixelT, MaskPixelT, VariancePixelT> lsst::afw::image::Exposure<ImageT, MaskT, VarianceT>::getCutout(const lsst::geom::SpherePoint&, const Extent2I&) const [with ImageT = float; MaskT = int; VarianceT = float; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
          Point (149.3815315446, +1.8309280706) lies at pixel (-90.6345, 212.302), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176)) {0}
      lsst::pex::exceptions::InvalidParameterError: 'Point (149.3815315446, +1.8309280706) lies at pixel (-90.6345, 212.302), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176))'
       
      lsst.zooniverseCutouts ERROR: InvalidParameterError processing diaSourceId 506831910732048:
        File "src/image/Exposure.cc", line 215, in lsst::afw::image::Exposure<ImagePixelT, MaskPixelT, VariancePixelT> lsst::afw::image::Exposure<ImageT, MaskT, VarianceT>::getCutout(const lsst::geom::SpherePoint&, const Extent2I&) const [with ImageT = float; MaskT = int; VarianceT = float; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
          Point (150.2557111406, +1.6788628358) lies at pixel (-296.011, 830.432), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176)) {0}
      lsst::pex::exceptions::InvalidParameterError: 'Point (150.2557111406, +1.6788628358) lies at pixel (-296.011, 830.432), which lies outside Exposure Box2I(Point2I(0, 0), Extent2I(2048, 4176))'
      

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment - - edited

            Thinking about this one a bit: we should be looking at the source Footprints, to see what the actual sources that were detected on are. I'm going to try to make cutouts using those footprints (that should be independent of the x/y centroids). I can also show you how to work with SourceCatalog footprints, assuming we can figure out how to go back from DiaSourceId into the SourceCatalog id.

            Show
            Parejkoj John Parejko added a comment - - edited Thinking about this one a bit: we should be looking at the source Footprints, to see what the actual sources that were detected on are. I'm going to try to make cutouts using those footprints (that should be independent of the x/y centroids). I can also show you how to work with SourceCatalog footprints, assuming we can figure out how to go back from DiaSourceId into the SourceCatalog id.
            Hide
            Parejkoj John Parejko added a comment -

            For the record, I count 175572 objects in the December sprint APDB with this query, only 8 of them with negative x/y:

            session.query(table).where((table.columns['x'] < 0) |
                                       (table.columns['y'] < 0) | 
                                       (table.columns['x'] > 2048) | 
                                       (table.columns['y'] > 4096)).count()
            

            Rejecting sources with off-chip centroids seems reasonable (my centerAll flag proposed for DM-37386 won't help, because the centroid is off chip!), but I'd like to look at some footprint cutouts, first. I'm trying to make that work, but struggling with flag name changes caused by the switch to HSM. I think I've got most of the relevant footprint-getting code figured out, though. I'll post here when I have some example cutouts, for you to compare with what you've done Brianna Smart.

            Show
            Parejkoj John Parejko added a comment - For the record, I count 175572 objects in the December sprint APDB with this query, only 8 of them with negative x/y: session.query(table).where((table.columns['x'] < 0) | (table.columns['y'] < 0) | (table.columns['x'] > 2048) | (table.columns['y'] > 4096)).count() Rejecting sources with off-chip centroids seems reasonable (my centerAll flag proposed for DM-37386 won't help, because the centroid is off chip!), but I'd like to look at some footprint cutouts, first. I'm trying to make that work, but struggling with flag name changes caused by the switch to HSM. I think I've got most of the relevant footprint-getting code figured out, though. I'll post here when I have some example cutouts, for you to compare with what you've done Brianna Smart .
            Hide
            sullivan Ian Sullivan added a comment -

            Coming to this discussion a little late, I think it is quite safe to reject any sources with off-chip centroids. If a real trailed source extended off the chip, the centroid of that partial trailed source would still be on the chip.

            Show
            sullivan Ian Sullivan added a comment - Coming to this discussion a little late, I think it is quite safe to reject any sources with off-chip centroids. If a real trailed source extended off the chip, the centroid of that partial trailed source would still be on the chip.
            Hide
            sullivan Ian Sullivan added a comment -

            The detector images for the five bad sources seem to have been deleted in the final notebook.
            I understand you can't display a proper cutout of the bad sources in the notebook, since they are all off the chip, but a visualization of the region near the bad source would be helpful. I am thinking a ~200x200 pixel box centered on the bad source location, with any pixels with real data filled and the rest zeroes. Perhaps mark the location of the (nonexistent) source with a symbol. The idea would be to provide some visual context for the "source".

            Show
            sullivan Ian Sullivan added a comment - The detector images for the five bad sources seem to have been deleted in the final notebook. I understand you can't display a proper cutout of the bad sources in the notebook, since they are all off the chip, but a visualization of the region near the bad source would be helpful. I am thinking a ~200x200 pixel box centered on the bad source location, with any pixels with real data filled and the rest zeroes. Perhaps mark the location of the (nonexistent) source with a symbol. The idea would be to provide some visual context for the "source".
            Hide
            bsmart Brianna Smart added a comment -

            We have found that the Sdss centroid and peak centroid do not match up. We need to set max dist to peak to 0.5 for direct images. On the difference images we should experiment with 0.5 to 5-10 distances. This is important for trailed sources where the brightest pixel may not be the source center. We need to ensure doFootprintCheck is True.

             

            SdssCentroid calls it maxDistToPeak and CentroidUtilities.CentroidChecker calls it maxDistFromPeak. These names should be the same. 

            Show
            bsmart Brianna Smart added a comment - We have found that the Sdss centroid and peak centroid do not match up. We need to set max dist to peak to 0.5 for direct images. On the difference images we should experiment with 0.5 to 5-10 distances. This is important for trailed sources where the brightest pixel may not be the source center. We need to ensure doFootprintCheck is True.   SdssCentroid calls it maxDistToPeak and CentroidUtilities.CentroidChecker calls it maxDistFromPeak. These names should be the same. 

              People

              Assignee:
              bsmart Brianna Smart
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Ian Sullivan
              Watchers:
              Brianna Smart, Eric Bellm, Erin Howard, Ian Sullivan, John Parejko, Kenneth Herner
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.