Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20535

Inspect light curves of flag-filtered DIAObjects

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Story Points:
      6
    • Sprint:
      AP F19-1, AP F19-2, AP F19-4 (2H September)
    • Team:
      Alert Production

      Description

      Following DM-20435, we have a working set of flags used to filter out bad DIASources. See if the problems we saw before with highly variable nonsense totFlux (and also psFlux, apFlux) in the PPDB still appears with the remaining flag-filtered DIAObjects. Think about what we need to do to fix any remaining weirdness. (Some metric we can compute from flux data in the PPDB/diasrc/src to determine if a DIAObject is "really variable" or not? Some flux cut or SNR cut? Some measure of how far the position of the DIASource is from the DIAObject?)

        Attachments

          Activity

          Hide
          ebellm Eric Bellm added a comment -

          raDecCov is not populated yet for DIAObjects, but we wonder (in discussions 7/15/19) if calculating the variance (etc.) in DIASource distances from the DIAObject would identify the sorts of centroid problems that lead to bad forced flux measurements.

          Show
          ebellm Eric Bellm added a comment - raDecCov is not populated yet for DIAObjects, but we wonder (in discussions 7/15/19) if calculating the variance (etc.) in DIASource distances from the DIAObject would identify the sorts of centroid problems that lead to bad forced flux measurements.
          Hide
          mrawls Meredith Rawls added a comment -

          The work for this ticket is entirely in the jupyter notebook on the PR.

          Show
          mrawls Meredith Rawls added a comment - The work for this ticket is entirely in the jupyter notebook on the PR.
          Hide
          mrawls Meredith Rawls added a comment -

          After chatting with Sophie and learning she'll be on vacation for a bit, I'm going to assign this to a new reviewer after I make some updates to the notebook prompted from discussions during a recent ap_pipe meeting.

          Show
          mrawls Meredith Rawls added a comment - After chatting with Sophie and learning she'll be on vacation for a bit, I'm going to assign this to a new reviewer after I make some updates to the notebook prompted from discussions during a recent ap_pipe meeting.
          Hide
          mrawls Meredith Rawls added a comment -

          I have revamped the notebook for this ticket extensively. I've copied the intro text block from the notebook here for easy reference.

           

          In this notebook:

          • We use Ian's new high-res CompareWarp template ap_pipe rerun as the latest and greatest.
          • We verify the `totFlux` light curves (which are forced photometry on the calexp at the DIA Source location) for the flag-filtered "good" sources still show lots more scatter (variability), much of which probably isn't real.
          • We inspect histograms showing object counts as a function of number of sources as well as `gPSFluxMean`. Each histogram shows a comparison between all sources vs. "good" flag-filtered sources. We note that throwing out sources with the shape flag throws out all negative sources, whoops.
          • We do a quick plot of all the "good" sources on the sky as well as a zoomed mini-region with various flagged sources plotted in red, and a corresponding flag tally bar chart. No surprises here (except for the recent point that the new high-res CompareWarp templates got rid of a lot of junk! hooray!).
          • We plot light curves for 10 "good" objects and explore the idea of computing a metric that can differentiate real variability from bogus variability (spoiler: this is more complicated than originally hoped)
          • We plot postage stamp cutouts of template images for the same 10 objects with object and source locations overlaid. These are instructive in seeing how the source location dances around, how the PSF size varies, and how many of the sources comprising a nominally "good" object are in fact not so good.
          • We plot postage stamps of *all* the calexps and diffims for the same 10 objects.
          • We very briefly explore trying the SimpleShape plugin during diffim measurement for a small data subset (instead of SdssShape) to see if it allows negative sources to be un-flagged (spoiler: yes!).

          Some key takeaways:

          • The high-res templates continue to be much better than the older lower-res templates.
          • The general shape failure flag is set for all negative sources in the difference images, eeeek!
          • Before we fixed the background subtraction problems, the "shape" flag (from the SdssShape algorithm) was a useful diagnostic, and we didn't even realize it was throwing out all the negative sources. Now we have removed nearly all of the background junk, and this problem is more obvious.
          • There are large-scale lines (satellites?) that appear in the all-good-sources-on-the-sky plot now.
          • As we (mostly) knew, the DIA Source location does move around a decent amount, but now we can visualize that with little thumbnails of each differencing template overlaid with Object and Source positions. Hooray!
          • PSF sizes can vary for lots of reasons, and none of the analysis here takes into count S/N or seeing, but it is still somewhat illustrative to plot the Source positions with circles illustrating their PSF sizes (determinant radii).
          • The criteria for calling a DIA Object "good" means it is composed of one or more "good" DIA Sources. This by definition means it can also be composed of a few "bad" DIA Sources.
          • Plotting some light curves does suggest a correlation between erroneous flux trends and how far a source is from the object location; individual source diffim cutouts also continue to show some funky shapes that are hard to fit a PSF location to.
          • There is no immediately obvious way to make a cut on which `totFlux` light curves points are likely to be discrepant.
            * It is possible to use the SimpleShape plugin for difference imaging source measurement instead of SdssShape without a failure flag happening for all negative detections. More work needs to be done to determine what is causing SdssShape to throw a flag for negative detections in the first place and whether it we want to use a different shape algorithm or not.
          Show
          mrawls Meredith Rawls added a comment - I have revamped the notebook for this ticket extensively. I've copied the intro text block from the notebook here for easy reference.   In this notebook: We use Ian's new high-res CompareWarp template ap_pipe rerun as the latest and greatest. We verify the `totFlux` light curves (which are forced photometry on the calexp at the DIA Source location) for the flag-filtered "good" sources still show lots more scatter (variability), much of which probably isn't real. We inspect histograms showing object counts as a function of number of sources as well as `gPSFluxMean`. Each histogram shows a comparison between all sources vs. "good" flag-filtered sources. We note that throwing out sources with the shape flag throws out all negative sources, whoops. We do a quick plot of all the "good" sources on the sky as well as a zoomed mini-region with various flagged sources plotted in red, and a corresponding flag tally bar chart. No surprises here (except for the recent point that the new high-res CompareWarp templates got rid of a lot of junk! hooray!). We plot light curves for 10 "good" objects and explore the idea of computing a metric that can differentiate real variability from bogus variability (spoiler: this is more complicated than originally hoped) We plot postage stamp cutouts of template images for the same 10 objects with object and source locations overlaid. These are instructive in seeing how the source location dances around, how the PSF size varies, and how many of the sources comprising a nominally "good" object are in fact not so good. We plot postage stamps of * all * the calexps and diffims for the same 10 objects. We very briefly explore trying the SimpleShape plugin during diffim measurement for a small data subset (instead of SdssShape) to see if it allows negative sources to be un-flagged (spoiler: yes!). Some key takeaways: The high-res templates continue to be much better than the older lower-res templates. The general shape failure flag is set for all negative sources in the difference images, eeeek! Before we fixed the background subtraction problems, the "shape" flag (from the SdssShape algorithm) was a useful diagnostic, and we didn't even realize it was throwing out all the negative sources. Now we have removed nearly all of the background junk, and this problem is more obvious. There are large-scale lines (satellites?) that appear in the all-good-sources-on-the-sky plot now. As we (mostly) knew, the DIA Source location does move around a decent amount, but now we can visualize that with little thumbnails of each differencing template overlaid with Object and Source positions. Hooray! PSF sizes can vary for lots of reasons, and none of the analysis here takes into count S/N or seeing, but it is still somewhat illustrative to plot the Source positions with circles illustrating their PSF sizes (determinant radii). The criteria for calling a DIA Object "good" means it is composed of one or more "good" DIA Sources. This by definition means it can also be composed of a few "bad" DIA Sources. Plotting some light curves does suggest a correlation between erroneous flux trends and how far a source is from the object location; individual source diffim cutouts also continue to show some funky shapes that are hard to fit a PSF location to. There is no immediately obvious way to make a cut on which `totFlux` light curves points are likely to be discrepant. * It is possible to use the SimpleShape plugin for difference imaging source measurement instead of SdssShape without a failure flag happening for all negative detections. More work needs to be done to determine what is causing SdssShape to throw a flag for negative detections in the first place and whether it we want to use a different shape algorithm or not.
          Hide
          ebellm Eric Bellm added a comment -

          This looks great--I don't know what to say about the fact that using the scatter in DIASource separations doesn't seem to clean up the totFlux measurements. From the thumbnails it's clear that the subtractions are still garbage, though, so I think that means greater focus on the diffim is still needed...

          Show
          ebellm Eric Bellm added a comment - This looks great--I don't know what to say about the fact that using the scatter in DIASource separations doesn't seem to clean up the totFlux measurements. From the thumbnails it's clear that the subtractions are still garbage, though, so I think that means greater focus on the diffim is still needed...

            People

            • Assignee:
              mrawls Meredith Rawls
              Reporter:
              mrawls Meredith Rawls
              Reviewers:
              Eric Bellm
              Watchers:
              Eric Bellm, Meredith Rawls, Sophie Reed
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel