Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34539

Add detection significance to DIASources and apdb

    XMLWordPrintable

    Details

      Description

      DM-31811 calculated the significance of detected peaks in footprints. This ticket is to pull those values out of the peaks and put them into the DIASource catalogs and apdb.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Eric Bellm: Are you able to review this small-ish ticket? There are two PRs with about 80 lines of changes. Given that this is writing to a dataframe, there may be a faster approach than my getSignificance closure there, but this works fine.

            Note that I filed DM-35331 because I realized it would be too much of a pain to add tests of sky sources on this ticket: I'm aware that I'm not testing that path in the significance code, but I did run ap_verify_ci_hits2015, which I think does use that path.

            Show
            Parejkoj John Parejko added a comment - Eric Bellm : Are you able to review this small-ish ticket? There are two PRs with about 80 lines of changes. Given that this is writing to a dataframe, there may be a faster approach than my getSignificance closure there, but this works fine. https://github.com/lsst/meas_base/pull/215 https://github.com/lsst/ap_association/pull/154 Note that I filed DM-35331 because I realized it would be too much of a pain to add tests of sky sources on this ticket: I'm aware that I'm not testing that path in the significance code, but I did run ap_verify_ci_hits2015, which I think does use that path.
            Hide
            ebellm Eric Bellm added a comment -

            I think there's an indexing bug in ap_association, but otherwise it looks good.

            Show
            ebellm Eric Bellm added a comment - I think there's an indexing bug in ap_association, but otherwise it looks good.
            Hide
            Parejkoj John Parejko added a comment - - edited

            Thank you for catching that bug: I've incorporated your fix and verified that it works on ap_verify cosmos (identical results to main, for the metrics we measure).

            Which reminds me: should we have a metric on this detection significance for diaSources? (added on another ticket, probably)

            New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36968/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited Thank you for catching that bug: I've incorporated your fix and verified that it works on ap_verify cosmos (identical results to main, for the metrics we measure). Which reminds me: should we have a metric on this detection significance for diaSources? (added on another ticket, probably) New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36968/pipeline
            Hide
            ebellm Eric Bellm added a comment -

            I don't think it needs a metric, myself--it's already a column in the APDB. What might be more interesting is a metric which compares (in aggregate) the difference between the detection SNR and the measurement SNR. But I think we should work on notebook-level exploration to understand the behavior first.

            Show
            ebellm Eric Bellm added a comment - I don't think it needs a metric, myself--it's already a column in the APDB. What might be more interesting is a metric which compares (in aggregate) the difference between the detection SNR and the measurement SNR. But I think we should work on notebook-level exploration to understand the behavior first.
            Hide
            ebellm Eric Bellm added a comment -

            Thanks John! Looks good now.

            Show
            ebellm Eric Bellm added a comment - Thanks John! Looks good now.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Eric Bellm
              Watchers:
              Adam Snyder, Eric Bellm, Ian Sullivan, John Parejko, Nima Sedaghat
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.