Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4878

Propagate flags from individual visit measurements to coadd measurements

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      It is useful to be able to identify suitable PSF stars from a coadd catalogue. However, the PSF is not determined on the coadd, but from all the inputs. Add a mechanism for propagating flags from the input catalogues to the coadd catalogue indicating stars that were used for measuring the PSF.

      Make the inclusion fraction threshold configurable so we can tweak it (so we only get stars that were consistently used for the PSF model; the threshold might be set it to 0 for "or", 1 for "all" and something in between for "some").

      Make the task sufficiently general that it can be used for propagating arbitrary flags.

      This is a port of work carried out on HSC-1052 and (part of) HSC-1293.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            I'm cleaning up the clutter, and hope you will find it improved in the next version.

            Show
            price Paul Price added a comment - I'm cleaning up the clutter, and hope you will find it improved in the next version.
            Hide
            price Paul Price added a comment -

            Russell Owen, now that the commits are rearranged and there's more documentation than code, would you mind having another look please? I hope you'll find that what you considered complicated before is now much simpler (and it turns out that the complication was there prior to this ticket).

            price@price-laptop:~/LSST/pipe/tasks (tickets/DM-4878=) $ git sub
            commit 2f3f40535da66ec17ab668f764f7602e4c969624
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Dec 18 22:41:51 2014 -0500
             
                add task to propagate flags from CCDs to coadds
                
                This is used in processCoadd and the detection and measurement parts
                of the multiband processing scheme to flag stars on the coadd as
                having been used for PSF determination.
                
                Based on code written by Hironao Miyatake.
             
             python/lsst/pipe/tasks/multiBand.py           |   5 +
             python/lsst/pipe/tasks/processCoadd.py        |   6 +
             python/lsst/pipe/tasks/propagateVisitFlags.py | 197 ++++++++++++++++++++++++++
             3 files changed, 208 insertions(+)
             
            commit 8f42c159f2057679cb7505dc7bcc61ff293075b6
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Feb 3 15:49:38 2016 -0500
             
                AssembleCoaddTask: make assembleMetadata clearer
                
                Reorganised the metadata reading loop into a list comprehension for
                clarity (it's now closer to appropriate the comment).  Removed the 'bbox'
                variable in favor of specifying the bounding box explicitly, so it's clear
                that it's only reading a single pixel.
                
                Added an assert to guard against vector length mismatches
             
             python/lsst/pipe/tasks/assembleCoadd.py | 18 ++++++++++--------
             1 file changed, 10 insertions(+), 8 deletions(-)
             
            commit 1b5babd7693a08ef71f28200e2fbd8f5ea7e20d2
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Feb 3 15:50:19 2016 -0500
             
                make coadd input catalogs contiguous
                
                We know how many inputs (both CCDs and visits) we have for the warp
                ("tempExp") and for the coadd, so reserve space in the catalogs for
                that many so they can be contiguous. This prevents errors when
                propagating flags to the coadd due to non-contiguous catalogs.
             
             python/lsst/pipe/tasks/assembleCoadd.py      | 3 +++
             python/lsst/pipe/tasks/coaddInputRecorder.py | 9 ++++++---
             python/lsst/pipe/tasks/makeCoaddTempExp.py   | 2 +-
             3 files changed, 10 insertions(+), 4 deletions(-)
             
            commit 4139813fcb8a090d2f3d4000f54c66ae7e4948c9
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Fri Jul 24 16:19:46 2015 -0400
             
                PropagateVisitFlagsTask: guard against non-contiguous ccdInputs
                
                It is safer to read the ccdInputs value by value instead of by column,
                since the latter way fails if the Catalog is non-contiguous. Since we
                don't expect that this catalog is large, this should not have too bad
                an effect on the performance in comparison with the other operations.
             
             python/lsst/pipe/tasks/propagateVisitFlags.py | 6 ++++--
             1 file changed, 4 insertions(+), 2 deletions(-)
             
            commit 7a88d2f0a431723d1f6c4de548592258429097ff
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Feb 3 15:11:51 2016 -0500
             
                CoaddTempExpInputRecorder: add docstring for __init__
             
             python/lsst/pipe/tasks/coaddInputRecorder.py | 7 +++++++
             1 file changed, 7 insertions(+)
            

            Show
            price Paul Price added a comment - Russell Owen , now that the commits are rearranged and there's more documentation than code, would you mind having another look please? I hope you'll find that what you considered complicated before is now much simpler (and it turns out that the complication was there prior to this ticket). price@price-laptop:~/LSST/pipe/tasks (tickets/DM-4878=) $ git sub commit 2f3f40535da66ec17ab668f764f7602e4c969624 Author: Paul Price <price@astro.princeton.edu> Date: Thu Dec 18 22:41:51 2014 -0500   add task to propagate flags from CCDs to coadds This is used in processCoadd and the detection and measurement parts of the multiband processing scheme to flag stars on the coadd as having been used for PSF determination. Based on code written by Hironao Miyatake.   python/lsst/pipe/tasks/multiBand.py | 5 + python/lsst/pipe/tasks/processCoadd.py | 6 + python/lsst/pipe/tasks/propagateVisitFlags.py | 197 ++++++++++++++++++++++++++ 3 files changed, 208 insertions(+)   commit 8f42c159f2057679cb7505dc7bcc61ff293075b6 Author: Paul Price <price@astro.princeton.edu> Date: Wed Feb 3 15:49:38 2016 -0500   AssembleCoaddTask: make assembleMetadata clearer Reorganised the metadata reading loop into a list comprehension for clarity (it's now closer to appropriate the comment). Removed the 'bbox' variable in favor of specifying the bounding box explicitly, so it's clear that it's only reading a single pixel. Added an assert to guard against vector length mismatches   python/lsst/pipe/tasks/assembleCoadd.py | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)   commit 1b5babd7693a08ef71f28200e2fbd8f5ea7e20d2 Author: Paul Price <price@astro.princeton.edu> Date: Wed Feb 3 15:50:19 2016 -0500   make coadd input catalogs contiguous We know how many inputs (both CCDs and visits) we have for the warp ("tempExp") and for the coadd, so reserve space in the catalogs for that many so they can be contiguous. This prevents errors when propagating flags to the coadd due to non-contiguous catalogs.   python/lsst/pipe/tasks/assembleCoadd.py | 3 +++ python/lsst/pipe/tasks/coaddInputRecorder.py | 9 ++++++--- python/lsst/pipe/tasks/makeCoaddTempExp.py | 2 +- 3 files changed, 10 insertions(+), 4 deletions(-)   commit 4139813fcb8a090d2f3d4000f54c66ae7e4948c9 Author: Paul Price <price@astro.princeton.edu> Date: Fri Jul 24 16:19:46 2015 -0400   PropagateVisitFlagsTask: guard against non-contiguous ccdInputs It is safer to read the ccdInputs value by value instead of by column, since the latter way fails if the Catalog is non-contiguous. Since we don't expect that this catalog is large, this should not have too bad an effect on the performance in comparison with the other operations.   python/lsst/pipe/tasks/propagateVisitFlags.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)   commit 7a88d2f0a431723d1f6c4de548592258429097ff Author: Paul Price <price@astro.princeton.edu> Date: Wed Feb 3 15:11:51 2016 -0500   CoaddTempExpInputRecorder: add docstring for __init__   python/lsst/pipe/tasks/coaddInputRecorder.py | 7 +++++++ 1 file changed, 7 insertions(+)
            Hide
            rowen Russell Owen added a comment -

            I like your cleanup. This looks good to merge.

            Show
            rowen Russell Owen added a comment - I like your cleanup. This looks good to merge.
            Hide
            price Paul Price added a comment -

            Thanks Russell!

            I found a test failure when running Jenkins before merging, which I quashed by disabling flag propagation (there's no 'src' dataset in the test, nor flags). Now it passes.

            Merged to master.

            Show
            price Paul Price added a comment - Thanks Russell! I found a test failure when running Jenkins before merging, which I quashed by disabling flag propagation (there's no 'src' dataset in the test, nor flags). Now it passes. Merged to master.
            Show
            swinbank John Swinbank added a comment - Added to release notes: https://confluence.lsstcorp.org/display/DM/Data+Release+Production+WIP+W16+release+notes#DataReleaseProductionWIPW16releasenotes-Propagateflagsfromindividualvisitstocoadds

              People

              Assignee:
              price Paul Price
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Russell Owen
              Watchers:
              John Swinbank, Paul Price, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.