Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9812

Clean up outputs from CharacterizeImageTask and CalibrateTask

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Team:
      Data Release Production

      Description

      We're writing the icExp files in CharacterizeImageTask, which we don't use, but greatly increases our disk usage.

      We want the denormalized match catalogs from CalibrateTask and from multiband processing.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            We need srcMatchFull (denormalized matches) from single-frame processing.
            Rename in the mapper: deepCoadd_srcMatch --> deepCoadd_measMatch
            We need deepCoadd_measMatchFull (denormalized matches) from multiband processing.

            Show
            price Paul Price added a comment - We need srcMatchFull (denormalized matches) from single-frame processing. Rename in the mapper: deepCoadd_srcMatch --> deepCoadd_measMatch We need deepCoadd_measMatchFull (denormalized matches) from multiband processing.
            Hide
            price Paul Price added a comment -

            RFC-306 proposes the rename deepCoadd_srcMatch to deepCoadd_measMatch. I'm going to move that to a separate ticket so this work can be merged.

            Show
            price Paul Price added a comment - RFC-306 proposes the rename deepCoadd_srcMatch to deepCoadd_measMatch . I'm going to move that to a separate ticket so this work can be merged.
            Hide
            price Paul Price added a comment -

            DM-9885 will do the rename folllowing RFC-306's acceptance.

            Show
            price Paul Price added a comment - DM-9885 will do the rename folllowing RFC-306 's acceptance.
            Hide
            price Paul Price added a comment -

            This has passed Jenkins.

            Show
            price Paul Price added a comment - This has passed Jenkins .
            Hide
            price Paul Price added a comment -

            Lauren MacArthur, could you review this please? We're hoping to get this into the HSC release real soon, so please let me know if you can't get to it today and I'll find someone else.

            price@pap-laptop:~/LSST/obs_base (tickets/DM-9812=) $ git sub
            commit 1578275f7cd25844a318e5a29868db66dc65b06a
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Mar 20 09:36:44 2017 -0400
             
                datasets: add denormalized matches
                
                Denormalized matches (srcMatchFull) are the matches (srcMatch)
                written in a denormalized format that uses more space but is
                more convenient to read.
             
             policy/datasets.yaml | 12 ++++++++++++
             1 file changed, 12 insertions(+)
             
             
            price@pap-laptop:~/LSST/meas_astrom (tickets/DM-9812=) $ git sub
            commit cfec1e75a0ce8ac9211ef8b8ca4a309eb1faf2a7
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Fri Mar 17 15:51:26 2017 -0400
             
                add function to convert matches to denormalized catalog
                
                This produces persisted catalogs that are easier to read,
                even though they take more space.
             
             python/lsst/meas/astrom/__init__.py            |  1 +
             python/lsst/meas/astrom/denormalizedMatches.py | 58 +++++++++++++++++++++++
             tests/testDenormalizedMatches.py               | 65 ++++++++++++++++++++++++++
             3 files changed, 124 insertions(+)
             
             
            price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-9812=) $ git sub
            commit a178704c2e7800dc148b735cd8619589b2acb4d9
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Mar 15 14:43:40 2017 +0900
             
                processCcd: disable exposure writing from charImage
                
                Writing exposures from charImage is unnecessary because we're going to
                write the calexp from calibrate.
             
             python/lsst/pipe/tasks/processCcd.py | 1 +
             tests/testProcessCcd.py              | 3 ++-
             2 files changed, 3 insertions(+), 1 deletion(-)
             
            commit f2a01131439ca0f0b756f4cac20fa5fd26731a82
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Mar 20 10:13:24 2017 -0400
             
                calibrate, measureCoaddSources: add option to write denormalized matches
                
                This writes the matches in a format that uses more disk space but
                is much more convenient to read.
             
             python/lsst/pipe/tasks/calibrate.py | 12 +++++++++++-
             python/lsst/pipe/tasks/multiBand.py | 12 +++++++++++-
             2 files changed, 22 insertions(+), 2 deletions(-)
             
             
            price@pap-laptop:~/LSST/obs_subaru (tickets/DM-9812=) $ git sub
            commit e7322b7782037f401bf2c43e25e59507b297e36e
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Mar 20 09:39:51 2017 -0400
             
                HscMapper: add denormalized matches
                
                Denormalized matches (srcMatchFull) are the matches (srcMatch)
                written in a denormalized format that uses more space but is
                more convenient to read.
             
             policy/HscMapper.paf | 6 ++++++
             1 file changed, 6 insertions(+)
             
            commit 53d43a4a818b303ce288fe9f4ffbec8843dd1fb5
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Mar 20 10:12:22 2017 -0400
             
                config: enable writing of denormalized matches
             
             config/measureCoaddSources.py | 2 ++
             config/processCcd.py          | 2 ++
             2 files changed, 4 insertions(+)
             
             
            price@pap-laptop:~/LSST/ci_hsc (tickets/DM-9812=) $ git sub
            commit d06bb4598a241264c075318d9567f4423e60ad8e
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Mar 20 10:31:28 2017 -0400
             
                validate denormalized matches
                
                We want denormalized matches to be written, so ensure they're
                there.
             
             python/lsst/ci/hsc/validate.py | 11 +++++++++++
             1 file changed, 11 insertions(+)
            

            Show
            price Paul Price added a comment - Lauren MacArthur , could you review this please? We're hoping to get this into the HSC release real soon, so please let me know if you can't get to it today and I'll find someone else. price@pap-laptop:~/LSST/obs_base (tickets/DM-9812=) $ git sub commit 1578275f7cd25844a318e5a29868db66dc65b06a Author: Paul Price <price@astro.princeton.edu> Date: Mon Mar 20 09:36:44 2017 -0400   datasets: add denormalized matches Denormalized matches (srcMatchFull) are the matches (srcMatch) written in a denormalized format that uses more space but is more convenient to read.   policy/datasets.yaml | 12 ++++++++++++ 1 file changed, 12 insertions(+)     price@pap-laptop:~/LSST/meas_astrom (tickets/DM-9812=) $ git sub commit cfec1e75a0ce8ac9211ef8b8ca4a309eb1faf2a7 Author: Paul Price <price@astro.princeton.edu> Date: Fri Mar 17 15:51:26 2017 -0400   add function to convert matches to denormalized catalog This produces persisted catalogs that are easier to read, even though they take more space.   python/lsst/meas/astrom/__init__.py | 1 + python/lsst/meas/astrom/denormalizedMatches.py | 58 +++++++++++++++++++++++ tests/testDenormalizedMatches.py | 65 ++++++++++++++++++++++++++ 3 files changed, 124 insertions(+)     price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-9812=) $ git sub commit a178704c2e7800dc148b735cd8619589b2acb4d9 Author: Paul Price <price@astro.princeton.edu> Date: Wed Mar 15 14:43:40 2017 +0900   processCcd: disable exposure writing from charImage Writing exposures from charImage is unnecessary because we're going to write the calexp from calibrate.   python/lsst/pipe/tasks/processCcd.py | 1 + tests/testProcessCcd.py | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)   commit f2a01131439ca0f0b756f4cac20fa5fd26731a82 Author: Paul Price <price@astro.princeton.edu> Date: Mon Mar 20 10:13:24 2017 -0400   calibrate, measureCoaddSources: add option to write denormalized matches This writes the matches in a format that uses more disk space but is much more convenient to read.   python/lsst/pipe/tasks/calibrate.py | 12 +++++++++++- python/lsst/pipe/tasks/multiBand.py | 12 +++++++++++- 2 files changed, 22 insertions(+), 2 deletions(-)     price@pap-laptop:~/LSST/obs_subaru (tickets/DM-9812=) $ git sub commit e7322b7782037f401bf2c43e25e59507b297e36e Author: Paul Price <price@astro.princeton.edu> Date: Mon Mar 20 09:39:51 2017 -0400   HscMapper: add denormalized matches Denormalized matches (srcMatchFull) are the matches (srcMatch) written in a denormalized format that uses more space but is more convenient to read.   policy/HscMapper.paf | 6 ++++++ 1 file changed, 6 insertions(+)   commit 53d43a4a818b303ce288fe9f4ffbec8843dd1fb5 Author: Paul Price <price@astro.princeton.edu> Date: Mon Mar 20 10:12:22 2017 -0400   config: enable writing of denormalized matches   config/measureCoaddSources.py | 2 ++ config/processCcd.py | 2 ++ 2 files changed, 4 insertions(+)     price@pap-laptop:~/LSST/ci_hsc (tickets/DM-9812=) $ git sub commit d06bb4598a241264c075318d9567f4423e60ad8e Author: Paul Price <price@astro.princeton.edu> Date: Mon Mar 20 10:31:28 2017 -0400   validate denormalized matches We want denormalized matches to be written, so ensure they're there.   python/lsst/ci/hsc/validate.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
            Hide
            price Paul Price added a comment -

            Lauren MacArthur is busy preparing for the same release this is aimed at, so asking Yusra AlSayyad to review instead.

            Show
            price Paul Price added a comment - Lauren MacArthur is busy preparing for the same release this is aimed at, so asking Yusra AlSayyad to review instead.
            Hide
            yusra Yusra AlSayyad added a comment -

            Looks good. 2 things

            • Why no srcMatchFull path for obs_cfht, obs_decam, obs_lsstSim, obs_sdss?
            • I mentioned this as a comment in the PR. It was clear to me what denormalizeMatches means, but unclear to me what matchesToCatalog means. I offered some replacements there.
            Show
            yusra Yusra AlSayyad added a comment - Looks good. 2 things Why no srcMatchFull path for obs_cfht, obs_decam, obs_lsstSim, obs_sdss? I mentioned this as a comment in the PR. It was clear to me what denormalizeMatches means, but unclear to me what matchesToCatalog means. I offered some replacements there.
            Hide
            price Paul Price added a comment -

            I'm going to put off adding srcMatchFull until DM-9885 (planning to do that later this week after RFC-306 is accepted), where I'll be touching all the obs packages anyway. I just added a note on that ticket so I don't forget.

            I'm renaming the function to denormalizedMatches (matches the filename, and properly conveys the intent), and fixing the docstring.

            Show
            price Paul Price added a comment - I'm going to put off adding srcMatchFull until DM-9885 (planning to do that later this week after RFC-306 is accepted), where I'll be touching all the obs packages anyway. I just added a note on that ticket so I don't forget. I'm renaming the function to denormalizedMatches (matches the filename, and properly conveys the intent), and fixing the docstring.
            Hide
            jbosch Jim Bosch added a comment -

            Did you mean denormalizeMatches? Seems like it ought to start with a present-tense verb, not an adjective (and coding guidelines agree).

            Show
            jbosch Jim Bosch added a comment - Did you mean denormalizeMatches ? Seems like it ought to start with a present-tense verb, not an adjective (and coding guidelines agree).
            Hide
            price Paul Price added a comment -

            denormalizeMatches it is.

            Show
            price Paul Price added a comment - denormalizeMatches it is.
            Hide
            price Paul Price added a comment -

            Revised changes pass Jenkins. I'm about to merge.

            Show
            price Paul Price added a comment - Revised changes pass Jenkins . I'm about to merge.
            Hide
            yusra Yusra AlSayyad added a comment -

            Show
            yusra Yusra AlSayyad added a comment -
            Hide
            price Paul Price added a comment -

            Thanks Yusra!

            Merged to master.

            Show
            price Paul Price added a comment - Thanks Yusra! Merged to master.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Jim Bosch, Lauren MacArthur, Paul Price, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.