Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13110

Support ingestion of compressed images

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw, ci_hsc, pipe_tasks
    • Labels:
    • Team:
      External

      Description

      When running ingestImages.py on compressed (fpack-ed) images, the code explicitly looks in hdu=0 which is the primary header which has no useful information. We need to use the default behaviour (go to the first HDU if it's compressed).

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          Real quick review for you Simon Krughoff, if you don't mind:

          pprice@tiger-sumire:/tigress/HSC/PAP-TEST/pipe_tasks (tickets/DM-13110=) $ git sub-patch
          commit 2acf0b43209c56bcc6aad10d986761b7667b2fd5
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Dec 21 15:31:57 2017 -0500
           
              ingest.py: read metadata from default HDU by default
              
              When the input file has been FITS tile-compressed, hdu=0 (which is the
              default) is the PHU which has no useful information. We need to specify
              hdu=INT_MIN, but that's not possible in python so we just need a
              mechanism to signal to leave off the 'hdu' parameter; choose hdu <= 0.
           
          diff --git a/python/lsst/pipe/tasks/ingest.py b/python/lsst/pipe/tasks/ingest.py
          index 9fdcfe6..651f154 100644
          --- a/python/lsst/pipe/tasks/ingest.py
          +++ b/python/lsst/pipe/tasks/ingest.py
          @@ -45,7 +45,7 @@ class ParseConfig(Config):
                                       doc="Properties and name of translator method")
               defaults = DictField(keytype=str, itemtype=str, default={},
                                    doc="Default values if header is not present")
          -    hdu = Field(dtype=int, default=0, doc="HDU to read for metadata")
          +    hdu = Field(dtype=int, default=-1, doc="HDU to read for metadata")
               extnames = ListField(dtype=str, default=[], doc="Extension names to search for")
           
           
          @@ -63,7 +63,8 @@ class ParseTask(Task):
                   @param filename    Name of file to inspect
                   @return File properties; list of file properties for each extension
                   """
          -        md = afwImage.readMetadata(filename, self.config.hdu)
          +        md = (afwImage.readMetadata(filename) if self.config.hdu < 0 else
          +              afwImage.readMetadata(filename, self.config.hdu))
                   phuInfo = self.getInfoFromMetadata(md)
                   if len(self.config.extnames) == 0:
                       # No extensions to worry about
          

          Show
          price Paul Price added a comment - Real quick review for you Simon Krughoff , if you don't mind: pprice@tiger-sumire:/tigress/HSC/PAP-TEST/pipe_tasks (tickets/DM-13110=) $ git sub-patch commit 2acf0b43209c56bcc6aad10d986761b7667b2fd5 Author: Paul Price <price@astro.princeton.edu> Date: Thu Dec 21 15:31:57 2017 -0500   ingest.py: read metadata from default HDU by default When the input file has been FITS tile-compressed, hdu=0 (which is the default) is the PHU which has no useful information. We need to specify hdu=INT_MIN, but that's not possible in python so we just need a mechanism to signal to leave off the 'hdu' parameter; choose hdu <= 0.   diff --git a/python/lsst/pipe/tasks/ingest.py b/python/lsst/pipe/tasks/ingest.py index 9fdcfe6..651f154 100644 --- a/python/lsst/pipe/tasks/ingest.py +++ b/python/lsst/pipe/tasks/ingest.py @@ -45,7 +45,7 @@ class ParseConfig(Config): doc="Properties and name of translator method") defaults = DictField(keytype=str, itemtype=str, default={}, doc="Default values if header is not present") - hdu = Field(dtype=int, default=0, doc="HDU to read for metadata") + hdu = Field(dtype=int, default=-1, doc="HDU to read for metadata") extnames = ListField(dtype=str, default=[], doc="Extension names to search for") @@ -63,7 +63,8 @@ class ParseTask(Task): @param filename Name of file to inspect @return File properties; list of file properties for each extension """ - md = afwImage.readMetadata(filename, self.config.hdu) + md = (afwImage.readMetadata(filename) if self.config.hdu < 0 else + afwImage.readMetadata(filename, self.config.hdu)) phuInfo = self.getInfoFromMetadata(md) if len(self.config.extnames) == 0: # No extensions to worry about
          Hide
          price Paul Price added a comment -

          Robert Lupton suggested a better (more complete) way to do this. Since the changes have expanded beyond what I promised Simon Krughoff, I'm taking him off the hook. Robert Lupton, would you mind reviewing the changes?

          price@pap-laptop:~/LSST/afw (tickets/DM-13110 %=) $ git sub
          commit c620cd3bf44bc014f3665314eb71488c28694d4c (HEAD -> tickets/DM-13110, origin/tickets/DM-13110)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Dec 21 23:27:08 2017 -0500
           
              export magic value to select default HDU
              
              We have used INT_MIN to indicate that the first non-empty HDU should
              be selected, but never exported the symbol to python. Changed the name
              (but kept the value) so it's clear what it is, and exported it to
              python.
           
           include/lsst/afw/fits.h                       |  7 ++++---
           include/lsst/afw/fitsDefaults.h               | 20 ++++++++++++++++++++
           include/lsst/afw/image/Image.h                | 27 ++++++++++++++-------------
           include/lsst/afw/image/Mask.h                 | 23 +++++++++++++----------
           include/lsst/afw/image/Utils.h                |  7 ++++---
           include/lsst/afw/table/Catalog.h              | 11 +++++++----
           include/lsst/afw/table/Exposure.h             | 13 +++++++++----
           include/lsst/afw/table/Schema.h               |  7 ++++---
           include/lsst/afw/table/SortedCatalog.h        | 12 ++++++++----
           include/lsst/afw/table/io/Persistable.h       | 17 +++++++++--------
           include/lsst/afw/table/io/python.h            |  8 ++++----
           include/lsst/afw/table/python/catalog.h       |  4 ++--
           include/lsst/afw/table/python/sortedCatalog.h |  4 ++--
           python/lsst/afw/fits/fits.cc                  | 11 ++++++-----
           python/lsst/afw/image/image/image.cc          | 11 ++++++-----
           python/lsst/afw/image/mask/mask.cc            |  8 ++++----
           python/lsst/afw/image/readMetadata.cc         |  2 +-
           python/lsst/afw/math/backgroundList.py        |  5 ++---
           python/lsst/afw/table/exposure/exposure.cc    |  4 ++--
           python/lsst/afw/table/schema/schema.cc        |  4 ++--
           src/fits.cc                                   |  4 ++--
           src/formatters/TanWcsFormatter.cc             |  2 +-
           src/formatters/WcsFormatter.cc                |  2 +-
           src/image/MaskedImage.cc                      |  4 ++--
           tests/ramFitsIO.cc                            |  5 +++--
           tests/test_spanSets.cc                        |  2 +-
           tests/test_tableArchives.cc                   |  5 +++--
           tests/test_ticket2352.py                      |  4 +++-
           28 files changed, 139 insertions(+), 94 deletions(-)
           
          price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13110=) $ git sub
          commit c830ea656298d52708786637e49ba951075c1318 (HEAD -> tickets/DM-13110, origin/tickets/DM-13110)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Dec 21 15:31:57 2017 -0500
           
              ingest.py: read metadata from default HDU by default
              
              When the input file has been FITS tile-compressed, hdu=0 (which is the
              default) is the PHU which has no useful information. We need to specify
              hdu=DEFAULT_HDU, which tells readMetadata to skip to the first non-empty
              HDU.
           
           python/lsst/pipe/tasks/ingest.py | 3 ++-
           1 file changed, 2 insertions(+), 1 deletion(-)
          

          Show
          price Paul Price added a comment - Robert Lupton suggested a better (more complete) way to do this. Since the changes have expanded beyond what I promised Simon Krughoff , I'm taking him off the hook. Robert Lupton , would you mind reviewing the changes? price@pap-laptop:~/LSST/afw (tickets/DM-13110 %=) $ git sub commit c620cd3bf44bc014f3665314eb71488c28694d4c (HEAD -> tickets/DM-13110, origin/tickets/DM-13110) Author: Paul Price <price@astro.princeton.edu> Date: Thu Dec 21 23:27:08 2017 -0500   export magic value to select default HDU We have used INT_MIN to indicate that the first non-empty HDU should be selected, but never exported the symbol to python. Changed the name (but kept the value) so it's clear what it is, and exported it to python.   include/lsst/afw/fits.h | 7 ++++--- include/lsst/afw/fitsDefaults.h | 20 ++++++++++++++++++++ include/lsst/afw/image/Image.h | 27 ++++++++++++++------------- include/lsst/afw/image/Mask.h | 23 +++++++++++++---------- include/lsst/afw/image/Utils.h | 7 ++++--- include/lsst/afw/table/Catalog.h | 11 +++++++---- include/lsst/afw/table/Exposure.h | 13 +++++++++---- include/lsst/afw/table/Schema.h | 7 ++++--- include/lsst/afw/table/SortedCatalog.h | 12 ++++++++---- include/lsst/afw/table/io/Persistable.h | 17 +++++++++-------- include/lsst/afw/table/io/python.h | 8 ++++---- include/lsst/afw/table/python/catalog.h | 4 ++-- include/lsst/afw/table/python/sortedCatalog.h | 4 ++-- python/lsst/afw/fits/fits.cc | 11 ++++++----- python/lsst/afw/image/image/image.cc | 11 ++++++----- python/lsst/afw/image/mask/mask.cc | 8 ++++---- python/lsst/afw/image/readMetadata.cc | 2 +- python/lsst/afw/math/backgroundList.py | 5 ++--- python/lsst/afw/table/exposure/exposure.cc | 4 ++-- python/lsst/afw/table/schema/schema.cc | 4 ++-- src/fits.cc | 4 ++-- src/formatters/TanWcsFormatter.cc | 2 +- src/formatters/WcsFormatter.cc | 2 +- src/image/MaskedImage.cc | 4 ++-- tests/ramFitsIO.cc | 5 +++-- tests/test_spanSets.cc | 2 +- tests/test_tableArchives.cc | 5 +++-- tests/test_ticket2352.py | 4 +++- 28 files changed, 139 insertions(+), 94 deletions(-)   price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13110=) $ git sub commit c830ea656298d52708786637e49ba951075c1318 (HEAD -> tickets/DM-13110, origin/tickets/DM-13110) Author: Paul Price <price@astro.princeton.edu> Date: Thu Dec 21 15:31:57 2017 -0500   ingest.py: read metadata from default HDU by default When the input file has been FITS tile-compressed, hdu=0 (which is the default) is the PHU which has no useful information. We need to specify hdu=DEFAULT_HDU, which tells readMetadata to skip to the first non-empty HDU.   python/lsst/pipe/tasks/ingest.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
          Hide
          price Paul Price added a comment - - edited

          Jenkins appears to be passing (one failure in that set was due to not having pushed afw changes). To demonstrate that things are working, I compressed half the raw data in ci_hsc. Robert Lupton, this is ready for review if you are able, please.

          price@pap-laptop:~/LSST/ci_hsc (tickets/DM-13110=) $ git sub
          commit 9ae12443219209c9248b277bc7e3e6fc22739beb (HEAD -> tickets/DM-13110, origin/tickets/DM-13110)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Dec 27 11:26:35 2017 -0500
           
              compress raw r-band data
              
              In order to demonstrate that we can deal with FITS tile-compressed raw
              data, compress the r-band raw data (fpack -F raw/HSC9033[34]*.fits).
              Explicitly not compressing the i-band raw data, to ensure we can still
              protect that mode of operating too.
           
           raw/HSCA90333426.fits | 4 ++--
           raw/HSCA90333427.fits | 4 ++--
           raw/HSCA90333430.fits | 4 ++--
           raw/HSCA90333431.fits | 4 ++--
           raw/HSCA90333625.fits | 4 ++--
           raw/HSCA90333629.fits | 4 ++--
           raw/HSCA90333824.fits | 4 ++--
           raw/HSCA90333828.fits | 4 ++--
           raw/HSCA90334231.fits | 4 ++--
           raw/HSCA90334336.fits | 4 ++--
           raw/HSCA90334342.fits | 4 ++--
           raw/HSCA90334537.fits | 4 ++--
           raw/HSCA90334543.fits | 4 ++--
           raw/HSCA90334553.fits | 4 ++--
           raw/HSCA90334738.fits | 4 ++--
           raw/HSCA90334744.fits | 4 ++--
           raw/HSCA90334754.fits | 4 ++--
           17 files changed, 34 insertions(+), 34 deletions(-)
          

          Show
          price Paul Price added a comment - - edited Jenkins appears to be passing (one failure in that set was due to not having pushed afw changes). To demonstrate that things are working, I compressed half the raw data in ci_hsc. Robert Lupton , this is ready for review if you are able, please. price@pap-laptop:~/LSST/ci_hsc (tickets/DM-13110=) $ git sub commit 9ae12443219209c9248b277bc7e3e6fc22739beb (HEAD -> tickets/DM-13110, origin/tickets/DM-13110) Author: Paul Price <price@astro.princeton.edu> Date: Wed Dec 27 11:26:35 2017 -0500   compress raw r-band data In order to demonstrate that we can deal with FITS tile-compressed raw data, compress the r-band raw data (fpack -F raw/HSC9033[34]*.fits). Explicitly not compressing the i-band raw data, to ensure we can still protect that mode of operating too.   raw/HSCA90333426.fits | 4 ++-- raw/HSCA90333427.fits | 4 ++-- raw/HSCA90333430.fits | 4 ++-- raw/HSCA90333431.fits | 4 ++-- raw/HSCA90333625.fits | 4 ++-- raw/HSCA90333629.fits | 4 ++-- raw/HSCA90333824.fits | 4 ++-- raw/HSCA90333828.fits | 4 ++-- raw/HSCA90334231.fits | 4 ++-- raw/HSCA90334336.fits | 4 ++-- raw/HSCA90334342.fits | 4 ++-- raw/HSCA90334537.fits | 4 ++-- raw/HSCA90334543.fits | 4 ++-- raw/HSCA90334553.fits | 4 ++-- raw/HSCA90334738.fits | 4 ++-- raw/HSCA90334744.fits | 4 ++-- raw/HSCA90334754.fits | 4 ++-- 17 files changed, 34 insertions(+), 34 deletions(-)
          Hide
          price Paul Price added a comment -

          Robert Lupton signed off on the GitHub PRs.

          Show
          price Paul Price added a comment - Robert Lupton signed off on the GitHub PRs.
          Hide
          price Paul Price added a comment -

          Merged to master.

          Show
          price Paul Price added a comment - Merged to master.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Robert Lupton
              Watchers:
              Paul Price, Robert Lupton, Simon Krughoff
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel