Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21452

Deal with undefined values in ingest

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
    • Team:
      External

      Description

      ParseTask.getInfoFromMetadata does not handle the case that a header value is present but undefined, leading to errors such as TypeError: float() argument must be a string or a number, not 'NoneType'.

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          Nate Lust, would you please have a look at this small change?

          price@price-laptop:~/LSST/pipe_tasks (tickets/DM-21452=) $ git sub-patch
          commit c81602f91f4759ce624398b39adb12c998e9e1dd (HEAD -> tickets/DM-21452, origin/tickets/DM-21452)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Sep 26 13:23:15 2019 -0400
           
              ingest: deal with undefined values
              
              PropertySet/PropertyList now handles undefined values in the
              FITS headers, but the ingest script treats them as None and
              blows up. Handle these cases explicitly.
           
          diff --git a/python/lsst/pipe/tasks/ingest.py b/python/lsst/pipe/tasks/ingest.py
          index 039c4371..b542efbb 100644
          --- a/python/lsst/pipe/tasks/ingest.py
          +++ b/python/lsst/pipe/tasks/ingest.py
          @@ -133,8 +133,8 @@ class ParseTask(Task):
                   if info is None:
                       info = {}
                   for p, h in self.config.translation.items():
          -            if md.exists(h):
          -                value = md.getScalar(h)
          +            value = md.get(h, None)
          +            if value is not None:
                           if isinstance(value, str):
                               value = value.strip()
                           info[p] = value
          

          Show
          price Paul Price added a comment - Nate Lust , would you please have a look at this small change? price@price-laptop:~/LSST/pipe_tasks (tickets/DM-21452=) $ git sub-patch commit c81602f91f4759ce624398b39adb12c998e9e1dd (HEAD -> tickets/DM-21452, origin/tickets/DM-21452) Author: Paul Price <price@astro.princeton.edu> Date: Thu Sep 26 13:23:15 2019 -0400   ingest: deal with undefined values PropertySet/PropertyList now handles undefined values in the FITS headers, but the ingest script treats them as None and blows up. Handle these cases explicitly.   diff --git a/python/lsst/pipe/tasks/ingest.py b/python/lsst/pipe/tasks/ingest.py index 039c4371..b542efbb 100644 --- a/python/lsst/pipe/tasks/ingest.py +++ b/python/lsst/pipe/tasks/ingest.py @@ -133,8 +133,8 @@ class ParseTask(Task): if info is None: info = {} for p, h in self.config.translation.items(): - if md.exists(h): - value = md.getScalar(h) + value = md.get(h, None) + if value is not None: if isinstance(value, str): value = value.strip() info[p] = value
          Hide
          price Paul Price added a comment -

          The change isn't as small as it was before, as I encountered errors when running Jenkins. The errors come because the defects FITS files that get generated have no FILTER set, but a filter value is required (if only to maintain consistency with the other calib types; I believe this is a design problem with ingestCalibs.py). I worked around this by modifying the configuration for ingestDefects.py to set a default value for filter.

          Jenkins is green.

          Show
          price Paul Price added a comment - The change isn't as small as it was before, as I encountered errors when running Jenkins. The errors come because the defects FITS files that get generated have no FILTER set, but a filter value is required (if only to maintain consistency with the other calib types; I believe this is a design problem with ingestCalibs.py ). I worked around this by modifying the configuration for ingestDefects.py to set a default value for filter . Jenkins is green.
          Hide
          nlust Nate Lust added a comment -

          looks fine

          Show
          nlust Nate Lust added a comment - looks fine
          Hide
          price Paul Price added a comment -

          Thanks Nate!

          Merged to master.

          Show
          price Paul Price added a comment - Thanks Nate! Merged to master.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Nate Lust
              Watchers:
              Nate Lust, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel