# Deal with undefined values in ingest

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Team:
External

## Description

ParseTask.getInfoFromMetadata does not handle the case that a header value is present but undefined, leading to errors such as TypeError: float() argument must be a string or a number, not 'NoneType'.

## Activity

Hide
Paul Price added a comment -

Nate Lust, would you please have a look at this small change?

 price@price-laptop:~/LSST/pipe_tasks (tickets/DM-21452=) $git sub-patch commit c81602f91f4759ce624398b39adb12c998e9e1dd (HEAD -> tickets/DM-21452, origin/tickets/DM-21452) Author: Paul Price  Date: Thu Sep 26 13:23:15 2019 -0400    ingest: deal with undefined values    PropertySet/PropertyList now handles undefined values in the  FITS headers, but the ingest script treats them as None and  blows up. Handle these cases explicitly.   diff --git a/python/lsst/pipe/tasks/ingest.py b/python/lsst/pipe/tasks/ingest.py index 039c4371..b542efbb 100644 --- a/python/lsst/pipe/tasks/ingest.py +++ b/python/lsst/pipe/tasks/ingest.py @@ -133,8 +133,8 @@ class ParseTask(Task):  if info is None:  info = {}  for p, h in self.config.translation.items(): - if md.exists(h): - value = md.getScalar(h) + value = md.get(h, None) + if value is not None:  if isinstance(value, str):  value = value.strip()  info[p] = value  Show Paul Price added a comment - Nate Lust , would you please have a look at this small change? price@price-laptop:~/LSST/pipe_tasks (tickets/DM-21452=)$ git sub-patch commit c81602f91f4759ce624398b39adb12c998e9e1dd (HEAD -> tickets/DM-21452, origin/tickets/DM-21452) Author: Paul Price <price@astro.princeton.edu> Date: Thu Sep 26 13:23:15 2019 -0400   ingest: deal with undefined values PropertySet/PropertyList now handles undefined values in the FITS headers, but the ingest script treats them as None and blows up. Handle these cases explicitly.   diff --git a/python/lsst/pipe/tasks/ingest.py b/python/lsst/pipe/tasks/ingest.py index 039c4371..b542efbb 100644 --- a/python/lsst/pipe/tasks/ingest.py +++ b/python/lsst/pipe/tasks/ingest.py @@ -133,8 +133,8 @@ class ParseTask(Task): if info is None: info = {} for p, h in self.config.translation.items(): - if md.exists(h): - value = md.getScalar(h) + value = md.get(h, None) + if value is not None: if isinstance(value, str): value = value.strip() info[p] = value
Hide
Paul Price added a comment -

The change isn't as small as it was before, as I encountered errors when running Jenkins. The errors come because the defects FITS files that get generated have no FILTER set, but a filter value is required (if only to maintain consistency with the other calib types; I believe this is a design problem with ingestCalibs.py). I worked around this by modifying the configuration for ingestDefects.py to set a default value for filter.

Jenkins is green.

Show
Paul Price added a comment - The change isn't as small as it was before, as I encountered errors when running Jenkins. The errors come because the defects FITS files that get generated have no FILTER set, but a filter value is required (if only to maintain consistency with the other calib types; I believe this is a design problem with ingestCalibs.py ). I worked around this by modifying the configuration for ingestDefects.py to set a default value for filter . Jenkins is green.
Hide
Nate Lust added a comment -

looks fine

Show
Nate Lust added a comment - looks fine
Hide
Paul Price added a comment -

Thanks Nate!

Merged to master.

Show
Paul Price added a comment - Thanks Nate! Merged to master.

## People

• Assignee:
Paul Price
Reporter:
Paul Price
Reviewers:
Nate Lust
Watchers:
Nate Lust, Paul Price