Details
-
Type:
Improvement
-
Status: To Do
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: pex_config
-
Labels:None
-
Story Points:2
-
Team:Data Access and Database
Description
In the process of trying to figure out how to fix DM-9909, I realized I needed some function to search the configuration hierarchy of a task for a given parameter, to know exactly which subtask/subsubtask/etc. to point to to change a particular config switch. I threw together the following function and it helped me, so perhaps it might help others if it were adapted to be a method of the Task base class? I'm imaging something like being able to call myTask.searchConfig('myKeyword') and have it return all matches (showing the proper hierarchy) as well as their values.
from lsst.pipe.tasks.processCcd import ProcessCcdTask |
import re |
|
def dict_generator(indict, pre=None): |
"""Walk through a dictionary and return a list of lists |
|
Slightly modified from
|
http://stackoverflow.com/questions/12507206/python-recommended-way-to-walk-complex-dictionary-structures-imported-from-json
|
|
The last item in each list is the value.
|
"""
|
pre = pre[:] if pre else [] |
if isinstance(indict, dict): |
for key, value in indict.items(): |
if isinstance(value, dict) and len(value) > 0: |
for d in dict_generator(value, [key] + pre): |
yield d |
elif isinstance(value, list) or isinstance(value, tuple): |
for v in value: |
for d in dict_generator(v, [key] + pre): |
yield d |
else: |
yield pre + [key, value] |
else: |
yield indict |
|
def search_config(config, key): |
"""Returns any config options and values matching key |
|
Config is instance of a Config object. |
|
Searching by key name:
|
|
>>> search_config(ProcessCcdTask.ConfigClass(), 'filterMap') |
{'astromRefObjLoader.calibrate.filterMap': {}, |
'photoRefObjLoader.calibrate.filterMap': {}, |
'refObjLoader.charImage.filterMap': {}} |
|
Or by task name:
|
|
>>> search_config(ProcessCcdTask.ConfigClass(), 'photoRefObjLoader') |
{'photoRefObjLoader.calibrate.defaultFilter': '', |
'photoRefObjLoader.calibrate.filterMap': {}, |
'photoRefObjLoader.calibrate.pixelMargin': 50} |
|
No need for exact matches: |
|
>>> search_config(ProcessCcdTask.ConfigClass(), 'ObjLoader') |
{'astromRefObjLoader.calibrate.defaultFilter': '', |
'astromRefObjLoader.calibrate.filterMap': {}, |
'astromRefObjLoader.calibrate.pixelMargin': 50, |
'photoRefObjLoader.calibrate.defaultFilter': '', |
'photoRefObjLoader.calibrate.filterMap': {}, |
'photoRefObjLoader.calibrate.pixelMargin': 50, |
'refObjLoader.charImage.defaultFilter': '', |
'refObjLoader.charImage.filterMap': {}, |
'refObjLoader.charImage.pixelMargin': 50} |
|
"""
|
# Import 'nan' because otherwise eval command will fail |
from numpy import nan |
d = eval(str(config)) |
|
lfound = [] |
for l in dict_generator(d): |
try: |
for s in l: |
if re.search(key, str(s)): |
lfound.append(l)
|
# if key in l:
|
# lfound.append(l)
|
except TypeError: |
pass |
|
# The last item in each list is the value |
return {'.'.join(l[:-1]):l[-1] for l in lfound} |
Attachments
Issue Links
- mentioned in
-
Page Loading...
We discussed this at the DRP standup of 2017-03-30.
We agreed that this is a nice convenience feature. We wondered if it could be achieved with even less code by re-using parts of the machinery already built for command line task.
Fritz Mueller, this pex_config is nominally in the DAX bailiwick. Do you have any opinions here? Objections to DRP just going ahead and merging (some variant on) Tim's patch above?