Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10021

Create a convenience method to search Config hierarchy

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: pex_config
    • Labels:
      None
    • Story Points:
      2
    • Team:
      Data Access and Database

      Description

      In the process of trying to figure out how to fix DM-9909, I realized I needed some function to search the configuration hierarchy of a task for a given parameter, to know exactly which subtask/subsubtask/etc. to point to to change a particular config switch. I threw together the following function and it helped me, so perhaps it might help others if it were adapted to be a method of the Task base class? I'm imaging something like being able to call myTask.searchConfig('myKeyword') and have it return all matches (showing the proper hierarchy) as well as their values.

      from lsst.pipe.tasks.processCcd import ProcessCcdTask
      import re
       
      def dict_generator(indict, pre=None):
          """Walk through a dictionary and return a list of lists
          
          Slightly modified from  
          http://stackoverflow.com/questions/12507206/python-recommended-way-to-walk-complex-dictionary-structures-imported-from-json
       
          The last item in each list is the value.
          """
          pre = pre[:] if pre else []
          if isinstance(indict, dict):
              for key, value in indict.items():
                  if isinstance(value, dict) and len(value) > 0:
                      for d in dict_generator(value, [key] + pre):
                          yield d
                  elif isinstance(value, list) or isinstance(value, tuple):
                      for v in value:
                          for d in dict_generator(v, [key] + pre):
                              yield d
                  else:
                      yield pre + [key, value]
          else:
              yield indict
       
      def search_config(config, key):
          """Returns any config options and values matching key
          
          Config is instance of a Config object.
          
          Searching by key name:
          
              >>> search_config(ProcessCcdTask.ConfigClass(), 'filterMap')
                  {'astromRefObjLoader.calibrate.filterMap': {},
                   'photoRefObjLoader.calibrate.filterMap': {},
                   'refObjLoader.charImage.filterMap': {}}
          
          Or by task name:
          
              >>> search_config(ProcessCcdTask.ConfigClass(), 'photoRefObjLoader')
                  {'photoRefObjLoader.calibrate.defaultFilter': '',
                   'photoRefObjLoader.calibrate.filterMap': {},
                   'photoRefObjLoader.calibrate.pixelMargin': 50}    
          
          No need for exact matches:
          
              >>> search_config(ProcessCcdTask.ConfigClass(), 'ObjLoader')
                  {'astromRefObjLoader.calibrate.defaultFilter': '',
                   'astromRefObjLoader.calibrate.filterMap': {},
                   'astromRefObjLoader.calibrate.pixelMargin': 50,
                   'photoRefObjLoader.calibrate.defaultFilter': '',
                   'photoRefObjLoader.calibrate.filterMap': {},
                   'photoRefObjLoader.calibrate.pixelMargin': 50,
                   'refObjLoader.charImage.defaultFilter': '',
                   'refObjLoader.charImage.filterMap': {},
                   'refObjLoader.charImage.pixelMargin': 50}    
          
          """
          # Import 'nan' because otherwise eval command will fail 
          from numpy import nan
          d = eval(str(config))
          
          lfound = []
          for l in dict_generator(d):
              try:
                  for s in l:
                      if re.search(key, str(s)):
                          lfound.append(l)
      #             if key in l:
      #                 lfound.append(l)
              except TypeError:
                  pass
              
          # The last item in each list is the value
          return {'.'.join(l[:-1]):l[-1] for l in lfound}
      

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            We discussed this at the DRP standup of 2017-03-30.

            We agreed that this is a nice convenience feature. We wondered if it could be achieved with even less code by re-using parts of the machinery already built for command line task.

            Fritz Mueller, this pex_config is nominally in the DAX bailiwick. Do you have any opinions here? Objections to DRP just going ahead and merging (some variant on) Tim's patch above?

            Show
            swinbank John Swinbank added a comment - We discussed this at the DRP standup of 2017-03-30. We agreed that this is a nice convenience feature. We wondered if it could be achieved with even less code by re-using parts of the machinery already built for command line task. Fritz Mueller , this pex_config is nominally in the DAX bailiwick. Do you have any opinions here? Objections to DRP just going ahead and merging (some variant on) Tim's patch above?
            Hide
            salnikov Andy Salnikov added a comment -

            My feeling is this addition should probably go to pex_config rather than pipe_base as it does not need to depend on any Task stuff. And it should be based on pex_config.Config iteration rather than depending on eval(str(config)); or maybe just replace eval(str()) with config.toDict() if dict-of-values is what you need (as opposed to the dict-of-fields).

            Show
            salnikov Andy Salnikov added a comment - My feeling is this addition should probably go to pex_config rather than pipe_base as it does not need to depend on any Task stuff. And it should be based on pex_config.Config iteration rather than depending on eval(str(config)) ; or maybe just replace eval(str()) with config.toDict() if dict-of-values is what you need (as opposed to the dict-of-fields).
            Hide
            swinbank John Swinbank added a comment -

            My feeling is this addition should probably go to pex_config rather than pipe_base as it does not need to depend on any Task stuff

            Show
            swinbank John Swinbank added a comment - My feeling is this addition should probably go to pex_config rather than pipe_base as it does not need to depend on any Task stuff
            Hide
            fritzm Fritz Mueller added a comment -

            Is this enhancement still desired?

            Show
            fritzm Fritz Mueller added a comment - Is this enhancement still desired?
            Hide
            ktl Kian-Tat Lim added a comment -

            As I understand it, this is a debugging feature, not something that would be used in normal Task code. I agree that it belongs in pex_config and that it should not use eval. It doesn't seem to be of high priority, however.

            Show
            ktl Kian-Tat Lim added a comment - As I understand it, this is a debugging feature, not something that would be used in normal Task code. I agree that it belongs in pex_config and that it should not use eval . It doesn't seem to be of high priority, however.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              tmorton Tim Morton [X] (Inactive)
              Watchers:
              Andy Salnikov, Fritz Mueller, Jim Bosch, John Swinbank, Kian-Tat Lim, Tim Morton [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.