Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10021

Create a convenience method to search Config hierarchy

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: pex_config
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Team:
      Data Access and Database

      Description

      In the process of trying to figure out how to fix DM-9909, I realized I needed some function to search the configuration hierarchy of a task for a given parameter, to know exactly which subtask/subsubtask/etc. to point to to change a particular config switch. I threw together the following function and it helped me, so perhaps it might help others if it were adapted to be a method of the Task base class? I'm imaging something like being able to call myTask.searchConfig('myKeyword') and have it return all matches (showing the proper hierarchy) as well as their values.

      from lsst.pipe.tasks.processCcd import ProcessCcdTask
      import re
       
      def dict_generator(indict, pre=None):
          """Walk through a dictionary and return a list of lists
          
          Slightly modified from  
          http://stackoverflow.com/questions/12507206/python-recommended-way-to-walk-complex-dictionary-structures-imported-from-json
       
          The last item in each list is the value.
          """
          pre = pre[:] if pre else []
          if isinstance(indict, dict):
              for key, value in indict.items():
                  if isinstance(value, dict) and len(value) > 0:
                      for d in dict_generator(value, [key] + pre):
                          yield d
                  elif isinstance(value, list) or isinstance(value, tuple):
                      for v in value:
                          for d in dict_generator(v, [key] + pre):
                              yield d
                  else:
                      yield pre + [key, value]
          else:
              yield indict
       
      def search_config(config, key):
          """Returns any config options and values matching key
          
          Config is instance of a Config object.
          
          Searching by key name:
          
              >>> search_config(ProcessCcdTask.ConfigClass(), 'filterMap')
                  {'astromRefObjLoader.calibrate.filterMap': {},
                   'photoRefObjLoader.calibrate.filterMap': {},
                   'refObjLoader.charImage.filterMap': {}}
          
          Or by task name:
          
              >>> search_config(ProcessCcdTask.ConfigClass(), 'photoRefObjLoader')
                  {'photoRefObjLoader.calibrate.defaultFilter': '',
                   'photoRefObjLoader.calibrate.filterMap': {},
                   'photoRefObjLoader.calibrate.pixelMargin': 50}    
          
          No need for exact matches:
          
              >>> search_config(ProcessCcdTask.ConfigClass(), 'ObjLoader')
                  {'astromRefObjLoader.calibrate.defaultFilter': '',
                   'astromRefObjLoader.calibrate.filterMap': {},
                   'astromRefObjLoader.calibrate.pixelMargin': 50,
                   'photoRefObjLoader.calibrate.defaultFilter': '',
                   'photoRefObjLoader.calibrate.filterMap': {},
                   'photoRefObjLoader.calibrate.pixelMargin': 50,
                   'refObjLoader.charImage.defaultFilter': '',
                   'refObjLoader.charImage.filterMap': {},
                   'refObjLoader.charImage.pixelMargin': 50}    
          
          """
          # Import 'nan' because otherwise eval command will fail 
          from numpy import nan
          d = eval(str(config))
          
          lfound = []
          for l in dict_generator(d):
              try:
                  for s in l:
                      if re.search(key, str(s)):
                          lfound.append(l)
      #             if key in l:
      #                 lfound.append(l)
              except TypeError:
                  pass
              
          # The last item in each list is the value
          return {'.'.join(l[:-1]):l[-1] for l in lfound}
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tmorton Tim Morton
              Watchers:
              Andy Salnikov, Jim Bosch, John Swinbank, Tim Morton
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Summary Panel