Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17044

Make searching packages for PipelineTasks optional in command-line activator

    Details

    • Story Points:
      0.5
    • Sprint:
      BG3_F18_11
    • Team:
      Data Access and Database

      Description

      The process of importing all modules in lsst.pipe.tasks to search for PipelineTask is slow enough that it's not always a convenience; while it's very nice to have for the list subcommand, it'd be nice to disable it if the user provides fully-qualified names for the Tasks to be run.

      Another option might be to cache the results of the search to a temporary file so it's only slow the first time the activator command-line tool is invoked in (e.g.) a terminal session, but it seems to me that it'd be difficult to know when to invalidate that cache.

        Attachments

          Activity

          Hide
          salnikov Andy Salnikov added a comment -

          Thanks for review! Merged and done. 

          Show
          salnikov Andy Salnikov added a comment - Thanks for review! Merged and done. 
          Hide
          jbosch Jim Bosch added a comment -

          Looks good. I hadn't realized the partial module name would reduce the search space so well already.

          Show
          jbosch Jim Bosch added a comment - Looks good. I hadn't realized the partial module name would reduce the search space so well already.
          Hide
          salnikov Andy Salnikov added a comment -

          I have added a trivial fix that requires module names to be included with the task names.

          Show
          salnikov Andy Salnikov added a comment - I have added a trivial fix that requires module names to be included with the task names.
          Hide
          salnikov Andy Salnikov added a comment -

          Here is how "complete" task name is searched with module name:

          $ pipetask -L debug build -t multiBand.DetectCoaddSourcesTask
          ctrl.mpexec.taskLoader DEBUG: load_task_class: will look for 'multiBand.DetectCoaddSourcesTask'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: try module 'multiBand'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: try module 'lsst.ctrl.mpexec.examples.multiBand'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: try module 'lsst.pipe.tasks.multiBand'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: imported 'lsst.pipe.tasks.multiBand'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found 'DetectCoaddSourcesTask' in 'lsst.pipe.tasks.multiBand', kind: PipelineTask
          

          and here is what it it needs to import when module name is not given:

          $ pipetask -L debug build -t DetectCoaddSourcesTask
          ctrl.mpexec.taskLoader DEBUG: load_task_class: will look for 'DetectCoaddSourcesTask'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing package 'lsst.ctrl.mpexec.examples'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.calexpToCoaddTask' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.calexpToCoaddTask'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.make_example_qgraph' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.make_example_qgraph'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.patchSkyMapTask' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.patchSkyMapTask'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.rawToCalexpTask' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.rawToCalexpTask'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.test1task' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.test1task'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.test2task' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.test2task'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing package 'lsst.pipe.tasks'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.assembleCoadd' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.assembleCoadd'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.calibrate' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.calibrate'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.characterizeImage' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.characterizeImage'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.coaddBase' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.coaddBase'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.coaddHelpers' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.coaddHelpers'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.coaddInputRecorder' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.coaddInputRecorder'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.colorterms' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.colorterms'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.dcrAssembleCoadd' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.dcrAssembleCoadd'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.dcrMultiBand' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.dcrMultiBand'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.deblendCoaddSourcesPipeline' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.deblendCoaddSourcesPipeline'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.exampleCmdLineTask' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.exampleCmdLineTask'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.exampleStatsTasks' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.exampleStatsTasks'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.fakes' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.fakes'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.getRepositoryData' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.getRepositoryData'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.imageDifference' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.imageDifference'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.ingest' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.ingest'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.ingestCalibs' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.ingestCalibs'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.ingestPgsql' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.ingestPgsql'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.interpImage' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.interpImage'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.makeCoaddTempExp' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.makeCoaddTempExp'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.makeDiscreteSkyMap' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.makeDiscreteSkyMap'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.makeSkyMap' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.makeSkyMap'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.matchBackgrounds' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.matchBackgrounds'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.measurePsf' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.measurePsf'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.mergeDetections' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.mergeDetections'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.mergeMeasurements' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.mergeMeasurements'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.mocks' ispkg=True
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.multiBand' ispkg=False
          ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.multiBand'
          ctrl.mpexec.taskLoader DEBUG: load_task_class: found class 'DetectCoaddSourcesTask' kind: PipelineTask
          

          Latter is not healthy of course, too much stuff to import (plus all indirect imports too like matplotlib).

          Show
          salnikov Andy Salnikov added a comment - Here is how "complete" task name is searched with module name: $ pipetask -L debug build -t multiBand.DetectCoaddSourcesTask ctrl.mpexec.taskLoader DEBUG: load_task_class: will look for 'multiBand.DetectCoaddSourcesTask' ctrl.mpexec.taskLoader DEBUG: load_task_class: try module 'multiBand' ctrl.mpexec.taskLoader DEBUG: load_task_class: try module 'lsst.ctrl.mpexec.examples.multiBand' ctrl.mpexec.taskLoader DEBUG: load_task_class: try module 'lsst.pipe.tasks.multiBand' ctrl.mpexec.taskLoader DEBUG: load_task_class: imported 'lsst.pipe.tasks.multiBand' ctrl.mpexec.taskLoader DEBUG: load_task_class: found 'DetectCoaddSourcesTask' in 'lsst.pipe.tasks.multiBand', kind: PipelineTask and here is what it it needs to import when module name is not given: $ pipetask -L debug build -t DetectCoaddSourcesTask ctrl.mpexec.taskLoader DEBUG: load_task_class: will look for 'DetectCoaddSourcesTask' ctrl.mpexec.taskLoader DEBUG: load_task_class: importing package 'lsst.ctrl.mpexec.examples' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.calexpToCoaddTask' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.calexpToCoaddTask' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.make_example_qgraph' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.make_example_qgraph' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.patchSkyMapTask' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.patchSkyMapTask' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.rawToCalexpTask' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.rawToCalexpTask' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.test1task' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.test1task' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.ctrl.mpexec.examples.test2task' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.ctrl.mpexec.examples.test2task' ctrl.mpexec.taskLoader DEBUG: load_task_class: importing package 'lsst.pipe.tasks' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.assembleCoadd' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.assembleCoadd' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.calibrate' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.calibrate' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.characterizeImage' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.characterizeImage' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.coaddBase' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.coaddBase' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.coaddHelpers' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.coaddHelpers' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.coaddInputRecorder' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.coaddInputRecorder' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.colorterms' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.colorterms' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.dcrAssembleCoadd' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.dcrAssembleCoadd' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.dcrMultiBand' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.dcrMultiBand' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.deblendCoaddSourcesPipeline' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.deblendCoaddSourcesPipeline' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.exampleCmdLineTask' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.exampleCmdLineTask' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.exampleStatsTasks' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.exampleStatsTasks' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.fakes' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.fakes' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.getRepositoryData' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.getRepositoryData' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.imageDifference' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.imageDifference' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.ingest' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.ingest' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.ingestCalibs' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.ingestCalibs' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.ingestPgsql' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.ingestPgsql' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.interpImage' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.interpImage' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.makeCoaddTempExp' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.makeCoaddTempExp' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.makeDiscreteSkyMap' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.makeDiscreteSkyMap' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.makeSkyMap' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.makeSkyMap' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.matchBackgrounds' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.matchBackgrounds' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.measurePsf' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.measurePsf' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.mergeDetections' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.mergeDetections' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.mergeMeasurements' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.mergeMeasurements' ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.mocks' ispkg=True ctrl.mpexec.taskLoader DEBUG: load_task_class: found module 'lsst.pipe.tasks.multiBand' ispkg=False ctrl.mpexec.taskLoader DEBUG: load_task_class: importing module 'lsst.pipe.tasks.multiBand' ctrl.mpexec.taskLoader DEBUG: load_task_class: found class 'DetectCoaddSourcesTask' kind: PipelineTask Latter is not healthy of course, too much stuff to import (plus all indirect imports too like matplotlib).
          Hide
          jbosch Jim Bosch added a comment -

          maybe we should just require full task name to be given to run?

          I'd be happy with this solution. I think if we want incomplete names to be performant enough, we'd need to explicitly make a registry/cache to look up rather than using Python imports as part of the search. And I don't think we need incomplete names right now.

          Show
          jbosch Jim Bosch added a comment - maybe we should just require full task name to be given to run? I'd be happy with this solution. I think if we want incomplete names to be performant enough, we'd need to explicitly make a registry/cache to look up rather than using Python imports as part of the search. And I don't think we need incomplete names right now.
          Hide
          salnikov Andy Salnikov added a comment -

          If full task name is given to run then it should not search but will instead try to import from a small set of packages, maybe we should just require full task name to be given to run?

          Show
          salnikov Andy Salnikov added a comment - If full task name is given to run then it should not search but will instead try to import from a small set of packages, maybe we should just require full task name to be given to run ?

            People

            • Assignee:
              salnikov Andy Salnikov
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Jim Bosch
              Watchers:
              Andy Salnikov, Jim Bosch, Nate Lust, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel