Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3368

Port HSC MPI driver for single-visit processing

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      6
    • Sprint:
      Science Pipelines DM-W16-6, DRP X16-1, DRP X16-2
    • Team:
      Data Release Production

      Description

      Transfer the reduceFrames.py script and the ProcessExposureTask it utilizes from hscPipe to a new package in the LSST stack (RFC-68 proposes calling this new package pool_tasks, but this isn't set in stone).

      We should probably rename either the driver script or the task (or both), so they agree; the lack of consistency is a historical artifact on the HSC side, and I think it's time to change that.

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment -

            merged to master

            Show
            nlust Nate Lust added a comment - merged to master
            Hide
            nlust Nate Lust added a comment -

            Paul Price, The config file that I built for cfht already has that in it, so the only overrides Dominique Boutigny should need are run specific things if he wants to change an option between runs or for a specific purpose.

            Show
            nlust Nate Lust added a comment - Paul Price , The config file that I built for cfht already has that in it, so the only overrides Dominique Boutigny should need are run specific things if he wants to change an option between runs or for a specific purpose.
            Hide
            price Paul Price added a comment -

            Dominique Boutigny, put in your obs package's config/singleFrameDriver.py:

            import os.path
            from lsst.utils import getPackageDir
            config.processCcd.load(os.path.join(getPackageDir("obs_subaru"), "config", "processCcd.py"))
            

            This ensures you're using the same overrides in the singleFrameDriver as you use for processCcd.

            If you've got an override file that's not being automatically loaded from your obs package, you can put this at the top:

            try:
                config = config.processCcd
            except:
                pass
            

            With this, you can use the overrides with either the singleFrameDriver or processCcd.

            Show
            price Paul Price added a comment - Dominique Boutigny , put in your obs package's config/singleFrameDriver.py: import os.path from lsst.utils import getPackageDir config.processCcd.load(os.path.join(getPackageDir("obs_subaru"), "config", "processCcd.py")) This ensures you're using the same overrides in the singleFrameDriver as you use for processCcd. If you've got an override file that's not being automatically loaded from your obs package, you can put this at the top: try: config = config.processCcd except: pass With this, you can use the overrides with either the singleFrameDriver or processCcd.
            Hide
            nlust Nate Lust added a comment -

            You should be able to do it just the same way, just noting that processccd is a subtask (hence sub configuration) of SingleFrameDriver test. So your configuration file would be something like config.processCcd.<configuration variable> = <value> instead of just config.<configuration variable> = <value>. Everything will just be relative to the configuration of the task you are running.

            Show
            nlust Nate Lust added a comment - You should be able to do it just the same way, just noting that processccd is a subtask (hence sub configuration) of SingleFrameDriver test. So your configuration file would be something like config.processCcd.<configuration variable> = <value> instead of just config.<configuration variable> = <value>. Everything will just be relative to the configuration of the task you are running.
            Hide
            boutigny Dominique Boutigny added a comment -

            I confirm that this is now running and ending... with the latest mods in obs_cfht.
            Just one question :

            • How can I pass a configuration parameter file as I do with processCcd with the --configfile parameter ?
            Show
            boutigny Dominique Boutigny added a comment - I confirm that this is now running and ending... with the latest mods in obs_cfht. Just one question : How can I pass a configuration parameter file as I do with processCcd with the --configfile parameter ?

              People

              • Assignee:
                nlust Nate Lust
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Russell Owen
                Watchers:
                Dominique Boutigny, Hsin-Fang Chiang, Jim Bosch, John Swinbank, Nate Lust, Paul Price, Russell Owen
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel