Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3368

Port HSC MPI driver for single-visit processing

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      Science Pipelines DM-W16-6, DRP X16-1, DRP X16-2
    • Team:
      Data Release Production

      Description

      Transfer the reduceFrames.py script and the ProcessExposureTask it utilizes from hscPipe to a new package in the LSST stack (RFC-68 proposes calling this new package pool_tasks, but this isn't set in stone).

      We should probably rename either the driver script or the task (or both), so they agree; the lack of consistency is a historical artifact on the HSC side, and I think it's time to change that.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-5681 Provide single-visit processing capability as required by HSC

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-2983 Backport HSC parallelization code

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-4255 Determine scope and requirements for CalibrateTask redesign

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-5284 Port HSC meas_extensions_simpleShape package to LSST

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-5837 Document pipe_drivers

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            14 older comments
            Hide
            Permalink
            boutigny Dominique Boutigny added a comment -

            I confirm that this is now running and ending... with the latest mods in obs_cfht.
            Just one question :

            • How can I pass a configuration parameter file as I do with processCcd with the --configfile parameter ?
            Show
            boutigny Dominique Boutigny added a comment - I confirm that this is now running and ending... with the latest mods in obs_cfht. Just one question : How can I pass a configuration parameter file as I do with processCcd with the --configfile parameter ?
            Hide
            Permalink
            nlust Nate Lust added a comment -

            You should be able to do it just the same way, just noting that processccd is a subtask (hence sub configuration) of SingleFrameDriver test. So your configuration file would be something like config.processCcd.<configuration variable> = <value> instead of just config.<configuration variable> = <value>. Everything will just be relative to the configuration of the task you are running.

            Show
            nlust Nate Lust added a comment - You should be able to do it just the same way, just noting that processccd is a subtask (hence sub configuration) of SingleFrameDriver test. So your configuration file would be something like config.processCcd.<configuration variable> = <value> instead of just config.<configuration variable> = <value>. Everything will just be relative to the configuration of the task you are running.
            Hide
            Permalink
            price Paul Price added a comment -

            Dominique Boutigny, put in your obs package's config/singleFrameDriver.py:

            import os.path
            		 
            from lsst.utils import getPackageDir
            		 
            config.processCcd.load(os.path.join(getPackageDir("obs_subaru"), "config", "processCcd.py"))

            This ensures you're using the same overrides in the singleFrameDriver as you use for processCcd.

            If you've got an override file that's not being automatically loaded from your obs package, you can put this at the top:

            try:
            		 
                config = config.processCcd
            		 
            except:
            		 
                pass

            With this, you can use the overrides with either the singleFrameDriver or processCcd.

            Show
            price Paul Price added a comment - Dominique Boutigny , put in your obs package's config/singleFrameDriver.py: import os.path from lsst.utils import getPackageDir config.processCcd.load(os.path.join(getPackageDir("obs_subaru"), "config", "processCcd.py")) This ensures you're using the same overrides in the singleFrameDriver as you use for processCcd. If you've got an override file that's not being automatically loaded from your obs package, you can put this at the top: try: config = config.processCcd except: pass With this, you can use the overrides with either the singleFrameDriver or processCcd.
            Hide
            Permalink
            nlust Nate Lust added a comment -

            Paul Price, The config file that I built for cfht already has that in it, so the only overrides Dominique Boutigny should need are run specific things if he wants to change an option between runs or for a specific purpose.

            Show
            nlust Nate Lust added a comment - Paul Price , The config file that I built for cfht already has that in it, so the only overrides Dominique Boutigny should need are run specific things if he wants to change an option between runs or for a specific purpose.
            Hide
            Permalink
            nlust Nate Lust added a comment -

            merged to master

            Show
            nlust Nate Lust added a comment - merged to master

              People

              Assignee:
              nlust Nate Lust
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Russell Owen
              Watchers:
              Dominique Boutigny, Hsin-Fang Chiang, Jim Bosch, John Swinbank, Nate Lust, Paul Price, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.