Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1761

Provide input data for exampleCmdLineTask.py

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      Science Pipelines DM-S15-4, Science Pipelines DM-S15-5
    • Team:
      Alert Production

      Description

      pipe_tasks/examples/exampleCmdLineTask.py reads data from a repository. The comments in pipe_tasks/python/lsst/pipe/tasks/exampleCmdLineTask.py suggest that

      # The following will work on an NCSA lsst* computer:
      examples/exampleCmdLineTask.py /lsst8/krughoff/diffim_data/sparse_diffim_output_v7_2 --id visit=6866601

      There are a few problems with that:

      • External contributors don't have access to lsst*;
      • Even though that data exists now, it's unclear how long it will remain there, or what steps are being taken to preserve it;
      • The mention of this data is fairly well buried – it does appear in the documentation, but it's certainly not the first thing a new user will stumble upon.

      At least the first two points could be addressed by referring to a publicly available data repository. For example, the following works once afwdata has been set up:

      examples/exampleCmdLineTask.py ${AFWDATA_DIR}/ImSim --id visit=85408556

      Although this has the downside of only providing a single image.

        Attachments

          Activity

          Hide
          rowen Russell Owen added a comment -

          obs_test has a suitable repository (it did not exist when I wrote the example) that contains a few images, though at this time it only contains raw images, no calexp. Note that the code in question is just an example and will work with any repository.

          Show
          rowen Russell Owen added a comment - obs_test has a suitable repository (it did not exist when I wrote the example) that contains a few images, though at this time it only contains raw images, no calexp. Note that the code in question is just an example and will work with any repository.
          Hide
          swinbank John Swinbank added a comment -

          Thanks! I'll push a tweak to the docstrings to point to obs_test when I get a moment (unless you beat me to it). Since it's just a minor doc update, I won't bother sending it for review unless you request it.

          (It was an educational experience for me to establish that it will not work with any repository; it took me a while to work out what this meant:

          $ python examples/exampleCmdLineTask.py ../lsst_dm_stack_demo/input/
          : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/exampleTask.py'
          : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/sdss/exampleTask.py'
          : input=/Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input
          : calib=None
          : output=None
          CameraMapper: Loading registry registry from /Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input/registry.sqlite3
          usage: exampleCmdLineTask.py input [options]
          exampleCmdLineTask.py: error: 'Cannot get keys for datasetType raw at level None'

          In the end, that's something I needed to learn, but it's probably not the best experience for folks struggling to get up to speed.)

          Show
          swinbank John Swinbank added a comment - Thanks! I'll push a tweak to the docstrings to point to obs_test when I get a moment (unless you beat me to it). Since it's just a minor doc update, I won't bother sending it for review unless you request it. (It was an educational experience for me to establish that it will not work with any repository; it took me a while to work out what this meant: $ python examples/exampleCmdLineTask.py ../lsst_dm_stack_demo/input/ : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/exampleTask.py' : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/sdss/exampleTask.py' : input=/Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input : calib=None : output=None CameraMapper: Loading registry registry from /Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input/registry.sqlite3 usage: exampleCmdLineTask.py input [options] exampleCmdLineTask.py: error: 'Cannot get keys for datasetType raw at level None' In the end, that's something I needed to learn, but it's probably not the best experience for folks struggling to get up to speed.)
          Hide
          swinbank John Swinbank added a comment -

          Actually, I'm a bit confused by the use of the calexp here – is there a reason to prefer it to raw? The comments refer to a "data reference for a calibrated science exposure", but, in fact, simply calling dataRef.get() returns a raw, contrary to expectations.

          The easy way to make all this work would be to simply remove the reference to calexp from exampleCmdLineTask and just work on raw data from obs_test. I'm not sure I understand the logic behind what's there now well enough to know if that's a smart thing to do though.

          Show
          swinbank John Swinbank added a comment - Actually, I'm a bit confused by the use of the calexp here – is there a reason to prefer it to raw? The comments refer to a "data reference for a calibrated science exposure", but, in fact, simply calling dataRef.get() returns a raw, contrary to expectations. The easy way to make all this work would be to simply remove the reference to calexp from exampleCmdLineTask and just work on raw data from obs_test . I'm not sure I understand the logic behind what's there now well enough to know if that's a smart thing to do though.
          Hide
          rowen Russell Owen added a comment -

          Implemented in pipe_tasks tickets/DM-1761. I took your suggestion of unpersisting raw exposures and simplified the example text to use obs_test's data repository.

          Show
          rowen Russell Owen added a comment - Implemented in pipe_tasks tickets/ DM-1761 . I took your suggestion of unpersisting raw exposures and simplified the example text to use obs_test's data repository.
          Hide
          swinbank John Swinbank added a comment -

          This is nice, thanks. Good to merge.

          Show
          swinbank John Swinbank added a comment - This is nice, thanks. Good to merge.
          Hide
          rowen Russell Owen added a comment -

          Thanks!

          Show
          rowen Russell Owen added a comment - Thanks!

            People

            • Assignee:
              rowen Russell Owen
              Reporter:
              swinbank John Swinbank
              Reviewers:
              John Swinbank
              Watchers:
              John Swinbank, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel