# Provide input data for exampleCmdLineTask.py

XMLWordPrintable

## Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Sprint:
Science Pipelines DM-S15-4, Science Pipelines DM-S15-5
• Team:

## Description

 # The following will work on an NCSA lsst* computer: examples/exampleCmdLineTask.py /lsst8/krughoff/diffim_data/sparse_diffim_output_v7_2 --id visit=6866601

There are a few problems with that:

• Even though that data exists now, it's unclear how long it will remain there, or what steps are being taken to preserve it;
• The mention of this data is fairly well buried – it does appear in the documentation, but it's certainly not the first thing a new user will stumble upon.

At least the first two points could be addressed by referring to a publicly available data repository. For example, the following works once afwdata has been set up:

 examples/exampleCmdLineTask.py ${AFWDATA_DIR}/ImSim --id visit=85408556 Although this has the downside of only providing a single image. ## Attachments ## Activity Hide Russell Owen added a comment - obs_test has a suitable repository (it did not exist when I wrote the example) that contains a few images, though at this time it only contains raw images, no calexp. Note that the code in question is just an example and will work with any repository. Show Russell Owen added a comment - obs_test has a suitable repository (it did not exist when I wrote the example) that contains a few images, though at this time it only contains raw images, no calexp. Note that the code in question is just an example and will work with any repository. Hide John Swinbank added a comment - Thanks! I'll push a tweak to the docstrings to point to obs_test when I get a moment (unless you beat me to it). Since it's just a minor doc update, I won't bother sending it for review unless you request it. (It was an educational experience for me to establish that it will not work with any repository; it took me a while to work out what this meant:  $ python examples/exampleCmdLineTask.py ../lsst_dm_stack_demo/input/ : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/exampleTask.py' : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/sdss/exampleTask.py' : input=/Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input : calib=None : output=None CameraMapper: Loading registry registry from /Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input/registry.sqlite3 usage: exampleCmdLineTask.py input [options] exampleCmdLineTask.py: error: 'Cannot get keys for datasetType raw at level None'

In the end, that's something I needed to learn, but it's probably not the best experience for folks struggling to get up to speed.)

Show
John Swinbank added a comment - Thanks! I'll push a tweak to the docstrings to point to obs_test when I get a moment (unless you beat me to it). Since it's just a minor doc update, I won't bother sending it for review unless you request it. (It was an educational experience for me to establish that it will not work with any repository; it took me a while to work out what this meant: \$ python examples/exampleCmdLineTask.py ../lsst_dm_stack_demo/input/ : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/exampleTask.py' : Config override file does not exist: '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/obs_sdss/10.0+1/config/sdss/exampleTask.py' : input=/Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input : calib=None : output=None CameraMapper: Loading registry registry from /Users/jds/Projects/Astronomy/LSST/src/lsst_dm_stack_demo/input/registry.sqlite3 usage: exampleCmdLineTask.py input [options] exampleCmdLineTask.py: error: 'Cannot get keys for datasetType raw at level None' In the end, that's something I needed to learn, but it's probably not the best experience for folks struggling to get up to speed.)
Hide
John Swinbank added a comment -

Actually, I'm a bit confused by the use of the calexp here – is there a reason to prefer it to raw? The comments refer to a "data reference for a calibrated science exposure", but, in fact, simply calling dataRef.get() returns a raw, contrary to expectations.

The easy way to make all this work would be to simply remove the reference to calexp from exampleCmdLineTask and just work on raw data from obs_test. I'm not sure I understand the logic behind what's there now well enough to know if that's a smart thing to do though.

Show
John Swinbank added a comment - Actually, I'm a bit confused by the use of the calexp here – is there a reason to prefer it to raw? The comments refer to a "data reference for a calibrated science exposure", but, in fact, simply calling dataRef.get() returns a raw, contrary to expectations. The easy way to make all this work would be to simply remove the reference to calexp from exampleCmdLineTask and just work on raw data from obs_test . I'm not sure I understand the logic behind what's there now well enough to know if that's a smart thing to do though.
Hide
Russell Owen added a comment -

Implemented in pipe_tasks tickets/DM-1761. I took your suggestion of unpersisting raw exposures and simplified the example text to use obs_test's data repository.

Show
Russell Owen added a comment - Implemented in pipe_tasks tickets/ DM-1761 . I took your suggestion of unpersisting raw exposures and simplified the example text to use obs_test's data repository.
Hide
John Swinbank added a comment -

This is nice, thanks. Good to merge.

Show
John Swinbank added a comment - This is nice, thanks. Good to merge.
Hide
Russell Owen added a comment -

Thanks!

Show
Russell Owen added a comment - Thanks!

## People

• Assignee:
Russell Owen
Reporter:
John Swinbank
Reviewers:
John Swinbank
Watchers:
John Swinbank, Russell Owen