# Remove /repo from paths to datasets

XMLWordPrintable

#### Details

• Type: RFC
• Resolution: Unresolved
• Component/s:
• Labels:
None

#### Description

The current convention at NCSA is that the data for camera XXX should be accessed via

 butler = dafPersist.Butler(/datasets/XXX/repo)

I'd like to propose that we move this up one level, so users can simply say

 butler = dafPersist.Butler(/datasets/XXX)

It's a small thing, but small things add up.

#### Activity

Hide
Robert Lupton added a comment -

I believe that this has been adopted, but we will delay implementation until after the initial HSC reprocessing.

The final accepted RFC is:

Datasets at NCSA may be passed to the butler as /datasets/cameraName (e.g. /datasets/hsc)

I.e. no trailing repo.

Show
Robert Lupton added a comment - I believe that this has been adopted, but we will delay implementation until after the initial HSC reprocessing. The final accepted RFC is: Datasets at NCSA may be passed to the butler as /datasets/cameraName (e.g. /datasets/hsc ) I.e. no trailing repo .
Hide
Hsin-Fang Chiang added a comment -

I've started the implementation, first in the DECam repo /datasets/decam (DM-11381).

I created a new folder /datasets/decam/_internal to have anything that's not in Butler/Mapper directory structure. Right now it looks like this:

 [hchiang2@lsst-dev01 _internal]$pwd /datasets/decam/_internal [hchiang2@lsst-dev01 _internal]$ ls * calib: bpmDes cp201302 preprocessed: dey_cosmos README saha_bulge raw: allen_neo cosmos nidever_smash README saha_bulge

I can run CmdLineTask like this:

 processCcd.py /datasets/decam/ --calib /datasets/decam/_internal/calib/cp201302/ -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --rerun private/hchiang2/ --id visit=177344 ccdnum=1 This uses CP MasterCal. It looks a bit weird to me that the task is given a calibration repo at /datasets/decam/_internal/calib/cp201302/ (as DecamMapper does not yet have this hack.) New Butler probably can do it more elegantly. I'll live with it for now. Show Hsin-Fang Chiang added a comment - I've started the implementation, first in the DECam repo /datasets/decam ( DM-11381 ). I created a new folder /datasets/decam/_internal to have anything that's not in Butler/Mapper directory structure. Right now it looks like this: [hchiang2 @lsst -dev01 _internal]$ pwd /datasets/decam/_internal [hchiang2 @lsst -dev01 _internal]$ls * calib: bpmDes cp201302 preprocessed: dey_cosmos README saha_bulge raw: allen_neo cosmos nidever_smash README saha_bulge I can run CmdLineTask like this: processCcd.py /datasets/decam/ --calib /datasets/decam/_internal/calib/cp201302/ -C$OBS_DECAM_DIR/config/processCcdCpIsr.py --rerun private /hchiang2/ --id visit= 177344 ccdnum= 1 This uses CP MasterCal. It looks a bit weird to me that the task is given a calibration repo at /datasets/decam/_internal/calib/cp201302/ (as DecamMapper does not yet have this hack .) New Butler probably can do it more elegantly. I'll live with it for now.
Hide
Simon Krughoff added a comment -

This RFC means that when implementing the README from RFC-447, it will need to go in /datasets/decam which is the root level of the repository. That is different from all the other datasets, but I suppose that's OK. I think that README is to be added by Hsin-Fang Chiang.

Show
Simon Krughoff added a comment - This RFC means that when implementing the README from RFC-447 , it will need to go in /datasets/decam which is the root level of the repository. That is different from all the other datasets, but I suppose that's OK. I think that README is to be added by Hsin-Fang Chiang .
Hide
John Swinbank added a comment -

Implementing this RFC requires updating the Developer Guide to describe the new layout before that new layout is applied to datasets. Created DM-14098 to cover that work; will add it as a blocker to existing implementation tickets.

Show
John Swinbank added a comment - Implementing this RFC requires updating the Developer Guide to describe the new layout before that new layout is applied to datasets. Created DM-14098 to cover that work; will add it as a blocker to existing implementation tickets.
Hide
Simon Krughoff added a comment -

I didn't realize before that this RFC is for all datasets. I thought it was asking for an exception for decam. If this is implemented, it makes it less obvious where to put the readme. I'm a little worried about that.

Show
Simon Krughoff added a comment - I didn't realize before that this RFC is for all datasets. I thought it was asking for an exception for decam . If this is implemented, it makes it less obvious where to put the readme. I'm a little worried about that.

#### People

Assignee:
Robert Lupton
Reporter:
Robert Lupton
Watchers:
Hsin-Fang Chiang, John Parejko, John Swinbank, Kian-Tat Lim, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness