Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-308

Remove /repo from paths to datasets

    Details

    • Type: RFC
    • Status: Adopted
    • Resolution: Unresolved
    • Component/s: DM
    • Labels:
      None

      Description

      The current convention at NCSA is that the data for camera XXX should be accessed via

      butler = dafPersist.Butler(/datasets/XXX/repo)
      

      I'd like to propose that we move this up one level, so users can simply say

      butler = dafPersist.Butler(/datasets/XXX)
      

      It's a small thing, but small things add up.

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment -

            I believe that this has been adopted, but we will delay implementation until after the initial HSC reprocessing.

            The final accepted RFC is:

            Datasets at NCSA may be passed to the butler as /datasets/cameraName (e.g. /datasets/hsc)

            I.e. no trailing repo.

            Show
            rhl Robert Lupton added a comment - I believe that this has been adopted, but we will delay implementation until after the initial HSC reprocessing. The final accepted RFC is: Datasets at NCSA may be passed to the butler as /datasets/cameraName (e.g. /datasets/hsc ) I.e. no trailing repo .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I've started the implementation, first in the DECam repo /datasets/decam (DM-11381).

            I created a new folder /datasets/decam/_internal to have anything that's not in Butler/Mapper directory structure. Right now it looks like this:

            [hchiang2@lsst-dev01 _internal]$ pwd
            /datasets/decam/_internal
            [hchiang2@lsst-dev01 _internal]$ ls *
            calib:
            bpmDes  cp201302
             
            preprocessed:
            dey_cosmos  README  saha_bulge
             
            raw:
            allen_neo  cosmos  nidever_smash  README  saha_bulge
            

            I can run CmdLineTask like this:

            processCcd.py /datasets/decam/ --calib /datasets/decam/_internal/calib/cp201302/ -C $OBS_DECAM_DIR/config/processCcdCpIsr.py  --rerun private/hchiang2/ --id visit=177344 ccdnum=1
            

            This uses CP MasterCal. It looks a bit weird to me that the task is given a calibration repo at /datasets/decam/_internal/calib/cp201302/ (as DecamMapper does not yet have this hack.) New Butler probably can do it more elegantly. I'll live with it for now.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I've started the implementation, first in the DECam repo /datasets/decam ( DM-11381 ). I created a new folder /datasets/decam/_internal to have anything that's not in Butler/Mapper directory structure. Right now it looks like this: [hchiang2 @lsst -dev01 _internal]$ pwd /datasets/decam/_internal [hchiang2 @lsst -dev01 _internal]$ ls * calib: bpmDes cp201302   preprocessed: dey_cosmos README saha_bulge   raw: allen_neo cosmos nidever_smash README saha_bulge I can run CmdLineTask like this: processCcd.py /datasets/decam/ --calib /datasets/decam/_internal/calib/cp201302/ -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --rerun private /hchiang2/ --id visit= 177344 ccdnum= 1 This uses CP MasterCal. It looks a bit weird to me that the task is given a calibration repo at /datasets/decam/_internal/calib/cp201302/ (as DecamMapper does not yet have this hack .) New Butler probably can do it more elegantly. I'll live with it for now.
            Hide
            krughoff Simon Krughoff added a comment -

            This RFC means that when implementing the README from RFC-447, it will need to go in /datasets/decam which is the root level of the repository. That is different from all the other datasets, but I suppose that's OK. I think that README is to be added by Hsin-Fang Chiang.

            Show
            krughoff Simon Krughoff added a comment - This RFC means that when implementing the README from RFC-447 , it will need to go in /datasets/decam which is the root level of the repository. That is different from all the other datasets, but I suppose that's OK. I think that README is to be added by Hsin-Fang Chiang .
            Hide
            swinbank John Swinbank added a comment -

            Implementing this RFC requires updating the Developer Guide to describe the new layout before that new layout is applied to datasets. Created DM-14098 to cover that work; will add it as a blocker to existing implementation tickets.

            Show
            swinbank John Swinbank added a comment - Implementing this RFC requires updating the Developer Guide to describe the new layout before that new layout is applied to datasets. Created DM-14098 to cover that work; will add it as a blocker to existing implementation tickets.
            Hide
            krughoff Simon Krughoff added a comment -

            I didn't realize before that this RFC is for all datasets. I thought it was asking for an exception for decam. If this is implemented, it makes it less obvious where to put the readme. I'm a little worried about that.

            Show
            krughoff Simon Krughoff added a comment - I didn't realize before that this RFC is for all datasets. I thought it was asking for an exception for decam . If this is implemented, it makes it less obvious where to put the readme. I'm a little worried about that.

              People

              • Assignee:
                rhl Robert Lupton
                Reporter:
                rhl Robert Lupton
                Watchers:
                Hsin-Fang Chiang, John Parejko, John Swinbank, Kian-Tat Lim, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Planned End:

                  Summary Panel