Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7884

Standardize Coadd and Differencing datasets

    Details

    • Story Points:
      4
    • Epic Link:
    • Sprint:
      DRP F16-5
    • Team:
      Data Release Production

      Description

      Implementation for what is decided by RFC-231.

      Make differencing and coadd datasets consistent wherever possible. Move common datasets to daf_butlerUtils. Write a guide of changes for each camera.

        Attachments

          Issue Links

            Activity

            Hide
            pgee Perry Gee added a comment -

            Jim Bosch Sorry if that last comment was not clear. I think that some of the coadd and difference datasets only contain filter, tract, and patch, while others contain camera dependent columns. Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base?

            In general, obs_base will have whatever HscMapper has.
            .

            Show
            pgee Perry Gee added a comment - Jim Bosch Sorry if that last comment was not clear. I think that some of the coadd and difference datasets only contain filter, tract, and patch, while others contain camera dependent columns. Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base? In general, obs_base will have whatever HscMapper has. .
            Hide
            jbosch Jim Bosch added a comment -

            Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base?

            Yes.

            In general, obs_base will have whatever HscMapper has.

            Agreed. If you do run into any cases where all of the other mappers agree with each other but disagree with HscMapper, feel free to make HscMapper the one that has to override the default.

            Show
            jbosch Jim Bosch added a comment - Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base? Yes. In general, obs_base will have whatever HscMapper has. Agreed. If you do run into any cases where all of the other mappers agree with each other but disagree with HscMapper, feel free to make HscMapper the one that has to override the default.
            Hide
            pgee Perry Gee added a comment -

            Jim Bosch I chatted with K-T about raw_visit and raw_skyTile. He says that raw_skyTile is completely obsolete, so I removed all raw_skyTile entries from the obs_base datasets.

            I also looked over registry.sqlite3 files for all of the mappers, and none has a raw_visit file which has columns which are not in raw and which are used by any datasets. Since at least in theory, raw_visit is derived from raw, I think it is safe to remove/replace raw_visit with raw.

            I have added these changes since the review. Let me know if you approve.

            Also, any changes which involve hand editing individual paf files will be moved to the RFC-232-237, that is, the RFC for the individual mapper. I expect that all to be done next week. I've added an issue for those changes. This ticket is specifically to cover global changes involving obs_base.

            Show
            pgee Perry Gee added a comment - Jim Bosch I chatted with K-T about raw_visit and raw_skyTile. He says that raw_skyTile is completely obsolete, so I removed all raw_skyTile entries from the obs_base datasets. I also looked over registry.sqlite3 files for all of the mappers, and none has a raw_visit file which has columns which are not in raw and which are used by any datasets. Since at least in theory, raw_visit is derived from raw, I think it is safe to remove/replace raw_visit with raw. I have added these changes since the review. Let me know if you approve. Also, any changes which involve hand editing individual paf files will be moved to the RFC-232 -237, that is, the RFC for the individual mapper. I expect that all to be done next week. I've added an issue for those changes. This ticket is specifically to cover global changes involving obs_base.
            Hide
            pgee Perry Gee added a comment -

            Merged, with individual paf modifications moved to RFC-232-237

            Show
            pgee Perry Gee added a comment - Merged, with individual paf modifications moved to RFC-232 -237
            Hide
            jbosch Jim Bosch added a comment -

            Ok. I completely agree with removing raw_skyTile, and while I'm a little nervous about dropping raw_visit from HSC in particular, with both your research and K-T's recommendation behind it I'm okay with it.

            Show
            jbosch Jim Bosch added a comment - Ok. I completely agree with removing raw_skyTile, and while I'm a little nervous about dropping raw_visit from HSC in particular, with both your research and K-T's recommendation behind it I'm okay with it.

              People

              • Assignee:
                pgee Perry Gee
                Reporter:
                pgee Perry Gee
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Michael Wood-Vasey, Perry Gee
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel