Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7884

Standardize Coadd and Differencing datasets

    XMLWordPrintable

Details

    • 4
    • DRP F16-5
    • Data Release Production

    Description

      Implementation for what is decided by RFC-231.

      Make differencing and coadd datasets consistent wherever possible. Move common datasets to daf_butlerUtils. Write a guide of changes for each camera.

      Attachments

        1. datasets.yaml.new
          1.0 kB
        2. exposures.yaml.new
          0.4 kB
        3. obskeys.yaml
          2 kB

        Issue Links

          Activity

            pgee Perry Gee added a comment -

            jbosch Sorry if that last comment was not clear. I think that some of the coadd and difference datasets only contain filter, tract, and patch, while others contain camera dependent columns. Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base?

            In general, obs_base will have whatever HscMapper has.
            .

            pgee Perry Gee added a comment - jbosch Sorry if that last comment was not clear. I think that some of the coadd and difference datasets only contain filter, tract, and patch, while others contain camera dependent columns. Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base? In general, obs_base will have whatever HscMapper has. .
            jbosch Jim Bosch added a comment -

            Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base?

            Yes.

            In general, obs_base will have whatever HscMapper has.

            Agreed. If you do run into any cases where all of the other mappers agree with each other but disagree with HscMapper, feel free to make HscMapper the one that has to override the default.

            jbosch Jim Bosch added a comment - Would you like to have obs_base contain the template in the former case, so that an override is only needed if the template differs from what is in obs_base? Yes. In general, obs_base will have whatever HscMapper has. Agreed. If you do run into any cases where all of the other mappers agree with each other but disagree with HscMapper, feel free to make HscMapper the one that has to override the default.
            pgee Perry Gee added a comment -

            jbosch I chatted with K-T about raw_visit and raw_skyTile. He says that raw_skyTile is completely obsolete, so I removed all raw_skyTile entries from the obs_base datasets.

            I also looked over registry.sqlite3 files for all of the mappers, and none has a raw_visit file which has columns which are not in raw and which are used by any datasets. Since at least in theory, raw_visit is derived from raw, I think it is safe to remove/replace raw_visit with raw.

            I have added these changes since the review. Let me know if you approve.

            Also, any changes which involve hand editing individual paf files will be moved to the RFC-232-237, that is, the RFC for the individual mapper. I expect that all to be done next week. I've added an issue for those changes. This ticket is specifically to cover global changes involving obs_base.

            pgee Perry Gee added a comment - jbosch I chatted with K-T about raw_visit and raw_skyTile. He says that raw_skyTile is completely obsolete, so I removed all raw_skyTile entries from the obs_base datasets. I also looked over registry.sqlite3 files for all of the mappers, and none has a raw_visit file which has columns which are not in raw and which are used by any datasets. Since at least in theory, raw_visit is derived from raw, I think it is safe to remove/replace raw_visit with raw. I have added these changes since the review. Let me know if you approve. Also, any changes which involve hand editing individual paf files will be moved to the RFC-232 -237, that is, the RFC for the individual mapper. I expect that all to be done next week. I've added an issue for those changes. This ticket is specifically to cover global changes involving obs_base.
            pgee Perry Gee added a comment -

            Merged, with individual paf modifications moved to RFC-232-237

            pgee Perry Gee added a comment - Merged, with individual paf modifications moved to RFC-232 -237
            jbosch Jim Bosch added a comment -

            Ok. I completely agree with removing raw_skyTile, and while I'm a little nervous about dropping raw_visit from HSC in particular, with both your research and K-T's recommendation behind it I'm okay with it.

            jbosch Jim Bosch added a comment - Ok. I completely agree with removing raw_skyTile, and while I'm a little nervous about dropping raw_visit from HSC in particular, with both your research and K-T's recommendation behind it I'm okay with it.

            People

              pgee Perry Gee
              pgee Perry Gee
              Jim Bosch
              Jim Bosch, John Parejko, John Swinbank, Michael Wood-Vasey, Perry Gee
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.