Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-192

Single source for camera data

    Details

    • Type: RFC
    • Status: Adopted
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Camera geometry used to be defined using PAF (policy) files, which are now deprecated. As part of the transition to the refactored camera geometry scheme, scripts were introduced to convert from the PAF files to the new camera geometry configuration scheme which uses FITS files and a python file to describe the camera. These scripts ("genCameraRepository.py" in obs_cfht, "makeLsstCameraRepository.py" in obs_lsstSim, "makeDecamCameraRepository.py" in obs_decam) are still part of the obs_* packages, and some people rely on them for making changes to the camera description. On the other hand, the generated FITS files and python file are also first-class members of the obs_* packages. This means that we have two sources of the same information, which is dangerous. What should be viewed as the primary source? The options appear to be:

      1. The generated FITS files and python files are primary, and the conversion scripts were temporary, intended for the transition only. This is the approach we have taken in obs_subaru, where we have deleted the conversion script as a relic of the past and track only the FITS tables because that's what the CameraMapper uses; the disadvantage is that changes to the camera geometry require editing the binary FITS tables. If this option is accepted, we should delete the conversion scripts in all obs_* packages.

      2. The conversion scripts are primary, and the generated FITS files and python files are just a cache for the CameraMapper to read. This provides a readable and easily editable source, but requires additional steps to generate what the CameraMapper will use. If this option is accepted, we should delete the generated FITS files and python files from the obs_* packages and generate them at build time. We should also update the conversion scripts to use a different source than the deprecated PAF files (e.g., YAML).

      3. Some other source is primary. Maybe the FITS files weren't such a great idea because they're hard to edit, and we should specify our camera geometry purely in python. This is the approach taken in obs_monocam. However, Monocam has only one CCD, and maybe this wouldn't be feasible for cameras with a hundred CCDs.

      It may be that this choice could be different for different obs_* packages depending on the preferences of the maintainer, but a choice needs to be made for each.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            I agree that the current situation is untenable. I think any of these solutions is acceptable and I suspect different projects may prefer different solutions.

            I wish we could use astropy tables instead of afw tables to hold the information (since the tools to manage those are a bit easier to use), but presently the information needs to be stored in a C++ object in the Detector, and I don't see that being changed soon.

            Show
            rowen Russell Owen added a comment - I agree that the current situation is untenable. I think any of these solutions is acceptable and I suspect different projects may prefer different solutions. I wish we could use astropy tables instead of afw tables to hold the information (since the tools to manage those are a bit easier to use), but presently the information needs to be stored in a C++ object in the Detector, and I don't see that being changed soon.
            Hide
            price Paul Price added a comment -

            I don't think I made an actual proposal for people to disagree with. I suggest adopting option 1, as I think that state is closest to what was envisioned in the cameraGeom refactoring (at least, my interpretation of what was envisioned) and it's the easiest to implement. I don't mind the other options, but they're a lot more work — they need to invent new configuration schemes, which is exactly one of the reasons we did the cameraGeom refactoring in the first place, and that invented the FITS files so I suggest we use those.

            I would prefer that the different obs_* packages all adopt the same solution, as it makes it easier to move between packages.

            Show
            price Paul Price added a comment - I don't think I made an actual proposal for people to disagree with. I suggest adopting option 1, as I think that state is closest to what was envisioned in the cameraGeom refactoring (at least, my interpretation of what was envisioned) and it's the easiest to implement. I don't mind the other options, but they're a lot more work — they need to invent new configuration schemes, which is exactly one of the reasons we did the cameraGeom refactoring in the first place, and that invented the FITS files so I suggest we use those. I would prefer that the different obs_* packages all adopt the same solution, as it makes it easier to move between packages.
            Hide
            jbosch Jim Bosch added a comment -

            I don't have terribly strong opinions here, but I do agree that this needs to be fixed somehow.

            I also think that camera descriptions will inevitably contain both machine-generated binary blobs and human-generated structured text. To me that suggests that we should have a YAML like format for human-generated, human-generated comment and structure that can delegate individual components to binary blobs stored some other way (such as FITS or HDF5).

            Show
            jbosch Jim Bosch added a comment - I don't have terribly strong opinions here, but I do agree that this needs to be fixed somehow. I also think that camera descriptions will inevitably contain both machine-generated binary blobs and human-generated structured text. To me that suggests that we should have a YAML like format for human-generated, human-generated comment and structure that can delegate individual components to binary blobs stored some other way (such as FITS or HDF5).
            Hide
            krughoff Simon Krughoff added a comment -

            I'll weigh in here. I really always meant the in memory object to be the first class product and that the on disk representation was an implementation detail, but I understand the pain with updating the fits files. They were meant to be a convenience for getting a camera to the butler. I didn't envision we'd need to update the camera model very often and when we did that it would be in memory.

            I think, in reality, we need to allow multiple of the options. The obs_lsstSim mechanism (conversion script), for example, is essentially dictated by how the image simulation software is configured. For cameras like monocam, having a python implementation makes a ton of sense.

            Given that I don't think we can choose a single option for all obs_* packages, we can eliminate some confusion by suggesting that any given obs package has only one of the options. I think we should also require that paf files disappear ASAP.

            Just my thoughts.

            Show
            krughoff Simon Krughoff added a comment - I'll weigh in here. I really always meant the in memory object to be the first class product and that the on disk representation was an implementation detail, but I understand the pain with updating the fits files. They were meant to be a convenience for getting a camera to the butler. I didn't envision we'd need to update the camera model very often and when we did that it would be in memory. I think, in reality, we need to allow multiple of the options. The obs_lsstSim mechanism (conversion script), for example, is essentially dictated by how the image simulation software is configured. For cameras like monocam, having a python implementation makes a ton of sense. Given that I don't think we can choose a single option for all obs_* packages, we can eliminate some confusion by suggesting that any given obs package has only one of the options. I think we should also require that paf files disappear ASAP. Just my thoughts.
            Hide
            price Paul Price added a comment -

            I've created DM-6349 to replace the PAF files. Simon Krughoff, could you please schedule this?

            I've created DM-6350 to delete the generated files and have them created at build time. I'm happy to handle that.

            Show
            price Paul Price added a comment - I've created DM-6349 to replace the PAF files. Simon Krughoff , could you please schedule this? I've created DM-6350 to delete the generated files and have them created at build time. I'm happy to handle that.
            Hide
            krughoff Simon Krughoff added a comment -

            Paul Price I'll schedule it.

            Show
            krughoff Simon Krughoff added a comment - Paul Price I'll schedule it.
            Hide
            price Paul Price added a comment -

            We agreed to remove the multiple options; this will be done in a different way for each obs_* package. Work has been scheduled.

            Show
            price Paul Price added a comment - We agreed to remove the multiple options; this will be done in a different way for each obs_* package. Work has been scheduled.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Watchers:
                Gregory Dubois-Felsmann, Jim Bosch, John Swinbank, Paul Price, Russell Owen, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel