Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6349

Replace cameraGeom PAF files

    XMLWordPrintable

Details

    • 10
    • Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9, Alert Production F16 - 10
    • Alert Production

    Description

      PAF files have long been deprecated, but continue to be used for describing the camera geometry. We need to replace the PAF cameraGeom files used for CFHT-MegaCam, DECam, LSSTSim and SDSS, and the scripts used to convert these files to FITS files for reading by the Mappers. They might be replaced by a configuration like YAML, or pure python.

      Attachments

        Issue Links

          Activity

            Parejkoj John Parejko added a comment -

            Taking this on while I'm blocked by other things.

            Parejkoj John Parejko added a comment - Taking this on while I'm blocked by other things.
            Parejkoj John Parejko added a comment -

            Notes from krughoff and my chat about this:

            Cameras that we need to deal with

            built exclusively in code

            • monocam
            • test

            Don't need to do anything: there is only one source, by definition. (But! check that the serialized files are not committed to the repos.)

            reads text files and generates FITS (and other) files
            lsstSim

            Delete persisted files, and ensure that makeLsstCameraRepository.py is run by scons and that it puts things where the butler expects them.

            parses PAF
            suprime
            hsc
            sdss

            Translate PAF to something else, then save that. Write a PAF to YAML mapper, serialize that, delete PAF, delete any FITS (etc.) persisted config, ensure scons runs FITS generator.

            Need to check these
            cfht
            decam

            If these use PAF, do the above. Otherwise, do something else.

            How to check that we got it right?

            • Butlerize old and new and do `for obj in dir(old): assert getattr(old, obj) == getattr(new, obj)` but that may not work since == isn't defined on all our SWIGed things.
            • Write the persistence formats and checksum/diff them (old vs. new).
            Parejkoj John Parejko added a comment - Notes from krughoff and my chat about this: Cameras that we need to deal with built exclusively in code monocam test Don't need to do anything: there is only one source, by definition. (But! check that the serialized files are not committed to the repos.) reads text files and generates FITS (and other) files lsstSim Delete persisted files, and ensure that makeLsstCameraRepository.py is run by scons and that it puts things where the butler expects them. parses PAF suprime hsc sdss Translate PAF to something else, then save that. Write a PAF to YAML mapper, serialize that, delete PAF, delete any FITS (etc.) persisted config, ensure scons runs FITS generator. Need to check these cfht decam If these use PAF, do the above. Otherwise, do something else. How to check that we got it right? Butlerize old and new and do `for obj in dir(old): assert getattr(old, obj) == getattr(new, obj)` but that may not work since == isn't defined on all our SWIGed things. Write the persistence formats and checksum/diff them (old vs. new).
            price Paul Price added a comment -

            Suprime-Cam and HSC (both in obs_subaru) do not use PAF, but are using the persisted files (camera.py, 1 FITS file per CCD) as the primary source.

            price Paul Price added a comment - Suprime-Cam and HSC (both in obs_subaru) do not use PAF, but are using the persisted files (camera.py, 1 FITS file per CCD) as the primary source.

            Do we need to do the same for the defects files, or can those just remain as FITS files?

            Parejkoj John Parejko added a comment - Do we need to do the same for the defects files, or can those just remain as FITS files?
            price Paul Price added a comment -

            I think the guiding principle should be that there is only one authoritative source, and anything else is built from that source.

            In obs_subaru, the defects FITS files and registry are built by scons from text-based descriptions.

            price Paul Price added a comment - I think the guiding principle should be that there is only one authoritative source, and anything else is built from that source. In obs_subaru, the defects FITS files and registry are built by scons from text-based descriptions.
            Parejkoj John Parejko added a comment - - edited

            Please review the obs_lsstSim commit. Note that you'll need sconsUtils DM-7179 in order to build with scons. The included tests pass, but its probably worth running with some other tests (suggestions are welcome), as there are some changes to the currently-committed camera descriptions, but I don't know if they matter or not.

            Parejkoj John Parejko added a comment - - edited Please review the obs_lsstSim commit. Note that you'll need sconsUtils DM-7179 in order to build with scons. The included tests pass, but its probably worth running with some other tests (suggestions are welcome), as there are some changes to the currently-committed camera descriptions, but I don't know if they matter or not.
            Parejkoj John Parejko added a comment -

            Confirming from my notes above:

            obs_monocam and obs_test do not contain any generated FITS files, so are already in compliance.

            Parejkoj John Parejko added a comment - Confirming from my notes above: obs_monocam and obs_test do not contain any generated FITS files, so are already in compliance.
            Parejkoj John Parejko added a comment -

            More details on the other surveys:

            • obs_sdss needs to generate the description files from the yanny files in etc/ at scons time.
            • obs_cfht could just use the persisted FITS files, but it's worth checking how feasible it is to scons-generate them from the amp info tables (e.g. what changes need to be made to the tables and/or generating code).
            • obs_decam should be able to generate the FITS files via makeDecamCameraRepository.py and the included DetectorLayoutFile (chipcenters.txt) and SegmentsFile (segmentfile.txt).
            • obs_subaru: the Camera.paf and Electronics.paf files do not exist in the repo. price: are you ok with the persisted FITS files being the sole source in this case, or can we get those two files? I can't see them in the repo history either.
            Parejkoj John Parejko added a comment - More details on the other surveys: obs_sdss needs to generate the description files from the yanny files in etc/ at scons time. obs_cfht could just use the persisted FITS files, but it's worth checking how feasible it is to scons-generate them from the amp info tables (e.g. what changes need to be made to the tables and/or generating code). obs_decam should be able to generate the FITS files via makeDecamCameraRepository.py and the included DetectorLayoutFile (chipcenters.txt) and SegmentsFile (segmentfile.txt). obs_subaru: the Camera.paf and Electronics.paf files do not exist in the repo. price : are you ok with the persisted FITS files being the sole source in this case, or can we get those two files? I can't see them in the repo history either.
            price Paul Price added a comment -

            I think we're happy with how obs_subaru is for now. The camera is quite stable at the moment, and I don't think we want to write a parser. If there is a generic parser available in the future, then maybe we could generate the FITS files at build time, but it's not necessary right now.

            price Paul Price added a comment - I think we're happy with how obs_subaru is for now. The camera is quite stable at the moment, and I don't think we want to write a parser. If there is a generic parser available in the future, then maybe we could generate the FITS files at build time, but it's not necessary right now.

            New question: should I unify where the camera description files live? There's several different directory structures.

            • decam lives in decam/cameraGeom
            • cfht lives in cfht/megacam
            • hsc, suprimecam live in hsc/camera and suprimecam/camera respectively
            • sdss and lsstSim lives in description/camera
            Parejkoj John Parejko added a comment - New question: should I unify where the camera description files live? There's several different directory structures. decam lives in decam/cameraGeom cfht lives in cfht/megacam hsc, suprimecam live in hsc/camera and suprimecam/camera respectively sdss and lsstSim lives in description/camera
            price Paul Price added a comment -

            To allow multiple cameras within the same obs package, if you're going to standardise (which doesn't appear to be necessary, but may be desirable), then I think it needs to live in a directory named after the camera.

            price Paul Price added a comment - To allow multiple cameras within the same obs package, if you're going to standardise (which doesn't appear to be necessary, but may be desirable), then I think it needs to live in a directory named after the camera.

            ... then I think it needs to live in a directory named after the camera.

            Yes, that's what I was thinking: use obs_BLAH/CAMERANAME/camera for everything (so obs_subaru doesn't have to change). Just would make it easier to find things.

            Parejkoj John Parejko added a comment - ... then I think it needs to live in a directory named after the camera. Yes, that's what I was thinking: use obs_BLAH/CAMERANAME/camera for everything (so obs_subaru doesn't have to change). Just would make it easier to find things.

            Parejkoj I looked the obs_lsstSim changes over. They generally seem fine. I am still wondering whether we need to tell people to just grab the calibration frames via globus. I think it's easier to just have people simulate the blank images since there appear to be no defects whatsoever. If there were defects we'd need the bias and flat frames to find them. Anyway, I trust your judgement. Feel free to procede.

            krughoff Simon Krughoff (Inactive) added a comment - Parejkoj I looked the obs_lsstSim changes over. They generally seem fine. I am still wondering whether we need to tell people to just grab the calibration frames via globus. I think it's easier to just have people simulate the blank images since there appear to be no defects whatsoever. If there were defects we'd need the bias and flat frames to find them. Anyway, I trust your judgement. Feel free to procede.

            krughoff Can you please checkout obs_lsstSim and try out this ticket branch with some higher-level tests? I'd like help checking whether I haven't broken anything, and you're well-placed to do so.

            Parejkoj John Parejko added a comment - krughoff Can you please checkout obs_lsstSim and try out this ticket branch with some higher-level tests? I'd like help checking whether I haven't broken anything, and you're well-placed to do so.

            Parejkoj I ran some twinkles data with the new obs_lsstSim and it worked fine. I'd say you are good to move ahead.

            krughoff Simon Krughoff (Inactive) added a comment - Parejkoj I ran some twinkles data with the new obs_lsstSim and it worked fine. I'd say you are good to move ahead.

            Back to "in progress" the partial review passed, and the design can go forward.

            Parejkoj John Parejko added a comment - Back to "in progress" the partial review passed, and the design can go forward.

            I have a yaml format that has successfully represented the ctio 0.9m and comCam, and I think will handle the full lsstCam. There's a ticket DM-11196 to move this to obs_base (although there is not yet any agreement that we will use it!).

            rhl Robert Lupton added a comment - I have a yaml format that has successfully represented the ctio 0.9m and comCam, and I think will handle the full lsstCam. There's a ticket DM-11196 to move this to obs_base (although there is not yet any agreement that we will use it!).

            Given the progress made in the last couple of years on YAMLCamera, obs_lsst, DM-11196, I think it's unlikely that we're going to do further work on this ticket. I'm marking this as “done” to reflect the (substantial!) work done on obs_lsstSim. Please reopen if you disagree.

            swinbank John Swinbank added a comment - Given the progress made in the last couple of years on YAMLCamera, obs_lsst, DM-11196 , I think it's unlikely that we're going to do further work on this ticket. I'm marking this as “done” to reflect the (substantial!) work done on obs_lsstSim. Please reopen if you disagree.

            People

              Parejkoj John Parejko
              price Paul Price
              Simon Krughoff (Inactive)
              James Chiang, John Parejko, John Swinbank, Paul Price, Robert Lupton, Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.