Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6349

Replace cameraGeom PAF files

    XMLWordPrintable

    Details

    • Story Points:
      10
    • Sprint:
      Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9, Alert Production F16 - 10
    • Team:
      Alert Production

      Description

      PAF files have long been deprecated, but continue to be used for describing the camera geometry. We need to replace the PAF cameraGeom files used for CFHT-MegaCam, DECam, LSSTSim and SDSS, and the scripts used to convert these files to FITS files for reading by the Mappers. They might be replaced by a configuration like YAML, or pure python.

        Attachments

          Issue Links

            Activity

            No builds found.
            price Paul Price created issue -
            price Paul Price made changes -
            Field Original Value New Value
            Link This issue is blocked by RFC-192 [ RFC-192 ]
            krughoff Simon Krughoff made changes -
            Team Alert Production [ 10300 ]
            Labels SciencePipelines
            Parejkoj John Parejko made changes -
            Assignee Simon Krughoff [ krughoff ] John Parejko [ parejkoj ]
            Hide
            Parejkoj John Parejko added a comment -

            Taking this on while I'm blocked by other things.

            Show
            Parejkoj John Parejko added a comment - Taking this on while I'm blocked by other things.
            Parejkoj John Parejko made changes -
            Epic Link DM-5691 [ 23683 ]
            Parejkoj John Parejko made changes -
            Sprint Alert Production F16 - 7 [ 236 ]
            Story Points 10
            Hide
            Parejkoj John Parejko added a comment -

            Notes from Simon Krughoff and my chat about this:

            Cameras that we need to deal with

            built exclusively in code

            • monocam
            • test

            Don't need to do anything: there is only one source, by definition. (But! check that the serialized files are not committed to the repos.)

            reads text files and generates FITS (and other) files
            lsstSim

            Delete persisted files, and ensure that makeLsstCameraRepository.py is run by scons and that it puts things where the butler expects them.

            parses PAF
            suprime
            hsc
            sdss

            Translate PAF to something else, then save that. Write a PAF to YAML mapper, serialize that, delete PAF, delete any FITS (etc.) persisted config, ensure scons runs FITS generator.

            Need to check these
            cfht
            decam

            If these use PAF, do the above. Otherwise, do something else.

            How to check that we got it right?

            • Butlerize old and new and do `for obj in dir(old): assert getattr(old, obj) == getattr(new, obj)` but that may not work since == isn't defined on all our SWIGed things.
            • Write the persistence formats and checksum/diff them (old vs. new).
            Show
            Parejkoj John Parejko added a comment - Notes from Simon Krughoff and my chat about this: Cameras that we need to deal with built exclusively in code monocam test Don't need to do anything: there is only one source, by definition. (But! check that the serialized files are not committed to the repos.) reads text files and generates FITS (and other) files lsstSim Delete persisted files, and ensure that makeLsstCameraRepository.py is run by scons and that it puts things where the butler expects them. parses PAF suprime hsc sdss Translate PAF to something else, then save that. Write a PAF to YAML mapper, serialize that, delete PAF, delete any FITS (etc.) persisted config, ensure scons runs FITS generator. Need to check these cfht decam If these use PAF, do the above. Otherwise, do something else. How to check that we got it right? Butlerize old and new and do `for obj in dir(old): assert getattr(old, obj) == getattr(new, obj)` but that may not work since == isn't defined on all our SWIGed things. Write the persistence formats and checksum/diff them (old vs. new).
            Hide
            price Paul Price added a comment -

            Suprime-Cam and HSC (both in obs_subaru) do not use PAF, but are using the persisted files (camera.py, 1 FITS file per CCD) as the primary source.

            Show
            price Paul Price added a comment - Suprime-Cam and HSC (both in obs_subaru) do not use PAF, but are using the persisted files (camera.py, 1 FITS file per CCD) as the primary source.
            krughoff Simon Krughoff made changes -
            Sprint Alert Production F16 - 7 [ 236 ] Alert Production F16 - 7, Alert Production F16 - 8 [ 236, 245 ]
            krughoff Simon Krughoff made changes -
            Rank Ranked higher
            Hide
            Parejkoj John Parejko added a comment -

            Do we need to do the same for the defects files, or can those just remain as FITS files?

            Show
            Parejkoj John Parejko added a comment - Do we need to do the same for the defects files, or can those just remain as FITS files?
            Hide
            price Paul Price added a comment -

            I think the guiding principle should be that there is only one authoritative source, and anything else is built from that source.

            In obs_subaru, the defects FITS files and registry are built by scons from text-based descriptions.

            Show
            price Paul Price added a comment - I think the guiding principle should be that there is only one authoritative source, and anything else is built from that source. In obs_subaru, the defects FITS files and registry are built by scons from text-based descriptions.
            Hide
            Parejkoj John Parejko added a comment - - edited

            Please review the obs_lsstSim commit. Note that you'll need sconsUtils DM-7179 in order to build with scons. The included tests pass, but its probably worth running with some other tests (suggestions are welcome), as there are some changes to the currently-committed camera descriptions, but I don't know if they matter or not.

            Show
            Parejkoj John Parejko added a comment - - edited Please review the obs_lsstSim commit. Note that you'll need sconsUtils DM-7179 in order to build with scons. The included tests pass, but its probably worth running with some other tests (suggestions are welcome), as there are some changes to the currently-committed camera descriptions, but I don't know if they matter or not.
            Parejkoj John Parejko made changes -
            Reviewers Simon Krughoff [ krughoff ]
            Status To Do [ 10001 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-7181 [ DM-7181 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-7179 [ DM-7179 ]
            Hide
            Parejkoj John Parejko added a comment -

            Confirming from my notes above:

            obs_monocam and obs_test do not contain any generated FITS files, so are already in compliance.

            Show
            Parejkoj John Parejko added a comment - Confirming from my notes above: obs_monocam and obs_test do not contain any generated FITS files, so are already in compliance.
            Hide
            Parejkoj John Parejko added a comment -

            More details on the other surveys:

            • obs_sdss needs to generate the description files from the yanny files in etc/ at scons time.
            • obs_cfht could just use the persisted FITS files, but it's worth checking how feasible it is to scons-generate them from the amp info tables (e.g. what changes need to be made to the tables and/or generating code).
            • obs_decam should be able to generate the FITS files via makeDecamCameraRepository.py and the included DetectorLayoutFile (chipcenters.txt) and SegmentsFile (segmentfile.txt).
            • obs_subaru: the Camera.paf and Electronics.paf files do not exist in the repo. Paul Price: are you ok with the persisted FITS files being the sole source in this case, or can we get those two files? I can't see them in the repo history either.
            Show
            Parejkoj John Parejko added a comment - More details on the other surveys: obs_sdss needs to generate the description files from the yanny files in etc/ at scons time. obs_cfht could just use the persisted FITS files, but it's worth checking how feasible it is to scons-generate them from the amp info tables (e.g. what changes need to be made to the tables and/or generating code). obs_decam should be able to generate the FITS files via makeDecamCameraRepository.py and the included DetectorLayoutFile (chipcenters.txt) and SegmentsFile (segmentfile.txt). obs_subaru: the Camera.paf and Electronics.paf files do not exist in the repo. Paul Price : are you ok with the persisted FITS files being the sole source in this case, or can we get those two files? I can't see them in the repo history either.
            Hide
            price Paul Price added a comment -

            I think we're happy with how obs_subaru is for now. The camera is quite stable at the moment, and I don't think we want to write a parser. If there is a generic parser available in the future, then maybe we could generate the FITS files at build time, but it's not necessary right now.

            Show
            price Paul Price added a comment - I think we're happy with how obs_subaru is for now. The camera is quite stable at the moment, and I don't think we want to write a parser. If there is a generic parser available in the future, then maybe we could generate the FITS files at build time, but it's not necessary right now.
            Hide
            Parejkoj John Parejko added a comment -

            New question: should I unify where the camera description files live? There's several different directory structures.

            • decam lives in decam/cameraGeom
            • cfht lives in cfht/megacam
            • hsc, suprimecam live in hsc/camera and suprimecam/camera respectively
            • sdss and lsstSim lives in description/camera
            Show
            Parejkoj John Parejko added a comment - New question: should I unify where the camera description files live? There's several different directory structures. decam lives in decam/cameraGeom cfht lives in cfht/megacam hsc, suprimecam live in hsc/camera and suprimecam/camera respectively sdss and lsstSim lives in description/camera
            Hide
            price Paul Price added a comment -

            To allow multiple cameras within the same obs package, if you're going to standardise (which doesn't appear to be necessary, but may be desirable), then I think it needs to live in a directory named after the camera.

            Show
            price Paul Price added a comment - To allow multiple cameras within the same obs package, if you're going to standardise (which doesn't appear to be necessary, but may be desirable), then I think it needs to live in a directory named after the camera.
            Hide
            Parejkoj John Parejko added a comment -

            ... then I think it needs to live in a directory named after the camera.

            Yes, that's what I was thinking: use obs_BLAH/CAMERANAME/camera for everything (so obs_subaru doesn't have to change). Just would make it easier to find things.

            Show
            Parejkoj John Parejko added a comment - ... then I think it needs to live in a directory named after the camera. Yes, that's what I was thinking: use obs_BLAH/CAMERANAME/camera for everything (so obs_subaru doesn't have to change). Just would make it easier to find things.
            krughoff Simon Krughoff made changes -
            Epic Link DM-5691 [ 23683 ] DM-7362 [ 26448 ]
            Hide
            krughoff Simon Krughoff added a comment -

            John Parejko I looked the obs_lsstSim changes over. They generally seem fine. I am still wondering whether we need to tell people to just grab the calibration frames via globus. I think it's easier to just have people simulate the blank images since there appear to be no defects whatsoever. If there were defects we'd need the bias and flat frames to find them. Anyway, I trust your judgement. Feel free to procede.

            Show
            krughoff Simon Krughoff added a comment - John Parejko I looked the obs_lsstSim changes over. They generally seem fine. I am still wondering whether we need to tell people to just grab the calibration frames via globus. I think it's easier to just have people simulate the blank images since there appear to be no defects whatsoever. If there were defects we'd need the bias and flat frames to find them. Anyway, I trust your judgement. Feel free to procede.
            Hide
            Parejkoj John Parejko added a comment -

            Simon Krughoff Can you please checkout obs_lsstSim and try out this ticket branch with some higher-level tests? I'd like help checking whether I haven't broken anything, and you're well-placed to do so.

            Show
            Parejkoj John Parejko added a comment - Simon Krughoff Can you please checkout obs_lsstSim and try out this ticket branch with some higher-level tests? I'd like help checking whether I haven't broken anything, and you're well-placed to do so.
            krughoff Simon Krughoff made changes -
            Sprint Alert Production F16 - 7, Alert Production F16 - 8 [ 236, 245 ] Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9 [ 236, 245, 247 ]
            krughoff Simon Krughoff made changes -
            Rank Ranked higher
            Hide
            krughoff Simon Krughoff added a comment -

            John Parejko I ran some twinkles data with the new obs_lsstSim and it worked fine. I'd say you are good to move ahead.

            Show
            krughoff Simon Krughoff added a comment - John Parejko I ran some twinkles data with the new obs_lsstSim and it worked fine. I'd say you are good to move ahead.
            krughoff Simon Krughoff made changes -
            Sprint Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9 [ 236, 245, 247 ] Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9, Alert Production F16 - 10 [ 236, 245, 247, 284 ]
            krughoff Simon Krughoff made changes -
            Rank Ranked higher
            krughoff Simon Krughoff made changes -
            Sprint Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9, Alert Production F16 - 10 [ 236, 245, 247, 284 ] Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9, Alert Production F16 - 10, Alert Production F16 - 11 [ 236, 245, 247, 284, 289 ]
            Parejkoj John Parejko made changes -
            Sprint Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9, Alert Production F16 - 10, Alert Production F16 - 11 [ 236, 245, 247, 284, 289 ] Alert Production F16 - 7, Alert Production F16 - 8, Alert Production F16 - 9, Alert Production F16 - 10 [ 236, 245, 247, 284 ]
            Parejkoj John Parejko made changes -
            Rank Ranked lower
            Hide
            Parejkoj John Parejko added a comment -

            Back to "in progress" the partial review passed, and the design can go forward.

            Show
            Parejkoj John Parejko added a comment - Back to "in progress" the partial review passed, and the design can go forward.
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] In Progress [ 3 ]
            krughoff Simon Krughoff made changes -
            Epic Link DM-7362 [ 26448 ] DM-8472 [ 28104 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggered by RFC-192 [ RFC-192 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by RFC-192 [ RFC-192 ]
            krughoff Simon Krughoff made changes -
            Epic Link DM-8472 [ 28104 ] DM-9680 [ 30785 ]
            Hide
            rhl Robert Lupton added a comment -

            I have a yaml format that has successfully represented the ctio 0.9m and comCam, and I think will handle the full lsstCam. There's a ticket DM-11196 to move this to obs_base (although there is not yet any agreement that we will use it!).

            Show
            rhl Robert Lupton added a comment - I have a yaml format that has successfully represented the ctio 0.9m and comCam, and I think will handle the full lsstCam. There's a ticket DM-11196 to move this to obs_base (although there is not yet any agreement that we will use it!).
            swinbank John Swinbank made changes -
            Epic Link DM-9680 [ 30785 ] DM-10068 [ 31628 ]
            swinbank John Swinbank made changes -
            Epic Link DM-10068 [ 31628 ] DM-11798 [ 34281 ]
            swinbank John Swinbank made changes -
            Epic Link DM-11798 [ 34281 ] DM-12728 [ 36327 ]
            swinbank John Swinbank made changes -
            Epic Link DM-12728 [ 36327 ] DM-14447 [ 80385 ]
            swinbank John Swinbank made changes -
            Epic Link DM-14447 [ 80385 ] DM-16722 [ 235355 ]
            frossie Frossie Economou made changes -
            Status Admin Review [ 3 ] In Progress [ 11605 ]
            frossie Frossie Economou made changes -
            Status Review [ 11605 ] In Progress [ 3 ]
            Hide
            swinbank John Swinbank added a comment -

            Given the progress made in the last couple of years on YAMLCamera, obs_lsst, DM-11196, I think it's unlikely that we're going to do further work on this ticket. I'm marking this as “done” to reflect the (substantial!) work done on obs_lsstSim. Please reopen if you disagree.

            Show
            swinbank John Swinbank added a comment - Given the progress made in the last couple of years on YAMLCamera, obs_lsst, DM-11196 , I think it's unlikely that we're going to do further work on this ticket. I'm marking this as “done” to reflect the (substantial!) work done on obs_lsstSim. Please reopen if you disagree.
            swinbank John Swinbank made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              price Paul Price
              Reviewers:
              Simon Krughoff
              Watchers:
              James Chiang, John Parejko, John Swinbank, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.