# Replace cameraGeom PAF files

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
10
• Sprint:
• Team:

#### Description

PAF files have long been deprecated, but continue to be used for describing the camera geometry. We need to replace the PAF cameraGeom files used for CFHT-MegaCam, DECam, LSSTSim and SDSS, and the scripts used to convert these files to FITS files for reading by the Mappers. They might be replaced by a configuration like YAML, or pure python.

#### Activity

Hide
John Parejko added a comment -

Taking this on while I'm blocked by other things.

Show
John Parejko added a comment - Taking this on while I'm blocked by other things.
Hide
John Parejko added a comment -

Cameras that we need to deal with

built exclusively in code

• monocam
• test

Don't need to do anything: there is only one source, by definition. (But! check that the serialized files are not committed to the repos.)

reads text files and generates FITS (and other) files
lsstSim

Delete persisted files, and ensure that makeLsstCameraRepository.py is run by scons and that it puts things where the butler expects them.

parses PAF
suprime
hsc
sdss

Translate PAF to something else, then save that. Write a PAF to YAML mapper, serialize that, delete PAF, delete any FITS (etc.) persisted config, ensure scons runs FITS generator.

Need to check these
cfht
decam

If these use PAF, do the above. Otherwise, do something else.

How to check that we got it right?

• Butlerize old and new and do for obj in dir(old): assert getattr(old, obj) == getattr(new, obj) but that may not work since == isn't defined on all our SWIGed things.
• Write the persistence formats and checksum/diff them (old vs. new).
Show
John Parejko added a comment - Notes from Simon Krughoff and my chat about this: Cameras that we need to deal with built exclusively in code monocam test Don't need to do anything: there is only one source, by definition. (But! check that the serialized files are not committed to the repos.) reads text files and generates FITS (and other) files lsstSim Delete persisted files, and ensure that makeLsstCameraRepository.py is run by scons and that it puts things where the butler expects them. parses PAF suprime hsc sdss Translate PAF to something else, then save that. Write a PAF to YAML mapper, serialize that, delete PAF, delete any FITS (etc.) persisted config, ensure scons runs FITS generator. Need to check these cfht decam If these use PAF, do the above. Otherwise, do something else. How to check that we got it right? Butlerize old and new and do for obj in dir(old): assert getattr(old, obj) == getattr(new, obj) but that may not work since == isn't defined on all our SWIGed things. Write the persistence formats and checksum/diff them (old vs. new).
Hide
Paul Price added a comment -

Suprime-Cam and HSC (both in obs_subaru) do not use PAF, but are using the persisted files (camera.py, 1 FITS file per CCD) as the primary source.

Show
Paul Price added a comment - Suprime-Cam and HSC (both in obs_subaru) do not use PAF, but are using the persisted files (camera.py, 1 FITS file per CCD) as the primary source.
Hide
John Parejko added a comment -

Do we need to do the same for the defects files, or can those just remain as FITS files?

Show
John Parejko added a comment - Do we need to do the same for the defects files, or can those just remain as FITS files?
Hide
Paul Price added a comment -

I think the guiding principle should be that there is only one authoritative source, and anything else is built from that source.

In obs_subaru, the defects FITS files and registry are built by scons from text-based descriptions.

Show
Paul Price added a comment - I think the guiding principle should be that there is only one authoritative source, and anything else is built from that source. In obs_subaru, the defects FITS files and registry are built by scons from text-based descriptions.
Hide
John Parejko added a comment - - edited

Please review the obs_lsstSim commit. Note that you'll need sconsUtils DM-7179 in order to build with scons. The included tests pass, but its probably worth running with some other tests (suggestions are welcome), as there are some changes to the currently-committed camera descriptions, but I don't know if they matter or not.

Show
John Parejko added a comment - - edited Please review the obs_lsstSim commit. Note that you'll need sconsUtils DM-7179 in order to build with scons. The included tests pass, but its probably worth running with some other tests (suggestions are welcome), as there are some changes to the currently-committed camera descriptions, but I don't know if they matter or not.
Hide
John Parejko added a comment -

Confirming from my notes above:

obs_monocam and obs_test do not contain any generated FITS files, so are already in compliance.

Show
John Parejko added a comment - Confirming from my notes above: obs_monocam and obs_test do not contain any generated FITS files, so are already in compliance.
Hide
John Parejko added a comment -

More details on the other surveys:

• obs_sdss needs to generate the description files from the yanny files in etc/ at scons time.
• obs_cfht could just use the persisted FITS files, but it's worth checking how feasible it is to scons-generate them from the amp info tables (e.g. what changes need to be made to the tables and/or generating code).
• obs_decam should be able to generate the FITS files via makeDecamCameraRepository.py and the included DetectorLayoutFile (chipcenters.txt) and SegmentsFile (segmentfile.txt).
• obs_subaru: the Camera.paf and Electronics.paf files do not exist in the repo. Paul Price: are you ok with the persisted FITS files being the sole source in this case, or can we get those two files? I can't see them in the repo history either.
Show
John Parejko added a comment - More details on the other surveys: obs_sdss needs to generate the description files from the yanny files in etc/ at scons time. obs_cfht could just use the persisted FITS files, but it's worth checking how feasible it is to scons-generate them from the amp info tables (e.g. what changes need to be made to the tables and/or generating code). obs_decam should be able to generate the FITS files via makeDecamCameraRepository.py and the included DetectorLayoutFile (chipcenters.txt) and SegmentsFile (segmentfile.txt). obs_subaru: the Camera.paf and Electronics.paf files do not exist in the repo. Paul Price : are you ok with the persisted FITS files being the sole source in this case, or can we get those two files? I can't see them in the repo history either.
Hide
Paul Price added a comment -

I think we're happy with how obs_subaru is for now. The camera is quite stable at the moment, and I don't think we want to write a parser. If there is a generic parser available in the future, then maybe we could generate the FITS files at build time, but it's not necessary right now.

Show
Paul Price added a comment - I think we're happy with how obs_subaru is for now. The camera is quite stable at the moment, and I don't think we want to write a parser. If there is a generic parser available in the future, then maybe we could generate the FITS files at build time, but it's not necessary right now.
Hide
John Parejko added a comment -

New question: should I unify where the camera description files live? There's several different directory structures.

• decam lives in decam/cameraGeom
• cfht lives in cfht/megacam
• hsc, suprimecam live in hsc/camera and suprimecam/camera respectively
• sdss and lsstSim lives in description/camera
Show
John Parejko added a comment - New question: should I unify where the camera description files live? There's several different directory structures. decam lives in decam/cameraGeom cfht lives in cfht/megacam hsc, suprimecam live in hsc/camera and suprimecam/camera respectively sdss and lsstSim lives in description/camera
Hide
Paul Price added a comment -

To allow multiple cameras within the same obs package, if you're going to standardise (which doesn't appear to be necessary, but may be desirable), then I think it needs to live in a directory named after the camera.

Show
Paul Price added a comment - To allow multiple cameras within the same obs package, if you're going to standardise (which doesn't appear to be necessary, but may be desirable), then I think it needs to live in a directory named after the camera.
Hide
John Parejko added a comment -

... then I think it needs to live in a directory named after the camera.

Yes, that's what I was thinking: use obs_BLAH/CAMERANAME/camera for everything (so obs_subaru doesn't have to change). Just would make it easier to find things.

Show
John Parejko added a comment - ... then I think it needs to live in a directory named after the camera. Yes, that's what I was thinking: use obs_BLAH/CAMERANAME/camera for everything (so obs_subaru doesn't have to change). Just would make it easier to find things.
Hide
Simon Krughoff added a comment -

John Parejko I looked the obs_lsstSim changes over. They generally seem fine. I am still wondering whether we need to tell people to just grab the calibration frames via globus. I think it's easier to just have people simulate the blank images since there appear to be no defects whatsoever. If there were defects we'd need the bias and flat frames to find them. Anyway, I trust your judgement. Feel free to procede.

Show
Simon Krughoff added a comment - John Parejko I looked the obs_lsstSim changes over. They generally seem fine. I am still wondering whether we need to tell people to just grab the calibration frames via globus. I think it's easier to just have people simulate the blank images since there appear to be no defects whatsoever. If there were defects we'd need the bias and flat frames to find them. Anyway, I trust your judgement. Feel free to procede.
Hide
John Parejko added a comment -

Simon Krughoff Can you please checkout obs_lsstSim and try out this ticket branch with some higher-level tests? I'd like help checking whether I haven't broken anything, and you're well-placed to do so.

Show
John Parejko added a comment - Simon Krughoff Can you please checkout obs_lsstSim and try out this ticket branch with some higher-level tests? I'd like help checking whether I haven't broken anything, and you're well-placed to do so.
Hide
Simon Krughoff added a comment -

John Parejko I ran some twinkles data with the new obs_lsstSim and it worked fine. I'd say you are good to move ahead.

Show
Simon Krughoff added a comment - John Parejko I ran some twinkles data with the new obs_lsstSim and it worked fine. I'd say you are good to move ahead.
Hide
John Parejko added a comment -

Back to "in progress" the partial review passed, and the design can go forward.

Show
John Parejko added a comment - Back to "in progress" the partial review passed, and the design can go forward.
Hide
Robert Lupton added a comment -

I have a yaml format that has successfully represented the ctio 0.9m and comCam, and I think will handle the full lsstCam. There's a ticket DM-11196 to move this to obs_base (although there is not yet any agreement that we will use it!).

Show
Robert Lupton added a comment - I have a yaml format that has successfully represented the ctio 0.9m and comCam, and I think will handle the full lsstCam. There's a ticket DM-11196 to move this to obs_base (although there is not yet any agreement that we will use it!).
Hide
John Swinbank added a comment -

Given the progress made in the last couple of years on YAMLCamera, obs_lsst, DM-11196, I think it's unlikely that we're going to do further work on this ticket. I'm marking this as “done” to reflect the (substantial!) work done on obs_lsstSim. Please reopen if you disagree.

Show
John Swinbank added a comment - Given the progress made in the last couple of years on YAMLCamera, obs_lsst, DM-11196 , I think it's unlikely that we're going to do further work on this ticket. I'm marking this as “done” to reflect the (substantial!) work done on obs_lsstSim. Please reopen if you disagree.

#### People

Assignee:
John Parejko
Reporter:
Paul Price
Reviewers:
Simon Krughoff
Watchers:
James Chiang, John Parejko, John Swinbank, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness