# Change interpretation of HDU for fits I/O routines

XMLWordPrintable

#### Details

• Type: RFC
• Status: Implemented
• Resolution: Done
• Component/s:
• Labels:
None

#### Description

Long long ago we decided to use 1-based indexing for HDUs in FITS files, based on the long-time Fortran legacy of FITS files (some say that the F stands for Fortran; they are only technically wrong), and partly on a reading of the cfitsio manual (details at the end of this RFC).

pyfits (and presumably its astropy.io successor), on the other hand, uses 0-indexing which is what the world expects these days.

This RFC proposes that we switch to using 0-indexing. The current code special cases 0 to mean, "the first non-empty HDU" – we'll have to change this default to e.g. -1 as part of this RFC. This change, too, will reduce confusion.

It'll be a minor change to the obs packages that handle LSST-style MEF data (we'll need to do this anyway as part of this summer's obs package cleanup), and possibly to a few other places in the stack but I'd be surprised if it's many – the butler mostly insulates us from this sort of worry.

We need to go into this code anyway, as the special-case code that skips empty PDUs when reading images is applied to reading metadata, which makes it impossible to read LSST test stand headers (well, without falling back on astropy.io which is what obs_comCam does). See DM-9854.

cfitsio details:

The cfitsio manual says the first HDU is 1 in section 4.3 (https://heasarc.gsfc.nasa.gov/docs/software/fitsio/quick/node8.html)

The first routine moves to the specified absolute HDU number in the FITS file (the first HDU = 1)

but elsewhere they say that cfitsio is 0 indexed e.g. section 10.7
(https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node93.html)

The HDU may be specified either by absolute position number, starting with 0 for the primary array,

Experiment reveals that they are in fact 0-indexed (my guess is that the author of 4.3 was assuming a separate PDU).

#### Activity

Hide
Simon Krughoff added a comment -

Robert Lupton phosim produces a file per amp. As far as I know, our galsim system doesn't produce amp level images at all yet. I think, by default they will do what phosim does so we can reduce both simulations with the same obs package, but we could probably inform that decision.

Show
Simon Krughoff added a comment - Robert Lupton phosim produces a file per amp. As far as I know, our galsim system doesn't produce amp level images at all yet. I think, by default they will do what phosim does so we can reduce both simulations with the same obs package, but we could probably inform that decision.
Hide
Tim Jenness added a comment -

All triggered work has been completed. Should this RFC be marked as implemented now?

Show
Tim Jenness added a comment - All triggered work has been completed. Should this RFC be marked as implemented now?
Hide
Robert Lupton added a comment -

Yes; but I don't know how. Should there be a button along with Reopen RFC' etc.? Or does, "We shipped it" mean "mark as implemented"?

Show
Robert Lupton added a comment - Yes; but I don't know how. Should there be a button along with Reopen RFC' etc.? Or does, "We shipped it" mean "mark as implemented"?
Hide
John Swinbank added a comment -

Or does, "We shipped it" mean "mark as implemented"?

It does. It's even documented. It's a shame the button doesn't have a more obvious label, though.

Anyway, to short circuit further discussion, I will now press it...

Show
John Swinbank added a comment - Or does, "We shipped it" mean "mark as implemented"? It does. It's even documented . It's a shame the button doesn't have a more obvious label, though. Anyway, to short circuit further discussion, I will now press it...
Hide
Robert Lupton added a comment -

It is implicit in this discussion, but not explicit: We now use INT_MIN to mean "The first non-empty HDU"

Show
Robert Lupton added a comment - It is implicit in this discussion, but not explicit: We now use INT_MIN to mean "The first non-empty HDU"

#### People

Assignee:
Robert Lupton
Reporter:
Robert Lupton
Watchers:
David Shupe, Gregory Dubois-Felsmann, John Parejko, John Swinbank, Robert Lupton, Simon Krughoff, Tim Jenness, Xiuqin Wu [X] (Inactive)