Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21263

Implement packaging of spectrograph output

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Sprint:
      TSSW Sprint - Sep 2 - Sep 14, TSSW Sprint - Sep 16 - Sep 28, TSSW Sprint - Sep 29 - Oct 13, TSSW Sprint - Oct 14 - Oct 27
    • Team:
      Telescope and Site

      Description

      The output of the spectrograph is currently going to be stored in the Large File Annex (LFA) as FITS files. The CSC needs to be able to push data to the LFA after it is read out and packaged: this ticket is for the work to manage that communication.

      My goal here is to make this flexible enough that it is easy to swap it out if we decide we want to use some other storage method (e.g. direct to database).

      Where do I find documentation about how to communicate with the (an?) LFA? How should we handle testing of that communication? I can obviously stand up a mock as I'm doing for the spectrograph itself, but it would be good to have a "real" system I can talk to for validation.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Given the above discussion, are you willing to mark this ticket and the PR `Reviewed`, Tim Jenness?

            Show
            Parejkoj John Parejko added a comment - Given the above discussion, are you willing to mark this ticket and the PR `Reviewed`, Tim Jenness ?
            Hide
            tjenness Tim Jenness added a comment -

            I am happy with the implementation and I will do the updated WCS storage in DM-21797

            Show
            tjenness Tim Jenness added a comment - I am happy with the implementation and I will do the updated WCS storage in DM-21797
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            John Parejko Is the data array (HDU 0 in your example above) one-dimensional, or is it two-dimensional with the list of wavelengths applying to one of the two axes (i.e., the dispersion direction in a chip image)?

            Show
            gpdf Gregory Dubois-Felsmann added a comment - John Parejko Is the data array (HDU 0 in your example above) one-dimensional, or is it two-dimensional with the list of wavelengths applying to one of the two axes (i.e., the dispersion direction in a chip image)?
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            I see that https://github.com/spacetelescope/jwst/issues/2235 discusses the implementation of astropy support for WAVE-TAB via GWCS.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - I see that https://github.com/spacetelescope/jwst/issues/2235 discusses the implementation of astropy support for WAVE-TAB via GWCS.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment - - edited

            John Parejko I believe that the way to just get the two arrays from a file in WAVE-TAB format would look something like this:

            hdulist = astropy.io.fits.open(filename)
            instFlux = hdulist[0].data
            wavelength = hdulist[1].data.field('WaveCoord')[0]
            

            (You are extracting the array-valued data in the 'WaveCoord' column in the 1st and only row of a one-row binary table.)

            This is what you would do if you were in control of how the file was written and you knew for sure that the binary table was in the first extension and that the column name for the wavelength list was 'WaveCoord'. If you wanted to write more bulletproof FITS-standard-compliant code, you could look up the extension name and column name; it might look like this:

            hdulist = astropy.io.fits.open(filename)
            instFlux = hdulist[0].data
            assert hdulist[0].header['NAXIS']==1
            assert hdulist[0].header['CTYPE1']=='WAVE-TAB'
            waveext = hdulist[0].header['PS1_0']
            wavecol = hdulist[0].header['PS1_1']
            wavelength = hdulist[waveext].data.field(wavecol)[0]
            

            Show
            gpdf Gregory Dubois-Felsmann added a comment - - edited John Parejko I believe that the way to just get the two arrays from a file in WAVE-TAB format would look something like this: hdulist = astropy.io.fits. open (filename) instFlux = hdulist[ 0 ].data wavelength = hdulist[ 1 ].data.field( 'WaveCoord' )[ 0 ] (You are extracting the array-valued data in the 'WaveCoord' column in the 1st and only row of a one-row binary table.) This is what you would do if you were in control of how the file was written and you knew for sure that the binary table was in the first extension and that the column name for the wavelength list was 'WaveCoord'. If you wanted to write more bulletproof FITS-standard-compliant code, you could look up the extension name and column name; it might look like this: hdulist = astropy.io.fits. open (filename) instFlux = hdulist[ 0 ].data assert hdulist[ 0 ].header[ 'NAXIS' ] = = 1 assert hdulist[ 0 ].header[ 'CTYPE1' ] = = 'WAVE-TAB' waveext = hdulist[ 0 ].header[ 'PS1_0' ] wavecol = hdulist[ 0 ].header[ 'PS1_1' ] wavelength = hdulist[waveext].data.field(wavecol)[ 0 ]

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Clements, Gregory Dubois-Felsmann, John Parejko, Merlin Fisher-Levine, Patrick Ingraham, Russell Owen, Tiago Ribeiro, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.