Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7382

Table creation from hdu data fails without copying

    XMLWordPrintable

    Details

      Description

      This line in meas_algorithms fails under python 3:
      https://github.com/lsst/meas_algorithms/blob/3f92e4c18e582ae77eba88b0ee4b522d1298dff7/python/lsst/meas/algorithms/readFitsCatalogTask.py#L115

      with the error:

      Traceback (most recent call last):
        File "tests/testReadFitsCatalog.py", line 150, in testBadColumnName
          task.run(FitsPath)
        File "/Users/nate/repos_lsst/meas_algorithms/python/lsst/meas/algorithms/readFitsCatalogTask.py", line 115, in run
          table = Table(hdu.data, copy=False)
        File "/Users/nate/lsstsw/python3/lsstsw/miniconda/lib/python3.5/site-packages/astropy/table/table.py", line 360, in __init__
          init_func(data, names, dtype, n_cols, copy)
        File "/Users/nate/lsstsw/python3/lsstsw/miniconda/lib/python3.5/site-packages/astropy/table/table.py", line 643, in _init_from_ndarray
          newdata = data.view(dtype).ravel()
        File "/Users/nate/lsstsw/python3/lsstsw/miniconda/lib/python3.5/site-packages/numpy/core/records.py", line 480, in __setattr__
          raise exctype(value)
      ValueError: new type not compatible with array.
      

      as can be shown by running the readFitsCatalog unit test.

      The problem can be worked around by copying, but in the future this should be reported upstream, to see if the copying is a hard requirement due to type limitations or if it can be fixed so we can again create a table without copying.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Which version of astropy do you have installed?

            Show
            tjenness Tim Jenness added a comment - Which version of astropy do you have installed?
            Hide
            swinbank John Swinbank added a comment -

            It wasn't obvious to me on first reading, so to be explicit:

            On DM-7289, Nate Lust has (assuming it passes review, which it is undergoing at time of writing) modified readFitsCatalogTask.py, changing the code that was at line 115:

            table = Table(hdu.data, copy=False)
            

            to this

            # This Table creation fails with an astropy/numpy error complaining about incompatable types when
            # copy is set to False. Future work may want to get and upstream fix for this
            table = Table(hdu.data, copy=True)
            

            That means the unit test passes for now. As and when an upstream fix is available, this should be reverted.

            Show
            swinbank John Swinbank added a comment - It wasn't obvious to me on first reading, so to be explicit: On DM-7289 , Nate Lust has (assuming it passes review, which it is undergoing at time of writing) modified readFitsCatalogTask.py , changing the code that was at line 115: table = Table(hdu.data, copy=False) to this # This Table creation fails with an astropy/numpy error complaining about incompatable types when # copy is set to False. Future work may want to get and upstream fix for this table = Table(hdu.data, copy=True) That means the unit test passes for now. As and when an upstream fix is available, this should be reverted.
            Hide
            swinbank John Swinbank added a comment -

            This is an Astropy issue, in code we're no longer using. It might have been fixed upstream, but since it's no longer relevant for us I think we can't afford the time to go back and check.

            Show
            swinbank John Swinbank added a comment - This is an Astropy issue, in code we're no longer using. It might have been fixed upstream, but since it's no longer relevant for us I think we can't afford the time to go back and check.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              nlust Nate Lust
              Watchers:
              John Swinbank, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.