Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13390

Consider developing the ability to read numpy .npy/.npz files

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Firefly, SUIT
    • Labels:
    • Team:
      Science User Interface

      Description

      In a Python data analysis environment, the numpy library is of central importance. It has a native persistence form for numeric arrays: .npy files (compressed .npz). It would be useful to have the ability to read, and perhaps write, files in this format.

      https://docs.scipy.org/doc/numpy/neps/npy-format.html

      This could be done in Java code, as the file format is documented (though some of it may be difficult to reproduce exactly in Java), or via a Python bridge as we will do for the native LSST datatypes.

      This ticket at the moment is only intended to stick a pin in the idea, not to mandate its implementation. It may be that we should focus our attention on the cross-language Apache Arrow project instead.

        Attachments

          Activity

          Hide
          gpdf Gregory Dubois-Felsmann added a comment -

          I saw this was assigned to Cindy Wang [X]; at the moment I wouldn't go beyond just Googling to see if there is a decent Java library available.

          We need to make a strategic decision about file format support before doing any real development.

          Show
          gpdf Gregory Dubois-Felsmann added a comment - I saw this was assigned to Cindy Wang [X] ; at the moment I wouldn't go beyond just Googling to see if there is a decent Java library available. We need to make a strategic decision about file format support before doing any real development.
          Hide
          gpdf Gregory Dubois-Felsmann added a comment -

          Closing this as "Won't Fix".  This is pretty clearly not worth the effort.  Parquet support in Firefly would be far more useful in the contemporary data science landscape.

          Show
          gpdf Gregory Dubois-Felsmann added a comment - Closing this as "Won't Fix".  This is pretty clearly not worth the effort.  Parquet support in Firefly would be far more useful in the contemporary data science landscape.
          Hide
          gpdf Gregory Dubois-Felsmann added a comment -

          Parquet support in Firefly is requested in IPAC ticket FIREFLY-538.

          Show
          gpdf Gregory Dubois-Felsmann added a comment - Parquet support in Firefly is requested in IPAC ticket FIREFLY-538.

            People

            Assignee:
            gpdf Gregory Dubois-Felsmann
            Reporter:
            gpdf Gregory Dubois-Felsmann
            Watchers:
            David Shupe, Gregory Dubois-Felsmann, Trey Roby, Xiuqin Wu [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.