Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17381

ISR pipelineTask cannot create thumbnail files

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: daf_butler
    • Labels:
      None
    • Team:
      Data Release Production

      Description

      The gen3 butler does not support the writing of PNG files from numpy ndarrays, and this prevents the ISR thumbnails from be created.  The issue appears to be a lack of a valid formatter for this file type.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Is the intent of this ticket to write a formatter that will allow you to put numpy arrays into a butler repository as PNGs? You want to be able to retrieve them as well and retrieve the numpy array? How does gen2 handle this at the moment? What metadata is stored in the PNG?

            Show
            tjenness Tim Jenness added a comment - Is the intent of this ticket to write a formatter that will allow you to put numpy arrays into a butler repository as PNGs? You want to be able to retrieve them as well and retrieve the numpy array? How does gen2 handle this at the moment? What metadata is stored in the PNG?
            Hide
            czw Christopher Waters added a comment -

            Currently, numpy arrays are passed to afw.display.rgb.writeRGB(filename, array), with the filename calculated by the butler (see ip_isr/python/ip/isr/isrQa.py for the current code).  I don't believe these are ever reread in Gen2, and don't think adding that ability is useful.  I don't think the PNG contains any metadata, and uses the numpy array values directly as RGB pixel intensities.  

            Show
            czw Christopher Waters added a comment - Currently, numpy arrays are passed to afw.display.rgb.writeRGB(filename, array), with the filename calculated by the butler (see ip_isr/python/ip/isr/isrQa.py for the current code).  I don't believe these are ever reread in Gen2, and don't think adding that ability is useful.  I don't think the PNG contains any metadata, and uses the numpy array values directly as RGB pixel intensities.  
            Hide
            tjenness Tim Jenness added a comment -

            It looks suspiciously like the thumbnails aren't registered with the butler repository but use the butler solely to get a filename. Am I wrong? Gen 3 doesn't work like that and relies solely on a registered formatter to do the right thing since you aren't supposed to know where the data end up (it might not be local). The put of the thumbnail being a PNG, JPEG, or FITS file is entirely down to the butler configuration file.

            Show
            tjenness Tim Jenness added a comment - It looks suspiciously like the thumbnails aren't registered with the butler repository but use the butler solely to get a filename. Am I wrong? Gen 3 doesn't work like that and relies solely on a registered formatter to do the right thing since you aren't supposed to know where the data end up (it might not be local). The put of the thumbnail being a PNG, JPEG, or FITS file is entirely down to the butler configuration file.
            Hide
            czw Christopher Waters added a comment -

            Yes, that's my understanding as well.  In the ISR/Gen3/pipelineTask ticket (DM-16467), the thumbnails are being considered as output datatypes.  I'm happy to have the fix for this be a formatter that accepts the numpy array and stores it on disk in some alternate format (determined by the butler configuration), as long as it's available to view in the future.  I don't know that that "view" needs to reconstruct the initial numpy array, however.

            Show
            czw Christopher Waters added a comment - Yes, that's my understanding as well.  In the ISR/Gen3/pipelineTask ticket ( DM-16467 ), the thumbnails are being considered as output datatypes.  I'm happy to have the fix for this be a formatter that accepts the numpy array and stores it on disk in some alternate format (determined by the butler configuration), as long as it's available to view in the future.  I don't know that that "view" needs to reconstruct the initial numpy array, however.
            Hide
            jbosch Jim Bosch added a comment -

            I think we want to implement the put properly in Gen3 with a real formatter but not worry about get (we'll assume people will call getUri and then poke at files, assuming PosixDatastore, because in practice that's what they'll want to do).

            Show
            jbosch Jim Bosch added a comment - I think we want to implement the put properly in Gen3 with a real formatter but not worry about get (we'll assume people will call getUri and then poke at files, assuming PosixDatastore, because in practice that's what they'll want to do).

              People

              • Assignee:
                Unassigned
                Reporter:
                czw Christopher Waters
                Watchers:
                Christopher Waters, Jim Bosch, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel