Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20018

Functors expect multi-level ParquetTable; correct to work for single-level

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      0.5
    • Epic Link:
    • Team:
      Data Release Production

      Description

      Starting to set up a demo notebook for Quansight, I realize that I can't call a simple functor (e.g., a labeler) on a single-level parquet table (e.g., a analysisCoaddTable_forced):

      ---------------------------------------------------------------------------
      		 
      TypeError                                 Traceback (most recent call last)
      		 
      <ipython-input-19-2b4d44d689db> in <module>
      		 
            1 labeller = StarGalaxyLabeller()
      		 
      ----> 2 label = labeller(coaddParq)
      		 
       
      		 
      ~/qa_explorer/python/lsst/qa/explorer/functors.py in __call__(self, parq, dropna, **kwargs)
      		 
          671 
          672     def __call__(self, parq, dropna=False, **kwargs):
      		 
      --> 673         return super().__call__(parq, dropna=False, **kwargs)
      		 
          674 
          675 
       
      		 
      ~/qa_explorer/python/lsst/qa/explorer/functors.py in __call__(self, parq, dropna)
      		 
          140 
          141     def __call__(self, parq, dropna=False):
      		 
      --> 142         df = self._get_cols(parq)
      		 
          143 
          144         vals = self._func(df)
      		 
       
      		 
      ~/qa_explorer/python/lsst/qa/explorer/functors.py in _get_cols(self, parq)
      		 
          127             columns = self.columns
      		 
          128 
      --> 129         df = parq.toDataFrame(columns=columns, droplevels=False)
      		 
          130         df = self._setLevels(df)
      		 
          131         return df
      		 
       
      		 
      TypeError: toDataFrame() got an unexpected keyword argument 'droplevels'

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-20015 Get Quansight developers up to speed with test dataset & LSST stack

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tmorton Tim Morton [X] (Inactive) added a comment -

            OK, the following now works:

            from lsst.daf.persistence import Butler
            		 
            from lsst.qa.explorer.functors import StarGalaxyLabeller, Magnitude 
             
            		 
            butler = Butler('/project/tmorton/tickets/DM-19684')
            		 
             
            		 
            tract = 9813
            		 
            filt = 'HSC-I'
            		 
            coaddParq = butler.get('analysisCoaddTable_forced', tract=tract, filter=filt)
            		 
             
            		 
             
            		 
            labeller = StarGalaxyLabeller()
            		 
             
            		 
            labels = labeller(coaddParq)
            		 
             
            		 
            labels.describe()

            Gives the output:

            count     1615018
            		 
            unique          3
            		 
            top        galaxy
            		 
            freq      1175532
            		 
            Name: label, dtype: object

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - OK, the following now works: from lsst.daf.persistence import Butler from lsst.qa.explorer.functors import StarGalaxyLabeller, Magnitude   butler = Butler( '/project/tmorton/tickets/DM-19684' )   tract = 9813 filt = 'HSC-I' coaddParq = butler.get( 'analysisCoaddTable_forced' , tract = tract, filter = filt)     labeller = StarGalaxyLabeller()   labels = labeller(coaddParq)   labels.describe() Gives the output: count 1615018 unique 3 top galaxy freq 1175532 Name: label, dtype: object
            Hide
            Permalink
            tmorton Tim Morton [X] (Inactive) added a comment -

            Oh shoot Yusra AlSayyad, I just jumped the gun on merging this; I saw the github review comments and assumed the review was complete.  Hopefully all is OK; I will update tonight if need be.

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - Oh shoot Yusra AlSayyad , I just jumped the gun on merging this; I saw the github review comments and assumed the review was complete.  Hopefully all is OK; I will update tonight if need be.

              People

              Assignee:
              tmorton Tim Morton [X] (Inactive)
              Reporter:
              tmorton Tim Morton [X] (Inactive)
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.