Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-807

Modify column names in Source/Object DRP Parquet Tables



    • RFC
    • Status: Implemented
    • Resolution: Done
    • DM
    • None


      We are making a column name consistency push for w_2021_40 with an eye on what users will see in the Parquet Tables with DP0.2. There is a prototype implementation on ticket DM-31825.

      Two major users:

      • DAX for ingest into qserv and
      • science users loading columns into notebooks for analysis.

      Tables in use currently are Source and Object which require an RFC.
      sourceTable_visit in particular is input to jointcal and FGCM which will require modifications.

      In your comments, save your fingers from typing: "I think everything should be consistent." We all agree on that! What is under debate here is, where names are not
      consistent, which should yield. What is on offer here is an iteration of of edits to our Parquet specification yaml files,
      transform* tasks, any tasks in pipe_tasks/postprocess.py. I am not offering: to edit all our
      afw_table source catalogs and plugin names and fields, to change implementation on how fields (such as ccdVisitId) are computed.



      • Start all columns lowercase
      • ixxPsf -> ixxPSF (and friends) to match DIASourceTable specification.


      • Galaxy fluxes:
        • remove ugStd, grStd, riStd, izStd, zyStd. We've already argued that nanojansky fluxes are the way to go. Magdiff cols have redundant and less information than the model fluxes.
        • direct users to use `g_CModelFlux`/`g_GaapFlux_1_0` or use a signifier alias that folks will look for: `g_ModelFlux`.
      • Add an underscore between band and column name, and if the column is in Source or ForcedSource, make it start lower case.
        e.g. (g_ra, g_psfFlux) instead of current (gRa, gPsFlux). Benefit is that you can reuse your code snippets between ForcedSource/Source and Object tables by just prepending `g_` instead of messing around with capitals.
      • Remove all apCorr columns.
      • Add more GaaP apertures (Arun responsible for ticket)
      • Add more aperture flux apertures
      • There are a lot of other changes that are not consistency/naming issues, but content issues and are not enumerated here


      • psFlux --> psfFlux. Everyone is in the habit of calling it psfFlux. Multifit is dead. Let's not fight expectation here.
      • One boolean column per flag in all Parquet tables (Leave bitpacked flags in the APDB). 
      • Set the DataFrame index to the primary key which is not replicated as a column.
        e.g. to get objectId from the Object table, you object.index rather than object['objectId'] unless you do object.reset_index() first
      • Replace filterName with band and physicalFilter.
      • (visit, detector, tract, patch`) instead of ({{visitId, detectorId, tractId, patchId)
      • Do not include base_PixelFlags_flag. No one knows what this is, and I've even seen it trick savvy pipelines folks.
        Spoiler alert: it is True if the plugin that sets the PixelFlags fails. Too meta


      Major changes not in the basic proposal because they would be tougher to get done in a week, but I'd push for if I hear overwhelming enthusiasm:

      • Proposal to rename the CcdVisit table to DetectorVisit with a primary key of detectorVisit.
      • Proposal to change all column names to snake_case.

      For more info on the basic proposal (without changing all cols to snake_case the CcdVisit Table to DetectorVisit Table)


        Issue Links


            yusra Yusra AlSayyad created issue -
            yusra Yusra AlSayyad made changes -
            Field Original Value New Value
            Link This issue is triggering DM-31825 [ DM-31825 ]
            yusra Yusra AlSayyad made changes -
            Planned End 27/Sep/21 2:21 PM 27/Sep/21 7:30 PM
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31061 ]
            tjenness Tim Jenness made changes -
            Status Proposed [ 10805 ] Flagged [ 10606 ]
            tjenness Tim Jenness made changes -
            Status Flagged [ 10606 ] Board Recommended [ 11405 ]
            yusra Yusra AlSayyad made changes -
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31131 ]
            ebellm Eric Bellm made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31153 ]
            yusra Yusra AlSayyad made changes -
            Status Board Recommended [ 11405 ] Adopted [ 10806 ]
            yusra Yusra AlSayyad made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31887 ]


              yusra Yusra AlSayyad
              yusra Yusra AlSayyad
              Colin Slater, Eli Rykoff, Gregory Dubois-Felsmann, John Parejko, Krzysztof Findeisen, Tim Jenness, Yusra AlSayyad
              0 Vote for this issue
              7 Start watching this issue


                Planned End:


                  No builds found.