Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31950

Add descriptions for DP0.2 and HSC Object/Source schema columns

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Because of the complete change-over of column names in the object table, I programmatically built the schema files and thus completely destroyed the all the descriptions in sdm_schemas.

      Not all columns had descriptions. This ticket will add back where they were accurate and add descriptions where they were missing.

        Attachments

          Activity

          Hide
          yusra Yusra AlSayyad added a comment -

          OK, one PR is open and ready for a content review: https://github.com/lsst/sdm_schemas/pull/57

          I grabbed the field descriptions from the plugins. Discovered from deficiencies there. e.g. DM-35007, need to file tickets for sky_source, and  base_LocalWcs_CDMatrix__.

          Decisions for Object Table:

          • Added "forced on z band" if from deepCoadd_forced_src
          • Added "measured on z band" if from deepCoadd_meas 

           

          TO DO:

          • Units for both Forced Source. 
          • Rebase on u/yusra/backport-main-dp02
          • PR for main
          • I accidentally blew away the capitalization for descriptions on Object, Source, ForcedSource*. Will fix on next round. 

          What I could use feedback on NOW, is whether there are any improvements that can be made to these programmatically generated docs, before we embark on the stage where we start editing by hand. 

           

          Show
          yusra Yusra AlSayyad added a comment - OK, one PR is open and ready for a content review: https://github.com/lsst/sdm_schemas/pull/57 I grabbed the field descriptions from the plugins. Discovered from deficiencies there. e.g. DM-35007 , need to file tickets for sky_source, and  base_LocalWcs_CDMatrix_ _ . Decisions for Object Table: Added "forced on z band" if from deepCoadd_forced_src Added "measured on z band" if from deepCoadd_meas    TO DO: Units for both Forced Source.  Rebase on u/yusra/backport-main-dp02 PR for main I accidentally blew away the capitalization for descriptions on Object, Source, ForcedSource*. Will fix on next round.  What I could use feedback on NOW, is whether there are any improvements that can be made to these programmatically generated docs, before we embark on the stage where we start editing by hand.   
          Hide
          ctslater Colin Slater added a comment -

          I'm kind of ambivalent about the "forced on [ugrizy] band" string after each column; sometimes it makes the description read more awkwardly, like "General failure flag forced on g band". It might help to at least put a period after each description (if it doesn't already have one), prior to appending the "forced on" text, so it's clearly a separate statement. And then to capitalize forced.

          Show
          ctslater Colin Slater added a comment - I'm kind of ambivalent about the "forced on [ugrizy] band" string after each column; sometimes it makes the description read more awkwardly, like "General failure flag forced on g band". It might help to at least put a period after each description (if it doesn't already have one), prior to appending the "forced on" text, so it's clearly a separate statement. And then to capitalize forced.
          Hide
          yusra Yusra AlSayyad added a comment - - edited

          OK officially up for review. I made your first round changes:

          1) "measured on z band" --> ". Measured on z-band."  (Can change it back or do a ", measured on z-band" if you don't like this)
          2) search replaced 000000 -> 0 so  the apertures read a more tame 3.0

          3) I'll enclose them in quotes if you really want, but wanted to make sure you were sure and it wasn't a "just in case."  I had stripped out any apostrophes or colons or anything that wouldn't make it valid yaml. If you're just reading it in as yaml, it should be exactly the same as enclosing it in quotes. I'll add them if you need them, but just wanted to make sure you needed them.

           Jenkins passes on main. https://ci.lsst.codes/job/stack-os-matrix/36760/display/redirect

          Need to figure out why daf_butler on v23.0.x failed on my backport jenkins run. 

          Show
          yusra Yusra AlSayyad added a comment - - edited OK officially up for review. I made your first round changes: 1) "measured on z band" --> ". Measured on z-band."  (Can change it back or do a ", measured on z-band" if you don't like this) 2) search replaced 000000 -> 0 so  the apertures read a more tame 3.0 3) I'll enclose them in quotes if you really want, but wanted to make sure you were sure and it wasn't a "just in case."  I had stripped out any apostrophes or colons or anything that wouldn't make it valid yaml. If you're just reading it in as yaml, it should be exactly the same as enclosing it in quotes. I'll add them if you need them, but just wanted to make sure you needed them.  Jenkins passes on main. https://ci.lsst.codes/job/stack-os-matrix/36760/display/redirect Need to figure out why daf_butler on v23.0.x failed on my backport jenkins run. 
          Hide
          yusra Yusra AlSayyad added a comment -

          Other issues with sdm_schemas. 

          1) The CI action was failing: https://github.com/lsst/sdm_schemas/runs/6741661644?check_suite_focus=true.   git clone -q git://github.com/lsst-dm/felis.git also hangs when I try myself.

          Christine Banek , is it cool that I changed it to git+http like it is on main?

          2) One of my commits copies a lot from dp02_dc2.yaml to imsim.yaml on main also. Colin, you and I spent many hours giving talks about how we should automate the checks that the pipelines are producing the expected schemas. We did it. We had a nice chain of checks going. When people copy dp02_dc2.yaml and only edit that, they brake that chain. If breaking the chain  was on purpose, let's iterate, but to me they looked like modifications that we'll always need... CcdVisit, Visit schemas, capitalized table names, colon in the units key, longer char's for the skymaps.... Would you remind folks that imsim.yaml is the source of truth schema that gets checked in CI and the pipelines folks maintain?

          Show
          yusra Yusra AlSayyad added a comment - Other issues with sdm_schemas.  1) The CI action was failing:  https://github.com/lsst/sdm_schemas/runs/6741661644?check_suite_focus=true.    git clone -q git://github.com/lsst-dm/felis.git also hangs when I try myself. Christine Banek  , is it cool that I changed it to git+http like it is on main? 2) One of my commits copies a lot from dp02_dc2.yaml to imsim.yaml on main also. Colin, you and I spent many hours giving talks about how we should automate the checks that the pipelines are producing the expected schemas. We did it. We had a nice chain of checks going. When people copy dp02_dc2.yaml and only edit that, they brake that chain. If breaking the chain  was on purpose, let's iterate, but to me they looked like modifications that we'll always need... CcdVisit, Visit schemas, capitalized table names, colon in the units key, longer char's for the skymaps.... Would you remind folks that imsim.yaml is the source of truth schema that gets checked in CI and the pipelines folks maintain?
          Hide
          ctslater Colin Slater added a comment -

          This is really great. So many descriptions! It looks like DiaObject didn't get any descriptions though? And dp02_dc2.yaml needs the same set of descriptions added. Other than that, it's good to go.

          I'm fine without quotes if you've already checked for characters that would cause an issue.

          Show
          ctslater Colin Slater added a comment - This is really great. So many descriptions! It looks like DiaObject didn't get any descriptions though? And dp02_dc2.yaml needs the same set of descriptions added. Other than that, it's good to go. I'm fine without quotes if you've already checked for characters that would cause an issue.
          Hide
          yusra Yusra AlSayyad added a comment -

          Yeah, forgot to mention that I'm skipping DiaObject on this ticket because it can't be auto-generated like the others. The tables I did on this ticket was done by extracting docstrings directly from the measurement plugins.  We need a new ticket to dig through the code that did the diaObject creation (Chris M would have been the obvious but alas). Will file a new ticket. 

          dp02_dc2.yaml pushed to the branch. As of this ticket its an exact copy of 23.0.x/imsim.yaml (aside the schema name) 

          Show
          yusra Yusra AlSayyad added a comment - Yeah, forgot to mention that I'm skipping DiaObject on this ticket because it can't be auto-generated like the others. The tables I did on this ticket was done by extracting docstrings directly from the measurement plugins.  We need a new ticket to dig through the code that did the diaObject creation (Chris M would have been the obvious but alas). Will file a new ticket.  dp02_dc2.yaml pushed to the branch. As of this ticket its an exact copy of 23.0.x/imsim.yaml (aside the schema name) 

            People

            Assignee:
            yusra Yusra AlSayyad
            Reporter:
            yusra Yusra AlSayyad
            Reviewers:
            Colin Slater
            Watchers:
            Colin Slater, Hsin-Fang Chiang, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.