Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6655

Webpage of flags produced by various stack products

    XMLWordPrintable

    Details

      Description

      SDSS has a handy webpage with descriptions of all of their bitmask flags:

      http://www.sdss.org/dr12/algorithms/bitmasks/#ListofBitmasks

      It would be exceptionally useful for LSST to produce a similar webpage. I could see it being auto-built from our current flags documetation, which would also help us identify places where our current docstrings are lacking (which many of them are).

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Since I originally filed this: I was specifically looking for a web page that documents the _flag fields that get set in our catalogs. We should be able to extract that from a "typical" run on some data (say, ci_hsc). More broadly, it would be very useful if that web page had descriptions of all of the fields our "typical" catalogs contain.

            It seems silly to me that a user has to go and read in a table and look at its schema to understand what sorts of fields the LSST software could produce.

            Show
            Parejkoj John Parejko added a comment - Since I originally filed this: I was specifically looking for a web page that documents the _flag fields that get set in our catalogs. We should be able to extract that from a "typical" run on some data (say, ci_hsc). More broadly, it would be very useful if that web page had descriptions of all of the fields our "typical" catalogs contain. It seems silly to me that a user has to go and read in a table and look at its schema to understand what sorts of fields the LSST software could produce.
            Hide
            jsick Jonathan Sick added a comment -

            Got it. I think I have that covered.

            Show
            jsick Jonathan Sick added a comment - Got it. I think I have that covered.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            It seems to me that we definitely need both Task-level documentation and LSST-data-model-level documentation on this. There's no guarantee that any packed flag words in the released data model will be the outputs of single algorithmic Tasks - we may well combine flags from multiple Tasks. (Of course, as a matter of good implementation practice the combiner itself might be a Task, but its selections of what to combine would be more likely to be configuration.)

            Ideally we could point back from the data model documentation, for most (perhaps all, depending on whether any combining is done) flags, to the Task documentation that defines it algorithmic meaning.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - It seems to me that we definitely need both Task-level documentation and LSST-data-model-level documentation on this. There's no guarantee that any packed flag words in the released data model will be the outputs of single algorithmic Tasks - we may well combine flags from multiple Tasks. (Of course, as a matter of good implementation practice the combiner itself might be a Task, but its selections of what to combine would be more likely to be configuration.) Ideally we could point back from the data model documentation, for most (perhaps all, depending on whether any combining is done) flags, to the Task documentation that defines it algorithmic meaning.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            I'm taking another look at this; where do we stand on the production of flag values in the output of the SDM standardization? Has documentation of those flags been thought about as part of that work? Colin Slater, maybe?

            Show
            gpdf Gregory Dubois-Felsmann added a comment - I'm taking another look at this; where do we stand on the production of flag values in the output of the SDM standardization? Has documentation of those flags been thought about as part of that work? Colin Slater , maybe?
            Hide
            jbosch Jim Bosch added a comment -

            As I just did with DM-28280, I'm linking DM-37544 and DM-33034 as being relevant for how I see this happening. I won't repeat everything I said there, but basically I see us declaring important catalog dataset types in pipelines, then looking at the schema files for those catalogs to find flags when generating pipelines.lsst.io docs for packages with "leaf" pipelines like drp_pipe. That will reveal (as John Swinbank said ages ago) that a lot of the docs for those flags are not very good, but that's a separate problem; we at least already have places to put that documentation.

            It's worth noting, however, that we do not have a way to propagate schema information (let alone docs) from afw.table schema datasets to parquet. I think we could now save schema information for parquet files as initOutputs, thanks to Eli Rykoff's work on parquet formatters. But we need to come up with conventions for doing so, and figure out the relationship between that task-written schema information and the stuff in sdm_schemas. It may be that it will work better to write docs for standardized schemas directly in sdm_schemas YAML files, but I worry that that's too "far away" from the code that sets the flags to be maintainable.

            Show
            jbosch Jim Bosch added a comment - As I just did with DM-28280 , I'm linking DM-37544 and DM-33034 as being relevant for how I see this happening. I won't repeat everything I said there, but basically I see us declaring important catalog dataset types in pipelines, then looking at the schema files for those catalogs to find flags when generating pipelines.lsst.io docs for packages with "leaf" pipelines like drp_pipe. That will reveal (as John Swinbank said ages ago) that a lot of the docs for those flags are not very good, but that's a separate problem; we at least already have places to put that documentation. It's worth noting, however, that we do not have a way to propagate schema information (let alone docs) from afw.table schema datasets to parquet. I think we could now save schema information for parquet files as initOutputs, thanks to Eli Rykoff 's work on parquet formatters. But we need to come up with conventions for doing so, and figure out the relationship between that task-written schema information and the stuff in sdm_schemas. It may be that it will work better to write docs for standardized schemas directly in sdm_schemas YAML files, but I worry that that's too "far away" from the code that sets the flags to be maintainable.

              People

              Assignee:
              jsick Jonathan Sick
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Colin Slater, Eric Bellm, Gregory Dubois-Felsmann, Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Jonathan Sick, Krzysztof Suberlak, Leanne Guy, Simon Krughoff, Zeljko Ivezic
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.