Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36425

Discuss purpose of tables produced in postprocess.py

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Discuss the purpose of tables such as the visit summary table that are produced in postprocess.py with DRP developers. Include a discussion of versioning of the contents, and consistency and redundancy between tables.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment - - edited

            Summary of my conversation with Yusra and Jim:

            We should treat VisitSummary as a code interface, not a data product, because it is a temporary internal product. We do have an explicit guarantee that you can read data produced with older code with new code; we do not have a guarantee that you can process old data with new code (for example, running difference imaging with old coadds, or postprocess/analysis_tools on old calexps/tables).

            In the context of this ticket, this means that developers can add fields to VisitSummary and assume in downstream code-released concurrently with or after the addition-that those fields will exist, while the removal of fields from VisitSummary or significant changes to meanings requires an RFC and potentially a deprecation period. This also means that VisitSummary is explicitly not versioned: its effective version is tied directly to the version of Science Pipelines that was used to produce it and use it.

            I don't know if the above is fully captured in RFC-699: that RFC mostly discusses reading of old data, but I had interpreted some of it to refer to further processing of older intermediate data products.

            Show
            Parejkoj John Parejko added a comment - - edited Summary of my conversation with Yusra and Jim: We should treat VisitSummary as a code interface, not a data product, because it is a temporary internal product. We do have an explicit guarantee that you can read data produced with older code with new code; we do not have a guarantee that you can process old data with new code (for example, running difference imaging with old coadds, or postprocess/analysis_tools on old calexps/tables). In the context of this ticket, this means that developers can add fields to VisitSummary and assume in downstream code-released concurrently with or after the addition-that those fields will exist, while the removal of fields from VisitSummary or significant changes to meanings requires an RFC and potentially a deprecation period. This also means that VisitSummary is explicitly not versioned: its effective version is tied directly to the version of Science Pipelines that was used to produce it and use it. I don't know if the above is fully captured in RFC-699 : that RFC mostly discusses reading of old data, but I had interpreted some of it to refer to further processing of older intermediate data products.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              sullivan Ian Sullivan
              Watchers:
              Eli Rykoff, Ian Sullivan, Jim Bosch, John Parejko, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.