Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36240

Setup repository demonstrating we can read historical files

    XMLWordPrintable

    Details

    • Type: Story
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      8
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      RFC-243 requested a standardized set of data that can evolve as software releases are made. The DM CCB require that we ask permission before we make a change that would stop us being able to read older data.

      Currently we do not have a way of checking if we can still read old files. Sometimes this happens by chance because an integration test is using a file that was created years ago as an input, but there is no way to ensure that we are covering all the file versions we want to support.

      What we need to do is:

      • Create a new git LFS repo containing representative files from pipelines releases (v24, v23, v22, possibly older). This should be at least a calexp and a coadd of some kind and a source catalog.
      • Write test code to read these files and compare them with reference values.
      • Ideally the reference values would be specified in an external file (YAML or JSON) rather than embedded directly into the python test code.

      This package should be added to lsst_ci so that it is run as part of the Jenkins defaults.

      One question is how we read files. Are we testing that we can read FITS files with a current version of afw, or are we testing that we can read them with the current version of a butler datastore formatter?

      When do we add new files? Only for major releases or whatever weekly we know we are changing the file content?

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            RFC-699 is alive!

            Show
            erykoff Eli Rykoff added a comment - RFC-699 is alive!

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Eli Rykoff, Gregory Dubois-Felsmann, John Parejko, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.