Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18181

Provide tool to validate datastore template configurations

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: daf_butler
    • Labels:
      None

      Description

      DM-17025 added the ability to validate a DatasetRef against a selected file template. Now we need the ability to be able to test that every single registered DatasetType is consistent with the corresponding file template (for datastores that support file templates).

      We also need to be able to form a DatasetRef for every DatasetType and every instrument to check that instrument specializations also conform.

      It should probably also check that every one of those DatasetRefs has a corresponding formatter assigned.

      Chained datastores will have to forward the validation to each contained datastore noting that not all datastores have to support all datasetTypes in a chained datastore.

      I am assuming this should be a command that is run on demand rather than something that occurs every time the butler is instantiated.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Andy Salnikov thanks for agreeing to review this.

            Show
            tjenness Tim Jenness added a comment - Andy Salnikov thanks for agreeing to review this.
            Hide
            salnikov Andy Salnikov added a comment -

            Looks OK, comments on PR.

             

            Show
            salnikov Andy Salnikov added a comment - Looks OK, comments on PR.  
            Hide
            tjenness Tim Jenness added a comment -

            Andy Salnikov I'm sorry but I've made extensive changes to the ticket to support some new validation requests from Michelle Gower – in particular I now also go through every template entry and ask whether it looks reasonable and matches a dataset type or storage class. It's effectively the reverse of what I was doing before.

            Show
            tjenness Tim Jenness added a comment - Andy Salnikov I'm sorry but I've made extensive changes to the ticket to support some new validation requests from Michelle Gower – in particular I now also go through every template entry and ask whether it looks reasonable and matches a dataset type or storage class. It's effectively the reverse of what I was doing before.
            Hide
            salnikov Andy Salnikov added a comment -

            I'll look at it after I finish my travel, give me 24h or more depending on my jetlag.

            Show
            salnikov Andy Salnikov added a comment - I'll look at it after I finish my travel, give me 24h or more depending on my jetlag.
            Hide
            salnikov Andy Salnikov added a comment -

            Looks OK from Europe  See my comments on PR.

            Show
            salnikov Andy Salnikov added a comment - Looks OK from Europe  See my comments on PR.
            Hide
            tjenness Tim Jenness added a comment -

            At some point we have to understand how to tell the difference between datasetTypes that are solely for ingestion (so won't need a formatter or template) and those that can be written out. At the moment you have to explicitly ask for raw to be ignored when using the command line.

            Show
            tjenness Tim Jenness added a comment - At some point we have to understand how to tell the difference between datasetTypes that are solely for ingestion (so won't need a formatter or template) and those that can be written out. At the moment you have to explicitly ask for raw to be ignored when using the command line.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                Andy Salnikov
                Watchers:
                Andy Salnikov, Jim Bosch, Michelle Gower, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel