Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27476

Add support for metadata sidecar files for ingest

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Team:
      Ops Middleware
    • Urgent?:
      No

      Description

      In order to support ingest of files from object store with any reasonable performance we need to be able to pre-calculate the ingest metadata and then use that metadata for ingest. Files will likely be ingested repeatedly and each time we do not want to have to download either the full file or the first N bytes to parse FITS headers.

      Mofdify butler ingest-raws so that it can read JSON metadata snippet. This can either be in the form of a .json file of the same name as the data file, or optionally an update of ButlerURI to get it to read metadata from the object. Kian-Tat Lim has suggested that we can use the object metadata (see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html but I'm not yet sure if the astronomy metadata fits in 2kB although we could probably encode it).

      Pre-calculating this metadata will be a different ticket.

      Since this is only really an issue for object stores, I am marking this as an IDF ticket.

        Attachments

          Issue Links

            Activity

            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Link This issue is blocked by DM-27477 [ DM-27477 ]
            tjenness Tim Jenness made changes -
            Component/s obs_base [ 10719 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-28844 [ DM-28844 ]
            tjenness Tim Jenness made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Reviewers John Parejko [ parejkoj ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-25965 [ DM-25965 ]
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            yusra Yusra AlSayyad made changes -
            Link This issue duplicates DM-11923 [ DM-11923 ]

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              John Parejko
              Watchers:
              Hsin-Fang Chiang, John Parejko, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.