Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16280

Update Avro schemas and estimate average packet size

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Story Points:
      4
    • Sprint:
      AP F18-5, AP F18-6, AP S19-1
    • Team:
      Alert Production

      Description

      Some of the fields in sample-avro-alert no longer appear in the DPDD. This ticket is to update the schemas accordingly and produce an average size estimate relevant to https://jira.lsstcorp.org/browse/RFC-538

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment - - edited

            In addition, I think it's important that the postage stamps not be naively packaged as blobs containing FITS file contents; the overhead of that seems very high. At a minimum, lossless gzip compression might be used to substantially reduce the header overhead.

            Show
            ktl Kian-Tat Lim added a comment - - edited In addition, I think it's important that the postage stamps not be naively packaged as blobs containing FITS file contents; the overhead of that seems very high. At a minimum, lossless gzip compression might be used to substantially reduce the header overhead.
            Hide
            ebellm Eric Bellm added a comment -

            Agreed--I'm planning to simply gzip compress them for the purposes of this ticket since the proportion of header/image data is pretty large.

            Show
            ebellm Eric Bellm added a comment - Agreed--I'm planning to simply gzip compress them for the purposes of this ticket since the proportion of header/image data is pretty large.
            Hide
            ebellm Eric Bellm added a comment -

            Kian-Tat Lim, can you take a look at this?

            I generated an avro-serialized alert packet with 106 DiaSources, 9 DiaForcedSources, and gzip-compressed image cutouts.

            Using ls -l for sizing, I obtain the following packet sizes:

            Avro with schema included: 126KB
            Avro with no schema: 82KB

            I note that gzip-compressing the above packets yields 68KB/65KB, which may suggest an avenue of further exploration.

            Show
            ebellm Eric Bellm added a comment - Kian-Tat Lim , can you take a look at this? I generated an avro-serialized alert packet with 106 DiaSources, 9 DiaForcedSources, and gzip-compressed image cutouts. Using ls -l for sizing, I obtain the following packet sizes: Avro with schema included: 126KB Avro with no schema: 82KB I note that gzip-compressing the above packets yields 68KB/65KB, which may suggest an avenue of further exploration.
            Hide
            ktl Kian-Tat Lim added a comment -

            One bug that may have caused packets to be slightly small (or might not be exercised; I didn't check), but otherwise looks good. Sorry it took so long for me to look it over.

            Show
            ktl Kian-Tat Lim added a comment - One bug that may have caused packets to be slightly small (or might not be exercised; I didn't check), but otherwise looks good. Sorry it took so long for me to look it over.
            Hide
            ebellm Eric Bellm added a comment -

            No change in the packet size after the review updates. I also added some more flexibility to the code to enable us to isolate how much of the packet comes from various components (e.g., the DIASource history, etc.).

            Show
            ebellm Eric Bellm added a comment - No change in the packet size after the review updates. I also added some more flexibility to the code to enable us to isolate how much of the packet comes from various components (e.g., the DIASource history, etc.).

              People

              Assignee:
              ebellm Eric Bellm
              Reporter:
              ebellm Eric Bellm
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Eric Bellm, Kian-Tat Lim
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.