Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-538

Update alert sizing in LDM-151

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      New estimates of the alert stream sizing have been developed. Jeff Kantor feels that LDM-151, which already has an estimate of the total alert data size per night, is an appropriate place to record the updated values. The edited text is available at https://github.com/lsst/LDM-151/pull/56 for now, soon to be merged to master and then uploaded to DocuShare. (I'm holding off a bit in case there are other LDM-151 updates that could be batched with this.)

      Some of the expansion (from 600 to 800 GB/night) comes from changes to the DPDD, especially the 12 months of history, including non-detections, and larger postage stamps (up to 50x50). The variable part of the expansion (from 800 to 1200 GB/night) comes from potential Avro overhead. While I wasn't able to generate a complete alert packet in the short time I've devoted to this, I came up with up to 50% overhead based on figures that Maria had provided and a few small test cases. It is possible that we may find a way to shrink this, but it will be quite difficult to reduce it below the 800 GB figure.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16280 Update Avro schemas and estimate average packet size

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15512 Provide estimated outbound bandwidth that may be allocated from the LDF for alert streams

          • Invalid
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            11 older comments
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            I think that this RFC has run its course since it's not requesting approval for an update of LDM-151, it seems to be requesting that we at some point update LDM-151. Can we mark it as implemented based on DM-16280 being done and then when Kian-Tat Lim does the update to LDM-151 that would become another RFC?

            Show
            tjenness Tim Jenness added a comment - I think that this RFC has run its course since it's not requesting approval for an update of LDM-151, it seems to be requesting that we at some point update LDM-151. Can we mark it as implemented based on DM-16280 being done and then when Kian-Tat Lim does the update to LDM-151 that would become another RFC?
            Hide
            Permalink
            swinbank John Swinbank added a comment -

            Per my comment above, we still haven't adequately addressed the todo list item which this RFC is (supposedly) in response to. Please don't close this RFC until either the full todo list item has been addressed or some other ticket has been filed to capture the action of documenting available bandwidth.

            Show
            swinbank John Swinbank added a comment - Per my comment above, we still haven't adequately addressed the todo list item which this RFC is (supposedly) in response to. Please don't close this RFC until either the full todo list item has been addressed or some other ticket has been filed to capture the action of documenting available bandwidth.
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            My plan is to modify LDM-151 with the per-stream bandwidth and LDM-148 with the total available baseline outbound alert network bandwidth (in text) and per-stream bandwidth (in figure). This RFC will then morph into proposing those two new document versions for CCB acceptance.

            I'm hoping to complete those modifications and publish new versions in DocuShare later this afternoon.

            Show
            ktl Kian-Tat Lim added a comment - My plan is to modify LDM-151 with the per-stream bandwidth and LDM-148 with the total available baseline outbound alert network bandwidth (in text) and per-stream bandwidth (in figure). This RFC will then morph into proposing those two new document versions for CCB acceptance. I'm hoping to complete those modifications and publish new versions in DocuShare later this afternoon.
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            New non-preferred draft versions of LDM-148 and LDM-151 have been uploaded to DocuShare.

            Show
            ktl Kian-Tat Lim added a comment - New non-preferred draft versions of LDM-148 and LDM-151 have been uploaded to DocuShare.
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            The DM-CCB is requested to approve these changes to LDM-148 and LDM-151.

            Show
            ktl Kian-Tat Lim added a comment - The DM-CCB is requested to approve these changes to LDM-148 and LDM-151.

              People

              Assignee:
              ktl Kian-Tat Lim
              Reporter:
              ktl Kian-Tat Lim
              Watchers:
              Eric Bellm, Jeff Kantor, John Swinbank, Kian-Tat Lim, Leanne Guy, Steve Pietrowicz, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.