Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-538

Update alert sizing in LDM-151

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      New estimates of the alert stream sizing have been developed. Jeff Kantor feels that LDM-151, which already has an estimate of the total alert data size per night, is an appropriate place to record the updated values. The edited text is available at https://github.com/lsst/LDM-151/pull/56 for now, soon to be merged to master and then uploaded to DocuShare. (I'm holding off a bit in case there are other LDM-151 updates that could be batched with this.)

      Some of the expansion (from 600 to 800 GB/night) comes from changes to the DPDD, especially the 12 months of history, including non-detections, and larger postage stamps (up to 50x50). The variable part of the expansion (from 800 to 1200 GB/night) comes from potential Avro overhead. While I wasn't able to generate a complete alert packet in the short time I've devoted to this, I came up with up to 50% overhead based on figures that Maria had provided and a few small test cases. It is possible that we may find a way to shrink this, but it will be quite difficult to reduce it below the 800 GB figure.

        Attachments

          Issue Links

            Activity

            ktl Kian-Tat Lim created issue -
            ktl Kian-Tat Lim made changes -
            Field Original Value New Value
            Risk Score 0
            ktl Kian-Tat Lim made changes -
            Description New estimates of the alert stream sizing have been developed. [~jkantor] feels that LDM-151, which already has an estimate of the total alert data size per night, is an appropriate place to record the updated values. The edited text is available at https://github.com/lsst/LDM-151/pull/56 for now, soon to be merged to master and then uploaded to DocuShare. (I'm holding off a bit in case there are other LDM-151 updates that could be batched with this.) New estimates of the alert stream sizing have been developed. [~jkantor] feels that LDM-151, which already has an estimate of the total alert data size per night, is an appropriate place to record the updated values. The edited text is available at https://github.com/lsst/LDM-151/pull/56 for now, soon to be merged to master and then uploaded to DocuShare. (I'm holding off a bit in case there are other LDM-151 updates that could be batched with this.)

            Some of the expansion (from 600 to 800 GB/night) comes from changes to the DPDD, especially the 12 months of history, including non-detections, and larger postage stamps (up to 50x50). The variable part of the expansion (from 800 to 1200 GB/night) comes from potential Avro overhead. While I wasn't able to generate a complete alert packet in the short time I've devoted to this, I came up with up to 50% overhead based on figures that Maria had provided and a few small test cases. It is possible that we may find a way to shrink this, but it will be quite difficult to reduce it below the 800 GB figure.
            ebellm Eric Bellm made changes -
            Watchers Jeff Kantor, John Swinbank, Kian-Tat Lim, Steve Pietrowicz [ Jeff Kantor, John Swinbank, Kian-Tat Lim, Steve Pietrowicz ] Jeff Kantor, John Swinbank, Kian-Tat Lim, Leanne Guy, Steve Pietrowicz [ Jeff Kantor, John Swinbank, Kian-Tat Lim, Leanne Guy, Steve Pietrowicz ]
            ktl Kian-Tat Lim made changes -
            Planned End 20/Oct/18 4:43 AM 27/Oct/18 4:43 AM
            tjenness Tim Jenness made changes -
            Planned End 27/Oct/18 4:43 AM 06/Nov/18 4:43 AM
            swinbank John Swinbank made changes -
            Link This issue relates to DM-15512 [ DM-15512 ]
            swinbank John Swinbank made changes -
            Link This issue relates to DM-16280 [ DM-16280 ]
            ktl Kian-Tat Lim made changes -
            Status Proposed [ 10805 ] Flagged [ 10606 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 19580 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 19610 ]
            ktl Kian-Tat Lim made changes -
            Status Flagged [ 10606 ] Board Recommended [ 11405 ]
            ktl Kian-Tat Lim made changes -
            Status Board Recommended [ 11405 ] Adopted [ 10806 ]
            ktl Kian-Tat Lim made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 19646 ]

              People

              Assignee:
              ktl Kian-Tat Lim
              Reporter:
              ktl Kian-Tat Lim
              Watchers:
              Eric Bellm, Jeff Kantor, John Swinbank, Kian-Tat Lim, Leanne Guy, Steve Pietrowicz, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.