Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22528

Update DMTN-091 with details about timescale for preserving processed datasets

    Details

    • Type: Story
    • Status: In Review
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Include timescale for preserving processed datasets in this document

        Attachments

          Issue Links

            Activity

            Hide
            lguy Leanne Guy added a comment - - edited
            • Keep a minimum of 2 LARGE size datasets, e.g PDR as well as 2 sets of corresponding master calibs to be used for subsequent processing. One of the 2 may be deleted prior to processing the next one if space is needed. There always needs to be a current and previous for comparison.
            • 12 monthly MEDIUM size datasets, e.g RC2/3 or DESC DC2
            • SMALL datasets need only be kept for a maximum of 1 month
            • There is not need to keep any CI datasets
            Show
            lguy Leanne Guy added a comment - - edited Keep a minimum of 2 LARGE size datasets, e.g PDR as well as 2 sets of corresponding master calibs to be used for subsequent processing. One of the 2 may be deleted prior to processing the next one if space is needed. There always needs to be a current and previous for comparison. 12 monthly MEDIUM size datasets, e.g RC2/3 or DESC DC2 SMALL datasets need only be kept for a maximum of 1 month There is not need to keep any CI datasets
            Hide
            lguy Leanne Guy added a comment -

            Proposal for timescale for preserving datasets added at:

            https://dmtn-091.lsst.io/v/DM-22528/index.html

            Show
            lguy Leanne Guy added a comment - Proposal for timescale for preserving datasets added at: https://dmtn-091.lsst.io/v/DM-22528/index.html
            Hide
            mbutler Michelle Butler added a comment -

            It's hard for me to look at this without seeing sizing.  Could we put what the sizes are for PDR2 (100GB?) and then sizes on the RC2/3.   I'm fine with the plan, but the sizing needs to be included.   Maybe Kian-Tat Lim could put some sizing to these?   

            Show
            mbutler Michelle Butler added a comment - It's hard for me to look at this without seeing sizing.  Could we put what the sizes are for PDR2 (100GB?) and then sizes on the RC2/3.   I'm fine with the plan, but the sizing needs to be included.   Maybe Kian-Tat Lim could put some sizing to these?   
            Hide
            ktl Kian-Tat Lim added a comment -

            All of this, including inputs and outputs, adds up to 1.8 PB in DMTN-135. This is assuming that all of DESC DC2 is kept on disk as potential input, though runs are only done on HSC PDR2-sized inputs. Without that, it's about 1.4 PB. It does assume deletion prior to new processing. Of this, the vast majority is PDR2; even bi-weekly RC2-sized runs are less than 100 TB.

            Show
            ktl Kian-Tat Lim added a comment - All of this, including inputs and outputs, adds up to 1.8 PB in DMTN-135. This is assuming that all of DESC DC2 is kept on disk as potential input, though runs are only done on HSC PDR2-sized inputs. Without that, it's about 1.4 PB. It does assume deletion prior to new processing. Of this, the vast majority is PDR2; even bi-weekly RC2-sized runs are less than 100 TB.
            Hide
            yusra Yusra AlSayyad added a comment -

            Question on github. Consider my half of the review done.

            Show
            yusra Yusra AlSayyad added a comment - Question on github. Consider my half of the review done.

              People

              • Assignee:
                lguy Leanne Guy
                Reporter:
                lguy Leanne Guy
                Reviewers:
                Michelle Butler
                Watchers:
                Kian-Tat Lim, Leanne Guy, Michelle Butler, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:

                  Summary Panel