Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18032

Update descriptions of DM L2 milestones

    Details

      Description

      Review LDM-503, and ensure that each L2 milestone has a coherent description of what it involves.

      For most of them, this will likely involve checking in with the relevant milestone ”owners” — see the list of action items from the Nov 2018 DMLT.

        Attachments

          Activity

          Hide
          swinbank John Swinbank added a comment -

          I have provided updated descriptions, based on test specs or other documentation, for:

          • LDM-503-01
          • LDM-503-02
          • LDM-503-03
          • LDM-503-04
          • LDM-503-04b
          • LDM-503-05
          • LDM-503-07
          • LDM-503-09
          • LDM-503-09a
          • LDM-503-11 (provisional; Robert Gruendl may wish to correct)
          • LDM-503-11b
          • LDM-503-12 (provisional; Robert Gruendl may wish to correct)
          • LDM-503-13
          • LDM-503-13a
          • LDM-503-14
          • LDM-503-15
          • LDM-503-15a
          • LDM-503-16
          • LDM-503-17
          • LDM-503-17a

          I have received the following contributions:

          The following are still outstanding:

          Show
          swinbank John Swinbank added a comment - I have provided updated descriptions, based on test specs or other documentation, for: LDM-503-01 LDM-503-02 LDM-503-03 LDM-503-04 LDM-503-04b LDM-503-05 LDM-503-07 LDM-503-09 LDM-503-09a LDM-503-11 (provisional; Robert Gruendl may wish to correct) LDM-503-11b LDM-503-12 (provisional; Robert Gruendl may wish to correct) LDM-503-13 LDM-503-13a LDM-503-14 LDM-503-15 LDM-503-15a LDM-503-16 LDM-503-17 LDM-503-17a I have received the following contributions: LDM-503-08 (from Michelle Butler ) The following are still outstanding: LDM-503-06 — Action item on Margaret Gelman from Nov 2018 DMLT. LDM-503-10 — Action item on Michelle Butler from Nov 2018 DMLT. LDM-503-10b — Action item on Michelle Butler from Nov 2018 DMLT. LDM-503-11a — Action item on Robert Gruendl from Nov 2018 DMLT. LDM-503-12a — Action item on Robert Gruendl from Nov 2018 DMLT.
          Hide
          swinbank John Swinbank added a comment -

          Michelle Butler sent the following for LDM-503-06:

          The milestone is to test the components of the DM subsystems to support a 1 raft size image from the DAQ at the summit to the data reliably stored at NCSA for viewing. The testing needs to include transferring 1 raft size data from the BDC to NCSA and being ingested into the
          DBB and Butler infrastructures. The DBB needs to insure that the data can't be lost and include provenance data. The raft size image must be a good image with proper headers and well-formed files. It must be transferred to NCSA through normal paths that will be used in first
          operations with scripts, not by hand. The DBB tests need to show that the data is stored and backed up on different locations and can be retrieved if lost. The RDMS that houses the Butler and DBB registries must also be robust and show that it can recover from data losses. The
          data must be made available for science evaluation through access through the LSP.

          Show
          swinbank John Swinbank added a comment - Michelle Butler sent the following for LDM-503-06: The milestone is to test the components of the DM subsystems to support a 1 raft size image from the DAQ at the summit to the data reliably stored at NCSA for viewing. The testing needs to include transferring 1 raft size data from the BDC to NCSA and being ingested into the DBB and Butler infrastructures. The DBB needs to insure that the data can't be lost and include provenance data. The raft size image must be a good image with proper headers and well-formed files. It must be transferred to NCSA through normal paths that will be used in first operations with scripts, not by hand. The DBB tests need to show that the data is stored and backed up on different locations and can be retrieved if lost. The RDMS that houses the Butler and DBB registries must also be robust and show that it can recover from data losses. The data must be made available for science evaluation through access through the LSP.
          Hide
          mbutler Michelle Butler added a comment -

          I will update the others on Monday.  Giving folks a bit of time to digest the ones that we were suppose to update since I just finished my pass today.   .  

           

          Show
          mbutler Michelle Butler added a comment - I will update the others on Monday.  Giving folks a bit of time to digest the ones that we were suppose to update since I just finished my pass today.   .    
          Show
          swinbank John Swinbank added a comment - Per https://github.com/lsst-dm/milestones/pull/10#pullrequestreview-208087150 , Robert Gruendl sounds happy.
          Hide
          swinbank John Swinbank added a comment -

          Text supplied by Joel Plutchak:

          LDM-503-06 Limited ComCam Interface Verification readiness

          Description: Demonstrate raft-scale archiving on the Level Test Stand.

          Comments:
          Demonstrate that a raft-scale Archiver can acquire pixel data from the DAQ system and place images into an OODS-like file system. Verify headers for correctness. This test will be exercised on the L1
          Complete Test Stand at NCSA.

          LDM-503-10 DAQ Validation

          Description: Test the extended DAQ network in Chile from the Summit to the Base Data Center.

          Comments:
          This milestone will take simulated data from the DAQ at the Summit and use the DWDM network environment to place data on DM machines at Base Data Center (BDC), essentially extending the DAQ extended
          network to the BDC. The test machines at the BDC will be a L1 handoff environment and a single forwarder.

          LDM-503-10b Verification Tests for Large Scale CCOB Data Access

          Description: Demonstrate the ability to transfer data from the CCOB with 21 rafts from SLAC and ingested at NCSA and make available through an instance of the LSP.

          Comments:
          This is a data transfer of data from SLAC or some other site with 21-raft-sized images to NCSA, ingest it into a Butler environment and place the file into file systems readable by the LSP. The CCOB
          device might NOT be available, but as 21 raft size data will be available from a test stand at SLAC, we will use a generic test stand data transfer method (e.g., rsync) to bring designated data to NCSA,
          ingest it, and place into appropriate filesystems, and make available through the LSP.

          LDM-503-09 Ops Rehearsal for Commissioning #1

          <retain current description and comments>

          LDM-503-11 Ops Rehearsal for Commissioning #2

          Description: Demonstrate ComCam archiving on the L1 Test Stand.

          Comment: This is a test on the L1 test stand at NCSA to verify the image acquisition and readout from the DAQ to the Archiver at 1 raft image scale and rate, and the L1 handoff from the archiver to the
          DBB service and OODS service.

          LDM-503-12

          <retain current description and comments>

          Show
          swinbank John Swinbank added a comment - Text supplied by Joel Plutchak : LDM-503-06 Limited ComCam Interface Verification readiness Description: Demonstrate raft-scale archiving on the Level Test Stand. Comments: Demonstrate that a raft-scale Archiver can acquire pixel data from the DAQ system and place images into an OODS-like file system. Verify headers for correctness. This test will be exercised on the L1 Complete Test Stand at NCSA. LDM-503-10 DAQ Validation Description: Test the extended DAQ network in Chile from the Summit to the Base Data Center. Comments: This milestone will take simulated data from the DAQ at the Summit and use the DWDM network environment to place data on DM machines at Base Data Center (BDC), essentially extending the DAQ extended network to the BDC. The test machines at the BDC will be a L1 handoff environment and a single forwarder. LDM-503-10b Verification Tests for Large Scale CCOB Data Access Description: Demonstrate the ability to transfer data from the CCOB with 21 rafts from SLAC and ingested at NCSA and make available through an instance of the LSP. Comments: This is a data transfer of data from SLAC or some other site with 21-raft-sized images to NCSA, ingest it into a Butler environment and place the file into file systems readable by the LSP. The CCOB device might NOT be available, but as 21 raft size data will be available from a test stand at SLAC, we will use a generic test stand data transfer method (e.g., rsync) to bring designated data to NCSA, ingest it, and place into appropriate filesystems, and make available through the LSP. LDM-503-09 Ops Rehearsal for Commissioning #1 <retain current description and comments> LDM-503-11 Ops Rehearsal for Commissioning #2 Description: Demonstrate ComCam archiving on the L1 Test Stand. Comment: This is a test on the L1 test stand at NCSA to verify the image acquisition and readout from the DAQ to the Archiver at 1 raft image scale and rate, and the L1 handoff from the archiver to the DBB service and OODS service. LDM-503-12 <retain current description and comments>
          Hide
          swinbank John Swinbank added a comment -

          All milestones from the list above now have updated descriptions.

          Note that:

          • I merged the LDM-503-06 descriptions provided by Michelle Butler & Joel Plutchak.
          • I provided my own descriptions for LDM-503-11a & 12a, but I think they are uncontroversial.

          Intend to merge this ticket as is.

          Show
          swinbank John Swinbank added a comment - All milestones from the list above now have updated descriptions. Note that: I merged the LDM-503-06 descriptions provided by Michelle Butler & Joel Plutchak . I provided my own descriptions for LDM-503-11a & 12a, but I think they are uncontroversial. Intend to merge this ticket as is.

            People

            • Assignee:
              swinbank John Swinbank
              Reporter:
              swinbank John Swinbank
              Watchers:
              John Swinbank, Michelle Butler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel