Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4557

Base site additions to simulator


    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Templates:
    • Story Points:
    • Epic Link:
    • Team:
      Data Facility


      Started using login on the Nebula cluster - set up some instances and used snapshot facility. Detected an error with launching from volume. Met with Chris Lindsey about issue.

      Started coding replicator and Foreman functionality.
      Began message taxonomy for comm between Foreman and Replicator.

      Installed log rotate, Rabbitmq, pip, pika and GCC on running instance. Successfully tested
      rabbitmq with another instance which had pika libs installed.
      Finished up prototyping of replicator and foreman code.

      Installed Nova Client on foreman instance so it can start and stop replicator instances
      as needed at runtime. Tested.

      Began a project to simulate realistic camera data from LSST to be used for various timings of DMCS prototype code. This was also a concentrated effort to learn about the nature of astronomical data, how it is represented, its various types of data, and practice with the C and Python libraries for building FITS files. Two sample files were built: One additively built and converted 16bit DES data into scaled 18bit data, then placed that binary representation in a 32bit integer, and the other was additively built up from PhoSim data. Files were built to represent a single LSST camera raft with 9 HDUs - one for each CCD. Files were over 600Mg. The code used in these files has been saved and parts turned into scripts for generating other sample files.

      Set up an environment for timing the imprint of these files as they are built and processed through the Base DMCS pipeline code. Initial timing was done on compression techniques.

      Wrote result paper on FITS file generation and timing. Errors due primarily to my own ignorance of this type of file format and how the data was truly represented were encountered several times. This project was personally extremely helpful in beginning to learn the vast amount of domain knowledge needed to complete future pieces of the L1 Base site code package.

      Added coding for setting up raw data to be moved and assembled into fits files in DMCS , simulating part of the Base DMCS data flow, into existing Base replicator/foreman prototype.




            • Assignee:
              jparsons James Parsons
              jparsons James Parsons
              James Parsons
            • Votes:
              0 Vote for this issue
              1 Start watching this issue


              • Created:

                Summary Panel