Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38090

Configure Watcher to escalate additional alarms

    XMLWordPrintable

    Details

    • Team:
      Telescope and Site
    • Urgent?:
      No

      Description

      Configure the Watcher to escalate specific critical alarms to specific squads and ask the users interested in those alarms to join the squads and configure hours during which they are willing to be woken up.

      Here are the ones I know about so far; please add others and discuss these in comments below:

      • ATCameraDewar: Patrick Ingraham is interested in this one
      • Heartbeat: Tiago Ribeiro is interested in some of these (which ones?, and see note below)

      Note: in order to support escalation of Heartbeat alarms we need to modify the Heartbeat rule to make a missing heartbeat CRITICAL instead of SERIOUS. This could be done for all of them, or make it a configurable option.

        Attachments

          Issue Links

            Activity

            Hide
            bstalder Brian Stalder added a comment -

            CCCamera: Vacuum, Ion Pump, BackBias Voltage, REB temps, CCD temps, cryotel temps (finger, water), cold plate temp, cryo plate, temp.  quadbox temps, water flow.  

            Dynalene Chillers (when available): set temp - coolant temp, flow/pressure output, flow/pressure input

            Glycol Chillers: water temps, function state, alarms/faults 

            Air Compressors: output pressures, function state, alarms/faults, compressor run time

            UMAs: set temp - output temp, function state, alarms/faults 

            Crack/Computer room: temperature, function state, alarms/faults 

            Power:  generator/source state, UPS life, generator run time, voltage in/out, power draw

            Network (when available):  LHN down, commodity down

            Weather Station (at night): dew point spread (and velocity), humidity 

            Lightning (at night): close event 

            TMA:  OSS fault, PS fault  

            Rotator: temps 

            Hexapods: temps 

            Main Dome environment (ESS) temps, humidity 

            AuxTel Dome environment (ESS) temps, humidity 

            Show
            bstalder Brian Stalder added a comment - CCCamera: Vacuum, Ion Pump, BackBias Voltage, REB temps, CCD temps, cryotel temps (finger, water), cold plate temp, cryo plate, temp.  quadbox temps, water flow.   Dynalene Chillers (when available): set temp - coolant temp, flow/pressure output, flow/pressure input Glycol Chillers: water temps, function state, alarms/faults  Air Compressors: output pressures, function state, alarms/faults, compressor run time UMAs: set temp - output temp, function state, alarms/faults  Crack/Computer room: temperature, function state, alarms/faults  Power:  generator/source state, UPS life, generator run time, voltage in/out, power draw Network (when available):  LHN down, commodity down Weather Station (at night): dew point spread (and velocity), humidity  Lightning (at night): close event  TMA:  OSS fault, PS fault   Rotator: temps  Hexapods: temps  Main Dome environment (ESS) temps, humidity  AuxTel Dome environment (ESS) temps, humidity 
            Hide
            rowen Russell Owen added a comment - - edited

            Comments on the above list:

            • We have no alarms for the cameras beyond the AT dewar alarm. I believe Tony Johnson is still thinking about how to make camera alarms visible to the Watcher. I suggest waiting for that before pursuing escalation of those alarms. Also Tony Johnson may prefer to escalate camera alarms directly to SquadCast and leave Watcher out of it.
            • We can easily add temperature alarms for any sensor that the ESS can see.
            • Air compressors: this will take some work because the warning conditions are more complicated (and checking for enabled is more than the Enabled rule can handle):
              • Warn if at least one of the two air compressors is not enabled.
              • Warn if either air compressor is in fault, but only if it not the "stop button" fault. This may be more work than it is worth.
              • Warn if the linePressure is < 8200 mBar from both air compressors. The pressure is OK if either instance reports linePressure >= 8200 mBar.
            • "Only at night": that can be managed in SquadCast, but not (at present) in the Watcher.
            • Lightning: we'll need a new rule. It will be simple.
            • Temperature and pressure anywhere: existing rules can be configured as long as the ESS is reading the data. It will require new rules to read these from other sources.
            Show
            rowen Russell Owen added a comment - - edited Comments on the above list: We have no alarms for the cameras beyond the AT dewar alarm. I believe Tony Johnson is still thinking about how to make camera alarms visible to the Watcher. I suggest waiting for that before pursuing escalation of those alarms. Also Tony Johnson may prefer to escalate camera alarms directly to SquadCast and leave Watcher out of it. We can easily add temperature alarms for any sensor that the ESS can see. Air compressors: this will take some work because the warning conditions are more complicated (and checking for enabled is more than the Enabled rule can handle): Warn if at least one of the two air compressors is not enabled. Warn if either air compressor is in fault, but only if it not the "stop button" fault. This may be more work than it is worth. Warn if the linePressure is < 8200 mBar from both air compressors. The pressure is OK if either instance reports linePressure >= 8200 mBar. "Only at night": that can be managed in SquadCast, but not (at present) in the Watcher. Lightning: we'll need a new rule. It will be simple. Temperature and pressure anywhere: existing rules can be configured as long as the ESS is reading the data. It will require new rules to read these from other sources.

              People

              Assignee:
              rowen Russell Owen
              Reporter:
              rowen Russell Owen
              Watchers:
              Brian Stalder, Patrick Ingraham, Russell Owen, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.