Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31767

Add Watcher Alarm(s) for the ATCamera

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      The ATCamera needs a monitor (with warnings) for the sensor temperature and vacuum pressure.

      The vacuum limit is a bit tricky because the system "burps". So basically the system should remain under 1e-6 at all times, but calculated as a moving average (median?) over a time interval. I think that time interval can be ~10-15 minutes.

      The CCD temperature limit should be -94C +/1 over a period of a few minutes.
      Coldplate can issue an alert when the temperature is greater than -110.
       The cryohead is harder, lets say it should have an alert when the temperature is greater than -140.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            Patrick Ingraham: please spell out which telemetry (and event?) topics and fields to use for the suggested inputs.

            If there are multiple sensors for a given item, how do you suggest combining them: average? median? most pessimistic? second-most pessimistic (as a crude form of outlier rejection)?

            Show
            rowen Russell Owen added a comment - Patrick Ingraham : please spell out which telemetry (and event?) topics and fields to use for the suggested inputs. If there are multiple sensors for a given item, how do you suggest combining them: average? median? most pessimistic? second-most pessimistic (as a crude form of outlier rejection)?
            Hide
            pingraham Patrick Ingraham added a comment -

            So all the data we're interested in can be seen in the following dashboard.
            https://chronograf-summit-efd.lsst.codes/sources/1/dashboards/14?refresh=Paused&lower=now%28%29%20-%202d

             
            The vacuum limit (lsst.sal.ATCamera.vacuum.vacuum) is a bit tricky because the system "burps". So basically the system should remain under 1e-6 at all times, but calculated as a moving median over a time interval. I think that time interval can be ~10-15 minutes
             
            The temperatures are easier.
            The CCD temperature(lsst.sal.ATCamera.vacuum.tempCCD) limit should be -94C +/1 over a period of a few minutes.
            Coldplate (lsst.sal.ATCamera.vacuum.tempColdPlate) can issue an alert when the temperature is greater than -110.
             The cryohead is harder (lsst.sal.ATCamera.vacuum.tempCryoHead), lets say it should have an alert when the temperature is greater than -140?

            Show
            pingraham Patrick Ingraham added a comment - So all the data we're interested in can be seen in the following dashboard. https://chronograf-summit-efd.lsst.codes/sources/1/dashboards/14?refresh=Paused&lower=now%28%29%20-%202d   The vacuum limit (lsst.sal.ATCamera.vacuum.vacuum) is a bit tricky because the system "burps". So basically the system should remain under 1e-6 at all times, but calculated as a moving median over a time interval. I think that time interval can be ~10-15 minutes   The temperatures are easier. The CCD temperature(lsst.sal.ATCamera.vacuum.tempCCD) limit should be -94C +/1 over a period of a few minutes. Coldplate (lsst.sal.ATCamera.vacuum.tempColdPlate) can issue an alert when the temperature is greater than -110.  The cryohead is harder (lsst.sal.ATCamera.vacuum.tempCryoHead), lets say it should have an alert when the temperature is greater than -140?
            Hide
            rowen Russell Owen added a comment - - edited

            More information: the mean cryo_head temperature is about -143 C and the mean cold plate temperature is about -112 C. Patrick suggests the following severity thresholds (which replace the ones suggested above, where there is a conflict). These will probably need to be tweaked as we learn more about the system:

            • max_warning_cold_plate_temp: -108
            • max_critical_cold_plate_temp: mean + 8 = -104
            • max_warning_cryo_head_temp: -140
            • max_critical_cryo_head_temp: mean + 15 = -128

            The CCD thresholds are based on a setpoint of -94 C. This is hard-coded in the controller and will not change, but our ranges may still need to be tweaked:

            • min_warning_ccd_temp: -95
            • min_critical_ccd_temp: -97
            • max_warning_ccd_temp: -93
            • max_critical_ccd_temp: -91

            Pull request: https://github.com/lsst-ts/ts_watcher/pull/43

            Show
            rowen Russell Owen added a comment - - edited More information: the mean cryo_head temperature is about -143 C and the mean cold plate temperature is about -112 C. Patrick suggests the following severity thresholds (which replace the ones suggested above, where there is a conflict). These will probably need to be tweaked as we learn more about the system: max_warning_cold_plate_temp: -108 max_critical_cold_plate_temp: mean + 8 = -104 max_warning_cryo_head_temp: -140 max_critical_cryo_head_temp: mean + 15 = -128 The CCD thresholds are based on a setpoint of -94 C. This is hard-coded in the controller and will not change, but our ranges may still need to be tweaked: min_warning_ccd_temp: -95 min_critical_ccd_temp: -97 max_warning_ccd_temp: -93 max_critical_ccd_temp: -91 Pull request: https://github.com/lsst-ts/ts_watcher/pull/43
            Hide
            pingraham Patrick Ingraham added a comment -

            Reviewed in GH. Comments can be found there.

            Show
            pingraham Patrick Ingraham added a comment - Reviewed in GH. Comments can be found there.
            Hide
            rowen Russell Owen added a comment -

            Merged to develop. I will hold off release for DM-33384

            Show
            rowen Russell Owen added a comment - Merged to develop. I will hold off release for DM-33384

              People

              Assignee:
              rowen Russell Owen
              Reporter:
              pingraham Patrick Ingraham
              Reviewers:
              Patrick Ingraham
              Watchers:
              Patrick Ingraham, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.