Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
Currently ssh access to worker nodes on the verification cluster (lsst-verify-worker*) is prohibited. I would like to change this policy to allow users of the verification cluster to ssh to worker nodes. It is very useful to be able to inspect running processes in situ to determine how multiple jobs are being packed onto the nodes.
This might be dangerous in scenarios where the users are not expected to behave themselves, but we have the luxury of having a relatively small and considerate user base for this specific resource. For these reasons, it seems relatively safe to allow this access.
Attachments
Activity
I've found it useful to be able to ssh into nodes on the Princeton clusters. It lets you poke around your processes when they're running, make sure they're behaving. It would be useful to have the same ability on the verification cluster.
Paul Price
added a comment - I've found it useful to be able to ssh into nodes on the Princeton clusters. It lets you poke around your processes when they're running, make sure they're behaving. It would be useful to have the same ability on the verification cluster. +1 This would be very useful for debugging and job monitoring.
Simon Krughoff do you need a comment from NCSA before adopting?
I think I do. Bill Glick [X] or Paul Domagala [X] can you comment on whether the LDF would be willing to support this RFC, if adopted?
Paul is away this week. Here is my initial thinking. I think it is useful for development, as long was we write a policy in the developers's guide that SSH usage must not interfere with the resource management of the batch system. I;e you cannot get a grip on batch node unless you rware working on the executing instance of software on that node, and ssh must cease when the batch job ends (or something like that).
Donald Petravick
added a comment - Paul is away this week. Here is my initial thinking. I think it is useful for development, as long was we write a policy in the developers's guide that SSH usage must not interfere with the resource management of the batch system. I;e you cannot get a grip on batch node unless you rware working on the executing instance of software on that node, and ssh must cease when the batch job ends (or something like that).
Excellent! Thanks Donald Petravick. I'll mark adopted and create an IHS ticket for implementation.
IHS-691 is the triggered issue.
If the developer batch queues were to be supported by a large batch "commons", it might be difficult to continue this kind of direct ssh access. But I don't think that's an argument against this RFC; instead, I think it's an argument against merging the developer batch with the commons. I can see reasons (e.g. testing of new versions of the batch system itself) why a separate batch system for development/developers would be desirable. On the other hand, it would also limit the maximum resources available to developers and the possibility for maximizing efficiency of compute resource utilization.