How is database authorization anticipated to be handled?

(Oh, just noticed the TBD note at the end... Hmm...)

Does this move free us at all from the POSIX filesystem ownership model for the various things a user or group can own in the Science Platform? I'm thinking specifically about butler-managed things, but that's obviously not everything (and it'd probably be a problem if different entities had different ownership models).

For example, something like PostgreSQL's role-based model feels a lot better for a lot of things, where:

• users and groups are both just roles
• roles can be granted implicit membership in another role (to automatically get all permissions of that role)
• roles can be granted explicit membership in another role (sudo-like; user can "become" a role and can then do the things that role can do).
• ownables have a single role that is their owner (not a user and a group in separate owner-like roles).

That makes recording ownership in e.g. a database easier (one role, not one user and one group), and I think it meshes much better with ACLs, both because there's only one kind of thing that can be in an ACL, and the space of possible ways to give your friends access to your things is smaller and hence less confusing.

I could go on, but I've probably said enough - my real question is whether something like this is at all viable, and we can have the debate about whether it's a good idea elsewhere.

The top-level goal is to not create users as authentication identities at the infrastructure level (so, for example, PostgreSQL databases, Google Cloud Identity, AWS IAM, NCSA LDAP, etc.) and instead track them only in our A&A level. Therefore, while we can implement our own role model in our services, we would not express users or their groups or roles directly in the underlying database.

My understanding is that we don't expect to have any use cases that require a user to talk directly to an underlying database using raw SQL (given DMTN-169). Authorization and access control decisions will happen one step higher at the service layer, and the service will use a connection pool to the database using its own credentials.

So, for example, qserv would use a similar model to DMTN-169 and DMTN-182 for Butler.

POSIX file systems used, for example, for user's home directories in the Notebook Aspect are a bit of an edge case. In the IDF, it's possible to use arbitrary UIDs and GIDs defined within our A&A system with file stores created for the IDF without needing to materialize users as underlying Google accounts, so there is no conflict. At NCSA as currently constructed there would be a conflict. My expectation is that the eventual USDF will look more like the IDF than like the current NCSA setup.

Could you explain how this protects in shell access and also in batch? I'm looking at bullet 4, of course. I have always imagined people would get shell and batch access. And assuming some people do have elevated privileges, how is it guaranteed they are who they say they are?

The top-level goal is to not create users as authentication identities at the infrastructure level (so, for example, PostgreSQL databases, Google Cloud Identity, AWS IAM, NCSA LDAP, etc.) and instead track them only in our A&A level. Therefore, while we can implement our own role model in our services, we would not express users or their groups or roles directly in the underlying database.

Understood; I didn't bring up PostgreSQL because I actually wanted to use PostgreSQL's system, but because if we can design our own ownership and role-membership model, I think PostgreSQL's system might be a better one to pattern ours after (even for ownership of file-like things), but so far the prevailing assumption has been that our ownership model would look a lot more like POSIX users+groups+ACLs, and I wasn't sure how much that was a constraint vs. a choice. It sounds like it at least may be a constraint, depending on how things shake out at USDF.

Mostly in response to Jim Bosch: we certainly can look at a wider range of models for how access controls would be configurable.  The existing POSIX-like model is in place (and mentioned in a number of change-controlled documents) in large part because an "impersonation-based" implementation, in which there are user accounts on the underlying infrastructure, was our architectural baseline for many years.

From a high-level requirements perspective, as long as both user-oriented and group-oriented access grants are possible, how that's actually modeled is not really important.  But we do need users to be able to say both things like "I am granting Yusra access to this file/table/whatever, and nobody else" and "I am creating a group XYZ and granting that group access to file (etc.) W, and I'm also granting control over membership of that group to Russ".

In response to Richard Dubois:

We have not been planning to provide anything like low-level shell / ssh access to users.  We will give them a shell prompt in a JupyterLab session, not directly on a physical or virtual host.  (I.e., quite unlike what SLAC provided for BaBar and, IIRC, Fermi users.)

Further to Richard Dubois : What Gregory Dubois-Felsmann said. One of the key drivers for deciding to build JupyterLab-as-a-service is to constrain shell access to nublado, which greatly simplifies the challenges of offering shell access to 8,000 users for the Data Facility. This is for Science Platform users of course, not project staff who need operational access.

Regarding how do we know people are who they say they are: We authenticate users though their institutional credentials (through InCommon federation) so I can be just as sure that GPDF from Caltech is using my services (and your infrastructure via my services) as Caltech is. However I do recognize that we do not have an accessible write-up to explain user onboarding, and plan to write a technote for easy reference.

Since this model is also intended for deployments of the RSP at iDACs I would like to express some concerns even if I'm not a developer of RSP services, which was the main intention of this RFC. I hope you don't mind.

I fully understand the motivations of this proposal and appreciate the flexibility and control this model would provide to the developers of the RSP services. However, I would adventure to say that making the RSP a hermetic bubble would prevent users to analyse datasets coming from several instruments (and living outside the RSP) using the services provided by the RSP and I think that would be a missed opportunity.

CC-IN2P3 users are not only well identified (they must have their own account which they use for several services) but are more often than not involved in several scientific projects (e.g. Rubin and EUCLID, Rubin and ZTF, Rubin and HSC, etc.). Each of those projects has their own specific tools and conventions, but one thing in common is that the permissions set on the datasets those individual need (and has the right) to access to do their science are based on their user identity. Their user identity is also used for using the share of batch farm allocated to each of those projects. I don't see this pattern change in the near future, for all those projects at the same time, in a shared facility such as CC-IN2P3.

From my understanding of this proposal, if user A, member of Rubin and EUCLID, needs to do their analysis using data from both instruments, they could not do that using the RSP services since their identity within the RSP would be unrelated to their identity one outside the RSP, which they use for working on and accessing data from EUCLID. The same constraints would be present for accessing files in their $HOME, which would be different from their $HOME within the RSP environment.

This "split the world" approach to user identification and authentication would be a recipe for frustration for the users and for nightmares for the service provider that I am. I suspect other prospective iDACs could be impacted in the same way from such an implementation of the RSP. I hope there will be possible ways to not completely disconnect the RSP from the underlying infrastructure.

Fabio Hernandez, let me paraphrase that back to see if I understand the concern. You're worried that, in an RSP deployment at CC-IN2P3, the local RSP deployment would invent its own identity structure independent of the CC-IN2P3 environment in which it's running and thus break the connection that you otherwise have between user authentication to CC-IN2P3 services outside the RSP and user authentication inside the RSP?

If that's correct, I don't think that will be a problem, although we should find a way to say that explicitly in a formal document. What we're saying is that our design will not depend on users having an external infrastructure identity for, say, Butler to work. We're not saying that a user cannot have an external infrastructure identity.

In other words, at CC-IN2P3 you should be able to configure the RSP A&A system to use the exact same identities, UIDs, and other authentication information that your other services use and the RSP A&A system should dutifully reflect that. You may have to do a bit of extra work on top of the normal RSP machinery to plumb your own credentials through into the system because, for example, the tokens the A&A system for the RSP will be using internally will be scoped only to the RSP deployment so people will have to use a separate token to access services outside the RSP. But this should all be doable and automatable with some library wrappers, and nothing about this design should prevent you from establishing those sorts of credentials inside the RSP.

at CC-IN2P3 you should be able to configure the RSP A&A system to use the exact same identities, UIDs, and other authentication information that your other services use and the RSP A&A system should dutifully reflect that.

Thanks Russ for this clarification: I'm relieved to see that my concerns were unfounded.

Just to clarify on a side comment Fabio Hernandez made - this proposal does not simplify the RSP services - in fact it makes more work for us. What it does is streamline infrastructure support and also address a number of security concerns that have been raised to us.

What it does not do:

• Prevent another data center from issuing infrastructure accounts to their users.
• Prevent you from adding user credentials that authorize users of the science platform to non-Rubin services

What it does do:

• Ensure that the RSP access model does not rely on users being granted infrastructure accounts.

To recap some offline discussions I've had - I understand that science users would be restricted to containers (presumably in batch as well) that would control which filesystems they can access. My concern remains with the shell access they'll have from the notebook/container environment. What can a malicious person do from a containerized shell prompt... 

We would also have a staff category of people who would have more/full privileges and who would be vetted. So, it's really the risk model for essentially letting anyone (for these purposes) into the science platform... will that security model be acceptable, is my worry.

Right, that is the raison d'être of the Science Platform, so we have to find a security model that makes that acceptable. One of the advantages of this approach is that it restricts the type of credentials that we have to give to users compared to the previous design of giving them infrastructure accounts at the underlying provider, and thus is a significant improvement on that front.

I think deeper discussion of the security model is probably off-topic for this RFC, but that's literally my job, so please feel free to reach out for further discussion! I wrote some preliminary thoughts a while back as SQR-041, which should provide an orientation and an introduction to the higher-level things like the threat model. Some of the specific details are now a bit out of date, and I'm hoping to revise this document sometime this year.

Has anyone designed user batch and the permissions?

• Where do people write data files? Do we give each user some scratch bucket?
• What code can they execute? How does that code get on the batch node?
• Can they run random shell scripts calling their own code or do they have to integrate into the gen3 pipeline infrastructure?
  • submitting override configurations is presumably fairly safe (albeit pex_config supports arbitrary python code execution).
• Can they use client/server butler directly within their batch code or only use butler via PipelineTask?

Tim Jenness:

Not really, no.

• Undetermined to date; I think we've been waiting to see how the filesystem model itself develops. Scratch: we have to, somehow, yes.
• I think we were only going to constrain this by terms-of-service, rather then engineering controls. Unknown.
• I don't think we can limit them to running Gen3 pipelines, because user analyses were not a meaningful source of requirements in the Gen3 design.
• Yes, they should be able to invoke a Butler directly, for the same reason as the previous bullet.

There are probably some slides / documents of Mario's or Leanne's that include some promises or implications; it might be worth checking a few high points.

My expectation has been:

• Users write to the same POSIX-filesystem-appearing space as their Notebook Aspect home directory, their Notebook Aspect shared project space, and their Notebook Aspect scratch space. This could be provided via an actual POSIX filesystem, VOSpace, WebDAV, or something else.
• They can execute any code they like compatible with the CPU architecture available. Executables would either be shared with batch nodes via shared filesystem or, preferably, copied to batch nodes. Those executables should ideally be containerized.
• If there's a compelling reason (including better user experience), we could require that users submit to an instance of the workflow service used for DRP (although quite likely not the same instance) rather than submit directly to the underlying batch service. But we cannot insist on Gen3 pipelines and BPS unless we can provide a generic PipelineTask that invokes a file-based executable or script, and even then the boilerplate overhead may be prohibitive.

I am a tad concerned that this RFC discussion is now revolving around a poorly defined service to which we don't have a prototype, a conceptual design, or a rich set of requirements. There are a lot of ways to service requirements for user ad-hoc analytics; some of them can be serviced science-platform side, some can be serviced factory-batch side, and I would be fairly stunned if access to the latter was not gatekeeped in some way that if the as-yet non-existent user-factory-batch system required infrastructure accounts it would not be a problem.

I'd like to bring this RFC to a close by constraining it to the Science Platform. The only Science Platform requirement relating to user batch is that nublado provide shell access to the user batch system.

I think you are saying two things: "we believe that user-provided batch code execution is not part of the RSP, besides shell access" (and indeed it was anticipated that project DACs would use an underlying infrastructure component provided by the facility) and "we have not yet worked out the details of user-provided batch code execution, but have reason to believe that either this approach (accounts in the RSP service layer) will work or else underlying infrastructure accounts can be provided to such users for authentication, authorization, and resource management."  Is that correct?

I agree with Frossie. Let's handle user batch separately - but soon, so that USDF can make a plan.

Adopting this in the narrow intended way of Science Platform services, excluding user batch.