From a security and a developer standpoint, I think HTTPS+Basic Auth or HTTPS+Application Tokens (or cookies for users logged in via a browser) is easiest, but that of course requires the unofficial route where either all clients are updated to trust the webserver or we go the official route and get an SSL Certificate, which does limit you on some flexibility. For other projects I'm working on, we are planning on using nginx as a load balancer and SSL/TLS endpoint and going the official route.
If we don't want to bother with SSL, I've implemented HMAC-based auth in clients before. It's very straightforward. I think I should be able to do it with Flask just fine as well (using that snippet code). It's similar to Digest Auth.
The basic implementation requires a shared key, a key id (which can be either a UUID or a username/user id). The user performs a digest of request attributes using the secret key, adds that digest and the key id to the request, and the server looks up the private key using the key id (this can be memoized/cached for a while). Every request is sent with a date, and the server always drops requests older than a few seconds. In one implementation I've written, I've allowed multiple UUID/key pairs to be assigned to a user, with each key having it's own expiration date. This slightly improves security, but you still need a way to communicate the keys securely (usually an application which uses HTTPS).
See Amazon's page on HMAC based auth.
See datacat documentation on authentication.
Finally, you can always just use iptables on the server if you trust the host, or have an equivalent application-level version of iptables as well. This is definitely the easy way forward for good-enough security if we want to defer making a decision for a bit, assuming the machines accessing the worker management service aren't open to just anyone.
For password storage, storing the secret somewhere under /etc, with proper permissions set, is how we usually do it as well.