I've been using black, isort, and mypy in my own development as much as possible recently (i.e. mostly just on new files), and we would really benefit from always using them consistently across our predominantly middleware packages:
I'm hesitating to include ctrl_bps here, because I very rarely work on it directly and I don't know if its somewhat distinct set of developers want it to be included here.
To be precise:
- I am proposing that we run mypy in PR CI in all of these packages (just as we do already in daf_butler). Many of the other packages already have some type annotations, and just like any other aspect of code that isn't tested, they're very often incorrect. Running mypy will fix that and make sure it stays fixed, without requiring type annotations in new code, even in those packages .
- I am proposing that we run black and isort once up-front on all files in these packages, and then at least check in PR CI that running black and isort would be no-ops; I'm open to the idea of running them in git commit hooks, too, but can't personally make a good case for or against.
I won't go into great detail about the general benefits of these tools - they're widely discussed on the broader internet. I'm proposing we adopt them now because:
- they now have broad adoption in the Python community, and despite some lingering "beta" labeling, they are in use in production code in organizations much larger than ours;
- the packages I am proposing using them in are primarily developed by a subset of the DM team that I think would be comfortable with this move;
- they are very much the kind of tools that need package-wide buy-in from all contributing developers; the utility of using them on an individual opt-in basis is much lower (and in some cases may even be negative).
I do also think Science Pipelines development would benefit from using these tools more, but I'd prefer to leave that for another RFC, especially after implementing this one has helped us smooth out any rough edges in the procedure. I will say that I now very much regret not weighing in strongly in favor of
RFC-649 when the question of black was brought up previously (I was ambivalent, having not tried it before). And I think conditions have changed enough for us to consider it again.