Basically it is tailored exactly to our needs, so there is "less to it". Everything we are doing is also technically possible with scons too (though certainly harder than say Parsl). Using any other system would likely still involve writing wrappers or customization classes to stream line use, which would end up at the total scale of code proposed.

I do see your point, and it is always a double edged sword between leveraging upstream work and adapting it, or doing everything in house, but you do it all. In this case by having something tailored, it is just a small bit of code focused on our needs, with a low learning / maintenance overhead.

However this also does not lock us in. As I mentioned if we did decide we needed the functionality of Parsl, or another system, we would likely write some helper code anyway, and would be able to leverage that under the hood later, without loosing the work any developers put in on top of this code.

This system has already been useful in both ci_hsc_gen3 as well as a new ci_imsim package, so it seems to fit our needs well.

Nate Lust I'd still like you to explain why we need the complexity at all rather than the shell script approach of pipelines_check. Most of the complexity is in the "make the graph and execute the graph" isn't it? Which is already external. Does your system automatically support rerunning a partially executed graph from that point? Why can't that be a part of pipetask directly?

1) Re-usability, I find it easier to share functionality and remix it using something like python
2) Edit-ability, It is my opinion, and is shared by some others that I talked with, that most people are much more comfortable making changes (especially in something more than a basic script) and maintaining something that is in python vs a shell script
3) Abstraction around maintaining filesystem checkpoints resetting and such, is better in python than shell

This system supports arbitrary checkpoints between commands. This is useful when working on say the pipelines bit of something, and don't want the "setting up the buttler" stuff yelling at you that things already exist for instance. We of course could handle graph failure/rerun things within pipetasks, but we can also split it up into different commands if you want basically instant results without graph queries to run and determine existence etc.

Nate Lust what do you want to do with this RFC? I think if you tweak the phrasing a bit you can adopt it since ci_imsim is already released and using it and it does not affect lsst_ci or lsst_distrib.