Details
-
Type:
RFC
-
Status: Withdrawn
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
The goal of this RFC is to add a new package to `lsst_distrib` which will be a library used in the creation of ci_* packages. The proposed name for this package is `ci_builder`.
This is meant to solve the issues that arise from using `scons` to orchestrate running ci jobs, as well as some other pain points developers have run into.
Currently the plan is to use this system to implement a new `ci_imsim` package and overhaul `ci_hsc_gen3`, but it is expected to be generally useful for many other ci tools, and potentially other tools in the future.
Attachments
Issue Links
Activity
Nate Lust I'd still like you to explain why we need the complexity at all rather than the shell script approach of pipelines_check. Most of the complexity is in the "make the graph and execute the graph" isn't it? Which is already external. Does your system automatically support rerunning a partially executed graph from that point? Why can't that be a part of pipetask directly?
1) Re-usability, I find it easier to share functionality and remix it using something like python
2) Edit-ability, It is my opinion, and is shared by some others that I talked with, that most people are much more comfortable making changes (especially in something more than a basic script) and maintaining something that is in python vs a shell script
3) Abstraction around maintaining filesystem checkpoints resetting and such, is better in python than shell
This system supports arbitrary checkpoints between commands. This is useful when working on say the pipelines bit of something, and don't want the "setting up the buttler" stuff yelling at you that things already exist for instance. We of course could handle graph failure/rerun things within pipetasks, but we can also split it up into different commands if you want basically instant results without graph queries to run and determine existence etc.
Nate Lust what do you want to do with this RFC? I think if you tweak the phrasing a bit you can adopt it since ci_imsim is already released and using it and it does not affect lsst_ci or lsst_distrib.
While I still think this would be a useful addition to the stack in many ways, There is no current need to have it included, and so I don't want to spend the energy advocating for it at this time. I will introduce another RFC if a new need arises
Basically it is tailored exactly to our needs, so there is "less to it". Everything we are doing is also technically possible with scons too (though certainly harder than say Parsl). Using any other system would likely still involve writing wrappers or customization classes to stream line use, which would end up at the total scale of code proposed.
I do see your point, and it is always a double edged sword between leveraging upstream work and adapting it, or doing everything in house, but you do it all. In this case by having something tailored, it is just a small bit of code focused on our needs, with a low learning / maintenance overhead.
However this also does not lock us in. As I mentioned if we did decide we needed the functionality of Parsl, or another system, we would likely write some helper code anyway, and would be able to leverage that under the hood later, without loosing the work any developers put in on top of this code.
This system has already been useful in both ci_hsc_gen3 as well as a new ci_imsim package, so it seems to fit our needs well.