Status: In Review
Fix Version/s: None
Sprint:AP S23-2 (January), AP S23-3 (February), AP S23-4 (March)
Add the DC2 real bogus trained model to the new CI dataset (
DM-34845). In addition to adding the data to the git-lfs repo, add the trained model to the ingest step of ap_verify. This will require defining the trained model for the butler, and it is suggested that we treat the model the same way as we treat our "curated calibration" datasets.
DM-36743 Add the real-bogus PipelineTask to ap_verify for the DC2 CI dataset
- is blocked by
DM-34845 Create ap_verify dataset for DC2
DM-37478 Design and implement "model-packages" and update existing code accordingly
- is duplicated by
DM-35754 Assist with creating DC2 testdata repository for CI
- relates to
DM-37157 Rewrite meas_transiNet git history to remove large file(s)
I actually missed the last part of the meeting. That's why I'm wondering about the discussed storage options for the model packages.
I can think of these options – and indeed I don't think we need to limit ourselves to only one:
1- local storage: model-packages that are local to the meas_TransiNet repository and are always fetched with it (e.g. the dummy model). We can probably access them just using a relative path :think:
2- neighbor git data-repository: I suppose these can also be accessed using a relative path, assuming the repository is always cloned into its default location.
3- butler repository: to be discussed.
I guess we are now waiting for John Parejko to get a pre-confirmation re. feasibility of these implementations.
I don't have the full context here, but I read Ian's summary as saying that local storage is out (I fully agree), neighbor git repository is in (accessed via eups environment variable, not relative path), and Butler repository seems unlikely (since this is more like code than data).
The relevant model package is now being stored in the neighboring package (rbClassify_data). Here's the relevant PR:
I think this addresses the purpose of this ticket. Please let me know if something is missing.
Am I correct in assuming that finishing DM-37478 will make this issue obsolete? If not, what is the new scope?
Whether an environment variable in a config field will get properly expanded depends a lot on how that path is actually evaluated – it's certainly not guaranteed.
I'd recommend testing it with your existing setup first.
Much later EDIT: a meaningless objection, since it's your code that decides how the config field is evaluated. Apologies for the noise!