Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
The current design model of scarlet is to include only core functionality in the package itself, with sample source classes and initialization scripts. The idea being that most users will be working with complicated data and their exact parameters for deblending will be too diverse to include in a command line function or set of parameters. Instead they are expected to use a subset (or modified version) of our initialization functions and use scarlet source classes as templates to create their own custom source classes.
As a result I created the `scarlet_extensions` package to be a place where high level functions and classes are stored for more specialized deblending. This is where I include the logic to initialize sources and (eventually) will contain our own specialized source classes for the DM science pipeline implementation.
In order to keep this code accessible to users not using the science pipelines I have kept it as a separate package which is copied into `meas_extensions_scarlet` when there are changes. A more streamlined approach is to just include `scarlet_extensions` in the conda environment, which is what this RFC proposes.
The only real dependencies for `scarlet_extensions` are `numpy`, `matplotlib`, and `scarlet`, although to run the regression tests also requires a `boto3` import that is hidden in the functions that require it.
Attachments
Issue Links
Activity
...I thought that the preferred method was to include 3rd party packages in the conda environment...
It might be worth unpacking that a little, just for clarity and future reference. Not sure if it changes the answer here.
First, there's a legalistic definition of what a “third party” package is. I'm not sure we have a hard line here. Since (per the text of the RFC) scarlet_extensions is a Fred creation, and since Fred is on the Rubin payroll, on the face of it it seems hard to argue that it's a third party package. However, were you to suggest that it's primarily being developed off-project (in your spare time, by your collaborators, etc), that might be a good approach.
More useful, perhaps, is a consideration of the technical and practical aspects of the packaging. If this is something that is already available through Conda, that is expected to need relatively infrequent changes, and that Rubin developers won't generally work on directly, then it seems appropriate to put it into Conda. If it's something that we'll often working on as part of regular Pipeline development, then it seems appropriate to package it for EUPS.
I think we're splitting hairs here. The idea behind scarlet_extensions is to... well extend the functionality of scarlet. So yes, right now all of the merged code has been done by me, on project time, but for example Remy has a branch with a great detection algorithm that will hopefully be merged soon and the idea is for Peter's students and other scarlet users to contribute their code to the repo. So in the near future I imagine that the majority of the repo will be written by non-Rubin collaborators. So in this way I don't see it as any different than scarlet in it's first days and I think that it fits into that same 3rd party repo territory.
In theory, I don't mind having a community developed package in the stack, but the more that I think about it the more this seems problematic. The main reason that I'd like it to be a 3rd party package is because DM has a very strict (and non-standard) style guide (for example using camel-case instead of underscores) that I think will create a barrier that might prevent many people from merging their code. For example, I don't think it's fair to require Peter's students (or Peter and Remy for that matter) or the graduate students currently using scarlet to have to adhere to the DM style-guide, which might result in useful features remaining on forks that we'll never see.
So my preference would be to either use the scarlet and proxmin model, where we keep the repo 3rd party and have an LSST fork, or to allow this package to diverge from the style guide. That latter seems like more of an exception, so I would recommend just using the scarlet/proxmin model that we already use for code related to this project.
Fred Moolekamp
added a comment - I think we're splitting hairs here. The idea behind scarlet_extensions is to... well extend the functionality of scarlet. So yes, right now all of the merged code has been done by me, on project time, but for example Remy has a branch with a great detection algorithm that will hopefully be merged soon and the idea is for Peter's students and other scarlet users to contribute their code to the repo. So in the near future I imagine that the majority of the repo will be written by non-Rubin collaborators. So in this way I don't see it as any different than scarlet in it's first days and I think that it fits into that same 3rd party repo territory.
In theory, I don't mind having a community developed package in the stack, but the more that I think about it the more this seems problematic. The main reason that I'd like it to be a 3rd party package is because DM has a very strict (and non-standard) style guide (for example using camel-case instead of underscores) that I think will create a barrier that might prevent many people from merging their code. For example, I don't think it's fair to require Peter's students (or Peter and Remy for that matter) or the graduate students currently using scarlet to have to adhere to the DM style-guide, which might result in useful features remaining on forks that we'll never see.
So my preference would be to either use the scarlet and proxmin model, where we keep the repo 3rd party and have an LSST fork, or to allow this package to diverge from the style guide. That latter seems like more of an exception, so I would recommend just using the scarlet/proxmin model that we already use for code related to this project. If this is not in conda but instead is a third-party EUPS package with an lsst-dev branch in a git fork, the DM-CCB is fine with this.
Ok, I'll mark this RFC as adopted and implement this in the few weeks once the processing of RC data with `meas_extensions_scarlet` is underway.
Fred Moolekamp
added a comment - Ok, I'll mark this RFC as adopted and implement this in the few weeks once the processing of RC data with `meas_extensions_scarlet` is underway. 
Ok, so we can just change this RFC to transferring `scarlet_extensions` to the lsst github account and making it a dependence for `meas_extensions_scarlet`.