John Swinbank: obs_subaru is already in lsst_distrib

I usually install the stack from source following the instructions at https://pipelines.lsst.io/install/newinstall.html so using:

eups distrib install -t v12_1 lsst_apps

In lsst_apps the package obs_subaru is definitely not included. See for instance what I get with my installation of v12.1:

$ eups list | grep obs_

obs_lsstSim           12.1       	b2294 current b2296 fe_test v12_1

obs_sdss              12.1       	current b2294 b2296 fe_test v12_1

obs_test              12.1       	current b2294 b2296 fe_test v12_1

I checked the instructions for installing the stack via conda and the documentation (https://pipelines.lsst.io/install/conda.html) tells me to use the command:

conda install lsst-distrib lsst-sims

Note that, unlike the installation from source, here we use lsst-distrib and not lsst_apps.

So I was wondering if there may be an inconsistency here: should I and do I get the same packages installed when installing from source or when installing via conda install? In other words, if the obs_* packages get included in lsst-distrib will they also be automagically included in lsst_apps?

I'm sorry if the answer to this is obvious, it only shows my my ignorance of the details of the packaging system.

should I and do I get the same packages installed when installing from source or when installing via conda install?

Yes, but you'll need to install the same top-level package. Try:

$ eups distrib install -t v12_1 lsst_distrib

I'm not aware of a really convenient way to check what the top-level packages will pull in other than checking their table files. As you can see from lsst_distrib's, it will give you all of lsst_apps, plus some processing middleware, a couple of measurement extensions, and obs_subaru.

I don't think (but I'm open to being corrected) that we have a documented and consistently-applied set of criteria for determining what goes in _apps and what goes in _distrib, and I don't think those names are helpful to newcomers trying to figure out what they're getting. I'd really like to see a root-and-branch rethinking of how this is organized, perhaps including the sort of system that Jim Bosch suggests above — but I'm worried that converging on a solution will take time, and hence my pushing to just include obs_cfht in _distrib for now, to make everybody's life easier in the short term.

...hence my pushing to just include obs_cfht in _distrib for now, to make everybody's life easier in the short term

From my user point of view, I see no problem with this solution. I was confused by the documentation of what top-level package I should be installing from source. I understand now that I should use lsst_distrib instead of lsst_apps.

Here's my suggestion: create an lsst_obs metapackage that contains all buildable obs_* packages other than those needed for the demo/tests. Explicitly state in lsst_obs that the packages within are supported for a given release only and that there is no guarantee that they are supported over all releases. Have lsst_distrib include lsst_obs.

We seem to have converged so I am marking this RFC as adopted. The main issue is whether this change will make it into the v13.0 release.