I support this. Defaults that usually work (and are easily overridden) are much better than defaults that never work (and require overriding).

It would be helpful to clearly define what "gaia dr2" and "ps3_pv1" actually mean as part of implementing this - i.e., paper citations and/or catalog release versions, columns used, where one can find these two specific ones, how to create a subset of them, etc., as the documentation on reference catalogs is sporadic at best.

For reference (ha), here are the CalibrateTask reference-catalog-related references I can quickly find:

https://developer.lsst.io/services/datasets.html#reference-catalogs

https://pipelines.lsst.io/modules/lsst.meas.algorithms/creating-a-reference-catalog.html

https://community.lsst.org/t/creating-and-using-new-style-reference-catalogs/1523

https://community.lsst.org/t/gaia-dr2-reference-catalog-in-lsst-format/3901 

It would be helpful to clearly define what "gaia dr2" and "ps3_pv1" actually mean as part of implementing this

The example setDefaults above notes the specific catalogs I am referring to. The documentation for those catalogs is provided at lsst-dev:/datasets/refcats/htm/README.txt (as noted in your first dev guide link) with details contained in a README.txt file in the specific catalog sub-directory.

Indeed, I'm very glad those READMEs exist, thanks for the reminder.

I am a little concerned about having defaults set that only work for people who use our software on lsst-dev. I don't want to block you from implementing this RFC, but I do think this is important, as many community members run CalibrateTask. Can the newly sanctioned default reference catalogs be hosted somewhere publicly visible and linked to in documentation?

Can the newly sanctioned default reference catalogs be hosted somewhere publicly visible and linked to in documentation?

This is a good point, but is orthogonal to this RFC. cal_ref_cat doesn't work for anyone, so at least this would be one step forward. The refcats are quite large (~400GB for PS1 and ~300GB for Gaia), so where we could host them is a good question. If this RFC is accepted, I can talk with SQuaRE or NCSA about hosting options.

I initially didn’t pay attention to this RFC, because obviously yes we want gaia and ps1 as default refcats. John Parejko pointed out during DM-25184 that it is actually proposing a relaxation of the "no filter names outside of obs packages" rule (a substantive change to how we handle filterMaps).

Currently, we consider filterMaps and filter names camera-specific. You won't see any filter names outside of obs_packages or one off unit tests LloadIndexReferenceObjects, currently sets a default ref_dataset_name but no default filterMap. We load those in all the obs_packages. This makes it obvious to users writing new obs packages, or tasks for existing obs_packages that they need to think about it.

So, I'm fully in support of:

astromRefObjLoader.ref_dataset_name = "gaia_dr2_20200414"

but not so much:

 astromRefObjLoader.filterMap = {"u": "phot_g_mean",

                                "g": "phot_g_mean",

                                "r": "phot_g_mean",

                                "i": "phot_g_mean",

                                "z": "phot_g_mean",

                                "y": "phot_g_mean"}

This is when Jim Bosch swoops in and assures me that filters are handled totally differently in gen3 and everything will be fine.

This is when Jim Bosch swoops in and assures me that filters are handled totally differently in gen3 and everything will be fine.

Not quite.  Gen3 is also plagued by pre-existing filter definition and mapping problems, possibly even more than Ge2n.  Rolling out Gen3 may force us to finally address them, but it's not something we already have fixed on our side.

I think my comment is wrong about us abiding by this principle thus far. There is 'B':'g' , 'V':'r', 'I':i' specified somewhere. We were just getting lucky that HSC has no filter named B/V/I and that PS1 DID have columns called g, r, i.

[yusra@lsst-dev01 w22reg]$ grep filterMap /datasets/hsc/repo/rerun/RC/w_2020_22/DM-25176-sfm/config/singleFrameDriver.py 

config.processCcd.charImage.refObjLoader.filterMap={'B': 'g', 'V': 'r', 'R': 'r', 'I': 'i', 'r2': 'r', 'i2': 'i', 'N387': 'g', 'N468': 'g', 'N515': 'g', 'N527': 'g', 'N656': 'r', 'N718': 'i', 'N816': 'i', 'N921': 'z', 'N926': 'z', 'N973': 'y', 'N1010': 'y', 'I945': 'z', 'HSC-G': 'g', 'HSC-R': 'r', 'HSC-R2': 'r', 'HSC-I': 'i', 'HSC-I2': 'i', 'HSC-Z': 'z', 'HSC-Y': 'y', 'NB0387': 'g', 'NB0468': 'g', 'NB0515': 'g', 'NB0527': 'g', 'NB0656': 'r', 'NB0718': 'i', 'NB0816': 'i', 'NB0921': 'z', 'NB0926': 'z', 'NB0973': 'y', 'NB1010': 'y', 'IB0945': 'z'}

<snip>

Yusra AlSayyad: those come from obs_subaru/config/filterMap.py, which I assume is there for the original SuprimeCam?

That said, I don't think the question of default filterMaps is a gen2 vs. gen3 issue: in both cases, the name used as a key is the abstract filter name, and those mappings should definitely be consistent from camera to camera, i.e. "u" should map to the same filter in a given refcat, whether it's the LSST or DECam or HSC "u". If that's not the case, our use of the term "abstract filter" is meaningless.

We should bear in mind that when we finally overhaul the filter singleton, we probably need to start using strings that include instrument names in maps like these. Certainly that's true in any context where something like "i2" might appear - there is no generic i2, so it shouldn't appear without an instrument name to disambiguate it; HSC-I2 is a particular i filter.

For the filterMaps, particularly for astrometry, "ugrizy" seem like they all have the same meaning for every camera we have and would have the same refcat filter that they should be matched to. I agree that e.g. "i2" should not appear in a default config.

I'd argue for a bit more than that:

• "i2" shouldn't appear anywhere (it should be "HSC-I2" or just "i")
• no dict that might ever have "HSC-I2" in its keys should ever have "i" in its keys (ditto for values - though a mapping from one to the other is of course fine).

In other words, those are different kind of filters and unless we're conforming very high level inputs from science users, we should not mix them.

Even though for these specific purposes (finding the refcat filter to use with the given survey filter, but not for doing any photometric conversions), "i" and "i2" are to be treated identically?

If you're saying that the keys of this dictionary are logically abstract filters, and this is basically a workaround for the fact that the afw Filter stuff gives us strings like "i2" instead of "i" sometimes, fine - but we should acknowledge that as a workaround, and otherwise be consistent about these always being camera-agnostic filter names.

Well, mostly fine - I'm a little worried that we're trading away flexibility to get convenience by saying that the keys here are camera agnostic. Should we always assume there is one abstract, instrument-agnostic filter associated with a single-epoch image, which does have a well-defined physical, instrument-specific filter? Gen3 does assume exactly that in its current data model, but I've long planned to eventually make that a many-to-many relationship with some kind of "filter system" that defines a particular mapping from physical->abstract. Put another way: we're starting with a well-defined physical filter when we want to use this mapping. If we need to obtain an abstract filter to use as a key for this mapping, we might need to also specify what filter system we're using.

A lot of these concerns are not really things we can do anything about yet - I'm just trying to make the point that all of this RFC is really building on sand because filters are such a mess overall, so we should aim to minimize disruption and effort in anything we do right now, rather than aiming to really "get it right".

I just wanted to chime in, and say this leads to the larger questions of how configs are organized in general. Up until now we have always associated a config with a python task in a 1:1 way. The built in assumption was that a task had one role to to, so there would be one configuration for that task, with overrides for instruments, and possibly a specific processing.

In keeping with the theme "gen 3 will fix everything", I want to highlight the role that pipelines will play in the near future and how they will fit in with configs. Pipelines are a way to define an action or outcome, and all the steps needed to achieve that, such as single frame processing or difference imaging.

Each of these actions are comprised of one or more tasks and their configurations, and you can still override pipeline configurations. This solves a problem that is hinted at with this rfc, and has long been debated without realizing the underlying problem is the 1:1 config overrides in obs packages to tasks. While there is certainly camera specific config options that must be set for a given instrument because of things like filters, or electronics or the nature of the data, we have fallen into the trap of putting things into camera configs because a) its a centralized location and b) they are related to the fact that data from those cameras are often used in to do specific actions, so configs for those actions were put in the obs package.

This model breaks down when tasks are not used in a standard way, or are used more than once, and causes the need for config override files that users maintain and pass around. One such example is the monthly rc processing that maintains its own flow and configs.

As mentioned pipeline files are tied to specific actions a user wants to do, and provide a way for configs needed for that action to be stored with the definition of that action. A good example of that can bee seen here https://github.com/lsst/cp_pipe/tree/master/pipelines. ISRTask is used in each of these pipelines, but it has a different role, and thus a different config, though there is still a notion of shared defaults at a camera level.

I said all that to say I hope we can start a discussion where we decide what is the bare minimum config to task configuration needed for a camera, and then start associating configs with pipelines. If there is a need to share configs between pipelines we would then do that in a separate config override file that each pipeline can read and override, rather than what we do now where most overrides are stored at the camera level or each user needing to maintain their own override files.

This may be long and rambling and unclear, so I am happy to clarify anything if people ask, but I wanted to get this comment out there.

After offline discussion with Jim Bosch, we decided that the best approach would be to use the defaultFilter config in the refcats for Gaia DR2, since there really isn't a "filter map", but rather "always use this filter, no matter what". This also should make it a bit more clear that Gaia is to be used for astrometry only, and we can make that more clear in the task documentation (e.g., add a note to the defaultFilter docs to the effect of "this option is useful for astrometric refcats, but should not be used for refcats that are used for photometry"). I think this takes care of Yusra AlSayyad and Jim Bosch's objections about defining default filterMaps, and it even further simplifies refcat configuration: if you're using this RFC's Gaia defaults, you won't have to specify anything at all in your obs package.

However, some experimenting with this in jointcal suggests that our defaultFilter implementation in the loader is broken: if defaultFilter is supplied without a filterMap, and the user specifies filter=something in loadSkyCircle() (as one normally does, because when you load the refcat you don't know how its filter mappings are configured), getRefFluxField() will fail. I think the fix is fairly trivial (try camFlux if x_camflux doesn't exist), but this is yet another point on the "our refcat filter system poorly specified and too complicated" list. Whether finally implementing RFC-624 will help with that is a separate question.

John Parejko if this RFC is still wanting discussion can you please change the end date?

Thanks for the reminder: I've modified the proposal to use anyFilterMapsToThis, which was added in DM-26007. That should alleviate the concerns about having filter names in obs_base, and makes it very clear what filter is to be used for any Gaia comparison.

Adopting as modified with anyFilterMapsToThis for Gaia.

John Parejko can you please assign a triggered DM ticket to this RFC?

John Parejko can you please clarify what DM ticket is triggered by this RFC?

Thank you Tim Jenness: I swore I'd made the implementation ticket, but apparently I forgot.

DM-27013

Has this RFC been implemented? The triggered ticket has been marked done.