Ok, I believe I've got things working as desired.  The commit messages and docs hopefully convey the function and purpose of these new datasets.  I have persisted tables for all 6 bands in RC2 tract 9813 (COSMOS) and have checked that they all have the same number and ordering of rows as the associated objectTable_tract dataset (NaNs have been inserted for missing patches for a given band, as is done for the objectTable_tract tables). They can be accessed via, e.g.:

In [1]: from lsst.daf.butler import Butler

In [2]: rootDir = "/project/hsc/gen3repo/rc2w02_ssw03"

   ...: butler = Butler(rootDir, collections="u/lauren/DM-28389")

In [3]: gQaTable_forced = butler.get("qaTractTable_forced", tract=9813, band="g")

In [4]: objectTable_tract = butler.get("objectTable_tract", tract=9813)

In [5]: gQaTable_forced.index.equals(objectTable_tract.index)

Out[5]: True

(the last line is the check on the matched number of rows and sorting of the catalogs, which is required to allow for ease of joint use, e.g. being able to use a single boolean expression to filter on both).

The pipeline command line to create these tables looks like:

pipetask run -b /project/hsc/gen3repo/rc2w02_ssw03 -i HSC/runs/RC2/w_2021_02 -o u/lauren/DM-28389 -p $OBS_SUBARU_DIR/pipelines/makeQaTractTables.yaml -d "instrument='HSC' AND tract=9813 AND band='r' AND skymap='hsc_rings_v1'"

(but leaving out the -i HSC/runs/RC2/w_2021_02 on subsequent runs.)

There is a generic pipeline in analysis_drp. The override in obs_subaru is to add a filterMap between the generic "band" names and the physical HSC filters (this is required as the _obj tables are indexed on the latter).

I'm putting you both as reviewers as I think Sophie Reed should comment on whether the tables provide what is needed for the the scripts she is working on for this repository, but I think maybe Jim Bosch (or someone else he can appoint that is suitable) should at least weigh in on the Gen3 aspects (it's my first Gen3 task, so I very likely have some less-than-ideal approaches in there!)

Already got some tips for improvement, so putting back to In Progress to clean thing up before sending it back for review (but the data products themselves won’t be changed, so feel free to play with them to see if they are of use).

Ok, back in for review.  I've added TODO s to remove the filterMap config song & dance once DM-28479 lands, but let me know if it's better to hold off merging this until that gets resolved.

I'm currently creating the tables for all three RC2 tracts (only 9813 is done so far, well, almost...still working on y, but they should all be done by tonight).  They can be accessed as indicated in the first comment on this ticket. 

I think I'm done reviewing (one final comment on the PR, in response to a question there). Thanks for handling all of my restructuring requests good-naturedly. I'll let Sophie Reed be the one to hit Reviewed in case she wants one more look.

Thanks, Jim!  I addressed your last comment and will now wait for a final review from Sophie Reed (which will also need approval of the obs_subaru PR if we still think it's necessary at that time).

Looks good to me.

Thanks to you both!  Merged and done.