OK, this is mostly a testing ticket. I built w40 source tables at u/yusra/RC2/w40 in gen3 /repo/main and private/yusra/RC2/DM-31825 in gen2 /datasets/hsc/repo.

So far I've tested:

gen3:

• jointcal: pipetask run -b /repo/main/butler.yaml -o u/yusra/RC2/w40 -d "" --register-dataset-types -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#jointcal &> /project/yusra/dp0tables/DM-31825/jointcal2.log
• fgcm: pipetask run -b /repo/main/butler.yaml -o u/yusra/RC2/w40 -d "" --register-dataset-types -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#fgcm &> /project/yusra/dp0tables/DM-31825/fgcmcal.log

gen2:

• jointcal:

  jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id ccd=0..8^10..103 visit=1202^1204^1206^1208^1210^1212^1214^1216^1218^1220^23692^23694^23704^23706^23716^23718 filter=HSC-R tract=9813 --no-versions &> /project/yusra/dp0tables/jointcal_gen2.log &

  jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id ccd=0..8^10..103 visit=1202^1208^1220^23716^23718 filter=HSC-R tract=9813 -j 4 --no-versions &> /project/yusra/dp0tables/jointcal_gen2_few_visits.log &

• fgcm: Lauren MacArthur or Eli Rykoff I need help with this one. I don't think I've ever ran this on gen2, and can't find instructions. I don't think this was right: {{fgcmBuildStarsTable.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y --no-versions &> build_stars_gen2.log }}}

Opened an initial jointcal PR. The "sourceId" in the columns list to be read from the parquet does no harm.
It's smart enough to recognize it, but bring it back as the index (same as if you have read the whole table). I don't really care if it's there or not. If you prefer gone, it's gone.

I don't THINK I need to make any changes to fgcmcal, but I want to test with gen2 before concluding that.

The effective instructions I follow for gen2 fgcm runs are in https://github.com/lsst-dm/gen2gen/blob/master/FGCM/all_steps.sh (you’ll need the configs files in https://github.com/lsst-dm/gen2gen/blob/master/FGCM/).  I’m not sure if it’s overkill to run the entire script…

So Yusra AlSayyad I did run your suggested command and the star ingestion worked fine (which is relevant for this ticket). And I have confirmed that I don't use sourceId in fgcmcal so this isn't a problem. But it then crashed later on because the default fgcmcal LUT that's in the gen2 repo is not compatible with the current code; we use a separate one for the RC2 runs (because of the different filters), and fixing this is not a high priority since gen2 is disappearing anyway and we're not running it on non-RC2 data.

I concur that the fgcmcal is good to go for gen2/gen3.

For reading in sourceId in the columns list for jointcal, I'd like to leave it there because it is a column we want to make sure we're reading. I am unhappy that we're depending on idiosyncrasies of pandas in our code, but that's a discussion for another day.

In that case, this is ready for review.

Already reviewed!

Thanks!