Working from the DM-15862 ticket branch, I've created a DM-16467 with an incomplete solution.  Currently, there are three issues:

1. raw data is loaded as a DecoratedImageU, but IsrTask expects it to be an Exposure.  Converting the received DecoratedImageU into an Exposure doesn't work, as the generated Exposure does not return a usable VisitInfo, and does not have a Detector associated.
2. Calibration data is not correctly returned by the query to the gen3 database.  A query/subquery of the form "select * from Exposure JOIN Dataset ON (valid_first < datetime_begin AND valid_last > datetime_end) WHERE Exposure.exposure = 903988 AND detector = 16 and dataset_type_name = 'bias';" needs to be done to get the correct (in this case) bias.  This will eventually supersede the IsrTask.readIsrData/IsrTask.getIsrExposure code, which currently fetches ~12 images/products.
3. Data queries that result in a single quanta to run (such as query two below, which restricts down to a single detector) fail to run as something in the code expects an array.  Removing the "Detector.detector=16" constraint allows it to continue to IsrTask.run().

Beyond these two issues, I can use the debugger to check that the code is reaching IsrTask.run(), and the graphs constructed for the raw input exposure look plausible based on my tests on ci_hsc.

Test commands for reference:

stac --clobber-config --clobber-versions -b ~/dev/DM-16467/ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" qgraph -t IsrTask --instrument-overrides --qgraph-dot ~/x.dot

stac --clobber-config --clobber-versions -b ~/dev/DM-16467/ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" run -t IsrTask --instrument-overrides

Last point was caused by a bug in passing arguments to registry.associate() method. Fixed by DM-16481, already in pipe_supertask master, you can clone it if you prefer not to wait until next weekly release.

For calibrations we need to add special handling of ExposureRange unit to pre-flight. I have made DM-16482 to work on that, it is of non-trivial kind but I hope to close it this week.

Christopher Waters, I have just merged daf_butler DM-16482 branch into master and that should take care of the calibration issue, flat/bias should now be correctly found and added to Quanta. Note that units for those should be:

bias: Instrument, Detector, ExposureRange

flat: Instrument, Detector, PhysicalFilter, ExposureRange

The complete ISR process does not work correctly at this time for this ticket branch:

• thumbnails do not have the needed daf_butler formatter to be written (DM-17381)
• fringe data for y-filter data is not read correctly (DM-17152)
• crosstalk sources are not generated for inter-CCD sources (DM-17169)

These issues do not prevent ci_hsc data from running, although additional calibration data must be ingested to add the brighter-fatter kernel and the HSC defects.  I have been adding those products using the attached python script, which will need to be superseded by a more permanent solution.  The commands to ingest products are:

calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/bfKernel/brighter_fatter_kernel.pkl `pwd`/ci_hsc/DATA bfKernel --valid_first '1977-01-01T00:00:00.000001' --valid_last '2095-01-01T00:00:00.000001' --instrument HSC --init

calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/defects/2013-01-31/defects_0.fits `pwd`/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --init --detector 0

calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/defects/2013-01-31/defects_1.fits `pwd`/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --detector 1

The gen2 behavior has not changed, and a comparison of output files shows that both generations have pixel-identical results, with the exception of changes in mask plane labeling.

Looks pretty good to me. I have put some comments in the PR. They are mostly suggestion level.

The most concerning thing to me is the hard coding of obs_subaru information, but, as is mentioned on this ticket, we are still shimming some aspect of things.

To be clear:  while testing ISR on `ci_hsc`, I often needed to start over from scratch and try again.  To make this easier and keep things consistent, I migrated as much as possible into script files.  This meant I could regenerate a usable `ci_hsc` gen3 repository for ISR by running:

#Clear old repo:

rm ./ci_hsc/DATA/gen3.sqlite3 ./ci_hsc/DATA/butler.yaml

#Regenerate repo:

cd ./ci_hsc && makeButlerRepo.py DATA && python ./bin/gen3.py

#Ingest calibs:

bash ./add_bfKernel.sh

bash ./add_defects.sh

 The two bash scripts encapsulate the `calibtool` commands for both calib types, and converts the brighter-fatter kernel to a python3 pickle using `pickletool`.