Eric Morganson [X] (Inactive) added a comment - 26/Mar/21 10:32 PM As a note: this is not specific to the z_band. That's just the error I copied and pasted.

Jim Bosch added a comment - 26/Mar/21 10:36 PM There may be a workaround in config-land (I'm not super familiar with this corner of it, so I'm not sure) but I think there must be a code bug, too; we should not be creating a FilterLabel with no bandLabel anywhere that I can think of.

Jim Bosch added a comment - 26/Mar/21 10:51 PM The problem is in LsstCamMapper: https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/lsstCamMapper.py#L462 Will have a branch shortly.

Jim Bosch added a comment - 26/Mar/21 10:57 PM Ok, branch tickets/ DM-29446 of obs_lsst should fix this, and I've cherry-picked that onto branch u/jbosch/w.2021.12+ DM-29446 , which has no other changes since w_2021_12 , in order to maximize consistency with everything else (I presume) you're running. Care to give that a try?

Tim Jenness added a comment - 26/Mar/21 11:00 PM Is this related to DM-29335 ?

Krzysztof Findeisen added a comment - 27/Mar/21 12:02 AM - edited I took a look at the branch, and I'm a bit confused that LsstCamMapper bypasses the filter handling in _standardizeExposure and then rolls its own. What's the reason for this? The best solution for DM-29313 (a more generic version of the filter population problem) may depend on how important it is for obs_lsst to not use generic filter code...

Eric Morganson [X] (Inactive) added a comment - 28/Mar/21 9:14 PM - edited Getting the same error. Used: . /software/lsstsw/stack/loadLSST.bash setup lsst_distrib -t w_2021_08 setup -k -v -r /home/emorgan2/SOFTWARE/obs_lsst singleFrameDriver.py /datasets/DC2/repoRun2.2i --rerun w_2021_08/ DM-28966 /sfm --batch-type slurm --mpiexec='-bind-to socket' --job dc2sfm08 --cores 24 --time 900 --id visit=254380^254381^254407^271920^400440^419000^419806^430094^466279^479264^480908^484236^484266^174534^177481^192355^204706^204708^211099^211100 --configfile /home/emorgan2/HSC/DC2/filter_config.py --clobber-config --clobber-versions; config.processCcd.calibrate.astromRefObjLoader.filterMap = {f'{_} sim_1.4': f'lsst {_}_smeared' for _ in 'ugrizy'} config.processCcd.calibrate.photoRefObjLoader.filterMap = {f'{_} sim_1.4': f'lsst {_}_smeared' for _ in 'ugrizy'} Where: cd /home/emorgan2/SOFTWARE/obs_lsst git branch master remotes/origin/tickets/ DM-29446 Sample log: https://lsst.ncsa.illinois.edu/~emorgan2/DM-28966/dc2sfm08.o49930 Includes many iterations of 24263 FATAL 2021-03-28T16:15:28.228-0500 singleFrameDriver: Failed on dataId= {'visit': 159507, 'run': '159507', 'raftName': 'R02', 'expId': 159507, 'detectorName': 'S11', 'detector': 13} : RuntimeError: FilterLabel(physical="g_sim_1.4") has no band. RuntimeError: FilterLabel(physical="g_sim_1.4") has no band.

Eric Morganson [X] (Inactive) added a comment - 29/Mar/21 2:06 PM I realized I don't really understand the -k option in setup. So I reran with setup -v -r /home/emorgan2/SOFTWARE/obs_lsst And got the same result.

Jim Bosch added a comment - 29/Mar/21 2:22 PM - edited Eric Morganson [X] , So, Lauren MacArthur spotted a bug in my code on the PR, which I've now fixed (on both tickets/ DM-29446 and u/jbosch/w.2021.12+ DM-29446 ). She reports that with that branch and this config override together, single-frame processing works for her (at least I think that's what she meant): DC2_filterMap = {band:'lsst_%s_smeared' % (band) for band in 'ugrizy'} config.calibrate.astromRefObjLoader.filterMap = DC2_filterMap config.calibrate.photoRefObjLoader.filterMap = DC2_filterMap To use that branch of obs_lsst, I would recommend doing the following: $ setup lsst_distrib -t w_2021_12 $ git clone https://github.com/lsst/obs_lsst.git $ cd obs_lsst $ git checkout u/jbosch/w.2021.12+DM-29446 $ setup -v -j -r . There are ways to do this with -k ("use already set-up packages for dependencies") instead of -j ("only set up this package and don't even try to set up its dependencies"), but I think it's harder to go wrong with -j . Krzysztof Findeisen , I'm a bit confused that LsstCamMapper bypasses the filter handling in _standardizeExposure and then rolls its own. What's the reason for this? I don't understand it fully, and I don't know if it has to roll its own filter stuff, but I at least wasn't surprised to find that it did, because LSST's raws are laid out quite differently on disk from most of our other cameras (one amp per HDU), and in Gen2 that's implemented via composites and a _raw dataset type, so the code path for loading raws is overall quite unique.

Eric Morganson [X] (Inactive) added a comment - 29/Mar/21 4:36 PM Thanks on the -j -k discussion. That actually is how I thought it worked. The documentation briefly convinced me that it these affected how many directories it dug down rather than dependencies. Running with filter conversion: DC2_filterMap = {band:'lsst_%s_smeared' % (band) for band in 'ugrizy'} config.processCcd.calibrate.astromRefObjLoader.filterMap = DC2_filterMap config.processCcd.calibrate.photoRefObjLoader.filterMap = DC2_filterMap Where ".processCcd" has been added. I assume this is just the vagaries of python module loading.

Lauren MacArthur added a comment - 29/Mar/21 6:32 PM I was running processCcd.py , so it can't be there...but it needs to be when running singleFrameDriver.py .

Jim Bosch added a comment - 30/Mar/21 2:06 AM Lauren MacArthur , since you've already looked at the code and checked that it works, could you sign off on the review for this? Eric Morganson [X] , I'm going to proceed on the assumption that this fix will work for you, too, and that if you run into a new problem it's best as a new ticket. But please let me know if this doesn't seem to be working.

Lauren MacArthur added a comment - 30/Mar/21 2:45 AM Yep...I confirm this change along with the filterMap  config override worked for me on a processCcd.py  run using w_2021_13 + your tickets/ DM-29446 branch.  Presumably the filterMap  issue needs resolving (on another ticket, though!)?

Eric Morganson [X] (Inactive) added a comment - 30/Mar/21 6:51 PM - edited This fix has not worked for me. To show current setup: setup lsst_distrib -t w_2021_08 # also lsst-distrib w_2021_12 setup -v -r /home/emorgan2/SOFTWARE/obs_lsst # Have also tried with -k and -j option singleFrameDriver.py /datasets/DC2/repoRun2.2i --rerun w_2021_08/DM-28966/sfm --batch-type slurm --mpiexec='-bind-to socket' --job dc2sfm08 --cores 24 --time 900 --id visit=159471^159491^159507^183772^183773^183818^183912^193780^193781^193827^194862^221574^221575^221577^221614^221615^221616^254358^254359^254379 --configfile /home/emorgan2/HSC/DC2/filter_config_2021_12.py --clobber-config --clobber-versions; filter_config_2021_12.py is: DC2_filterMap = {band:'lsst_%s_smeared' % (band) for band in 'ugrizy'} config.processCcd.calibrate.astromRefObjLoader.filterMap = DC2_filterMap config.processCcd.calibrate.photoRefObjLoader.filterMap = DC2_filterMap Have also tried: DC2_filterMap = {f'{_}_sim_1.4': f'lsst_{_}_smeared' for _ in 'ugrizy'} DC2_filterMap.update({_: f'lsst_{_}_smeared' for _ in 'ugrizy'}) config.processCcd.calibrate.astromRefObjLoader.filterMap = DC2_filterMap config.processCcd.calibrate.photoRefObjLoader.filterMap = DC2_filterMap And config.processCcd.calibrate.astromRefObjLoader.filterMap = {f'{_}_sim_1.4': f'lsst_{_}_smeared' for _ in 'ugrizy'} config.processCcd.calibrate.photoRefObjLoader.filterMap = {f'{_}_sim_1.4': f'lsst_{_}_smeared' for _ in 'ugrizy'} Produces error: RuntimeError: FilterLabel(physical="g_sim_1.4") has no band. Traceback (most recent call last): File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-15-g361d225+ff3912ea15/python/lsst/pipe/base/cmdLineTask.py", line 431, in __call__ result = self.runTask(task, dataRef, kwargs) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-15-g361d225+ff3912ea15/python/lsst/pipe/base/cmdLineTask.py", line 495, in runTask return task.runDataRef(dataRef, **kwargs) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_drivers/21.0.0-4-g3af6bfd+64d2251902/python/lsst/pipe/drivers/singleFrameDriver.py", line 83, in runDataRef result = self.processCcd.runDataRef(sensorRef) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-15-g361d225+ff3912ea15/python/lsst/pipe/base/timer.py", line 157, in wrapper res = func(self, *args, **keyArgs) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_tasks/21.0.0-58-g436064c8+549a115573/python/lsst/pipe/tasks/processCcd.py", line 181, in runDataRef exposure = self.isr.runDataRef(sensorRef).exposure File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-15-g361d225+ff3912ea15/python/lsst/pipe/base/timer.py", line 157, in wrapper res = func(self, *args, **keyArgs) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-10-g17396bd+2c5eba4060/python/lsst/ip/isr/isrTask.py", line 1719, in runDataRef isrData = self.readIsrData(sensorRef, ccdExposure) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-10-g17396bd+2c5eba4060/python/lsst/ip/isr/isrTask.py", line 1148, in readIsrData if self.config.doFringe and self.fringe.checkFilter(rawExposure) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-10-g17396bd+2c5eba4060/python/lsst/ip/isr/fringe.py", line 257, in checkFilter return checkFilter(exposure, self.config.filters, log=self.log) File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-10-g17396bd+2c5eba4060/python/lsst/ip/isr/isrFunctions.py", line 897, in checkFilter elif thisFilter.bandLabel in filterList: Sorry I have been late responding. I have been tweaking a lot of knobs.

Eric Morganson [X] (Inactive) added a comment - 30/Mar/21 7:51 PM Worked with Jim and my own git problems. Can confirm it works now.

Lauren MacArthur added a comment - 30/Mar/21 8:08 PM Phew...was just starting to try to reproduce (failing so far!)

Yusra AlSayyad added a comment - 21/Apr/21 6:15 AM Preops because DC2-specific