Yusra AlSayyad added a comment - 10/Jun/21 1:36 AM - edited FYI: I pushed a commit to ci_hsc_gen3:u/jbosch/ DM-30444 changing the required template area to 20% I plan to merge DM-30444 tonight, so your u/jbosch/ DM-30444 branch can get folded into tickets/ DM-30649 .

Jim Bosch added a comment - 15/Jun/21 2:27 PM Ok, I think this is ready for another round of review and testing.  Nate Lust , I believe I've updated pipe_base and ctrl_mpexec to reflect your review comments and our discussion.   ci_hsc_gen3 (which now has a  PR as well) now passes, a Jenkins ci_imsim run is underway, and I'm continuing to test with BPS.  So I don't think I need help testing, but if anyone wants to try it out they're more than welcome (note that the branches at /project/jbosch/tkt/ DM-30649 are intentionally not-recently-rebased, because I wanted to keep my BPS test environment as compatible with w24 as possible). Nate Lust and Yusra AlSayyad , could you take another look at the PRs (and Yusra AlSayyad , could you take ci_hsc_gen3)?

Jim Bosch added a comment - 15/Jun/21 3:34 PM My BPS testing efforts have been completely stymied by failed attempts to workaround DM-30426 (and instead breaking other things) and my own merge of an overlapping multi-package ticket ( DM-29370 ).  At this point I think I'm going to declare those tests as suffering from diminishing returns, and consider this working if ci_hsc_gen3 and ci_imsim look good in Jenkins, as the former at least should provide coverage of the new functionality now that we've removed the diffim workarounds there.  If anyone thinks otherwise, I'll just make an lsstsw stack for large-scaling testing, but I'd rather just wait for the weekly, especially because it's an odd-numbered one. I also took a look at adding some more unit tests (there's some coverage in existing ones, at least for "this ticket does no harm"), but we're operating at such a unit test deficit for GraphBuilder that my initial effort towards that should get moved to and finished on a new ticket instead (just created DM-30758 ).

Yusra AlSayyad added a comment - 16/Jun/21 2:02 AM I tested on a gen2 coaddDriver (works as expected). coaddDriver INFO: lsst-devl01:1822642: Start coadding DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '8,0'}, tag=set()) coaddDriver.assembleCoadd INFO: Coadding 1 exposures coaddDriver.assembleCoadd WARN: Could not find deepCoadd_directWarp DataId(initialdata={'visit': 29336, 'filter': 'HSC-G', 'patch': '8,0', 'tract': 9813}, tag=set()); skipping it coaddDriver.assembleCoadd INFO: Found 0 deepCoadd_directWarp coaddDriver.assembleCoadd WARN: No coadd temporary exposures found coaddDriver INFO: lsst-devl01:1822642: Finished coadding DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '8,0'}, tag=set()) I’ll assume you plan on running it on ci_hsc_gen2 too? (I stalked #34454 and #34453 ) I also tested a gen3 imageDifferenceTask,forcedPhotDiffim which worked as expected (lsst-scipipe) [yusra@lsst-devl01 pipe_tasks]$ pipetask run -b /repo/main -o u/yusra/w18-DM-30444 -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#imageDifference,forcedPhotDiffim -d "instrument='HSC' AND skymap='hsc_rings_v1' AND tract=9697 AND detector=71 AND visit=34412 > " ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 2 quanta for 2 tasks, graph ID: '1623780329.7827184-1544096' conda.common.io INFO: overtaking stderr and stdout conda.common.io INFO: stderr and stdout yielding back imageDifference INFO: Processing {instrument: 'HSC', skymap: 'hsc_rings_v1', detector: 71, visit: 34412, ...} imageDifference.getTemplate INFO: Using template input tract=9697, patch=6 imageDifference.getTemplate INFO: Using template input tract=9697, patch=7 imageDifference.getTemplate INFO: coadd dimensions=(4038, 2070) imageDifference.getTemplate INFO: template has 448218 good pixels (5.4%) ctrl.mpexec.singleQuantumExecutor INFO: Task 'imageDifference' on quantum {instrument: 'HSC', skymap: 'hsc_rings_v1', detector: 71, visit: 34412, ...} exited early: Insufficient Template Coverage. (5.2% < 10.0%) Not attempting subtraction. To force subtraction, set config requiredTemplateFraction=0. ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'imageDifference' on quantum {instrument: 'HSC', skymap: 'hsc_rings_v1', detector: 71, visit: 34412, ...} took 4.689 seconds ctrl.mpexec.mpGraphExecutor INFO: Executed 1 quanta, 1 remain out of total 2 quanta. ctrl.mpexec.singleQuantumExecutor INFO: Nothing to do for task 'forcedPhotDiffim' on quantum {instrument: 'HSC', skymap: 'hsc_rings_v1', detector: 71, tract: 9697, visit: 34412, ...}; saving metadata and skipping: ('forcedPhotDiffim', 'exposure', Input(name='goodSeeingDiff_differenceExp', storageClass='ExposureF', doc='Input exposure to perform photometry on.', multiple=False, dimensions=['instrument', 'visit', 'detector'], isCalibration=False, deferLoad=False, minimum=1)) ctrl.mpexec.mpGraphExecutor INFO: Executed 2 quanta, 0 remain out of total 2 quanta. I marked ip_isr and pipe_tasks as approved. ci_hsc_gen3 looks fine and so does obs_lsst (you can consider those mutually-reviewed as well). So I’m going to remove myself on the reviewer list.

Jim Bosch added a comment - 16/Jun/21 3:38 AM Thanks for the extra testing and the reminder about ci_hsc_gen2; I've got that and ci_cpp_gen3 going in https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34465/pipeline .