Details
-
Type:
Bug
-
Status: Invalid
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_pipe
-
Labels:None
-
Team:Alert Production
-
Urgent?:No
Description
When trying to do a pipetask run of the small 2021 diffim sprint dataset as part of DM-35283, I was seeing a significant number of the subtractImages tasks fail with an error of the following form:
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_42/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ip_diffim/g8a851bd635+1be
|
58ecf80/python/lsst/ip/diffim/subtractImages.py", line 512, in _validateExposures
|
assert template.wcs == science.wcs,\
|
AssertionError: Template and science exposure WCS are not identical.
|
I've never seen this processing this particular dataset before. I'm using w_2022_42 as the base release with no custom packages, in order to establish a baseline for comparison with future updates to the code, especially the convolve the science image case. It happens for multiple visits and multiple CCDs, so it's not specific to any particular visit. My pipeline yaml, in its entirety, is
description: End to end AP pipeline for running on the DC2 repo
|
|
|
instrument: lsst.obs.subaru.HyperSuprimeCam
|
imports:
|
- location: $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApPipeWithFakes.yaml
|
exclude:
|
- imageDifference
|
parameters:
|
# Pipeline configurable to run on both bestThird and deep templates, depending on dataset.
|
coaddName: goodSeeing
|
diff: goodSeeingDiff_differenceExp
|
tasks:
|
subtractImages:
|
class: lsst.ip.diffim.subtractImages.AlardLuptonSubtractTask
|
config:
|
connections.coaddName: parameters.coaddName
|
forceCompatibility: False
|
mode: convolveTemplate
|
detectAndMeasure:
|
class: lsst.ip.diffim.detectAndMeasure.DetectAndMeasureTask
|
config:
|
connections.coaddName: parameters.coaddName
|
doSkySources: true
|
diaPipe:
|
class: lsst.ap.association.DiaPipelineTask
|
config:
|
# # For a postgreslq DB, use only the following line:
|
apdb.db_url: 'postgresql://rubin@usdf-prompt-processing-dev.slac.stanford.edu/lsst-devl'
|
# # For a local sqlite file, use the following two lines:
|
# # apdb.isolation_level: 'READ_UNCOMMITTED'
|
# # apdb.db_url: 'sqlite:////project/kherner/PREOPS-598_reprocSprint/bps_PREOPS-598_defaults_Preconvolve.db'
|
apdb.namespace: 'kherner_dm35283_convolvetemplate'
|
connections.coaddName: parameters.coaddName
|
exclude:
|
- imageDifference
|
contracts:
|
# - detectAndMeasure.doSkySources == transformDiaSrcCat.doRemoveSkySources
|
# # Inputs and outputs must match
|
- subtractImages.connections.coaddName == detectAndMeasure.connections.coaddName
|
So everything is default except for turning off auto mode and forcing convolveTemplate, which is the right things to do for using the best third seeing templates. I do note that I did not remake the templates recently, so they are left over from my latest runs at NCSA.
Attachments
Issue Links
- relates to
-
DM-35283 Process Diffim sprint dataset with new Image Differencing
- Done
Activity
I'll cancel this ticket as it turned out to be a yaml misconfig.
Kenneth Herner
added a comment - I'll cancel this ticket as it turned out to be a yaml misconfig. 
Here's pipetask command for completeness (run on rubin-devl):
pipetask run -j 12 -i u/kherner/
DM-33911/templates_bestThirdSeeing -b /sdf/group/rubin/repo/main/butler.yaml -o u/kherner/DM-35283/baselineWithFakes_bestThirdSeeing_convolveTemplate -p /sdf/group/rubin/u/kherner/DM-35283/yaml/ApPipe_newDiffim_convolveTemplate.yaml -d 'exposure IN (11690, 11692, 11694, 11696, 11698, 11700, 11702, 11704, 11706, 11708, 11710, 29340, 1204, 1208, 1214, 1216, 1218, 1220, 23692, 23694, 23704, 23718) AND detector in (49, 50, 57, 58, 65, 66)' --register-dataset-types > pipetask1.out 2> pipetask1.err