Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_lsst, pipe_analysis
-
Labels:
-
Story Points:10
-
Epic Link:
-
Sprint:DRP S21b
-
Team:Data Release Production
-
Urgent?:No
Description
A few extra tweaks need to be made to get the gen2 vs. gen3 functionality working and it's finally time to adapt the colorAnalysis script for DC2 data. While in there, add a set of stellar locus plots that use the newly added GAaP colors (see DM-24737).
A few updates are also made to some (gen2 only) obs_lsst overrides and templates for plot persistence and to fix up some incorrect filterMap-ings.
Attachments
Activity
Having a looking at the plots now - comments so far:
The plots that end in "ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png" look good overall - with most of them centered around and close to 0. There appears to be a bias in a few patches (say, (2,5), (6,3) in plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png). I should look into it if it persists after the recent changes.
The plots that end in "ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-gals.png" have no utility afaik. They provide good consistency checks for stars though as the GAaP magnitudes and PSF magnitudes should be consistent with each other. I know there are produced as part of the loop, but it might be worth not saving them to avoid confusion by unfair comparison.
I can barely see anything but some blue points in "plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_PsfFlux_forced-sky-stars.png". Since this involves GAaP PSF mags, I'm going to wait and see if the recent changes make it better (or worse).
Thanks for the review, Arun. I've addressed/commented on all your comments on the pipe_analysis PR. I also edited the code such that the ext_gaap_*-sky-gals.png plots no longer get created. Did you want to have a look at the obs_lsst PR (or should I take the "Reviewed" status to mean it has your approval)? Since is has been quite a while since the previous run, I kicked off another Jenkins just to err on the safe side (it succeeded).
Ok, I went ahead and merged the pipe_analysis PR. As for the obs_lsst PR, I'm actually going to ask Yusra AlSayyad to have a look since I have added a few config overrides to persist the visit-level parquet tables as well as removing some obsolete filterMap-ings that we have been (way too) long been overriding with config file overrides on the command line (e.g. see https://github.com/lsst-dm/gen2gen/blob/master/filter_config_2021_12.py & https://github.com/lsst-dm/gen2gen/blob/master/filter_config_multi.py. I am also doing a major update to the files in that repo based on my recent DC2 Gen2 processing experience, and will remove the unnecessary config overrides from the scripts as a part of that once this is merged.)
I have confirmed that the parquet files are indeed created with a singleFrameDriver run with no extra overrides:
ls /datasets/DC2/repoRun2.2i/rerun/private/lauren/DM-31274/sfm/src/00159471-g/R01/ |
sourceTable_00159471-g-R01-S00-det000.parq source_00159471-g-R01-S00-det000.parq src_00159471-g-R01-S00-det000.fits |
sourceTable_00159471-g-R01-S01-det001.parq source_00159471-g-R01-S01-det001.parq src_00159471-g-R01-S01-det001.fits |
sourceTable_00159471-g-R01-S02-det002.parq source_00159471-g-R01-S02-det002.parq src_00159471-g-R01-S02-det002.fits |
sourceTable_00159471-g-R01-S10-det003.parq source_00159471-g-R01-S10-det003.parq src_00159471-g-R01-S10-det003.fits |
sourceTable_00159471-g-R01-S11-det004.parq source_00159471-g-R01-S11-det004.parq src_00159471-g-R01-S11-det004.fits |
sourceTable_00159471-g-R01-S12-det005.parq source_00159471-g-R01-S12-det005.parq src_00159471-g-R01-S12-det005.fits |
sourceTable_00159471-g-R01-S20-det006.parq source_00159471-g-R01-S20-det006.parq src_00159471-g-R01-S20-det006.fits |
sourceTable_00159471-g-R01-S21-det007.parq source_00159471-g-R01-S21-det007.parq src_00159471-g-R01-S21-det007.fits |
sourceTable_00159471-g-R01-S22-det008.parq source_00159471-g-R01-S22-det008.parq src_00159471-g-R01-S22-det008.fits |
and running consolidateSourceTable.py produces:
/datasets/DC2/repoRun2.2i/rerun/private/lauren/DM-31274/sfm/src/00159471-g/sourceTable_00159471-g.parq |
I have also checked that multibandDriver.py runs happily with no extra config overrides. The command I used was (i.e. a subset of patches pointing at the w_20201_26 run coadds):
$ multiBandDriver.py /datasets/DC2/repoRun2.2i --rerun w_2021_36/DM-31665/coadd:private/lauren/DM-31274/multi --batch-type=slurm --mpiexec='-bind-to socket' --job DM-31274_multi --nodes 2 --procs 6 --time 45000 --id filter=g^r^i^u^y^z patch=0,0^1,0^2,0^0,1^1,1^2,1 tract=3829 |
I note that these config changes/updates are Gen2-visible only.
Another Jenkins is green.
In the interest in closing out this long-marked-reviewed ticket, I am withdrawing my PR on obs_lsst here and instead including those commits on the PR in DM-30284.
Would you mind giving this a look? You don't need to officially review the code changes in pipe_analysis (but are more than welcome to, of course!) You can take as validation all the plots I've been generating and posting to tickets/meeting notes of late, and the full set of plots from the latest runs can be perused here for HSC RC2 outputs and here & here for LSSTCam-imSim/DC2 outputs.
Jenkins is
runninggreen.