# Update pipe_analysis scripts for DC2 (gen2 & gen3) and add GAaP color plots

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
10
• Sprint:
DRP S21b
• Team:
Data Release Production
• Urgent?:
No

#### Description

A few extra tweaks need to be made to get the gen2 vs. gen3 functionality working and it's finally time to adapt the colorAnalysis script for DC2 data.  While in there, add a set of stellar locus plots that use the newly added GAaP colors (see DM-24737).

A few updates are also made to some (gen2 only) obs_lsst overrides and templates for plot persistence and to fix up some incorrect filterMap-ings.

#### Activity

Hide
Lauren MacArthur added a comment - - edited

Would you mind giving this a look?  You don't need to officially review the code changes in pipe_analysis (but are more than welcome to, of course!)  You can take as validation all the plots I've been generating and posting to tickets/meeting notes of late, and the full set of plots from the latest runs can be perused here for HSC RC2 outputs and here & here for LSSTCam-imSim/DC2 outputs.

Jenkins is runninggreen.

Show
Lauren MacArthur added a comment - - edited Would you mind giving this a look?  You don't need to officially review the code changes in pipe_analysis (but are more than welcome to, of course!)  You can take as validation all the plots I've been generating and posting to tickets/meeting notes of late, and the full set of plots from the latest runs can be perused here for HSC RC2 outputs and here & here for LSSTCam-imSim/DC2 outputs. Jenkins is running green .
Hide

Having a looking at the plots now - comments so far:

The plots that end in "ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png" look good overall - with most of them centered around and close to 0. There appears to be a bias in a few patches (say, (2,5), (6,3) in plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png). I should look into it if it persists after the recent changes.

The plots that end in "ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-gals.png" have no utility afaik. They provide good consistency checks for stars though as the GAaP magnitudes and PSF magnitudes should be consistent with each other. I know there are produced as part of the loop, but it might be worth not saving them to avoid confusion by unfair comparison.

I can barely see anything but some blue points in "plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_PsfFlux_forced-sky-stars.png". Since this involves GAaP PSF mags, I'm going to wait and see if the recent changes make it better (or worse).

Show
Arun Kannawadi added a comment - Having a looking at the plots now - comments so far: The plots that end in " ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png " look good overall - with most of them centered around and close to 0. There appears to be a bias in a few patches (say, (2,5), (6,3) in plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png). I should look into it if it persists after the recent changes. The plots that end in " ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-gals.png " have no utility afaik. They provide good consistency checks for stars though as the GAaP magnitudes and PSF magnitudes should be consistent with each other. I know there are produced as part of the loop, but it might be worth not saving them to avoid confusion by unfair comparison. I can barely see anything but some blue points in "plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_PsfFlux_forced-sky-stars.png". Since this involves GAaP PSF mags, I'm going to wait and see if the recent changes make it better (or worse).
Hide
Lauren MacArthur added a comment - - edited

Thanks for the review, Arun. I've addressed/commented on all your comments on the pipe_analysis PR. I also edited the code such that the ext_gaap_*-sky-gals.png plots no longer get created. Did you want to have a look at the obs_lsst PR (or should I take the "Reviewed" status to mean it has your approval)? Since is has been quite a while since the previous run, I kicked off another Jenkins just to err on the safe side (it succeeded).

Show
Lauren MacArthur added a comment - - edited Thanks for the review, Arun. I've addressed/commented on all your comments on the pipe_analysis PR. I also edited the code such that the ext_gaap_*-sky-gals.png plots no longer get created. Did you want to have a look at the obs_lsst PR (or should I take the "Reviewed" status to mean it has your approval)? Since is has been quite a while since the previous run, I kicked off another Jenkins just to err on the safe side (it succeeded).
Hide
Lauren MacArthur added a comment -

Ok, I went ahead and merged the pipe_analysis PR.  As for the obs_lsst PR, I'm actually going to ask Yusra AlSayyad to have a look since I have added a few config overrides to persist the visit-level parquet tables as well as removing some obsolete filterMap-ings that we have been (way too) long been overriding with config file overrides on the command line (e.g. see https://github.com/lsst-dm/gen2gen/blob/master/filter_config_2021_12.py & https://github.com/lsst-dm/gen2gen/blob/master/filter_config_multi.py. I am also doing a major update to the files in that repo based on my recent DC2 Gen2 processing experience, and will remove the unnecessary config overrides from the scripts as a part of that once this is merged.)

I have confirmed that the parquet files are indeed created with a singleFrameDriver run with no extra overrides:

 ls /datasets/DC2/repoRun2.2i/rerun/private/lauren/DM-31274/sfm/src/00159471-g/R01/ sourceTable_00159471-g-R01-S00-det000.parq source_00159471-g-R01-S00-det000.parq src_00159471-g-R01-S00-det000.fits sourceTable_00159471-g-R01-S01-det001.parq source_00159471-g-R01-S01-det001.parq src_00159471-g-R01-S01-det001.fits sourceTable_00159471-g-R01-S02-det002.parq source_00159471-g-R01-S02-det002.parq src_00159471-g-R01-S02-det002.fits sourceTable_00159471-g-R01-S10-det003.parq source_00159471-g-R01-S10-det003.parq src_00159471-g-R01-S10-det003.fits sourceTable_00159471-g-R01-S11-det004.parq source_00159471-g-R01-S11-det004.parq src_00159471-g-R01-S11-det004.fits sourceTable_00159471-g-R01-S12-det005.parq source_00159471-g-R01-S12-det005.parq src_00159471-g-R01-S12-det005.fits sourceTable_00159471-g-R01-S20-det006.parq source_00159471-g-R01-S20-det006.parq src_00159471-g-R01-S20-det006.fits sourceTable_00159471-g-R01-S21-det007.parq source_00159471-g-R01-S21-det007.parq src_00159471-g-R01-S21-det007.fits sourceTable_00159471-g-R01-S22-det008.parq source_00159471-g-R01-S22-det008.parq src_00159471-g-R01-S22-det008.fits 

and running consolidateSourceTable.py produces:

 /datasets/DC2/repoRun2.2i/rerun/private/lauren/DM-31274/sfm/src/00159471-g/sourceTable_00159471-g.parq 

I have also checked that multibandDriver.py runs happily with no extra config overrides. The command I used was (i.e. a subset of patches pointing at the w_20201_26 run coadds):

 $multiBandDriver.py /datasets/DC2/repoRun2.2i --rerun w_2021_36/DM-31665/coadd:private/lauren/DM-31274/multi --batch-type=slurm --mpiexec='-bind-to socket' --job DM-31274_multi --nodes 2 --procs 6 --time 45000 --id filter=g^r^i^u^y^z patch=0,0^1,0^2,0^0,1^1,1^2,1 tract=3829  I note that these config changes/updates are Gen2-visible only. Another Jenkins is green. Show Lauren MacArthur added a comment - Ok, I went ahead and merged the pipe_analysis PR. As for the obs_lsst PR, I'm actually going to ask Yusra AlSayyad to have a look since I have added a few config overrides to persist the visit-level parquet tables as well as removing some obsolete filterMap-ings that we have been (way too) long been overriding with config file overrides on the command line (e.g. see https://github.com/lsst-dm/gen2gen/blob/master/filter_config_2021_12.py & https://github.com/lsst-dm/gen2gen/blob/master/filter_config_multi.py . I am also doing a major update to the files in that repo based on my recent DC2 Gen2 processing experience, and will remove the unnecessary config overrides from the scripts as a part of that once this is merged.) I have confirmed that the parquet files are indeed created with a singleFrameDriver run with no extra overrides: ls / datasets / DC2 / repoRun2. 2i / rerun / private / lauren / DM - 31274 / sfm / src / 00159471 - g / R01 / sourceTable_00159471 - g - R01 - S00 - det000.parq source_00159471 - g - R01 - S00 - det000.parq src_00159471 - g - R01 - S00 - det000.fits sourceTable_00159471 - g - R01 - S01 - det001.parq source_00159471 - g - R01 - S01 - det001.parq src_00159471 - g - R01 - S01 - det001.fits sourceTable_00159471 - g - R01 - S02 - det002.parq source_00159471 - g - R01 - S02 - det002.parq src_00159471 - g - R01 - S02 - det002.fits sourceTable_00159471 - g - R01 - S10 - det003.parq source_00159471 - g - R01 - S10 - det003.parq src_00159471 - g - R01 - S10 - det003.fits sourceTable_00159471 - g - R01 - S11 - det004.parq source_00159471 - g - R01 - S11 - det004.parq src_00159471 - g - R01 - S11 - det004.fits sourceTable_00159471 - g - R01 - S12 - det005.parq source_00159471 - g - R01 - S12 - det005.parq src_00159471 - g - R01 - S12 - det005.fits sourceTable_00159471 - g - R01 - S20 - det006.parq source_00159471 - g - R01 - S20 - det006.parq src_00159471 - g - R01 - S20 - det006.fits sourceTable_00159471 - g - R01 - S21 - det007.parq source_00159471 - g - R01 - S21 - det007.parq src_00159471 - g - R01 - S21 - det007.fits sourceTable_00159471 - g - R01 - S22 - det008.parq source_00159471 - g - R01 - S22 - det008.parq src_00159471 - g - R01 - S22 - det008.fits and running consolidateSourceTable.py produces: / datasets / DC2 / repoRun2. 2i / rerun / private / lauren / DM - 31274 / sfm / src / 00159471 - g / sourceTable_00159471 - g.parq I have also checked that multibandDriver.py runs happily with no extra config overrides. The command I used was (i.e. a subset of patches pointing at the w_20201_26 run coadds):$ multiBandDriver.py / datasets / DC2 / repoRun2. 2i - - rerun w_2021_36 / DM - 31665 / coadd:private / lauren / DM - 31274 / multi - - batch - type = slurm - - mpiexec = '-bind-to socket' - - job DM - 31274_multi - - nodes 2 - - procs 6 - - time 45000 - - id filter = g^r^i^u^y^z patch = 0 , 0 ^ 1 , 0 ^ 2 , 0 ^ 0 , 1 ^ 1 , 1 ^ 2 , 1 tract = 3829 I note that these config changes/updates are Gen2-visible only. Another Jenkins is green .
Hide
Lauren MacArthur added a comment -

In the interest in closing out this long-marked-reviewed ticket, I am withdrawing my PR on obs_lsst here and instead including those commits on the PR in DM-30284.

Show
Lauren MacArthur added a comment - In the interest in closing out this long-marked-reviewed ticket, I am withdrawing my PR on obs_lsst here and instead including those commits on the PR in DM-30284 .

#### People

Assignee:
Lauren MacArthur
Reporter:
Lauren MacArthur
Reviewers:
Watchers: