Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31274

Update pipe_analysis scripts for DC2 (gen2 & gen3) and add GAaP color plots

    XMLWordPrintable

    Details

    • Story Points:
      10
    • Epic Link:
    • Sprint:
      DRP S21b
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      A few extra tweaks need to be made to get the gen2 vs. gen3 functionality working and it's finally time to adapt the colorAnalysis script for DC2 data.  While in there, add a set of stellar locus plots that use the newly added GAaP colors (see DM-24737).

      A few updates are also made to some (gen2 only) obs_lsst overrides and templates for plot persistence and to fix up some incorrect filterMap-ings.

        Attachments

          Activity

          Hide
          lauren Lauren MacArthur added a comment - - edited

          Would you mind giving this a look?  You don't need to officially review the code changes in pipe_analysis (but are more than welcome to, of course!)  You can take as validation all the plots I've been generating and posting to tickets/meeting notes of late, and the full set of plots from the latest runs can be perused here for HSC RC2 outputs and here & here for LSSTCam-imSim/DC2 outputs.

          Jenkins is runninggreen.

          Show
          lauren Lauren MacArthur added a comment - - edited Would you mind giving this a look?  You don't need to officially review the code changes in pipe_analysis (but are more than welcome to, of course!)  You can take as validation all the plots I've been generating and posting to tickets/meeting notes of late, and the full set of plots from the latest runs can be perused here for HSC RC2 outputs and here & here for LSSTCam-imSim/DC2 outputs. Jenkins is running green .
          Hide
          kannawad Arun Kannawadi added a comment -

          Having a looking at the plots now - comments so far:

          The plots that end in "ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png" look good overall - with most of them centered around and close to 0. There appears to be a bias in a few patches (say, (2,5), (6,3) in plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png). I should look into it if it persists after the recent changes.

          The plots that end in "ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-gals.png" have no utility afaik. They provide good consistency checks for stars though as the GAaP magnitudes and PSF magnitudes should be consistent with each other. I know there are produced as part of the loop, but it might be worth not saving them to avoid confusion by unfair comparison.

          I can barely see anything but some blue points in "plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_PsfFlux_forced-sky-stars.png". Since this involves GAaP PSF mags, I'm going to wait and see if the recent changes make it better (or worse).

          Show
          kannawad Arun Kannawadi added a comment - Having a looking at the plots now - comments so far: The plots that end in " ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png " look good overall - with most of them centered around and close to 0. There appears to be a bias in a few patches (say, (2,5), (6,3) in plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-stars.png). I should look into it if it persists after the recent changes. The plots that end in " ext_gaap_GaapFlux_1_15x_Optimal_forced-sky-gals.png " have no utility afaik. They provide good consistency checks for stars though as the GAaP magnitudes and PSF magnitudes should be consistent with each other. I know there are produced as part of the loop, but it might be worth not saving them to avoid confusion by unfair comparison. I can barely see anything but some blue points in "plot-t3829-r-mag_ext_gaap_GaapFlux_1_15x_PsfFlux_forced-sky-stars.png". Since this involves GAaP PSF mags, I'm going to wait and see if the recent changes make it better (or worse).
          Hide
          lauren Lauren MacArthur added a comment - - edited

          Thanks for the review, Arun. I've addressed/commented on all your comments on the pipe_analysis PR. I also edited the code such that the ext_gaap_*-sky-gals.png plots no longer get created. Did you want to have a look at the obs_lsst PR (or should I take the "Reviewed" status to mean it has your approval)? Since is has been quite a while since the previous run, I kicked off another Jenkins just to err on the safe side (it succeeded).

          Show
          lauren Lauren MacArthur added a comment - - edited Thanks for the review, Arun. I've addressed/commented on all your comments on the pipe_analysis PR. I also edited the code such that the ext_gaap_*-sky-gals.png plots no longer get created. Did you want to have a look at the obs_lsst PR (or should I take the "Reviewed" status to mean it has your approval)? Since is has been quite a while since the previous run, I kicked off another Jenkins just to err on the safe side (it succeeded).
          Hide
          lauren Lauren MacArthur added a comment -

          Ok, I went ahead and merged the pipe_analysis PR.  As for the obs_lsst PR, I'm actually going to ask Yusra AlSayyad to have a look since I have added a few config overrides to persist the visit-level parquet tables as well as removing some obsolete filterMap-ings that we have been (way too) long been overriding with config file overrides on the command line (e.g. see https://github.com/lsst-dm/gen2gen/blob/master/filter_config_2021_12.py & https://github.com/lsst-dm/gen2gen/blob/master/filter_config_multi.py. I am also doing a major update to the files in that repo based on my recent DC2 Gen2 processing experience, and will remove the unnecessary config overrides from the scripts as a part of that once this is merged.)

          I have confirmed that the parquet files are indeed created with a singleFrameDriver run with no extra overrides:

          ls /datasets/DC2/repoRun2.2i/rerun/private/lauren/DM-31274/sfm/src/00159471-g/R01/
          sourceTable_00159471-g-R01-S00-det000.parq  source_00159471-g-R01-S00-det000.parq  src_00159471-g-R01-S00-det000.fits
          sourceTable_00159471-g-R01-S01-det001.parq  source_00159471-g-R01-S01-det001.parq  src_00159471-g-R01-S01-det001.fits
          sourceTable_00159471-g-R01-S02-det002.parq  source_00159471-g-R01-S02-det002.parq  src_00159471-g-R01-S02-det002.fits
          sourceTable_00159471-g-R01-S10-det003.parq  source_00159471-g-R01-S10-det003.parq  src_00159471-g-R01-S10-det003.fits
          sourceTable_00159471-g-R01-S11-det004.parq  source_00159471-g-R01-S11-det004.parq  src_00159471-g-R01-S11-det004.fits
          sourceTable_00159471-g-R01-S12-det005.parq  source_00159471-g-R01-S12-det005.parq  src_00159471-g-R01-S12-det005.fits
          sourceTable_00159471-g-R01-S20-det006.parq  source_00159471-g-R01-S20-det006.parq  src_00159471-g-R01-S20-det006.fits
          sourceTable_00159471-g-R01-S21-det007.parq  source_00159471-g-R01-S21-det007.parq  src_00159471-g-R01-S21-det007.fits
          sourceTable_00159471-g-R01-S22-det008.parq  source_00159471-g-R01-S22-det008.parq  src_00159471-g-R01-S22-det008.fits
          

          and running consolidateSourceTable.py produces:

          /datasets/DC2/repoRun2.2i/rerun/private/lauren/DM-31274/sfm/src/00159471-g/sourceTable_00159471-g.parq
          

          I have also checked that multibandDriver.py runs happily with no extra config overrides. The command I used was (i.e. a subset of patches pointing at the w_20201_26 run coadds):

          $ multiBandDriver.py /datasets/DC2/repoRun2.2i --rerun w_2021_36/DM-31665/coadd:private/lauren/DM-31274/multi --batch-type=slurm --mpiexec='-bind-to socket' --job DM-31274_multi --nodes 2 --procs 6 --time 45000 --id filter=g^r^i^u^y^z patch=0,0^1,0^2,0^0,1^1,1^2,1 tract=3829
          

          I note that these config changes/updates are Gen2-visible only.

          Another Jenkins is green.

          Show
          lauren Lauren MacArthur added a comment - Ok, I went ahead and merged the pipe_analysis PR.  As for the obs_lsst PR, I'm actually going to ask Yusra AlSayyad  to have a look since I have added a few config overrides to persist the visit-level parquet tables as well as removing some obsolete filterMap-ings that we have been (way too) long been overriding with config file overrides on the command line (e.g. see https://github.com/lsst-dm/gen2gen/blob/master/filter_config_2021_12.py  & https://github.com/lsst-dm/gen2gen/blob/master/filter_config_multi.py . I am also doing a major update to the files in that repo based on my recent DC2 Gen2 processing experience, and will remove the unnecessary config overrides from the scripts as a part of that once this is merged.) I have confirmed that the parquet files are indeed created with a singleFrameDriver run with no extra overrides: ls / datasets / DC2 / repoRun2. 2i / rerun / private / lauren / DM - 31274 / sfm / src / 00159471 - g / R01 / sourceTable_00159471 - g - R01 - S00 - det000.parq source_00159471 - g - R01 - S00 - det000.parq src_00159471 - g - R01 - S00 - det000.fits sourceTable_00159471 - g - R01 - S01 - det001.parq source_00159471 - g - R01 - S01 - det001.parq src_00159471 - g - R01 - S01 - det001.fits sourceTable_00159471 - g - R01 - S02 - det002.parq source_00159471 - g - R01 - S02 - det002.parq src_00159471 - g - R01 - S02 - det002.fits sourceTable_00159471 - g - R01 - S10 - det003.parq source_00159471 - g - R01 - S10 - det003.parq src_00159471 - g - R01 - S10 - det003.fits sourceTable_00159471 - g - R01 - S11 - det004.parq source_00159471 - g - R01 - S11 - det004.parq src_00159471 - g - R01 - S11 - det004.fits sourceTable_00159471 - g - R01 - S12 - det005.parq source_00159471 - g - R01 - S12 - det005.parq src_00159471 - g - R01 - S12 - det005.fits sourceTable_00159471 - g - R01 - S20 - det006.parq source_00159471 - g - R01 - S20 - det006.parq src_00159471 - g - R01 - S20 - det006.fits sourceTable_00159471 - g - R01 - S21 - det007.parq source_00159471 - g - R01 - S21 - det007.parq src_00159471 - g - R01 - S21 - det007.fits sourceTable_00159471 - g - R01 - S22 - det008.parq source_00159471 - g - R01 - S22 - det008.parq src_00159471 - g - R01 - S22 - det008.fits and running consolidateSourceTable.py produces: / datasets / DC2 / repoRun2. 2i / rerun / private / lauren / DM - 31274 / sfm / src / 00159471 - g / sourceTable_00159471 - g.parq I have also checked that multibandDriver.py runs happily with no extra config overrides. The command I used was (i.e. a subset of patches pointing at the w_20201_26 run coadds): $ multiBandDriver.py / datasets / DC2 / repoRun2. 2i - - rerun w_2021_36 / DM - 31665 / coadd:private / lauren / DM - 31274 / multi - - batch - type = slurm - - mpiexec = '-bind-to socket' - - job DM - 31274_multi - - nodes 2 - - procs 6 - - time 45000 - - id filter = g^r^i^u^y^z patch = 0 , 0 ^ 1 , 0 ^ 2 , 0 ^ 0 , 1 ^ 1 , 1 ^ 2 , 1 tract = 3829 I note that these config changes/updates are Gen2-visible only. Another Jenkins is green .
          Hide
          lauren Lauren MacArthur added a comment -

          In the interest in closing out this long-marked-reviewed ticket, I am withdrawing my PR on obs_lsst here and instead including those commits on the PR in DM-30284.

          Show
          lauren Lauren MacArthur added a comment - In the interest in closing out this long-marked-reviewed ticket, I am withdrawing my PR on obs_lsst here and instead including those commits on the PR in DM-30284 .

            People

            Assignee:
            lauren Lauren MacArthur
            Reporter:
            lauren Lauren MacArthur
            Reviewers:
            Arun Kannawadi, Yusra AlSayyad
            Watchers:
            Arun Kannawadi, Lauren MacArthur
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.