Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13519

Implement per-object Galactic Extinction correction in color analysis QA plots

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      5
    • Epic Link:
    • Sprint:
      DRP S18-3, DRP S18-4, DRP S18-5, DRP S18-6, DRP F18-1, DRP F18-2
    • Team:
      Data Release Production

      Description

      Implement a per-object Galactic Extinction correction for use with the color-analysis QA plots to replace the per-field placeholder included in DM-13154.  It looks like there is code in sims_photUtils (and dependencies) to do this, so this will be an attempt to get that working with the analysis scripts.

       

      Note that this requires the A_filter/E(B-V) extinction coefficients for the HSC filters (awaiting a response from the HSC team, the placeholder noted above is just using SDSS filter values).

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            The per-object Galactic Extinction correction currently requires lsst.sims.catUtils to be setup as it uses its EBVbase class (in lsst.sims.catUtils.dust.EBV) to obtain the color excess, E(B-V), at a given RA, DEC. The call is thus put here in a try/except to fall back to the per-field correction (currently only hard-coded for the five tracts of HSC RC2 + RC datasets) if the sims repo is not setup until we can access the EBVbase class from an lsst_distrib installation (to be discussed with the sims team).

            I made the sims code accessible to an lsst_distrib installation on lsst-dev with the following:

            $ mkdir -p ~/LSST/sims/ups_db
            $ export EUPS_PATH=~/LSST/sims:$EUPS_PATH
            $ eups distrib install sims_catUtils -t sims

            [where ~ evaluates to /home/lauren/ here]
            The installed sims package is version 2.8.0.

            And the package is setup as follows:

            $ setup -k sims_catUtils -t sims
            

            Hsin-Fang Chiang has confirmed that she can set this repo up herself and did a test run on the PDR1 data reprocessing (results can currently, but temporarily, be seen https://lsst-web.ncsa.illinois.edu/~hchiang2/lauren_DM-13519/ – this will be redone once this ticket is closed).  Thus, anyone on lsst-dev could setup the repo in my /home/lauren/LSST/sims directory to make use of it having setup an lsst_distrib installation.

            As a check on the extinction corrections applied I confirmed that the mean value matches that used for the "per-field" correction (currently hardwired for the 5 RC2 + old RC datasets), and I created plots of the galactic extinction applied to each object for all bands, e.g.

            The following example of the wPerp Principal Color fit:

            Note that there are "fit" and "wired" results displayed. The latter is the current "canonical" value that is hardwired and the former is a live fit to the current dataset. The wired coefficients came from the tract shown as a "happy medium" between the three RC2 tracts. These may be adjusted again based on the PDR1 reprocessing (after running the full run through colorAnalysis with the per-object GE correction applied, see DM-14590.)

            The full set of results using the per-object GE correction can currently be perused here: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22/

            The fit and wired Principal Color axes will be added to the figures in DM-14826.

            Show
            lauren Lauren MacArthur added a comment - - edited The per-object Galactic Extinction correction currently requires lsst.sims.catUtils to be setup as it uses its EBVbase class (in lsst.sims.catUtils.dust.EBV) to obtain the color excess, E(B-V), at a given RA, DEC. The call is thus put here in a try/except to fall back to the per-field correction (currently only hard-coded for the five tracts of HSC RC2 + RC datasets) if the sims repo is not setup until we can access the EBVbase class from an lsst_distrib installation (to be discussed with the sims team). I made the sims code accessible to an lsst_distrib installation on lsst-dev with the following: $ mkdir -p ~/LSST/sims/ups_db $ export EUPS_PATH=~/LSST/sims:$EUPS_PATH $ eups distrib install sims_catUtils -t sims [where ~ evaluates to /home/lauren/ here] The installed sims package is version 2.8.0. And the package is setup as follows: $ setup -k sims_catUtils -t sims Hsin-Fang Chiang has confirmed that she can set this repo up herself and did a test run on the PDR1 data reprocessing (results can currently, but temporarily, be seen https://lsst-web.ncsa.illinois.edu/~hchiang2/lauren_DM-13519/ – this will be redone once this ticket is closed).  Thus, anyone on lsst-dev could setup the repo in my /home/lauren/LSST/sims directory to make use of it having setup an lsst_distrib  installation. As a check on the extinction corrections applied I confirmed that the mean value matches that used for the "per-field" correction (currently hardwired for the 5 RC2 + old RC datasets), and I created plots of the galactic extinction applied to each object for all bands, e.g. The following example of the wPerp Principal Color fit: Note that there are "fit" and "wired" results displayed. The latter is the current "canonical" value that is hardwired and the former is a live fit to the current dataset. The wired coefficients came from the tract shown as a "happy medium" between the three RC2 tracts. These may be adjusted again based on the PDR1 reprocessing (after running the full run through colorAnalysis with the per-object GE correction applied, see DM-14590 .) The full set of results using the per-object GE correction can currently be perused here:  https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22/ The fit and wired Principal Color axes will be added to the figures in DM-14826 .
            Hide
            lauren Lauren MacArthur added a comment -

            Paul, would you be able to give this a look?  I'm pinging you first as I want you to at least look at the addition of the config files in obs_subaru to check that my implementation is suitable.  Let me know if you'd prefer to pass off the pipe_analysis part of the review.

            Show
            lauren Lauren MacArthur added a comment - Paul, would you be able to give this a look?  I'm pinging you first as I want you to at least look at the addition of the config files in obs_subaru to check that my implementation is suitable.  Let me know if you'd prefer to pass off the pipe_analysis part of the review.
            Hide
            price Paul Price added a comment -

            Changes in obs_subaru look fine. If you don't mind, I'll duck reviewing pipe_analysis as I'm digging out from the vacation backlog and have multiple other reviews pending.

            Show
            price Paul Price added a comment - Changes in obs_subaru look fine. If you don't mind, I'll duck reviewing pipe_analysis as I'm digging out from the vacation backlog and have multiple other reviews pending.
            Hide
            lauren Lauren MacArthur added a comment -

            Sophie Reed, would you mind having a look at the pipe_analysis changes here?  Paul Price has already approved the obs_subaru changes.

            Show
            lauren Lauren MacArthur added a comment - Sophie Reed , would you mind having a look at the pipe_analysis  changes here?   Paul Price has already approved the obs_subaru  changes.
            Hide
            lauren Lauren MacArthur added a comment -

            I've addressed all your comments to date and pushed the updated branch.  A new run of colorAnalysis.py on the RC2 tracts is going now.  I moved the previous one to https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22old2 so this run will be in https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22 when it finishes.

             

            Let me know if you see anything else that needs fixing.

            Show
            lauren Lauren MacArthur added a comment - I've addressed all your comments to date and pushed the updated branch.  A new run of colorAnalysis.py on the RC2 tracts is going now.  I moved the previous one to  https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22old2  so this run will be in  https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22  when it finishes.   Let me know if you see anything else that needs fixing.
            Hide
            sophiereed Sophie Reed added a comment -

            Looks good to me

            Show
            sophiereed Sophie Reed added a comment - Looks good to me
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks both.  Merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks both.  Merged to master.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Paul Price, Sophie Reed
              Watchers:
              John Swinbank, Lauren MacArthur, Paul Price, Sophie Reed
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.