Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13519

Implement per-object Galactic Extinction correction in color analysis QA plots

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      5
    • Epic Link:
    • Sprint:
      DRP S18-3, DRP S18-4, DRP S18-5, DRP S18-6, DRP F18-1, DRP F18-2
    • Team:
      Data Release Production

      Description

      Implement a per-object Galactic Extinction correction for use with the color-analysis QA plots to replace the per-field placeholder included in DM-13154.  It looks like there is code in sims_photUtils (and dependencies) to do this, so this will be an attempt to get that working with the analysis scripts.

       

      Note that this requires the A_filter/E(B-V) extinction coefficients for the HSC filters (awaiting a response from the HSC team, the placeholder noted above is just using SDSS filter values).

        Attachments

          Issue Links

            Activity

            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Epic Link DM-12735 [ 36334 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-13154 [ DM-13154 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP S18-3 [ 687 ] DRP S18-3, DRP S18-4 [ 687, 696 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP S18-3, DRP S18-4 [ 687, 696 ] DRP S18-3, DRP S18-4, DRP S18-5 [ 687, 696, 707 ]
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP S18-3, DRP S18-4, DRP S18-5 [ 687, 696, 707 ] DRP S18-3, DRP S18-4, DRP S18-5, DRP S18-6 [ 687, 696, 707, 727 ]
            yusra Yusra AlSayyad made changes -
            Risk Score 0
            lauren Lauren MacArthur made changes -
            Link This issue blocks DM-14556 [ DM-14556 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue blocks DM-14616 [ DM-14616 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP S18-3, DRP S18-4, DRP S18-5, DRP S18-6 [ 687, 696, 707, 727 ] DRP S18-3, DRP S18-4, DRP S18-5, DRP S18-6, DRP F18-1 [ 687, 696, 707, 727, 742 ]
            lauren Lauren MacArthur made changes -
            Hide
            lauren Lauren MacArthur added a comment - - edited

            The per-object Galactic Extinction correction currently requires lsst.sims.catUtils to be setup as it uses its EBVbase class (in lsst.sims.catUtils.dust.EBV) to obtain the color excess, E(B-V), at a given RA, DEC. The call is thus put here in a try/except to fall back to the per-field correction (currently only hard-coded for the five tracts of HSC RC2 + RC datasets) if the sims repo is not setup until we can access the EBVbase class from an lsst_distrib installation (to be discussed with the sims team).

            I made the sims code accessible to an lsst_distrib installation on lsst-dev with the following:

            $ mkdir -p ~/LSST/sims/ups_db
            $ export EUPS_PATH=~/LSST/sims:$EUPS_PATH
            $ eups distrib install sims_catUtils -t sims

            [where ~ evaluates to /home/lauren/ here]
            The installed sims package is version 2.8.0.

            And the package is setup as follows:

            $ setup -k sims_catUtils -t sims
            

            Hsin-Fang Chiang has confirmed that she can set this repo up herself and did a test run on the PDR1 data reprocessing (results can currently, but temporarily, be seen https://lsst-web.ncsa.illinois.edu/~hchiang2/lauren_DM-13519/ – this will be redone once this ticket is closed).  Thus, anyone on lsst-dev could setup the repo in my /home/lauren/LSST/sims directory to make use of it having setup an lsst_distrib installation.

            As a check on the extinction corrections applied I confirmed that the mean value matches that used for the "per-field" correction (currently hardwired for the 5 RC2 + old RC datasets), and I created plots of the galactic extinction applied to each object for all bands, e.g.

            The following example of the wPerp Principal Color fit:

            Note that there are "fit" and "wired" results displayed. The latter is the current "canonical" value that is hardwired and the former is a live fit to the current dataset. The wired coefficients came from the tract shown as a "happy medium" between the three RC2 tracts. These may be adjusted again based on the PDR1 reprocessing (after running the full run through colorAnalysis with the per-object GE correction applied, see DM-14590.)

            The full set of results using the per-object GE correction can currently be perused here: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22/

            The fit and wired Principal Color axes will be added to the figures in DM-14826.

            Show
            lauren Lauren MacArthur added a comment - - edited The per-object Galactic Extinction correction currently requires lsst.sims.catUtils to be setup as it uses its EBVbase class (in lsst.sims.catUtils.dust.EBV) to obtain the color excess, E(B-V), at a given RA, DEC. The call is thus put here in a try/except to fall back to the per-field correction (currently only hard-coded for the five tracts of HSC RC2 + RC datasets) if the sims repo is not setup until we can access the EBVbase class from an lsst_distrib installation (to be discussed with the sims team). I made the sims code accessible to an lsst_distrib installation on lsst-dev with the following: $ mkdir -p ~/LSST/sims/ups_db $ export EUPS_PATH=~/LSST/sims:$EUPS_PATH $ eups distrib install sims_catUtils -t sims [where ~ evaluates to /home/lauren/ here] The installed sims package is version 2.8.0. And the package is setup as follows: $ setup -k sims_catUtils -t sims Hsin-Fang Chiang has confirmed that she can set this repo up herself and did a test run on the PDR1 data reprocessing (results can currently, but temporarily, be seen https://lsst-web.ncsa.illinois.edu/~hchiang2/lauren_DM-13519/ – this will be redone once this ticket is closed).  Thus, anyone on lsst-dev could setup the repo in my /home/lauren/LSST/sims directory to make use of it having setup an lsst_distrib  installation. As a check on the extinction corrections applied I confirmed that the mean value matches that used for the "per-field" correction (currently hardwired for the 5 RC2 + old RC datasets), and I created plots of the galactic extinction applied to each object for all bands, e.g. The following example of the wPerp Principal Color fit: Note that there are "fit" and "wired" results displayed. The latter is the current "canonical" value that is hardwired and the former is a live fit to the current dataset. The wired coefficients came from the tract shown as a "happy medium" between the three RC2 tracts. These may be adjusted again based on the PDR1 reprocessing (after running the full run through colorAnalysis with the per-object GE correction applied, see DM-14590 .) The full set of results using the per-object GE correction can currently be perused here:  https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22/ The fit and wired Principal Color axes will be added to the figures in DM-14826 .
            Hide
            lauren Lauren MacArthur added a comment -

            Paul, would you be able to give this a look?  I'm pinging you first as I want you to at least look at the addition of the config files in obs_subaru to check that my implementation is suitable.  Let me know if you'd prefer to pass off the pipe_analysis part of the review.

            Show
            lauren Lauren MacArthur added a comment - Paul, would you be able to give this a look?  I'm pinging you first as I want you to at least look at the addition of the config files in obs_subaru to check that my implementation is suitable.  Let me know if you'd prefer to pass off the pipe_analysis part of the review.
            lauren Lauren MacArthur made changes -
            Reviewers Paul Price [ price ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            swinbank John Swinbank made changes -
            Epic Link DM-12735 [ 36334 ] DM-14402 [ 79665 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP S18-3, DRP S18-4, DRP S18-5, DRP S18-6, DRP F18-1 [ 687, 696, 707, 727, 742 ] DRP S18-3, DRP S18-4, DRP S18-5, DRP S18-6, DRP F18-1, DRP F18-2 [ 687, 696, 707, 727, 742, 756 ]
            Hide
            price Paul Price added a comment -

            Changes in obs_subaru look fine. If you don't mind, I'll duck reviewing pipe_analysis as I'm digging out from the vacation backlog and have multiple other reviews pending.

            Show
            price Paul Price added a comment - Changes in obs_subaru look fine. If you don't mind, I'll duck reviewing pipe_analysis as I'm digging out from the vacation backlog and have multiple other reviews pending.
            lauren Lauren MacArthur made changes -
            Reviewers Paul Price [ price ] Paul Price, Sophie Reed [ price, sophiereed ]
            Hide
            lauren Lauren MacArthur added a comment -

            Sophie Reed, would you mind having a look at the pipe_analysis changes here?  Paul Price has already approved the obs_subaru changes.

            Show
            lauren Lauren MacArthur added a comment - Sophie Reed , would you mind having a look at the pipe_analysis  changes here?   Paul Price has already approved the obs_subaru  changes.
            Hide
            lauren Lauren MacArthur added a comment -

            I've addressed all your comments to date and pushed the updated branch.  A new run of colorAnalysis.py on the RC2 tracts is going now.  I moved the previous one to https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22old2 so this run will be in https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22 when it finishes.

             

            Let me know if you see anything else that needs fixing.

            Show
            lauren Lauren MacArthur added a comment - I've addressed all your comments to date and pushed the updated branch.  A new run of colorAnalysis.py on the RC2 tracts is going now.  I moved the previous one to  https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22old2  so this run will be in  https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-13519/w_2018_22  when it finishes.   Let me know if you see anything else that needs fixing.
            Hide
            sophiereed Sophie Reed added a comment -

            Looks good to me

            Show
            sophiereed Sophie Reed added a comment - Looks good to me
            sophiereed Sophie Reed made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks both.  Merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks both.  Merged to master.
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                Paul Price, Sophie Reed
                Watchers:
                John Swinbank, Lauren MacArthur, Paul Price, Sophie Reed
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel