Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15569

Updates to comparison scripts in pipe_analysis

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      DRP F18-3, DRP F18-4, DRP F18-5
    • Team:
      Data Release Production

      Description

      Various updates are needed in the comparison scripts.  Namely, some plotting limits and missing comparison plots as well as fixing the task runner such that the output rerun directory can be specified by the --rerun inDir:outDir method which is currently broken due to the implementation of the --rerun2 in2Dir (i.e. the comparison directory).

        Attachments

          Activity

          Hide
          lauren Lauren MacArthur added a comment -

          In addition to the above, I have also added the following on this ticket:

          • plot against CModel flux (rather than PSF flux) for coadds as this is the best representation of flux for both stars and galaxies (they both follow each other very closely for stars).  CModel is not currently run in SFP, so we still plot agains PSF flux a the visit level.
          • add histograms for the full sample (stars + gals) to the scatter plots (an HSC request from long ago)
          • fix up some labelling of forced vs. unforced catalog usage

          I also fixed a bug in the compare scripts where it would fail if a patch was missing (which is annoying if one does not wish to pass a list of patches, but rather have the script simply skip any missing patches).

          Show
          lauren Lauren MacArthur added a comment - In addition to the above, I have also added the following on this ticket: plot against CModel flux (rather than PSF flux) for coadds as this is the best representation of flux for both stars and galaxies (they both follow each other very closely for stars).  CModel is not currently run in SFP, so we still plot agains PSF flux a the visit level. add histograms for the full sample (stars + gals) to the scatter plots (an HSC request from long ago) fix up some labelling of forced vs. unforced catalog usage I also fixed a bug in the compare scripts where it would fail if a patch was missing (which is annoying if one does not wish to pass a list of patches, but rather have the script simply skip any missing patches).
          Hide
          lauren Lauren MacArthur added a comment -

          Would you mind having a look at this?  With this branch, you should now also be able to omit the list of patches in your command line id list for the COSMOS tract 9813 compareCoaddAnalysis.py run.

           

          I have scrutinized the plots and they look good to me, but if you want to have a look for yourself, you can have a look at https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15569/UD_COSMOS_9813/plots/ (new ones are being created now, but what is there is largely representative of all but a minor labeling fix).  The comparison scripts had the following rerun specification:

          --rerun RC/w_2018_30/DM-15120:private/lauren/DM-15569/UD_COSMOS_9813 --rerun2 RC/w_2018_34/DM-15517-B/

          which is proof of concept that the output redirection is indeed working.

          Show
          lauren Lauren MacArthur added a comment - Would you mind having a look at this?  With this branch, you should now also be able to omit the list of patches in your command line id list for the COSMOS tract 9813 compareCoaddAnalysis.py run.   I have scrutinized the plots and they look good to me, but if you want to have a look for yourself, you can have a look at https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15569/UD_COSMOS_9813/plots/  (new ones are being created now, but what is there is largely representative of all but a minor labeling fix).  The comparison scripts had the following rerun specification: --rerun RC/w_2018_30/DM-15120:private/lauren/DM-15569/UD_COSMOS_9813 --rerun2 RC/w_2018_34/DM-15517-B/ which is proof of concept that the output redirection is indeed working.
          Hide
          lauren Lauren MacArthur added a comment - - edited

          Note that I have had in-person discussions with Yusra AlSayyad and Jim Bosch about the change involving plotting against CModel flux (instead of PSF flux) for the coadds.  They both agree this is an appropriate change.

          Show
          lauren Lauren MacArthur added a comment - - edited Note that I have had in-person discussions with Yusra AlSayyad and Jim Bosch about the change involving plotting against CModel flux (instead of PSF flux) for the coadds.  They both agree this is an appropriate change.
          Hide
          price Paul Price added a comment -

          Could you explain why? The nice thing about plotting against PsfMags is that it matches what we do for detection, so the selection effects are uncomplicated (a vertical line, modulo deblending). Now, if you're only concerned about the bright end, that doesn't matter; but if you're concerned about low-level effects at the faint end, then I think PsfMags is definitely what you want to use.

          Show
          price Paul Price added a comment - Could you explain why? The nice thing about plotting against PsfMags is that it matches what we do for detection, so the selection effects are uncomplicated (a vertical line, modulo deblending). Now, if you're only concerned about the bright end, that doesn't matter; but if you're concerned about low-level effects at the faint end, then I think PsfMags is definitely what you want to use.
          Hide
          lauren Lauren MacArthur added a comment -

          My motivation is that PSF fluxes are wrong for galaxies, but CModel fluxes are representative for both stars and galaxies (as evidenced for the former in the CModel - PSF flux comparison plots).  The galaxies are often plotted, but not used in any "metric" calculations as those are limited to "stellar" objects.  A while back, some HSC folk asked for me to add the total (star + galaxy) histogram to the scatter plots (currently it's just the the separate histograms being plotted).  I believe the motivation for this request was to get an eyeball on the total luminosity function...it didn't seem right to use the PSF fluxes for that.

          I could revert the change if you feel strongly about it...I could also add a plot(s) that is specific to a "detection threshold" type of diagnostic (and I would welcome suggestions on what would be the ideal plot for this!)

          Show
          lauren Lauren MacArthur added a comment - My motivation is that PSF fluxes are wrong for galaxies, but CModel fluxes are representative for both stars and galaxies (as evidenced for the former in the CModel - PSF flux comparison plots).  The galaxies are often plotted, but not used in any "metric" calculations as those are limited to "stellar" objects.  A while back, some HSC folk asked for me to add the total (star + galaxy) histogram to the scatter plots (currently it's just the the separate histograms being plotted).  I believe the motivation for this request was to get an eyeball on the total luminosity function...it didn't seem right to use the PSF fluxes for that. I could revert the change if you feel strongly about it...I could also add a plot(s) that is specific to a "detection threshold" type of diagnostic (and I would welcome suggestions on what would be the ideal plot for this!)
          Hide
          lauren Lauren MacArthur added a comment -

          We discussed the CModel vs PSF flux issue at the Monday morning meeting here in Princeton (including our Princeton-non local contingent via BlueJeans).  Consensus was reached that it makes sense to use the CModel fluxes for these diagnostic plots.  The understanding for anyone using the plots is that they should not be used to asses detections limits/thresholds (the selection function with CModel will be more of a slope than a simple vertical line).  Since we are not using these plots for such purposes, plotting the best measured flux (CModel for coadds) is desirable.

          If there is a specific interest in assessing detection limits and thresholding via the pipe_analysis scripts, a dedicated (set of) plot(s) should be created.

          Show
          lauren Lauren MacArthur added a comment - We discussed the CModel vs PSF flux issue at the Monday morning meeting here in Princeton (including our Princeton-non local contingent via BlueJeans).  Consensus was reached that it makes sense to use the CModel fluxes for these diagnostic plots.  The understanding for anyone using the plots is that they should not be used to asses detections limits/thresholds (the selection function with CModel will be more of a slope than a simple vertical line).  Since we are not using these plots for such purposes, plotting the best measured flux (CModel for coadds) is desirable. If there is a specific interest in assessing detection limits and thresholding via the pipe_analysis scripts, a dedicated (set of) plot(s) should be created.
          Hide
          lauren Lauren MacArthur added a comment -

          To facilitate a combined test, the current branch has been rebased against DM-15689 and I have setup the tickets/DM-15916 branch of meas_mosaic.  I have tried running every combination of pipeAnalysisTask.py vs. old rerun (RC/w_2018_32/DM-15184/, which is pre both calib_psfUsed->calib_psf_used & flux->instFlux) and vs. latest rerun (RC/w_2018_41/DM-16011), including old vs. new along with output redirection in the comparison scripts and all processed all the way through. The results can be viewed at w32: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/instFluxBack/ and w41: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/instFlux/

          Show
          lauren Lauren MacArthur added a comment - To facilitate a combined test, the current branch has been rebased against DM-15689 and I have setup the tickets/ DM-15916 branch of meas_mosaic .  I have tried running every combination of pipeAnalysisTask.py vs. old rerun ( RC/w_2018_32/ DM-15184 / , which is pre both calib_psfUsed -> calib_psf_used & flux -> instFlux ) and vs. latest rerun ( RC/w_2018_41/ DM-16011 ), including old vs. new along with output redirection in the comparison scripts and all processed all the way through. The results can be viewed at w32: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/instFluxBack/ and w41: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/instFlux/
          Hide
          lauren Lauren MacArthur added a comment -

          Hsin-Fang Chiang this is now ready for review...is it too much to ask that you do a combined DM-15916, DM-15869, DM-15569 review?  As stated above, the tickets/DM-15916 branch of meas_mosaic needs to be setup for this to work and I have rebased this branch on top of tickets/DM-155869 of pipe_analysis so it all works with current master.

          Show
          lauren Lauren MacArthur added a comment - Hsin-Fang Chiang this is now ready for review...is it too much to ask that you do a combined DM-15916 , DM-15869 , DM-15569 review?  As stated above, the  tickets/ DM-15916 branch of meas_mosaic needs to be setup for this to work and I have rebased this branch on top of tickets/DM-155869 of pipe_analysis so it all works with current master.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          LGTM. I ran a simple example and also used the branch to do w_2018_38, and all seemed to work fine so far.  I would think the rerun input/output feature be handled in the pipe_base base class, but your approach in pipe_analysis is probably fine too, given that both pipe_analysis and CmdLineTask are to be replaced soon. 

          Show
          hchiang2 Hsin-Fang Chiang added a comment - LGTM. I ran a simple example and also used the branch to do w_2018_38, and all seemed to work fine so far.  I would think the rerun input/output feature be handled in the pipe_base base class, but your approach in pipe_analysis is probably fine too, given that both pipe_analysis and CmdLineTask are to be replaced soon. 
          Hide
          lauren Lauren MacArthur added a comment -

          Thanks! Yeah, I don't think there's any impetus for rerun functionality changes in the main stack.

          Show
          lauren Lauren MacArthur added a comment - Thanks! Yeah, I don't think there's any impetus for rerun functionality changes in the main stack.

            People

            • Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Lauren MacArthur, Paul Price, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel