Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23069

Monthly January ap_pipe rerun

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe
    • Labels:
      None
    • Story Points:
      8
    • Sprint:
      AP S20-2 (January)
    • Team:
      Alert Production

      Description

      This is the ~monthly ap_pipe rerun of the HiTS 2015 dataset (all 3 fields). This time we should see if setting nImagesMax in the best seeing selector will work with the new slurm template generation. To be clear, this rerun should include generating new templates as well as running ap_pipe.

      Optional bonus level and/or spinoff ticket: adopt the templates generated as part of this rerun as the new default in the ap_verify datasets.

        Attachments

          Issue Links

            Activity

            Hide
            ebellm Eric Bellm added a comment -

            Additionally this ticket will use the correct (arcsecond) values for seeing as input to the BestSeeingSelector.

            Show
            ebellm Eric Bellm added a comment - Additionally this ticket will use the correct (arcsecond) values for seeing as input to the BestSeeingSelector .
            Hide
            mrawls Meredith Rawls added a comment -

            I generated new templates with the `BestSeeingSelector` using `maxPsfFwhm` of 1.4 arcseconds, and the results are much improved. Previously, I was errantly running the selector with excessively large maximum PSF sizes, thinking it was in pixels and not arcseconds. This caused it to select all (or nearly all) overlapping visits when assembling the coadd templates. Now it chooses significantly fewer, which results in higher quality templates. The ~3 problematic CCDs seen in DM-22729 look fine now, which makes me less inclined to spend much time on DM-22781.

            Show
            mrawls Meredith Rawls added a comment - I generated new templates with the `BestSeeingSelector` using `maxPsfFwhm` of 1.4 arcseconds, and the results are much improved. Previously, I was errantly running the selector with excessively large maximum PSF sizes, thinking it was in pixels and not arcseconds. This caused it to select all (or nearly all) overlapping visits when assembling the coadd templates. Now it chooses significantly fewer, which results in higher quality templates. The ~3 problematic CCDs seen in DM-22729 look fine now, which makes me less inclined to spend much time on DM-22781 .
            Hide
            mrawls Meredith Rawls added a comment -

            Steps taken

            • makeDiscreteSkyMap.py for HiTS 2014
            • makeCoaddTempExp.py on HiTS 2014
            • assembleCoadd.py on the result
            • ap_pipe.py on HiTS 2015 with the newly assembled coadds as templates

            The relevant repositories are on lsst-dev in /project/mrawls/hits2014/rerun/2020_01_coadds_full and /project/mrawls/hits2015/rerun/cw_2020_01.

            I chose not to mess with nImagesMax in the makeCoaddTempExp step with BestSeeingWcsSelectImagesTask this time, and instead left it large (1000) as before. It does take a day or so to make new warps for all 3 fields with only 2 parallel processes, but this isn't presently prohibitive.

            Instead, I adjusted maxPsfFwhm=1.4 arcsec (again in BestSeeingWcsSelectImagesTask). This is because it was previously set too high (> 4), a value which would make sense if it were in pixels but was errantly allowing essentially all of the images to be selected rather than just some with better seeing than others.

            There were no problems seen in the output, and the excess sources seen in the Dec 2019 rerun have disappeared.

            I noted an increase in the total DIA Source count and a decrease in the good DIA source count, which is attributable to fewer visits comprising the templates. This results in (1) slightly greater chip gap template coverage in December than in January as well as (2) larger regions masked SENSOR_EDGE in December compared to January. These effects are visible in the plots in this notebook.

            I recommend adopting these new templates for all our ap_verify HiTS datasets. I made DM-23311 for this.

            Show
            mrawls Meredith Rawls added a comment - Steps taken makeDiscreteSkyMap.py for HiTS 2014 makeCoaddTempExp.py on HiTS 2014 assembleCoadd.py on the result ap_pipe.py on HiTS 2015 with the newly assembled coadds as templates The relevant repositories are on lsst-dev in /project/mrawls/hits2014/rerun/2020_01_coadds_full and /project/mrawls/hits2015/rerun/cw_2020_01. I chose not to mess with nImagesMax in the makeCoaddTempExp step with BestSeeingWcsSelectImagesTask this time, and instead left it large ( 1000 ) as before. It does take a day or so to make new warps for all 3 fields with only 2 parallel processes, but this isn't presently prohibitive. Instead, I adjusted maxPsfFwhm=1.4 arcsec (again in BestSeeingWcsSelectImagesTask ). This is because it was previously set too high (> 4), a value which would make sense if it were in pixels but was errantly allowing essentially all of the images to be selected rather than just some with better seeing than others. There were no problems seen in the output, and the excess sources seen in the Dec 2019 rerun have disappeared. I noted an increase in the total DIA Source count and a decrease in the good DIA source count, which is attributable to fewer visits comprising the templates. This results in (1) slightly greater chip gap template coverage in December than in January as well as (2) larger regions masked SENSOR_EDGE in December compared to January. These effects are visible in the plots in this notebook. I recommend adopting these new templates for all our ap_verify HiTS datasets. I made DM-23311 for this.
            Hide
            sullivan Ian Sullivan added a comment -

            Looks very good!

             

            One minor comment, is to please make sure that your comparison histograms have the same binning, and that they are all semi-transparent if they are plotted on top of each other. This is only relevant for your one plot near the end of the notebook, and it is more a note to keep in mind for the future than something that needs to be changed now.

            Show
            sullivan Ian Sullivan added a comment - Looks very good!   One minor comment, is to please make sure that your comparison histograms have the same binning, and that they are all semi-transparent if they are plotted on top of each other. This is only relevant for your one plot near the end of the notebook, and it is more a note to keep in mind for the future than something that needs to be changed now.

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                mrawls Meredith Rawls
                Reviewers:
                Ian Sullivan
                Watchers:
                Eric Bellm, Gabor Kovacs, Ian Sullivan, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel