Monthly January ap_pipe rerun

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
8
• Sprint:
AP S20-2 (January)
• Team:

Description

This is the ~monthly ap_pipe rerun of the HiTS 2015 dataset (all 3 fields). This time we should see if setting nImagesMax in the best seeing selector will work with the new slurm template generation. To be clear, this rerun should include generating new templates as well as running ap_pipe.

Optional bonus level and/or spinoff ticket: adopt the templates generated as part of this rerun as the new default in the ap_verify datasets.

Activity

Hide
Eric Bellm added a comment -

Additionally this ticket will use the correct (arcsecond) values for seeing as input to the BestSeeingSelector.

Show
Eric Bellm added a comment - Additionally this ticket will use the correct (arcsecond) values for seeing as input to the BestSeeingSelector .
Hide
Meredith Rawls added a comment -

I generated new templates with the BestSeeingSelector using maxPsfFwhm of 1.4 arcseconds, and the results are much improved. Previously, I was errantly running the selector with excessively large maximum PSF sizes, thinking it was in pixels and not arcseconds. This caused it to select all (or nearly all) overlapping visits when assembling the coadd templates. Now it chooses significantly fewer, which results in higher quality templates. The ~3 problematic CCDs seen in DM-22729 look fine now, which makes me less inclined to spend much time on DM-22781.

Show
Meredith Rawls added a comment - I generated new templates with the BestSeeingSelector using maxPsfFwhm of 1.4 arcseconds, and the results are much improved. Previously, I was errantly running the selector with excessively large maximum PSF sizes, thinking it was in pixels and not arcseconds. This caused it to select all (or nearly all) overlapping visits when assembling the coadd templates. Now it chooses significantly fewer, which results in higher quality templates. The ~3 problematic CCDs seen in DM-22729 look fine now, which makes me less inclined to spend much time on DM-22781 .
Hide
Meredith Rawls added a comment -

Steps taken

• makeDiscreteSkyMap.py for HiTS 2014
• ap_pipe.py on HiTS 2015 with the newly assembled coadds as templates

The relevant repositories are on lsst-dev in /project/mrawls/hits2014/rerun/2020_01_coadds_full and /project/mrawls/hits2015/rerun/cw_2020_01.

I chose not to mess with nImagesMax in the makeCoaddTempExp step with BestSeeingWcsSelectImagesTask this time, and instead left it large (1000) as before. It does take a day or so to make new warps for all 3 fields with only 2 parallel processes, but this isn't presently prohibitive.

Instead, I adjusted maxPsfFwhm=1.4 arcsec (again in BestSeeingWcsSelectImagesTask). This is because it was previously set too high (> 4), a value which would make sense if it were in pixels but was errantly allowing essentially all of the images to be selected rather than just some with better seeing than others.

There were no problems seen in the output, and the excess sources seen in the Dec 2019 rerun have disappeared.

I noted an increase in the total DIA Source count and a decrease in the good DIA source count, which is attributable to fewer visits comprising the templates. This results in (1) slightly greater chip gap template coverage in December than in January as well as (2) larger regions masked SENSOR_EDGE in December compared to January. These effects are visible in the plots in this notebook.

I recommend adopting these new templates for all our ap_verify HiTS datasets. I made DM-23311 for this.

Show
Hide
Ian Sullivan added a comment -

Looks very good!

One minor comment, is to please make sure that your comparison histograms have the same binning, and that they are all semi-transparent if they are plotted on top of each other. This is only relevant for your one plot near the end of the notebook, and it is more a note to keep in mind for the future than something that needs to be changed now.

Show
Ian Sullivan added a comment - Looks very good!   One minor comment, is to please make sure that your comparison histograms have the same binning, and that they are all semi-transparent if they are plotted on top of each other. This is only relevant for your one plot near the end of the notebook, and it is more a note to keep in mind for the future than something that needs to be changed now.

People

Assignee:
Meredith Rawls
Reporter:
Meredith Rawls
Reviewers:
Ian Sullivan
Watchers:
Eric Bellm, Gabor Kovacs, Ian Sullivan, Meredith Rawls