# Process HiTS 2014, build template coadds

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
5
• Sprint:
AP F18-2, AP F18-3, AP F18-4, AP F18-5
• Team:

#### Description

Following DM-14259, we can run the AP pipeline on the VC using SLURM. Use that capability to process the HiTS 2014 data and build template coadds.

#### Attachments

1. object_compare_4ccds.png
109 kB

#### Activity

Hide
Meredith Rawls added a comment -

Processing is underway. These are the commands I am using on lsst-dev with the latest weekly, running everything from /project/mrawls/hits2014/ingested

 INGESTION $ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_04/* $ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_09/* $ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_10/* $ mkdir calib $mkdir calib/defects_2013-11-29 $ cp /datasets/decam/_internal/calib/bpmDes/2013-11-29/* calib/defects_2013-11-29/. [insert step here where I located and copied the relevant CP biases and flats for 2014 DECam into /project/mrawls/hits2014/preprocessed/ because the needed ones are not all in /datasets/decam/_internal/preprocessed/] $ingestCalibs.py . --calib calib --calibType defect calib/defects_2013-11-29/* --validity 0 --mode=skip $ ingestCalibs.py . --calib calib/project/mrawls/hits2014/preprocessed/* --validity 9999 --mode=link   PROCESSING $processCcd.py . --calib calib --rerun processed1 -C$OBS_DECAM_DIR/config/processCcdCpIsr.py --id object='Blind14A_04' $processCcd.py . --calib calib --rerun processed1 -C$OBS_DECAM_DIR/config/processCcdCpIsr.py --id object='Blind14A_09' $processCcd.py . --calib calib --rerun processed1 -C$OBS_DECAM_DIR/config/processCcdCpIsr.py --id object='Blind14A_10'

Show
Meredith Rawls added a comment - Processing is underway. These are the commands I am using on lsst-dev with the latest weekly, running everything from /project/mrawls/hits2014/ingested   INGESTION $ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_04/*$ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_09/* $ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_10/*$ mkdir calib $mkdir calib/defects_2013- 11 - 29$ cp /datasets/decam/_internal/calib/bpmDes/ 2013 - 11 - 29 /* calib/defects_2013- 11 - 29 /. [insert step here where I located and copied the relevant CP biases and flats for 2014 DECam into /project/mrawls/hits2014/preprocessed/ because the needed ones are not all in /datasets/decam/_internal/preprocessed/] $ingestCalibs.py . --calib calib --calibType defect calib/defects_2013- 11 - 29 /* --validity 0 --mode=skip$ ingestCalibs.py . --calib calib/project/mrawls/hits2014/preprocessed/* --validity 9999 --mode=link   PROCESSING $processCcd.py . --calib calib --rerun processed1 -C$OBS_DECAM_DIR/config/processCcdCpIsr.py --id object= 'Blind14A_04' $processCcd.py . --calib calib --rerun processed1 -C$OBS_DECAM_DIR/config/processCcdCpIsr.py --id object= 'Blind14A_09' $processCcd.py . --calib calib --rerun processed1 -C$OBS_DECAM_DIR/config/processCcdCpIsr.py --id object= 'Blind14A_10'
Hide
Meredith Rawls added a comment -

This ticket has been slow going due to elusive coadd issues.

I was initially following these steps to create best-seeing psf-matched coadds:

 makeDiscreteSkyMap.pyingested/rerun/processed1 --output coadds --id object='Blind14A_04'^'Blind14A_09'^'Blind14A_10'   makeCoaddTempExp.pyingested/rerun/processed1 --output coadds -C ~/ap_pipe/config/makeCoaddTempExp_goodSeeing.py --config select.nImagesMax=5 --selectId object='Blind14A_04'^'Blind14A_09'^'Blind14A_10' --id filter=g   assembleCoadd.py ingested/rerun/processed1 --output coadds --selectId object='Blind14A_04'^'Blind14A_09'^'Blind14A_10' --config doInterp=True warpType=psfMatched --id filter=g tract=0 patch=10,10^10,11^10,12^10,13^10,3^10,4^10,5^10,6^10,7^10,8^10,9^11,10^11,11^11,12^11,13^11,3^11,4^11,5^11,6^11,7^11,8^11,9^12,10^12,11^12,12^12,3^12,4^12,5^12,6^12,7^12,8^12,9^13,10^13,11^13,3^13,4^13,5^13,6^13,7^13,8^13,9^14,4^14,5^14,6^14,7^17,18^18,18^18,19^19,18^19,19^22,27^22,28^22,29^22,30^22,31^23,27^23,28^23,29^23,30^23,31^23,32^24,27^24,28^24,29^24,30^24,31^24,32^25,27^25,28^25,29^25,30^25,31^25,32^26,27^26,28^26,29^26,30^26,31^26,32^27,27^27,28^27,29^27,30^27,31^28,29^28,30^7,10^7,9^8,10^8,11^8,12^8,5^8,6^8,8^8,9^9,10^9,11^9,12^9,13^9,3^9,4^9,5^9,6^9,7^9,8^9,9

These steps appear to work, but the resulting coadds show many of the warps in the wrong place, lots of holes in the coadd where there aren't any images at all, and overall low-quality data with only 1-2 visits comprising the coadd in some cases. I re-ran with select.nImagesMax=10 to see if there was a difference, which helped me troubleshoot the issue.

Briefly, there are two problems.

(1) Some of the coadds are not landing in the correct place on the sky. In particular, a few errant chips want to land in a region where there clearly should not be any data, and some other chips may also be landing in more-subtly incorrect places. This is still under investigation, and for a while was conflated with the unrelated ds9-doesn't-do-wcs-mosaics-well issue discussed on community. It may ultimately be related to the other problem, namely

(2) The best seeing selector is not working as intended. Instead of choosing up to nImagesMax of overlapping visits, it is using the visit+ccdnum dataId to make its selections. As a result, some areas have zero images selected, and the problem is worse for lower values of nImagesMax. See DM-16191 for more on this; the problem is actually rather subtle and illustrative of divide between tract+patch vs. visit+ccd.

To proceed with this ticket, the plan is to do one or both of

• Just make psf-matched coadds using all the HiTS-2014 visits for now, because they at least shouldn't be worse than the old ones
• Use the best seeing selector differently, by specifying a maxPsfFwhm = 4.2 (which is what Eric did last time) and setting nImagesMax very large (say 1000)
Show
Hide
Meredith Rawls added a comment -

Can you please review this, Krzysztof Findeisen? There is no code, just a new set of coadds that now exist on lsst-dev. They live in /project/mrawls/hits2014 and the useful subdirectory there (as indicated by the README!) is coadds_good.

Show
Meredith Rawls added a comment - Can you please review this, Krzysztof Findeisen ? There is no code, just a new set of coadds that now exist on lsst-dev. They live in /project/mrawls/hits2014 and the useful subdirectory there (as indicated by the README!) is coadds_good .
Hide
Krzysztof Findeisen added a comment - - edited

Hmm... are you sure the new coadds are better than the old ones? Running from lsst-dev:/scratch/krzys001/hits2015/, before:

 $ap_pipe.py ingested/ --calib calibingested/ --template${AP_VERIFY_HITS2015_DIR}/templates \  --rerun old_templates/ --doraise -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \  -c associator.level1_db.db_name=ingested/rerun/old_templates/association.db \  --id visit=411758 ccdnum=35 filter=g ... apPipe.differencer.detection INFO: Detected 69 positive peaks in 50 footprints and 80 negative peaks in 55 footprints to 5 sigma apPipe.differencer INFO: Merging detections into 57 sources  and after:  $ ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \  --rerun new_templates/ --doraise -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \  -c associator.level1_db.db_name=ingested/rerun/new_templates/association.db \  --id visit=411758 ccdnum=35 filter=g ... apPipe.differencer.detection INFO: Detected 86 positive peaks in 72 footprints and 68 negative peaks in 58 footprints to 5 sigma apPipe.differencer INFO: Merging detections into 97 sources  Also, it would be nice if the (very congratulatory ) readme said that these are deepCoaddPsfMatched; the current description suggests they might be goodSeeingCoaddPsfMatched. Show Krzysztof Findeisen added a comment - - edited Hmm... are you sure the new coadds are better than the old ones? Running from lsst-dev:/scratch/krzys001/hits2015/ , before:$ ap_pipe.py ingested/ --calib calibingested/ --template ${AP_VERIFY_HITS2015_DIR}/templates \ --rerun old_templates/ --doraise -C${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/old_templates/association.db \ --id visit=411758 ccdnum=35 filter=g ... apPipe.differencer.detection INFO: Detected 69 positive peaks in 50 footprints and 80 negative peaks in 55 footprints to 5 sigma apPipe.differencer INFO: Merging detections into 57 sources and after: $ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \ --rerun new_templates/ --doraise -C${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/new_templates/association.db \ --id visit=411758 ccdnum=35 filter=g ... apPipe.differencer.detection INFO: Detected 86 positive peaks in 72 footprints and 68 negative peaks in 58 footprints to 5 sigma apPipe.differencer INFO: Merging detections into 97 sources Also, it would be nice if the (very congratulatory ) readme said that these are deepCoaddPsfMatched ; the current description suggests they might be goodSeeingCoaddPsfMatched .
Hide
Krzysztof Findeisen added a comment -

Hmm, with large-number statistics it looks a bit better:

 $nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template${AP_VERIFY_HITS2015_DIR}/templates \  --rerun old_templates_full/ -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \  -c associator.level1_db.db_name=ingested/rerun/old_templates_full/association.db \  --id visit=411758 filter=g -j 20 $ sqlite3 ingested/rerun/old_templates_full/association.db sqlite> select count(*) from dia_objects; 19334 

versus

 $nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \  --rerun new_templates_full/ -C${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \  -c associator.level1_db.db_name=ingested/rerun/new_templates_full/association.db \  --id visit=411758 filter=g -j 20 $sqlite3 ingested/rerun/new_templates_full/association.db sqlite> select count(*) from dia_objects; 16556  Any sense for how big of an improvement we were expecting? Show Krzysztof Findeisen added a comment - Hmm, with large-number statistics it looks a bit better:$ nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template ${AP_VERIFY_HITS2015_DIR}/templates \ --rerun old_templates_full/ -C${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/old_templates_full/association.db \ --id visit=411758 filter=g -j 20 $sqlite3 ingested/rerun/old_templates_full/association.db sqlite> select count(*) from dia_objects; 19334 versus$ nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \ --rerun new_templates_full/ -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/new_templates_full/association.db \ --id visit=411758 filter=g -j 20$ sqlite3 ingested/rerun/new_templates_full/association.db sqlite> select count(*) from dia_objects; 16556 Any sense for how big of an improvement we were expecting?
Hide
Krzysztof Findeisen added a comment -

Per Slack discussion with Meredith Rawls, the goal for this ticket is "not worse", which the new coadds definitely are.

Show
Krzysztof Findeisen added a comment - Per Slack discussion with Meredith Rawls , the goal for this ticket is "not worse", which the new coadds definitely are.
Hide
Meredith Rawls added a comment -

Thanks Krzysztof Findeisen.

For context, we discussed how a new run of ap_pipe (see DM-14762 for details) does demonstrate the new templates are at the very least "not worse." The larger number of sources in some regions is due to a larger number of edge detections with the new templates compared with the old ones, which is illustrated in the attached figure. (Blue = detected source with new template; Orange = detected source with old template)

Show
Meredith Rawls added a comment - Thanks Krzysztof Findeisen . For context, we discussed how a new run of ap_pipe (see DM-14762 for details) does demonstrate the new templates are at the very least "not worse." The larger number of sources in some regions is due to a larger number of edge detections with the new templates compared with the old ones, which is illustrated in the attached figure. (Blue = detected source with new template; Orange = detected source with old template)

#### People

Assignee:
Meredith Rawls
Reporter:
John Swinbank
Reviewers:
Krzysztof Findeisen
Watchers:
John Swinbank, Krzysztof Findeisen, Meredith Rawls