Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15080

Process HiTS 2014, build template coadds

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      5
    • Epic Link:
    • Sprint:
      AP F18-2, AP F18-3, AP F18-4, AP F18-5
    • Team:
      Alert Production

      Description

      Following DM-14259, we can run the AP pipeline on the VC using SLURM. Use that capability to process the HiTS 2014 data and build template coadds.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            Processing is underway. These are the commands I am using on lsst-dev with the latest weekly, running everything from /project/mrawls/hits2014/ingested

             

            INGESTION
            $ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_04/*
            $ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_09/*
            $ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_10/*
            $ mkdir calib
            $ mkdir calib/defects_2013-11-29
            $ cp /datasets/decam/_internal/calib/bpmDes/2013-11-29/* calib/defects_2013-11-29/.
            [insert step here where I located and copied the relevant CP biases and flats for 2014 DECam into /project/mrawls/hits2014/preprocessed/ because the needed ones are not all in /datasets/decam/_internal/preprocessed/]
            $ ingestCalibs.py . --calib calib --calibType defect calib/defects_2013-11-29/* --validity 0 --mode=skip
            $ ingestCalibs.py . --calib calib/project/mrawls/hits2014/preprocessed/* --validity 9999 --mode=link
             
            PROCESSING
            $ processCcd.py . --calib calib --rerun processed1 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --id object='Blind14A_04'
            $ processCcd.py . --calib calib --rerun processed1 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --id object='Blind14A_09'
            $ processCcd.py . --calib calib --rerun processed1 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --id object='Blind14A_10'

            Show
            mrawls Meredith Rawls added a comment - Processing is underway. These are the commands I am using on lsst-dev with the latest weekly, running everything from /project/mrawls/hits2014/ingested   INGESTION $ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_04/* $ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_09/* $ ingestImagesDecam.py . --mode=link /datasets/decam/_internal/raw/hits/Blind14A_10/* $ mkdir calib $ mkdir calib/defects_2013- 11 - 29 $ cp /datasets/decam/_internal/calib/bpmDes/ 2013 - 11 - 29 /* calib/defects_2013- 11 - 29 /. [insert step here where I located and copied the relevant CP biases and flats for 2014 DECam into /project/mrawls/hits2014/preprocessed/ because the needed ones are not all in /datasets/decam/_internal/preprocessed/] $ ingestCalibs.py . --calib calib --calibType defect calib/defects_2013- 11 - 29 /* --validity 0 --mode=skip $ ingestCalibs.py . --calib calib/project/mrawls/hits2014/preprocessed/* --validity 9999 --mode=link   PROCESSING $ processCcd.py . --calib calib --rerun processed1 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --id object= 'Blind14A_04' $ processCcd.py . --calib calib --rerun processed1 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --id object= 'Blind14A_09' $ processCcd.py . --calib calib --rerun processed1 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --id object= 'Blind14A_10'
            Hide
            mrawls Meredith Rawls added a comment -

            This ticket has been slow going due to elusive coadd issues.

            I was initially following these steps to create best-seeing psf-matched coadds:

            makeDiscreteSkyMap.pyingested/rerun/processed1 --output coadds --id object='Blind14A_04'^'Blind14A_09'^'Blind14A_10'
             
            makeCoaddTempExp.pyingested/rerun/processed1 --output coadds -C ~/ap_pipe/config/makeCoaddTempExp_goodSeeing.py --config select.nImagesMax=5 --selectId object='Blind14A_04'^'Blind14A_09'^'Blind14A_10' --id filter=g
             
            assembleCoadd.py ingested/rerun/processed1 --output coadds --selectId object='Blind14A_04'^'Blind14A_09'^'Blind14A_10' --config doInterp=True warpType=psfMatched --id filter=g tract=0 patch=10,10^10,11^10,12^10,13^10,3^10,4^10,5^10,6^10,7^10,8^10,9^11,10^11,11^11,12^11,13^11,3^11,4^11,5^11,6^11,7^11,8^11,9^12,10^12,11^12,12^12,3^12,4^12,5^12,6^12,7^12,8^12,9^13,10^13,11^13,3^13,4^13,5^13,6^13,7^13,8^13,9^14,4^14,5^14,6^14,7^17,18^18,18^18,19^19,18^19,19^22,27^22,28^22,29^22,30^22,31^23,27^23,28^23,29^23,30^23,31^23,32^24,27^24,28^24,29^24,30^24,31^24,32^25,27^25,28^25,29^25,30^25,31^25,32^26,27^26,28^26,29^26,30^26,31^26,32^27,27^27,28^27,29^27,30^27,31^28,29^28,30^7,10^7,9^8,10^8,11^8,12^8,5^8,6^8,8^8,9^9,10^9,11^9,12^9,13^9,3^9,4^9,5^9,6^9,7^9,8^9,9 

            These steps appear to work, but the resulting coadds show many of the warps in the wrong place, lots of holes in the coadd where there aren't any images at all, and overall low-quality data with only 1-2 visits comprising the coadd in some cases. I re-ran with select.nImagesMax=10 to see if there was a difference, which helped me troubleshoot the issue.

            Briefly, there are two problems.

            (1) Some of the coadds are not landing in the correct place on the sky. In particular, a few errant chips want to land in a region where there clearly should not be any data, and some other chips may also be landing in more-subtly incorrect places. This is still under investigation, and for a while was conflated with the unrelated ds9-doesn't-do-wcs-mosaics-well issue discussed on community. It may ultimately be related to the other problem, namely

            (2) The best seeing selector is not working as intended. Instead of choosing up to nImagesMax of overlapping visits, it is using the visit+ccdnum dataId to make its selections. As a result, some areas have zero images selected, and the problem is worse for lower values of nImagesMax. See DM-16191 for more on this; the problem is actually rather subtle and illustrative of divide between tract+patch vs. visit+ccd. 

            To proceed with this ticket, the plan is to do one or both of

            • Just make psf-matched coadds using all the HiTS-2014 visits for now, because they at least shouldn't be worse than the old ones
            • Use the best seeing selector differently, by specifying a maxPsfFwhm = 4.2 (which is what Eric did last time) and setting nImagesMax very large (say 1000)
            Show
            mrawls Meredith Rawls added a comment - This ticket has been slow going due to elusive coadd issues. I was initially following these steps to create best-seeing psf-matched coadds: makeDiscreteSkyMap.pyingested/rerun/processed1 --output coadds --id object= 'Blind14A_04' ^ 'Blind14A_09' ^ 'Blind14A_10'   makeCoaddTempExp.pyingested/rerun/processed1 --output coadds -C ~/ap_pipe/config/makeCoaddTempExp_goodSeeing.py --config select.nImagesMax= 5 --selectId object= 'Blind14A_04' ^ 'Blind14A_09' ^ 'Blind14A_10' --id filter=g   assembleCoadd.py ingested/rerun/processed1 --output coadds --selectId object= 'Blind14A_04' ^ 'Blind14A_09' ^ 'Blind14A_10'  --config doInterp=True warpType=psfMatched --id filter=g tract= 0 patch= 10 , 10 ^ 10 , 11 ^ 10 , 12 ^ 10 , 13 ^ 10 , 3 ^ 10 , 4 ^ 10 , 5 ^ 10 , 6 ^ 10 , 7 ^ 10 , 8 ^ 10 , 9 ^ 11 , 10 ^ 11 , 11 ^ 11 , 12 ^ 11 , 13 ^ 11 , 3 ^ 11 , 4 ^ 11 , 5 ^ 11 , 6 ^ 11 , 7 ^ 11 , 8 ^ 11 , 9 ^ 12 , 10 ^ 12 , 11 ^ 12 , 12 ^ 12 , 3 ^ 12 , 4 ^ 12 , 5 ^ 12 , 6 ^ 12 , 7 ^ 12 , 8 ^ 12 , 9 ^ 13 , 10 ^ 13 , 11 ^ 13 , 3 ^ 13 , 4 ^ 13 , 5 ^ 13 , 6 ^ 13 , 7 ^ 13 , 8 ^ 13 , 9 ^ 14 , 4 ^ 14 , 5 ^ 14 , 6 ^ 14 , 7 ^ 17 , 18 ^ 18 , 18 ^ 18 , 19 ^ 19 , 18 ^ 19 , 19 ^ 22 , 27 ^ 22 , 28 ^ 22 , 29 ^ 22 , 30 ^ 22 , 31 ^ 23 , 27 ^ 23 , 28 ^ 23 , 29 ^ 23 , 30 ^ 23 , 31 ^ 23 , 32 ^ 24 , 27 ^ 24 , 28 ^ 24 , 29 ^ 24 , 30 ^ 24 , 31 ^ 24 , 32 ^ 25 , 27 ^ 25 , 28 ^ 25 , 29 ^ 25 , 30 ^ 25 , 31 ^ 25 , 32 ^ 26 , 27 ^ 26 , 28 ^ 26 , 29 ^ 26 , 30 ^ 26 , 31 ^ 26 , 32 ^ 27 , 27 ^ 27 , 28 ^ 27 , 29 ^ 27 , 30 ^ 27 , 31 ^ 28 , 29 ^ 28 , 30 ^ 7 , 10 ^ 7 , 9 ^ 8 , 10 ^ 8 , 11 ^ 8 , 12 ^ 8 , 5 ^ 8 , 6 ^ 8 , 8 ^ 8 , 9 ^ 9 , 10 ^ 9 , 11 ^ 9 , 12 ^ 9 , 13 ^ 9 , 3 ^ 9 , 4 ^ 9 , 5 ^ 9 , 6 ^ 9 , 7 ^ 9 , 8 ^ 9 , 9 These steps appear to work, but the resulting coadds show many of the warps in the wrong place, lots of holes in the coadd where there aren't any images at all, and overall low-quality data with only 1-2 visits comprising the coadd in some cases. I re-ran with select.nImagesMax=10 to see if there was a difference, which helped me troubleshoot the issue. Briefly, there are two problems. (1) Some of the coadds are not landing in the correct place on the sky. In particular, a few errant chips want to land in a region where there clearly should not be any data, and some other chips may also be landing in more-subtly incorrect places. This is still under investigation, and for a while was conflated with the unrelated ds9-doesn't-do-wcs-mosaics-well issue discussed on community. It may ultimately be related to the other problem, namely (2) The best seeing selector is not working as intended. Instead of choosing up to nImagesMax of overlapping visits, it is using the visit+ccdnum dataId to make its selections. As a result, some areas have zero images selected, and the problem is worse for lower values of nImagesMax. See DM-16191 for more on this; the problem is actually rather subtle and illustrative of divide between tract+patch vs. visit+ccd.  To proceed with this ticket, the plan is to do one or both of Just make psf-matched coadds using all the HiTS-2014 visits for now, because they at least shouldn't be worse than the old ones Use the best seeing selector differently, by specifying a maxPsfFwhm = 4.2 (which is what Eric did last time) and setting nImagesMax very large (say 1000)
            Hide
            mrawls Meredith Rawls added a comment -

            Can you please review this, Krzysztof Findeisen? There is no code, just a new set of coadds that now exist on lsst-dev. They live in /project/mrawls/hits2014 and the useful subdirectory there (as indicated by the README!) is coadds_good.

            Show
            mrawls Meredith Rawls added a comment - Can you please review this, Krzysztof Findeisen ? There is no code, just a new set of coadds that now exist on lsst-dev. They live in /project/mrawls/hits2014 and the useful subdirectory there (as indicated by the README!) is coadds_good .
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Hmm... are you sure the new coadds are better than the old ones? Running from lsst-dev:/scratch/krzys001/hits2015/, before:

            $ ap_pipe.py ingested/ --calib calibingested/ --template ${AP_VERIFY_HITS2015_DIR}/templates \
                    --rerun old_templates/ --doraise -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \
                    -c associator.level1_db.db_name=ingested/rerun/old_templates/association.db \
                    --id visit=411758 ccdnum=35 filter=g
            ...
            apPipe.differencer.detection INFO: Detected 69 positive peaks in 50 footprints and 80 negative peaks in 55 footprints to 5 sigma
            apPipe.differencer INFO: Merging detections into 57 sources
            

            and after:

            $ ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \
                    --rerun new_templates/ --doraise -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \
                    -c associator.level1_db.db_name=ingested/rerun/new_templates/association.db \
                    --id visit=411758 ccdnum=35 filter=g
            ...
            apPipe.differencer.detection INFO: Detected 86 positive peaks in 72 footprints and 68 negative peaks in 58 footprints to 5 sigma
            apPipe.differencer INFO: Merging detections into 97 sources
            

            Also, it would be nice if the (very congratulatory ) readme said that these are deepCoaddPsfMatched; the current description suggests they might be goodSeeingCoaddPsfMatched.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Hmm... are you sure the new coadds are better than the old ones? Running from lsst-dev:/scratch/krzys001/hits2015/ , before: $ ap_pipe.py ingested/ --calib calibingested/ --template ${AP_VERIFY_HITS2015_DIR}/templates \ --rerun old_templates/ --doraise -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/old_templates/association.db \ --id visit=411758 ccdnum=35 filter=g ... apPipe.differencer.detection INFO: Detected 69 positive peaks in 50 footprints and 80 negative peaks in 55 footprints to 5 sigma apPipe.differencer INFO: Merging detections into 57 sources and after: $ ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \ --rerun new_templates/ --doraise -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/new_templates/association.db \ --id visit=411758 ccdnum=35 filter=g ... apPipe.differencer.detection INFO: Detected 86 positive peaks in 72 footprints and 68 negative peaks in 58 footprints to 5 sigma apPipe.differencer INFO: Merging detections into 97 sources Also, it would be nice if the (very congratulatory ) readme said that these are deepCoaddPsfMatched ; the current description suggests they might be goodSeeingCoaddPsfMatched .
            Hide
            krzys Krzysztof Findeisen added a comment -

            Hmm, with large-number statistics it looks a bit better:

            $ nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template ${AP_VERIFY_HITS2015_DIR}/templates \
                    --rerun old_templates_full/ -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \
                    -c associator.level1_db.db_name=ingested/rerun/old_templates_full/association.db \
                    --id visit=411758 filter=g -j 20
            $ sqlite3 ingested/rerun/old_templates_full/association.db
            sqlite> select count(*) from dia_objects;
            19334
            

            versus

            $ nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \
                    --rerun new_templates_full/ -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \
                    -c associator.level1_db.db_name=ingested/rerun/new_templates_full/association.db \
                    --id visit=411758 filter=g -j 20
            $ sqlite3 ingested/rerun/new_templates_full/association.db
            sqlite> select count(*) from dia_objects;
            16556
            

            Any sense for how big of an improvement we were expecting?

            Show
            krzys Krzysztof Findeisen added a comment - Hmm, with large-number statistics it looks a bit better: $ nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template ${AP_VERIFY_HITS2015_DIR}/templates \ --rerun old_templates_full/ -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/old_templates_full/association.db \ --id visit=411758 filter=g -j 20 $ sqlite3 ingested/rerun/old_templates_full/association.db sqlite> select count(*) from dia_objects; 19334 versus $ nice -n 10 ap_pipe.py ingested/ --calib calibingested/ --template /project/mrawls/hits2014/coadds_good/ \ --rerun new_templates_full/ -C ${AP_VERIFY_HITS2015_DIR}/config/apPipe.py \ -c associator.level1_db.db_name=ingested/rerun/new_templates_full/association.db \ --id visit=411758 filter=g -j 20 $ sqlite3 ingested/rerun/new_templates_full/association.db sqlite> select count(*) from dia_objects; 16556 Any sense for how big of an improvement we were expecting?
            Hide
            krzys Krzysztof Findeisen added a comment -

            Per Slack discussion with Meredith Rawls, the goal for this ticket is "not worse", which the new coadds definitely are.

            Show
            krzys Krzysztof Findeisen added a comment - Per Slack discussion with Meredith Rawls , the goal for this ticket is "not worse", which the new coadds definitely are.
            Hide
            mrawls Meredith Rawls added a comment -

            Thanks Krzysztof Findeisen.

            For context, we discussed how a new run of ap_pipe (see DM-14762 for details) does demonstrate the new templates are at the very least "not worse." The larger number of sources in some regions is due to a larger number of edge detections with the new templates compared with the old ones, which is illustrated in the attached figure. (Blue = detected source with new template; Orange = detected source with old template)

            Show
            mrawls Meredith Rawls added a comment - Thanks Krzysztof Findeisen . For context, we discussed how a new run of ap_pipe (see DM-14762 for details) does demonstrate the new templates are at the very least "not worse." The larger number of sources in some regions is due to a larger number of edge detections with the new templates compared with the old ones, which is illustrated in the attached figure. (Blue = detected source with new template; Orange = detected source with old template)

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Krzysztof Findeisen
              Watchers:
              John Swinbank, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: