# Run HiTS processing with doAddCalexpBackground = False

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
6
• Sprint:
AP F19-1, AP F19-2
• Team:

## Description

Yusra AlSayyad suggests from attempting HSC processing that we should try using the calexp background in our HiTS processing as well. She also adjusted the background order to 2. This ticket is to do so and compare the results to the current ap_pipe defaults.

## Attachments

1. ap_pipe_config.py
0.7 kB
2. run_ap_pipe.conf
4 kB
3. run_ap_pipe.sh
2 kB
4. run_ap_pipe.sl
0.5 kB

## Activity

Hide

 config.differencer.subtract['al'].kernel.active.spatialBgOrder = 2 config.differencer.doAddCalexpBackground = False 

spatialBgOrder was more important than doAddCalexpBackground, and I make no claims that 2 is optimal, just that it is much better than 0.

Show
Yusra AlSayyad added a comment - config.differencer.subtract['al'].kernel.active.spatialBgOrder = 2 config.differencer.doAddCalexpBackground = False spatialBgOrder was more important than doAddCalexpBackground, and I make no claims that 2 is optimal, just that it is much better than 0.
Hide
Meredith Rawls added a comment -

Rerun cw_processed6 in progress.

Both the config settings specified by Yusra were added. This is also the first full ap_pipe run with the new-style defects, and I updated my slurm script so it uses --longlog to enable eyeballing runtime metrics. The slurm script now also writes individual log files for each dataId (visit+ccd combination). Previously, the log file for any given ccd was overwritten when the subsequent visit was processed.

Show
Meredith Rawls added a comment - Rerun cw_processed6 in progress. Both the config settings specified by Yusra were added. This is also the first full ap_pipe run with the new-style defects, and I updated my slurm script so it uses  --longlog  to enable eyeballing runtime metrics. The slurm script now also writes individual log files for each dataId (visit+ccd combination). Previously, the log file for any given ccd was overwritten when the subsequent visit was processed.
Hide
Meredith Rawls added a comment -

The run completed successfully and it looks like the best ap_pipe run yet! Some 10-15% fewer sources with the shape flag, and at a qualitative glance, I think adding these settings completely solved DM-18417 (a posteriori).

At some point, we should figure out how to save all the config tweaks we have done to make ap_pipe on HiTS2015 better. So far it is just this and the photometric calibration reference selection magnitude limit.

The output (calexps, diffims, PPDB, etc.) are in /project/mrawls/hits2015/rerun/cw_processed6.

The logs are in /project/mrawls/hits2015/slurm/ap_pipe159876-[visit]-[task].out. Note that task is similar to ccdnum but not identical; it starts at 0 and does not skip 2 or 60.

I will adopt this as the latest and greatest ap_pipe run on the HiTS 2015 dataset.

Show
Meredith Rawls added a comment - The run completed successfully and it looks like the best ap_pipe run yet! Some 10-15% fewer sources with the shape flag, and at a qualitative glance, I think adding these settings completely solved DM-18417 (a posteriori). At some point, we should figure out how to save all the config tweaks we have done to make ap_pipe on HiTS2015 better. So far it is just this and the photometric calibration reference selection magnitude limit. The output (calexps, diffims, PPDB, etc.) are in /project/mrawls/hits2015/rerun/cw_processed6 . The logs are in /project/mrawls/hits2015/slurm/ap_pipe159876- [visit] - [task] .out . Note that task is similar to ccdnum but not identical; it starts at 0 and does not skip 2 or 60. I will adopt this as the latest and greatest ap_pipe run on the HiTS 2015 dataset.
Hide
Gabor Kovacs added a comment -

Looks all right. For meditation, does this mean that the background is not well subtracted by processCcd ? Shouldn't we use in general a polynomial background fit?

Show
Gabor Kovacs added a comment - Looks all right. For meditation, does this mean that the background is not well subtracted by processCcd ? Shouldn't we use in general a polynomial background fit?
Hide
Meredith Rawls added a comment -

The issue seems to lie in a yet-untraced bug (e.g., DM-20484). It has a bigger effect on HSC than DECam, I think. There are background offsets in the difference images and something isn't working with background matching. For now, these config settings are a reasonable workaround. If I'm missing something perhaps Yusra AlSayyad can chime in, but I'm going to mark this as Done.

Show
Meredith Rawls added a comment - The issue seems to lie in a yet-untraced bug (e.g., DM-20484 ). It has a bigger effect on HSC than DECam, I think. There are background offsets in the difference images and something isn't working with background matching. For now, these config settings are a reasonable workaround. If I'm missing something perhaps Yusra AlSayyad can chime in, but I'm going to mark this as Done.

## People

• Assignee:
Meredith Rawls
Reporter:
Eric Bellm
Reviewers:
Gabor Kovacs
Watchers:
Eric Bellm, Gabor Kovacs, Krzysztof Findeisen, Meredith Rawls, Yusra AlSayyad