Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20555

Run HiTS processing with doAddCalexpBackground = False

    Details

      Description

      Yusra AlSayyad suggests from attempting HSC processing that we should try using the calexp background in our HiTS processing as well. She also adjusted the background order to 2. This ticket is to do so and compare the results to the current ap_pipe defaults.

        Attachments

        1. ap_pipe_config.py
          0.7 kB
        2. run_ap_pipe.conf
          4 kB
        3. run_ap_pipe.sh
          2 kB
        4. run_ap_pipe.sl
          0.5 kB

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment -

            config.differencer.subtract['al'].kernel.active.spatialBgOrder = 2
            config.differencer.doAddCalexpBackground = False
            

            spatialBgOrder was more important than doAddCalexpBackground, and I make no claims that 2 is optimal, just that it is much better than 0.

            Show
            yusra Yusra AlSayyad added a comment - config.differencer.subtract['al'].kernel.active.spatialBgOrder = 2 config.differencer.doAddCalexpBackground = False spatialBgOrder was more important than doAddCalexpBackground, and I make no claims that 2 is optimal, just that it is much better than 0.
            Hide
            mrawls Meredith Rawls added a comment -

            Rerun cw_processed6 in progress.

            Both the config settings specified by Yusra were added. This is also the first full ap_pipe run with the new-style defects, and I updated my slurm script so it uses --longlog to enable eyeballing runtime metrics. The slurm script now also writes individual log files for each dataId (visit+ccd combination). Previously, the log file for any given ccd was overwritten when the subsequent visit was processed.

            Show
            mrawls Meredith Rawls added a comment - Rerun cw_processed6 in progress. Both the config settings specified by Yusra were added. This is also the first full ap_pipe run with the new-style defects, and I updated my slurm script so it uses  --longlog  to enable eyeballing runtime metrics. The slurm script now also writes individual log files for each dataId (visit+ccd combination). Previously, the log file for any given ccd was overwritten when the subsequent visit was processed.
            Hide
            mrawls Meredith Rawls added a comment -

            The run completed successfully and it looks like the best ap_pipe run yet! Some 10-15% fewer sources with the shape flag, and at a qualitative glance, I think adding these settings completely solved DM-18417 (a posteriori).

            At some point, we should figure out how to save all the config tweaks we have done to make ap_pipe on HiTS2015 better. So far it is just this and the photometric calibration reference selection magnitude limit.

            The output (calexps, diffims, PPDB, etc.) are in /project/mrawls/hits2015/rerun/cw_processed6.

            The logs are in /project/mrawls/hits2015/slurm/ap_pipe159876-[visit]-[task].out. Note that task is similar to ccdnum but not identical; it starts at 0 and does not skip 2 or 60.

            I will adopt this as the latest and greatest ap_pipe run on the HiTS 2015 dataset.

            Show
            mrawls Meredith Rawls added a comment - The run completed successfully and it looks like the best ap_pipe run yet! Some 10-15% fewer sources with the shape flag, and at a qualitative glance, I think adding these settings completely solved DM-18417 (a posteriori). At some point, we should figure out how to save all the config tweaks we have done to make ap_pipe on HiTS2015 better. So far it is just this and the photometric calibration reference selection magnitude limit. The output (calexps, diffims, PPDB, etc.) are in /project/mrawls/hits2015/rerun/cw_processed6 . The logs are in /project/mrawls/hits2015/slurm/ap_pipe159876- [visit] - [task] .out . Note that task is similar to ccdnum but not identical; it starts at 0 and does not skip 2 or 60. I will adopt this as the latest and greatest ap_pipe run on the HiTS 2015 dataset.
            Hide
            gkovacs Gabor Kovacs added a comment -

            Looks all right. For meditation, does this mean that the background is not well subtracted by processCcd ? Shouldn't we use in general a polynomial background fit?

            Show
            gkovacs Gabor Kovacs added a comment - Looks all right. For meditation, does this mean that the background is not well subtracted by processCcd ? Shouldn't we use in general a polynomial background fit?
            Hide
            mrawls Meredith Rawls added a comment -

            The issue seems to lie in a yet-untraced bug (e.g., DM-20484). It has a bigger effect on HSC than DECam, I think. There are background offsets in the difference images and something isn't working with background matching. For now, these config settings are a reasonable workaround. If I'm missing something perhaps Yusra AlSayyad can chime in, but I'm going to mark this as Done.

            Show
            mrawls Meredith Rawls added a comment - The issue seems to lie in a yet-untraced bug (e.g., DM-20484 ). It has a bigger effect on HSC than DECam, I think. There are background offsets in the difference images and something isn't working with background matching. For now, these config settings are a reasonable workaround. If I'm missing something perhaps Yusra AlSayyad can chime in, but I'm going to mark this as Done.

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                ebellm Eric Bellm
                Reviewers:
                Gabor Kovacs
                Watchers:
                Eric Bellm, Gabor Kovacs, Krzysztof Findeisen, Meredith Rawls, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel