Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30703

Reprocess DECam HiTS data from scratch with background fixes

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Once DM-30651 is sorted out, we need to reprocess alllllll /repo/main DECam data from scratch with the correct config settings in cp_pipe and ISR (including RunIsrForCrosstalkSources).

      As part of this ticket, get all the CHAINED and TAGGED collections for all the latest and greatest DECam datasets and runs defined correctly, and consider blowing away old/bad ones people shouldn't be using.

      This ticket definitely includes successfully completing single frame processing for hits2014 and hits2015. Ideally it also includes making templates from 2014 and running the full AP Pipeline with 2015 as science, as usual.

        Attachments

        1. 8363_nImage_mosaic.png
          8363_nImage_mosaic.png
          68 kB
        2. 9813_nImage_mosaic.png
          9813_nImage_mosaic.png
          88 kB
        3. ApPipe_DM30703_bps.yaml
          2 kB
        4. ApPipe_DM30703.png
          ApPipe_DM30703.png
          405 kB
        5. ApPipe_DM30703.yaml
          5 kB
        6. ApTemplate_DM30703_bps.yaml
          2 kB
        7. ApTemplate_DM30703.yaml
          0.4 kB

          Issue Links

            Activity

            Hide
            sullivan Ian Sullivan added a comment -

            Let's also add Fakes to this processing run.

            Show
            sullivan Ian Sullivan added a comment - Let's also add Fakes to this processing run.
            Hide
            mrawls Meredith Rawls added a comment - - edited

            Status update on this ticket

            • Made a pipeline that inserts fakes, builds good seeing templates, does multi-tract-friendly differencing, and finally association (DiaPipe) that writes to a postgres APDB. It works as intended with one major caveat: can only run one tract at a time due to how fakes work (see DM-31964). Getting this to work took lots of trial and error due to the new execution butler, new imageDifference related configs (DM-29310), plus DM-30210.
            • Built new crosstalk source images, median biases, and median flat frames for each night of HiTS 2014 and 2015, using the new DM-30651 recommended per-amp configs. Presently working on certifying them to /repo/main to DECam/calib/DM-30703, which I will chain prepend-style to DECam/calib.
            • Next step is to run the pipeline, and there are a few options for this worth considering.
            1. If DM-31964 takes a while, worth considering running tract-by-tract. Could uncover interesting issues and/or point of comparison with this option vs. running multiple tracts at the same time.
            2. Current pipeline I wrote (/project/mrawls/repo-main-logs/ApPipe_DM30703.yaml) builds templates as it goes, which I will take the liberty of calling "DRP-style," but in the past, I've built templates from one dataset (HiTS 2014) in "step 1" and run the AP Pipeline in "step 2" on a different dataset (HiTS 2015), which might be better characterized as "AP-style." Might be interesting to compare these two approaches.
            3. There's always the option of turning off fakes entirely to try multi-tract image differencing on a large scale in the interim, I don't know whether we've done any tests in true tract overlap regions.
            Show
            mrawls Meredith Rawls added a comment - - edited Status update on this ticket Made a pipeline that inserts fakes, builds good seeing templates, does multi-tract-friendly differencing, and finally association (DiaPipe) that writes to a postgres APDB. It works as intended with one major caveat: can only run one tract at a time due to how fakes work (see DM-31964 ). Getting this to work took lots of trial and error due to the new execution butler, new imageDifference related configs ( DM-29310 ), plus DM-30210 . Built new crosstalk source images, median biases, and median flat frames for each night of HiTS 2014 and 2015, using the new DM-30651 recommended per-amp configs. Presently working on certifying them to /repo/main to DECam/calib/ DM-30703 , which I will chain prepend-style to DECam/calib. Next step is to run the pipeline, and there are a few options for this worth considering. If DM-31964 takes a while, worth considering running tract-by-tract. Could uncover interesting issues and/or point of comparison with this option vs. running multiple tracts at the same time. Current pipeline I wrote (/project/mrawls/repo-main-logs/ApPipe_DM30703.yaml) builds templates as it goes, which I will take the liberty of calling "DRP-style," but in the past, I've built templates from one dataset (HiTS 2014) in "step 1" and run the AP Pipeline in "step 2" on a different dataset (HiTS 2015), which might be better characterized as "AP-style." Might be interesting to compare these two approaches. There's always the option of turning off fakes entirely to try multi-tract image differencing on a large scale in the interim, I don't know whether we've done any tests in true tract overlap regions.
            Hide
            ebellm Eric Bellm added a comment -

            After sprint planning discussion on 10/4, we agreed (on #2) to move to AP-style image differencing, which requires building separate templates. Once those templates are in hand, evaluate the status of DM-31964. If it is still a ways off, proceed with fake-injection differencing on single tracts at a time.

            Show
            ebellm Eric Bellm added a comment - After sprint planning discussion on 10/4, we agreed (on #2) to move to AP-style image differencing, which requires building separate templates. Once those templates are in hand, evaluate the status of DM-31964 . If it is still a ways off, proceed with fake-injection differencing on single tracts at a time.
            Hide
            mrawls Meredith Rawls added a comment -

            I got templates built and 16 ApPipe runs to successfully complete via bps with only a handful of expected errors in characterizeImage and/or calibrate on known bad detectors 2 and 61. There were minimal bps hiccups and I understand the execution Butler much better now!

            OK so the bad news... no errors were reported in diaPipe, but every quantum's log says "Nothing to do for task 'diaPipe'". I haven't figured out why yet. The logs are in /project/mrawls/repo-main-logs/bps/u/mrawls/DM-30703-DECam-fakes-TRACT, where you replace TRACT with one of 16 valid tracts that I ran processing on.

            Show
            mrawls Meredith Rawls added a comment - I got templates built and 16 ApPipe runs to successfully complete via bps with only a handful of expected errors in characterizeImage and/or calibrate on known bad detectors 2 and 61. There were minimal bps hiccups and I understand the execution Butler much better now! OK so the bad news... no errors were reported in diaPipe, but every quantum's log says "Nothing to do for task 'diaPipe'". I haven't figured out why yet. The logs are in /project/mrawls/repo-main-logs/bps/u/mrawls/ DM-30703 -DECam-fakes-TRACT , where you replace TRACT with one of 16 valid tracts that I ran processing on.
            Hide
            mrawls Meredith Rawls added a comment - - edited

            Postmortem for this ticket...

            • Collections exist in /repo/main with names like u/mrawls/DM-30703-DECam-fakes-9813 (where 9813 is the tract number; 16 collections exist for 16 different tracts, many of them with spatial overlap)
            • The calexps and source catalogs processed successfully, with all but a couple failures in detectors 2 and 61 as expected
            • This is good progress because we now have processed visit images with the new and improved calibrations!!

            However...

            • No difference images or later pipeline outputs exist due to a bug Jim and Nate are aware of (edit: nominally to be fixed on DM-31769); the new template retrieving task uses calexp.wcs and calexp.bbox as inputs instead of the whole calexp, and caused downstream quanta to try to run before their inputs were ready. As a result, all logs for image differencing, diaPipe, etc., state "Nothing to do for task X in quantum Y," yet bps reports this as a "success."
            • As a result, there is nothing much to analyze in the output collections, and no corresponding analysis notebook or plots worth sharing.
            Show
            mrawls Meredith Rawls added a comment - - edited Postmortem for this ticket... Collections exist in /repo/main with names like u/mrawls/ DM-30703 -DECam-fakes-9813 (where 9813 is the tract number; 16 collections exist for 16 different tracts, many of them with spatial overlap) The calexps and source catalogs processed successfully, with all but a couple failures in detectors 2 and 61 as expected This is good progress because we now have processed visit images with the new and improved calibrations!! However... No difference images or later pipeline outputs exist due to a bug Jim and Nate are aware of (edit: nominally to be fixed on DM-31769 ); the new template retrieving task uses calexp.wcs and calexp.bbox as inputs instead of the whole calexp, and caused downstream quanta to try to run before their inputs were ready. As a result, all logs for image differencing, diaPipe, etc., state "Nothing to do for task X in quantum Y," yet bps reports this as a "success." As a result, there is nothing much to analyze in the output collections, and no corresponding analysis notebook or plots worth sharing.
            Hide
            mrawls Meredith Rawls added a comment -

            Ian, with your signoff, I'm going to call this ticket done and try to run the full pipeline after Nate's fix on DM-32245.

            Show
            mrawls Meredith Rawls added a comment - Ian, with your signoff, I'm going to call this ticket done and try to run the full pipeline after Nate's fix on DM-32245 .
            Hide
            sullivan Ian Sullivan added a comment -

            Sounds like a plan. Please add a link to the ticket that is "Nate's fix", and also add that as a blocker to DM-32245.

            Show
            sullivan Ian Sullivan added a comment - Sounds like a plan. Please add a link to the ticket that is "Nate's fix", and also add that as a blocker to DM-32245 .
            Hide
            mrawls Meredith Rawls added a comment -

            Done - I also added the pipelines I used and a couple "nImage" tract plots I created from templates of 2 of the 16 tracts with the most data in all patches.

            Show
            mrawls Meredith Rawls added a comment - Done - I also added the pipelines I used and a couple "nImage" tract plots I created from templates of 2 of the 16 tracts with the most data in all patches.

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              mrawls Meredith Rawls
              Reviewers:
              Ian Sullivan
              Watchers:
              Eric Bellm, Ian Sullivan, Lee Kelvin, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.