Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22729

Monthly December ap_pipe rerun

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe
    • Labels:
      None
    • Story Points:
      8
    • Sprint:
      AP S20-1 (December), AP S20-2 (January)
    • Team:
      Alert Production

      Description

      This is the ~monthly ap_pipe rerun of the HiTS 2015 dataset (all 3 fields). For the first time, I will use templates with the DECam instrumental pixel scale (0.26"/pixel). The rerun will be in /project/mrawls/hits2015/rerun/cw_2019_12 and an accompanying "quick look" analysis notebook will be made.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            The rerun is in `/project/mrawls/hits2015/rerun/cw_2019_12v2`. (The non-v2 rerun had a problem detailed in DM-21330 pertaining to the templates, and specifically the good seeing selector). The notebook is on the PR.

            For the most part, the result looks good, but as detailed in the notebook and in DM-22781, there are 3-4 CCDs which have spurious sources that will need further investigation before we adopt the templates made for DM-21330.

            Gabor Kovacs, can you please review this by taking a look at the notebook as well as the output data products (difference images and/or APDB)? If you have any insights for DM-22781 I would be very interested to hear them! Please note I will be away until Jan 13.

            Show
            mrawls Meredith Rawls added a comment - The rerun is in `/project/mrawls/hits2015/rerun/cw_2019_12v2`. (The non-v2 rerun had a problem detailed in DM-21330 pertaining to the templates, and specifically the good seeing selector). The notebook is on the PR. For the most part, the result looks good, but as detailed in the notebook and in DM-22781 , there are 3-4 CCDs which have spurious sources that will need further investigation before we adopt the templates made for DM-21330 . Gabor Kovacs , can you please review this by taking a look at the notebook as well as the output data products (difference images and/or APDB)? If you have any insights for DM-22781 I would be very interested to hear them! Please note I will be away until Jan 13.
            Hide
            gkovacs Gabor Kovacs added a comment -

            Looks good.

             

            Two minor comments on the notebook:

             

            Which commands/tasks were run in parallel that produced the race condition in the failed run? If I understand correctly, ap_pipe was run parallelized on a per ccd basis. I think it'd be worth of adding some comment to the notebook how to reproduce the race condition if you can recall the exact scenario. Particularly, I can't really see why the race condidion actually happened. Apdb should be multiprocess safe. All raw exposure files, and calexps are in separate fits files per ccd, templates and refcats are read only, so running ap_pipe parallel on a per ccd basis should have been ok.

            Why is the skymap plot cut in "the patches vs. visit fields" plot ? I guess it's just a plotting range choice and there is no tract boundary there. Actually, it is worthwhile to note that GetCoaddAsTemplateTask in image differencing chooses the tract whose center is closest to the exposure (center) on a per exposure basis. This implies that if we are close to the edge of a tract then the neighboring tract may be chosen depending on overlapping.

             

            Show
            gkovacs Gabor Kovacs added a comment - Looks good.   Two minor comments on the notebook:   Which commands/tasks were run in parallel that produced the race condition in the failed run? If I understand correctly,  ap_pipe was run parallelized on a per ccd basis. I think it'd be worth of adding some comment to the notebook how to reproduce the race condition if you can recall the exact scenario. Particularly, I can't really see why the race condidion actually happened. Apdb should be multiprocess safe. All raw exposure files, and calexps are in separate fits files per ccd, templates and refcats are read only, so running ap_pipe parallel on a per ccd basis should have been ok. Why is the skymap plot cut in "the patches vs. visit fields" plot ? I guess it's just a plotting range choice and there is no tract boundary there. Actually, it is worthwhile to note that  GetCoaddAsTemplateTask in image differencing chooses the tract whose center is closest to the exposure (center) on a per exposure basis. This implies that if we are close to the edge of a tract then the neighboring tract may be chosen depending on overlapping.  
            Hide
            mrawls Meredith Rawls added a comment -

            Thanks. The failure wasn't exactly a race condition in my mind, but rather an issue with how the good seeing selector works. We have DM-16191 as well as DM-23068 to follow up on this; you can read more there if you are interested.

            I omitted the skymap plot for the "December fail" case because it is identical to the "December OK" case. (That is actually not exactly true - the initial skymap included a few errant patches smack in the middle of the tract due to a single 2014 visit having a garbage WCS; that visit was subsequently removed by not including the errant patches in the dataIds to process during assembly.)

            I think you are saying "tract" when you mean "patch," but indeed, there are challenges in how to choose the best constituent set of visits when the output template thinks in patches and both have different spatial extents. I again refer you to the good seeing selector tickets linked above  

            Show
            mrawls Meredith Rawls added a comment - Thanks. The failure wasn't exactly a race condition in my mind, but rather an issue with how the good seeing selector works. We have DM-16191 as well as DM-23068 to follow up on this; you can read more there if you are interested. I omitted the skymap plot for the "December fail" case because it is identical to the "December OK" case. (That is actually not exactly true - the initial skymap included a few errant patches smack in the middle of the tract due to a single 2014 visit having a garbage WCS; that visit was subsequently removed by not including the errant patches in the dataIds to process during assembly.) I think you are saying "tract" when you mean "patch," but indeed, there are challenges in how to choose the best constituent set of visits when the output template thinks in patches and both have different spatial extents. I again refer you to the good seeing selector tickets linked above  

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                mrawls Meredith Rawls
                Reviewers:
                Gabor Kovacs
                Watchers:
                Gabor Kovacs, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel