Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28168

Look into pipe_analysis script failures in RC2 w_2020_50 run

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP S21a (Dec Jan)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Two failure modes in the pipe_analysis scripts were noted on DM-28140, the latest RC2 processing run with weekly w_2020_50. Look into these and fix the issues. 

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-28140 RC2 reprocessing with w_2020_50 (Repeat)

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22266 Adapt pipe_analysis to run on post-processing parquet tables in Gen2

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            The issue with coaddAnalysis.py on only certain COSMOS (tract 9813) filters turned out to be an issue where some patches didn't have full filter coverage.  When this is the case, the deepCoadd_obj parquet tables only include a filter "level" for those filters for which data exists, so when looking for one that didn't exists, it failed to find those columns The fix is to add a check for the filter in consideration when reading in the deepCoadd_obj and removing that patch from the "patch exists" list for the missing patch.

            As for the compareCoaddAnalysis.py failure, that was simply as misplaced variable setting that needed to be moved outside of an if clause.

            Show
            lauren Lauren MacArthur added a comment - The issue with coaddAnalysis.py on only certain COSMOS (tract 9813) filters turned out to be an issue where some patches didn't have full filter coverage.  When this is the case, the  deepCoadd_obj parquet tables only include a filter "level" for those filters for which data exists, so when looking for one that didn't exists, it failed to find those columns The fix is to add a check for the filter in consideration when reading in the deepCoadd_obj and removing that patch from the "patch exists" list for the missing patch. As for the compareCoaddAnalysis.py failure, that was simply as misplaced variable setting that needed to be moved outside of an if clause.
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Eric, would you mind giving this a look?  I've run a bunch of the scripts that failed for you on DM-28140 and they are all now working.  Example plots can be found (for now) here and logs are at /home/lauren/tickets/DM-28168/logs/

            Show
            lauren Lauren MacArthur added a comment - Eric, would you mind giving this a look?  I've run a bunch of the scripts that failed for you on DM-28140 and they are all now working.  Example plots can be found (for now) here and logs are at /home/lauren/tickets/ DM-28168 /logs/
            Hide
            Permalink
            emorganson Eric Morganson [X] (Inactive) added a comment -

            Looks good. All 4 processed. This was within to 2020_50 run if you want to check out the results.

            Show
            emorganson Eric Morganson [X] (Inactive) added a comment - Looks good. All 4 processed. This was within to 2020_50 run if you want to check out the results.
            Hide
            Permalink
            emorganson Eric Morganson [X] (Inactive) added a comment -

            All good.

            Show
            emorganson Eric Morganson [X] (Inactive) added a comment - All good.
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Awesome, thanks!  Merged to master.

            Show
            lauren Lauren MacArthur added a comment - Awesome, thanks!  Merged to master.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Eric Morganson [X] (Inactive)
              Watchers:
              Eric Morganson [X] (Inactive), Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.