Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3911

HSC backport: avoid I/O race conditions config write out

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Sprint:
      Science Pipelines DM-W16-1, Science Pipelines DM-W16-2
    • Team:
      Data Release Production

      Description

      This is a port of HSC-1106

      When running tasks that write out config settings files (processCcd.py, for example), if multiple processes start simultaneously, an I/O race condition can occur in writing these files. This is solved here by writing to temp files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. config.py~1, config.py~2, ...), a --no-backup-config option (to be used with --clobber-config) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run.

        Attachments

          Issue Links

            Activity

            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            Aside from needing to move the new code in butler to a separate file it all looks good to me. Tim pointed out a couple things that can be cleaned up too, I guess.

            Show
            npease Nate Pease [X] (Inactive) added a comment - Aside from needing to move the new code in butler to a separate file it all looks good to me. Tim pointed out a couple things that can be cleaned up too, I guess.
            Hide
            lauren Lauren MacArthur added a comment -

            Great, thanks Nate Pease [X]. I have made all the changes and pushed tickets/DM-3911 branches. I am running a Jenkins now if you want to have a quick look. Otherwise, I'll assume I'm ok to merge once the Jenkins build is successful.

            Show
            lauren Lauren MacArthur added a comment - Great, thanks Nate Pease [X] . I have made all the changes and pushed tickets/ DM-3911 branches. I am running a Jenkins now if you want to have a quick look. Otherwise, I'll assume I'm ok to merge once the Jenkins build is successful.
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            Ok, I checked the most recent changes in daf_persistence, they look great. I'll let Tim Jenness look at the other packages if he wants to; it's ok by me for you to checkin after Jenkins passes.

            Show
            npease Nate Pease [X] (Inactive) added a comment - Ok, I checked the most recent changes in daf_persistence, they look great. I'll let Tim Jenness look at the other packages if he wants to; it's ok by me for you to checkin after Jenkins passes.
            Hide
            lauren Lauren MacArthur added a comment -

            Merged to master.

            Show
            lauren Lauren MacArthur added a comment - Merged to master.
            Hide
            swinbank John Swinbank added a comment -

            ...& added to release notes. Thanks!

            Show
            swinbank John Swinbank added a comment - ...& added to release notes. Thanks!

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Nate Pease [X] (Inactive)
              Watchers:
              John Swinbank, Kian-Tat Lim, Lauren MacArthur, Nate Pease [X] (Inactive), Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.