I'm hoping to get this into the weekly on Friday, so I figured I'd send this to you now even though I'm still testing my cleanups to obs_base (where most of the code is) before putting them on the tickets branch.  So the PRs that are ready now are:

• daf_butler: https://github.com/lsst/daf_butler/pull/158
• obs_subaru: https://github.com/lsst/obs_subaru/pull/201
• the new gen3-hsc-rc2 repo: https://github.com/lsst-dm/gen3-hsc-rc2/pull/1

The last is in lsst-dm, so it doesn't formally need a review, but I figure it's always good to get two eyes on any new code.

I'll post the obs_base PR as soon as that testing completes and I've moved my cleaned-up version from u/jbosch/ to tickets/.

Testing succeeded, and the obs_base PR is up: https://github.com/lsst/obs_base/pull/150.

Not sure I understand daf_butler & obs_base enough to say much about them, but in general looks okay so I'm marking it Reviewed. I'll take another look tomorrow. Not sure if you'd want somebody with better understanding of the design to take a look too.

Do you think anything in this ticket could possibly change the behavior in multiprocessing? When I test-ran pipetask with the ticket branches and -j 12 I got

  File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+31/python/lsst/pipe/base/pipelineTask.py", line 579, in runQuantum

    self.saveStruct(struct, outputDataRefs, butler)

  File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+31/python/lsst/pipe/base/pipelineTask.py", line 613, in saveStruct

    butler.put(data, dataRef.datasetType.name, dataRef.dataId)

  File "/home/hchiang2/stack/DM19638/daf_butler/python/lsst/daf/butler/core/utils.py", line 181, in inner

    return func(self, *args, **kwargs)

  File "/home/hchiang2/stack/DM19638/daf_butler/python/lsst/daf/butler/butler.py", line 365, in put

    raise TypeError("Butler is read-only.")

TypeError: Butler is read-only.

while running the same repo seemed to work fine with w_2019_19. However it doesn't look relevant from the code changes and I haven't tested carefully, so please just ignore this if it doesn't ring a bell.

Thanks for the quick review.  The obs_base changes aren't really a part of any larger design, so I don't think there necessarily is anyone better.  There might be some changes in daf_butler that Tim Jenness might be interested in, but as he's traveling I'll just leave this ping here but not hold up the ticket on him taking a look.

I'm afraid I don't think this ticket could be in play for the multiprocessing problem you're seeing - if it is, it'd have to be a very strange interaction.

I've created an lsst.log PR for the suggestion to move the log-level context manager to that package: https://github.com/lsst/log/pull/41.

Jenkins is running: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29843/pipeline

Hsin-Fang Chiang, sorry about my skepticism about your error report above being related to this ticket - it absolutely was, but it took breaking the weekly and some debugging by Nate Lust for me to figure it out.