# Create parent task/script for bootstrapping Gen3 repos

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
4
• Team:
Data Release Production

## Description

Building on recent work to enable bootstrapping the Gen3 ci_hsc dataset with minimal reliance on gen2convert (DM-19614, DM-19615, DM-19622, DM-19531, DM-19272) and improvements to gen2convert itself (DM-18023), write a task/script that can be used to similarly bootstrap other Gen3 repos, starting with RC2.

Steps will include:

• register instrument(s)
• add curated calibrations, via Instrument.writeCuratedCalibrations
• register skymaps(s)
• ingest raws, via RawIngestTask
• add Gen2-calib-repo calibrations, refcats, and BrightObjectMasks via explicit, lower-level calls to gen2convert code in daf_butler.

I do not currently plan to include actually creating the empty repository, which will save this task from having to somehow read in and apply overrides to butler yaml configuration.

## Activity

Hide
Jim Bosch added a comment -

I'm hoping to get this into the weekly on Friday, so I figured I'd send this to you now even though I'm still testing my cleanups to obs_base (where most of the code is) before putting them on the tickets branch.  So the PRs that are ready now are:

The last is in lsst-dm, so it doesn't formally need a review, but I figure it's always good to get two eyes on any new code.

I'll post the obs_base PR as soon as that testing completes and I've moved my cleaned-up version from u/jbosch/ to tickets/.

Show
Jim Bosch added a comment - I'm hoping to get this into the weekly on Friday, so I figured I'd send this to you now even though I'm still testing my cleanups to obs_base (where most of the code is) before putting them on the tickets branch.  So the PRs that are ready now are: daf_butler: https://github.com/lsst/daf_butler/pull/158 obs_subaru: https://github.com/lsst/obs_subaru/pull/201 the new gen3-hsc-rc2 repo: https://github.com/lsst-dm/gen3-hsc-rc2/pull/1 The last is in lsst-dm, so it doesn't formally need a review, but I figure it's always good to get two eyes on any new code. I'll post the obs_base PR as soon as that testing completes and I've moved my cleaned-up version from u/jbosch/ to tickets/.
Hide
Jim Bosch added a comment -

Testing succeeded, and the obs_base PR is up: https://github.com/lsst/obs_base/pull/150.

Show
Jim Bosch added a comment - Testing succeeded, and the obs_base PR is up: https://github.com/lsst/obs_base/pull/150 .
Hide
Hsin-Fang Chiang added a comment -

Not sure I understand daf_butler & obs_base enough to say much about them, but in general looks okay so I'm marking it Reviewed. I'll take another look tomorrow. Not sure if you'd want somebody with better understanding of the design to take a look too.

Show
Hsin-Fang Chiang added a comment - Not sure I understand daf_butler & obs_base enough to say much about them, but in general looks okay so I'm marking it Reviewed. I'll take another look tomorrow. Not sure if you'd want somebody with better understanding of the design to take a look too.
Hide
Hsin-Fang Chiang added a comment -

Do you think anything in this ticket could possibly change the behavior in multiprocessing? When I test-ran pipetask with the ticket branches and -j 12 I got

  File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+31/python/lsst/pipe/base/pipelineTask.py", line 579, in runQuantum  self.saveStruct(struct, outputDataRefs, butler)  File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+31/python/lsst/pipe/base/pipelineTask.py", line 613, in saveStruct  butler.put(data, dataRef.datasetType.name, dataRef.dataId)  File "/home/hchiang2/stack/DM19638/daf_butler/python/lsst/daf/butler/core/utils.py", line 181, in inner  return func(self, *args, **kwargs)  File "/home/hchiang2/stack/DM19638/daf_butler/python/lsst/daf/butler/butler.py", line 365, in put  raise TypeError("Butler is read-only.") TypeError: Butler is read-only. 

while running the same repo seemed to work fine with w_2019_19. However it doesn't look relevant from the code changes and I haven't tested carefully, so please just ignore this if it doesn't ring a bell.

Show
Hsin-Fang Chiang added a comment - Do you think anything in this ticket could possibly change the behavior in multiprocessing? When I test-ran pipetask with the ticket branches and -j 12 I got File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+31/python/lsst/pipe/base/pipelineTask.py", line 579, in runQuantum self.saveStruct(struct, outputDataRefs, butler) File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+31/python/lsst/pipe/base/pipelineTask.py", line 613, in saveStruct butler.put(data, dataRef.datasetType.name, dataRef.dataId) File "/home/hchiang2/stack/DM19638/daf_butler/python/lsst/daf/butler/core/utils.py", line 181, in inner return func(self, *args, **kwargs) File "/home/hchiang2/stack/DM19638/daf_butler/python/lsst/daf/butler/butler.py", line 365, in put raise TypeError("Butler is read-only.") TypeError: Butler is read-only. while running the same repo seemed to work fine with w_2019_19. However it doesn't look relevant from the code changes and I haven't tested carefully, so please just ignore this if it doesn't ring a bell.
Hide
Jim Bosch added a comment -

Thanks for the quick review.  The obs_base changes aren't really a part of any larger design, so I don't think there necessarily is anyone better.  There might be some changes in daf_butler that Tim Jenness might be interested in, but as he's traveling I'll just leave this ping here but not hold up the ticket on him taking a look.

I'm afraid I don't think this ticket could be in play for the multiprocessing problem you're seeing - if it is, it'd have to be a very strange interaction.

Show
Jim Bosch added a comment - Thanks for the quick review.  The obs_base changes aren't really a part of any larger design, so I don't think there necessarily is anyone better.  There might be some changes in daf_butler that Tim Jenness might be interested in, but as he's traveling I'll just leave this ping here but not hold up the ticket on him taking a look.   I'm afraid I don't think this ticket could be in play for the multiprocessing problem you're seeing - if it is, it'd have to be a very strange interaction.
Hide
Jim Bosch added a comment -

I've created an lsst.log PR for the suggestion to move the log-level context manager to that package: https://github.com/lsst/log/pull/41.

Show
Jim Bosch added a comment - I've created an lsst.log PR for the suggestion to move the log-level context manager to that package: https://github.com/lsst/log/pull/41. Jenkins is running: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29843/pipeline
Hide
Jim Bosch added a comment -

Hsin-Fang Chiang, sorry about my skepticism about your error report above being related to this ticket - it absolutely was, but it took breaking the weekly and some debugging by Nate Lust for me to figure it out.

Show
Jim Bosch added a comment - Hsin-Fang Chiang , sorry about my skepticism about your error report above being related to this ticket - it absolutely was, but it took breaking the weekly and some debugging by Nate Lust for me to figure it out.

## People

• Assignee:
Jim Bosch
Reporter:
Jim Bosch
Reviewers:
Hsin-Fang Chiang
Watchers:
Hsin-Fang Chiang, Jim Bosch