# DM-21724 unpickling error appears again

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
2
• Sprint:
DB_S20_02
• Team:
Data Access and Database
• Urgent?:
No

#### Description

Saving a QuantumGraph and run it with pipetask doesn't work. This is the same problem as DM-21724 but the fix got unfixed.

When trying to run a stored QuantumGraph pickle with
pipetask run --qgraph name.pickle -b butler.yaml the error is

 _pickle.UnpicklingError: DimensionUniverse with version '0' not found. Note that DimensionUniverse objects are not truly serialized; when using pickle to transfer them between processes, an equivalent instance with the same version must already exist in the receiving process.

#### Activity

Hide
Jim Bosch added a comment -

Looks good; only comment on the PR is about naming.

In case you're curious, part of the reason dimension-related things are this way w.r.t. pickling is because of the way Python's multiprocessing module uses pickling to pass objects between processes - it doesn't provide any way to use one shared object (like a DimensionUniverse) when unpickling others (pickle itself has this functionality, but multiprocessing doesn't provide any way to use it). So while DimensionUniverse isn't directly a singleton, and it isn't used at all as a singleton in most other contexts, there is a global dict containing all instances, and the unpickling code for all dimension-based objects ultimately looks in that.

Show
Jim Bosch added a comment - Looks good; only comment on the PR is about naming. In case you're curious, part of the reason dimension-related things are this way w.r.t. pickling is because of the way Python's multiprocessing module uses pickling to pass objects between processes - it doesn't provide any way to use one shared object (like a DimensionUniverse) when unpickling others (pickle itself has this functionality, but multiprocessing doesn't provide any way to use it). So while DimensionUniverse isn't directly a singleton, and it isn't used at all as a singleton in most other contexts, there is a global dict containing all instances, and the unpickling code for all dimension-based objects ultimately looks in that.
Hide
Andy Salnikov added a comment -

Thanks for review! Merged and closing.
I think I understand the reasoning behind DimensionUniverse, my worry now is that unit testing for even trivial things feels very involved, which potentially can lead to unexpected failures.

Show
Andy Salnikov added a comment - Thanks for review! Merged and closing. I think I understand the reasoning behind DimensionUniverse, my worry now is that unit testing for even trivial things feels very involved, which potentially can lead to unexpected failures.
Hide
Hsin-Fang Chiang added a comment -

Thanks for fixing this!

With the two new master branches from this ticket I'm still having issues using separate pipetask commands saving a qgraph and then running it. I suspect I use the commands wrong. What would be the right way to break the pipetask run as in https://github.com/lsst/ci_hsc_gen3/blob/master/bin/pipeline.sh into two steps? I thought the following would work but still running into errors like

 lsst.pipe.tasks.colorterms.ColortermNotFoundError: No colorterm found for filter 'HSC-R' with photoCatName 'ps1_pv3_3pi_20170110' = catalog 'ps1*'

 pipetask qgraph -q ci_hsc.pickle -d "patch = 69" -b "$2"/butler.yaml \ -i calib/hsc,raw/hsc,masks/hsc,ref_cats,skymaps,shared/ci_hsc -o "$COLLECTION" \ -p "$CI_HSC_GEN3_DIR"/pipelines/CiHsc.yaml pipetask run --qgraph ci_hsc.pickle -j "$1" -b "$2"/butler.yaml \ -i calib/hsc,raw/hsc,masks/hsc,ref_cats,skymaps,shared/ci_hsc -o "$COLLECTION" \ --register-dataset-types

If it's more appropriate to do this in a new ticket I can open one.

Show
Hsin-Fang Chiang added a comment - Thanks for fixing this! With the two new master branches from this ticket I'm still having issues using separate pipetask commands saving a qgraph and then running it. I suspect I use the commands wrong. What would be the right way to break the pipetask run as in https://github.com/lsst/ci_hsc_gen3/blob/master/bin/pipeline.sh into two steps? I thought the following would work but still running into errors like lsst.pipe.tasks.colorterms.ColortermNotFoundError: No colorterm found for filter 'HSC-R' with photoCatName 'ps1_pv3_3pi_20170110' = catalog 'ps1*' pipetask qgraph -q ci_hsc.pickle -d "patch = 69" -b "$2"/butler.yaml \ -i calib/hsc,raw/hsc,masks/hsc,ref_cats,skymaps,shared/ci_hsc -o "$COLLECTION" \ -p "$CI_HSC_GEN3_DIR"/pipelines/CiHsc.yaml pipetask run --qgraph ci_hsc.pickle -j "$1" -b "$2"/butler.yaml \ -i calib/hsc,raw/hsc,masks/hsc,ref_cats,skymaps,shared/ci_hsc -o "$COLLECTION" \ --register-dataset-types If it's more appropriate to do this in a new ticket I can open one.
Hide
Andy Salnikov added a comment -

Hsin-Fang Chiang, I think this deserves separate ticket, it's not obvious that it is related to what has been fixed on this ticket.

Show
Andy Salnikov added a comment - Hsin-Fang Chiang , I think this deserves separate ticket, it's not obvious that it is related to what has been fixed on this ticket.
Hide
Hsin-Fang Chiang added a comment -

Agree. DM-25113

Show
Hsin-Fang Chiang added a comment - Agree. DM-25113

#### People

Assignee:
Andy Salnikov
Reporter:
Hsin-Fang Chiang
Reviewers:
Jim Bosch
Watchers:
Andy Salnikov, Hsin-Fang Chiang, Jim Bosch, Tim Jenness