Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_lsst
-
Labels:
-
Story Points:2
-
Epic Link:
-
Sprint:DRP S21b
-
Team:Data Release Production
-
Urgent?:No
Description
As was noted on DM-30812, there were differences between the gen2 and gen3 middleware src catalog outputs in the following columns:
- id
- parent
- various ext_shapeHSM_HsmPsfMomentsDebiased_* columns
- the total number of finite entries
As Joshua Meyers pointed out on that ticket, all of these could stem from the id mismatch as it is used to seed a rng in the HsmPsfMomentsDebiased code. This ticket is to figure out where the mismatch is happening and sync them up (hopefully to the gen3 version) in hopes of achieving gen2/gen3 parity for the DC2 dataset.
Attachments
Issue Links
Activity
Yes, thank you! That is very helpful. I had vague recollections of that commit of mine and of Tim's Community post, but couldn't recall exactly where to look (I somehow thought it discussed solely on slack.).
James Chiang
added a comment - Yes, thank you! That is very helpful. I had vague recollections of that commit of mine and of Tim's Community post, but couldn't recall exactly where to look (I somehow thought it discussed solely on slack.). The changes look good to me. Based on the range of run numbers we used for DC2 and allowing for detector numbers up to 205, 34 bits should be fine. (I can't recall why I chose 38 last time...probably being overly conservative.)
James Chiang
added a comment - The changes look good to me. Based on the range of run numbers we used for DC2 and allowing for detector numbers up to 205, 34 bits should be fine. (I can't recall why I chose 38 last time...probably being overly conservative.) Great, thanks Jim. I just checked and no rebase is needed, so I think the original Jenkins run still holds. When you're ready, please hit the "Review Complete" button (you could also officially "approve" the PR) and I will then go ahead and merge.
Thanks for the speedy review. Merged & done.
Most of the details regarding the need for this change are in the ticket description and in
DM-30812. It boils down to our ambition to achieve exact parity between the gen2 & gen3 middlewares on the road to deprecating the former. The way this value is derived in gen3, it gets set to 34. As of your original commit (which looks like it was part ofDM-18551), gen2 has this set to 38. This changes the object ids and, since ids are often used as predictable seeds for random number generators in some measurement algorithms, this can lead to a difference in the outputs. I guess my assumption here is that gen3 is setting the "correct" (i.e. that will be used for all future processing – see this community post for details) and I am just looking to match the two while gen2 is still in play (parity between the two is greatly desired, so any simple changes to make that happen are being made, even if it's an update to the soon-to-be-deprecated gen2 paths). Does that make sense?