Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_lsst
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Team:Data Facility
Description
The expId calculation has changed and now it can be much larger, hence sometimes, with large visit IDs, expBits=32 is no longer enough:
processCcd FATAL: Failed on dataId={'visit': 439196, 'raftName': 'R11', 'detectorName': 'S11', 'run': '439196', 'detector': 40, 'snap': 0}: RuntimeError: expId=4391960040 uses 33 bits > expBits=32
|
So, I tried increasing ccdExposureId_bits to 36 bits, then it failed at processCcd characterizeImage detectMeasureAndEstimatePsf
File "/home/hchiang2/DC2/meas_algorithms/python/lsst/meas/algorithms/reserveSourcesTask.py", line 132, in select
|
rng = np.random.RandomState(self.config.seed + expId)
|
File "mtrand.pyx", line 644, in mtrand.RandomState.__init__
|
File "mtrand.pyx", line 680, in mtrand.RandomState.seed
|
ValueError: Seed must be between 0 and 2**32 - 1
|
It's okay to truncate the ID sent to rng because it just wants something deterministic and not-usually-the-same for different units of data.
Attachments
Activity
On the other hand we don't really need 4 digits of YYYY so it might not hurt to trim there. Tim Jenness what do you think?
Hsin-Fang Chiang
added a comment - On the other hand we don't really need 4 digits of YYYY so it might not hurt to trim there. Tim Jenness what do you think? Encoding a detector_exposure_id for inclusion in a source ID is not necessarily the same thing as creating a detector_exposure_id. For the date-based Ids (which will include LSSTCam and AuxTel as well as TS8) we can have an offset to subtract the 20 from the front. It doesn't really save us much though. I think I'm going to follow through with my threat and have a discussion on community about this.
For the time being I'm gonna merge this (36 bits), which would make imsim/DC2 processing happy. I doubt anybody is seriously processing AuxTel or TS8 data in the very near term and it can be adjusted later.
Hsin-Fang Chiang
added a comment - For the time being I'm gonna merge this (36 bits), which would make imsim/DC2 processing happy. I doubt anybody is seriously processing AuxTel or TS8 data in the very near term and it can be adjusted later. Merged after running Jenkins (https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29357/pipeline/ )
Hsin-Fang Chiang
added a comment - Merged after running Jenkins ( https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29357/pipeline/ ) 
That leaves us with 16 bits for sources, which is getting close to the edge of what I'd consider safe for full production with real data and crowded fields, but it would definitely be fine for all of the obs_lsst data we have now, and I may just be paranoid about that being on the edge in the future.