Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:2
-
Epic Link:
-
Sprint:DRP S21b
-
Team:Data Release Production
-
Urgent?:No
Description
It has long been noted that many raw files associated with the DC2 dataset defined for regular re-processings (DM-22954) were missing in the gen3 repo, i.e. the 2.2i/raw/test-med-1 collection of /repo/dc2. This should be fixed as of PREOPS-580. This ticket is to do a quick check (ahead of the w_2021_28 run) on a single visit that the missing raws are now found.
Attachments
Activity
Looks like the problem with ingest was just that I was looking for filenames with 7-digit exposure IDs, but that field is apparently not zero-padded, so I missed files from dp0-missing with 6-digit (or smaller) IDs. I've reopened PREOPS-579 to fix it.
A final follow-up. I do now see the previously missing 166, 167, 168 detectors in the gen3 repo:
$ butler query-datasets /repo/dc2 "raw" --collections 2.2i/defaults/test-med-1 --where "instrument='LSSTCam-imSim' AND exposure=457681 AND skymap='DC2' AND detector in (165..169)" |
py.warnings WARN: [...]
|
|
|
type run id band instrument detector physical_filter exposure |
---- ------------ ------------------------------------ ---- ------------- -------- --------------- -------- |
raw 2.2i/raw/all fab45ab5-e8c4-532c-91e4-0f6f9a0cfb37 i LSSTCam-imSim 165 i_sim_1.4 457681 |
raw 2.2i/raw/all ced4db0b-1bee-5a52-8d2e-f8e82d5a8227 i LSSTCam-imSim 166 i_sim_1.4 457681 |
raw 2.2i/raw/all 2957764d-fb13-594a-8e96-fff988997692 i LSSTCam-imSim 167 i_sim_1.4 457681 |
raw 2.2i/raw/all 613fd8ba-3f4f-573d-93f0-2ddcf95da9a0 i LSSTCam-imSim 168 i_sim_1.4 457681 |
raw 2.2i/raw/all 0974c7d7-c980-5e92-b7ac-8e8e9cfaa4fb i LSSTCam-imSim 169 i_sim_1.4 457681 |
And a rerun of the above now produces:
Holes are filled in and the new visit padding added detector 42 to the list of possible tract overlaps 
Let me know if this is good to close, or if you'd like a more thorough validation on this ticket (I've already added doing that to the description of DM-31070, so either way is fine with me!)
Checking at CC-IN2P3, those three raw files are indeed missing. I'm looking in /sps/lsstcest/datasets/desc/DC2/Run2.2i/sim . I'll do a full census of the raw data at CC-IN2P3.