# Do a quick check that the test-med-1 now has all necessary raws defined

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
2
• Sprint:
DRP S21b
• Team:
Data Release Production
• Urgent?:
No

#### Description

It has long been noted that many raw files associated with the DC2 dataset defined for regular re-processings (DM-22954) were missing in the gen3 repo, i.e. the 2.2i/raw/test-med-1 collection of /repo/dc2. This should be fixed as of PREOPS-580. This ticket is to do a quick check (ahead of the w_2021_28 run) on a single visit that the missing raws are now found.

#### Activity

Hide
James Chiang added a comment -

Checking at CC-IN2P3, those three raw files are indeed missing.  I'm looking in /sps/lsstcest/datasets/desc/DC2/Run2.2i/sim .  I'll do a full census of the raw data at CC-IN2P3.

Show
James Chiang added a comment - Checking at CC-IN2P3, those three raw files are indeed missing.  I'm looking in /sps/lsstcest/datasets/desc/DC2/Run2.2i/sim .  I'll do a full census of the raw data at CC-IN2P3.
Hide
Jim Bosch added a comment -

Looks like the problem with ingest was just that I was looking for filenames with 7-digit exposure IDs, but that field is apparently not zero-padded, so I missed files from dp0-missing with 6-digit (or smaller) IDs.  I've reopened PREOPS-579 to fix it.

Show
Jim Bosch added a comment - Looks like the problem with ingest was just that I was looking for filenames with 7-digit exposure IDs, but that field is apparently not zero-padded, so I missed files from dp0-missing with 6-digit (or smaller) IDs.  I've reopened PREOPS-579 to fix it.
Hide
Lauren MacArthur added a comment - - edited

A final follow-up. I do now see the previously missing 166, 167, 168 detectors in the gen3 repo:

 $butler query-datasets /repo/dc2 "raw" --collections 2.2i/defaults/test-med-1 --where "instrument='LSSTCam-imSim' AND exposure=457681 AND skymap='DC2' AND detector in (165..169)" py.warnings WARN: [...]   type run id band instrument detector physical_filter exposure ---- ------------ ------------------------------------ ---- ------------- -------- --------------- --------  raw 2.2i/raw/all fab45ab5-e8c4-532c-91e4-0f6f9a0cfb37 i LSSTCam-imSim 165 i_sim_1.4 457681  raw 2.2i/raw/all ced4db0b-1bee-5a52-8d2e-f8e82d5a8227 i LSSTCam-imSim 166 i_sim_1.4 457681  raw 2.2i/raw/all 2957764d-fb13-594a-8e96-fff988997692 i LSSTCam-imSim 167 i_sim_1.4 457681  raw 2.2i/raw/all 613fd8ba-3f4f-573d-93f0-2ddcf95da9a0 i LSSTCam-imSim 168 i_sim_1.4 457681  raw 2.2i/raw/all 0974c7d7-c980-5e92-b7ac-8e8e9cfaa4fb i LSSTCam-imSim 169 i_sim_1.4 457681  And a rerun of the above now produces: Holes are filled in and the new visit padding added detector 42 to the list of possible tract overlaps Show Lauren MacArthur added a comment - - edited A final follow-up. I do now see the previously missing 166, 167, 168 detectors in the gen3 repo:$ butler query - datasets / repo / dc2 "raw" - - collections 2.2i / defaults / test - med - 1 - - where "instrument='LSSTCam-imSim' AND exposure=457681 AND skymap='DC2' AND detector in (165..169)" py.warnings WARN: [...]   type run id band instrument detector physical_filter exposure - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - raw 2.2i / raw / all fab45ab5 - e8c4 - 532c - 91e4 - 0f6f9a0cfb37 i LSSTCam - imSim 165 i_sim_1. 4 457681 raw 2.2i / raw / all ced4db0b - 1bee - 5a52 - 8d2e - f8e82d5a8227 i LSSTCam - imSim 166 i_sim_1. 4 457681 raw 2.2i / raw / all 2957764d - fb13 - 594a - 8e96 - fff988997692 i LSSTCam - imSim 167 i_sim_1. 4 457681 raw 2.2i / raw / all 613fd8ba - 3f4f - 573d - 93f0 - 2ddcf95da9a0 i LSSTCam - imSim 168 i_sim_1. 4 457681 raw 2.2i / raw / all 0974c7d7 - c980 - 5e92 - b7ac - 8e8e9cfaa4fb i LSSTCam - imSim 169 i_sim_1. 4 457681 And a rerun of the above now produces: Holes are filled in and the new visit padding added detector 42 to the list of possible tract overlaps
Hide
Lauren MacArthur added a comment -

Let me know if this is good to close, or if you'd like a more thorough validation on this ticket (I've already added doing that to the description of DM-31070, so either way is fine with me!)

Show
Lauren MacArthur added a comment - Let me know if this is good to close, or if you'd like a more thorough validation on this ticket (I've already added doing that to the description of DM-31070 , so either way is fine with me!)
Hide
Jim Bosch added a comment -

I think this is enough for this ticket, thanks!

Show
Jim Bosch added a comment - I think this is enough for this ticket, thanks!

#### People

Assignee:
Lauren MacArthur
Reporter:
Lauren MacArthur
Reviewers:
Jim Bosch
Watchers:
Eli Rykoff, James Chiang, Jim Bosch, Lauren MacArthur, Yusra AlSayyad