Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31036

Do a quick check that the test-med-1 now has all necessary raws defined

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Sprint:
      DRP S21b
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      It has long been noted that many raw files associated with the DC2 dataset defined for regular re-processings (DM-22954) were missing in the gen3 repo, i.e. the 2.2i/raw/test-med-1 collection of /repo/dc2. This should be fixed as of PREOPS-580. This ticket is to do a quick check (ahead of the w_2021_28 run) on a single visit that the missing raws are now found.

        Attachments

          Activity

          Hide
          jchiang James Chiang added a comment -

          Checking at CC-IN2P3, those three raw files are indeed missing.  I'm looking in /sps/lsstcest/datasets/desc/DC2/Run2.2i/sim .  I'll do a full census of the raw data at CC-IN2P3.

          Show
          jchiang James Chiang added a comment - Checking at CC-IN2P3, those three raw files are indeed missing.  I'm looking in /sps/lsstcest/datasets/desc/DC2/Run2.2i/sim .  I'll do a full census of the raw data at CC-IN2P3.
          Hide
          jbosch Jim Bosch added a comment -

          Looks like the problem with ingest was just that I was looking for filenames with 7-digit exposure IDs, but that field is apparently not zero-padded, so I missed files from dp0-missing with 6-digit (or smaller) IDs.  I've reopened PREOPS-579 to fix it.

          Show
          jbosch Jim Bosch added a comment - Looks like the problem with ingest was just that I was looking for filenames with 7-digit exposure IDs, but that field is apparently not zero-padded, so I missed files from dp0-missing with 6-digit (or smaller) IDs.  I've reopened PREOPS-579 to fix it.
          Hide
          lauren Lauren MacArthur added a comment - - edited

          A final follow-up. I do now see the previously missing 166, 167, 168 detectors in the gen3 repo:

          $ butler query-datasets /repo/dc2 "raw" --collections 2.2i/defaults/test-med-1 --where "instrument='LSSTCam-imSim' AND exposure=457681 AND skymap='DC2' AND detector in (165..169)"
          py.warnings WARN: [...]
           
          type run id band instrument detector physical_filter exposure
          ---- ------------ ------------------------------------ ---- ------------- -------- --------------- --------
           raw 2.2i/raw/all fab45ab5-e8c4-532c-91e4-0f6f9a0cfb37 i LSSTCam-imSim 165 i_sim_1.4 457681
           raw 2.2i/raw/all ced4db0b-1bee-5a52-8d2e-f8e82d5a8227 i LSSTCam-imSim 166 i_sim_1.4 457681
           raw 2.2i/raw/all 2957764d-fb13-594a-8e96-fff988997692 i LSSTCam-imSim 167 i_sim_1.4 457681
           raw 2.2i/raw/all 613fd8ba-3f4f-573d-93f0-2ddcf95da9a0 i LSSTCam-imSim 168 i_sim_1.4 457681
           raw 2.2i/raw/all 0974c7d7-c980-5e92-b7ac-8e8e9cfaa4fb i LSSTCam-imSim 169 i_sim_1.4 457681
          

          And a rerun of the above now produces:

          Holes are filled in and the new visit padding added detector 42 to the list of possible tract overlaps

          Show
          lauren Lauren MacArthur added a comment - - edited A final follow-up. I do now see the previously missing 166, 167, 168 detectors in the gen3 repo: $ butler query - datasets / repo / dc2 "raw" - - collections 2.2i / defaults / test - med - 1 - - where "instrument='LSSTCam-imSim' AND exposure=457681 AND skymap='DC2' AND detector in (165..169)" py.warnings WARN: [...]   type run id band instrument detector physical_filter exposure - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - raw 2.2i / raw / all fab45ab5 - e8c4 - 532c - 91e4 - 0f6f9a0cfb37 i LSSTCam - imSim 165 i_sim_1. 4 457681 raw 2.2i / raw / all ced4db0b - 1bee - 5a52 - 8d2e - f8e82d5a8227 i LSSTCam - imSim 166 i_sim_1. 4 457681 raw 2.2i / raw / all 2957764d - fb13 - 594a - 8e96 - fff988997692 i LSSTCam - imSim 167 i_sim_1. 4 457681 raw 2.2i / raw / all 613fd8ba - 3f4f - 573d - 93f0 - 2ddcf95da9a0 i LSSTCam - imSim 168 i_sim_1. 4 457681 raw 2.2i / raw / all 0974c7d7 - c980 - 5e92 - b7ac - 8e8e9cfaa4fb i LSSTCam - imSim 169 i_sim_1. 4 457681 And a rerun of the above now produces: Holes are filled in and the new visit padding added detector 42 to the list of possible tract overlaps
          Hide
          lauren Lauren MacArthur added a comment -

          Let me know if this is good to close, or if you'd like a more thorough validation on this ticket (I've already added doing that to the description of DM-31070, so either way is fine with me!)

          Show
          lauren Lauren MacArthur added a comment - Let me know if this is good to close, or if you'd like a more thorough validation on this ticket (I've already added doing that to the description of DM-31070 , so either way is fine with me!)
          Hide
          jbosch Jim Bosch added a comment -

          I think this is enough for this ticket, thanks!

          Show
          jbosch Jim Bosch added a comment - I think this is enough for this ticket, thanks!

            People

            Assignee:
            lauren Lauren MacArthur
            Reporter:
            lauren Lauren MacArthur
            Reviewers:
            Jim Bosch
            Watchers:
            Eli Rykoff, James Chiang, Jim Bosch, Lauren MacArthur, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                CI Builds

                No builds found.