Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30730

DC2 Reprocessing with w_2021_24 (gen2)

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 8
    • Data Release Production
    • No

    Description

      Process the DC2 dataset with the w_2021_24 stack.  Will first attempt to follow the procedure described in the lsst-dm/gen2gen repo here.

      Attachments

        Issue Links

          Activity

            The output repo is linked here, where one can access the logs and view the validate_drp plots, e.g. PA1.

            lauren Lauren MacArthur added a comment - The output repo is linked here , where one can access the logs and view the validate_drp plots , e.g. PA1 .
            lauren Lauren MacArthur added a comment - - edited

            A slightly overdue update on this ticket.

            The metrics were re-dispatched using the following environment variable settings:

            export DATASET="DC2_test-med-1"
            export DATASET_REPO_URL="https://jira.lsstcorp.org/browse/DM-30730"
            export RUN_ID="DM-30730"
            export RUN_ID_URL="https://jira.lsstcorp.org/browse/DM-30730"
            export VERSION_TAG="w_2021_24"
            

            I've just setup an initial attempt at a chronograf dashboard (an edited clone of the existing "DRP metrics monthly DC2 test-med-1") called:
            DRP monthly metrics for DC2 (Gen2)
            I'm not sure why the PF1 measurements are not showing up (I may have messed up the query, but can't see where),
            nor why I seem to be picking up extra TE1 values (it seems they are present for two previous runs, whereas none of the others are?), nor why no i band data seem to be showing up (despite the json files being present and dispatched). But otherwise all looks ok

            Side note: should we be concerned that the matchedVisit_band.json files are of order 6.1G?

            I have been updating the pipe_analysis scripts so they all run on DC2 data (some plots can be seen here). I will finish that up and include the PRs on a separate ticket.

            lauren Lauren MacArthur added a comment - - edited A slightly overdue update on this ticket. The metrics were re-dispatched using the following environment variable settings: export DATASET = "DC2_test-med-1" export DATASET_REPO_URL = "https://jira.lsstcorp.org/browse/DM-30730" export RUN_ID = "DM-30730" export RUN_ID_URL = "https://jira.lsstcorp.org/browse/DM-30730" export VERSION_TAG = "w_2021_24" I've just setup an initial attempt at a chronograf dashboard  (an edited clone of the existing "DRP metrics monthly DC2 test-med-1") called: DRP monthly metrics for DC2 (Gen2) I'm not sure why the PF1 measurements are not showing up (I may have messed up the query, but can't see where), nor why I seem to be picking up extra TE1 values (it seems they are present for two previous runs, whereas none of the others are?), nor why no i band data seem to be showing up (despite the json files being present and dispatched). But otherwise all looks ok Side note: should we be concerned that the matchedVisit_band.json files are of order 6.1G? I have been updating the pipe_analysis scripts so they all run on DC2 data (some plots can be seen here ). I will finish that up and include the PRs on a separate ticket.

            Please let me know if you'd like to see anything else on this ticket. Otherwise, I think it's ready to be closed out.

            lauren Lauren MacArthur added a comment - Please let me know if you'd like to see anything else on this ticket. Otherwise, I think it's ready to be closed out.
            dtaranu Dan Taranu added a comment -

            The PF1 query looks fine; it looks like the measurements just didn't succeed.

            grep -C 2 'validate_drp.PA1"' /datasets/DC2/repoRun2.2i/rerun/w_2021_24/DM-30730/validateDrp/matchedVisitMetrics/3828/g/matchedVisit_g.json

            ... outputs:

                  "identifier": "784f5c4e871c457eab61af8c2b0985db",
                  "metric": "validate_drp.PA1",
                  "unit": "mmag",
                  "value": 9.469147892027074
            

            Meanwhile, the same grep for validate_drp.PF1_design_gri gets nothing. So it's probably a validate_drp bug that nobody is ever going to fix.

            I think the jsons are giant because validate_drp saves blobs with entire columns worth of data in ASCII.

            dtaranu Dan Taranu added a comment - The PF1 query looks fine; it looks like the measurements just didn't succeed. grep -C 2 'validate_drp.PA1"' /datasets/DC2/repoRun2.2i/rerun/w_2021_24/ DM-30730 /validateDrp/matchedVisitMetrics/3828/g/matchedVisit_g.json ... outputs: "identifier": "784f5c4e871c457eab61af8c2b0985db", "metric": "validate_drp.PA1", "unit": "mmag", "value": 9.469147892027074 Meanwhile, the same grep for validate_drp.PF1_design_gri gets nothing. So it's probably a validate_drp bug that nobody is ever going to fix. I think the jsons are giant because validate_drp saves blobs with entire columns worth of data in ASCII.
            lauren Lauren MacArthur added a comment - - edited

            Ah, right...thanks Dan.  I just checked some older logs for the PF1_design_gri entries...missing there too, so probably a long-standing (and, indeed, never-to-be-fixed) bug in validate_drp.

            lauren Lauren MacArthur added a comment - - edited Ah, right...thanks Dan.  I just checked some older logs for the PF1_design_gri entries...missing there too, so probably a long-standing (and, indeed, never-to-be-fixed) bug in validate_drp .

            People

              lauren Lauren MacArthur
              lauren Lauren MacArthur
              Yusra AlSayyad
              Dan Taranu, Jim Bosch, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.