Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10397

Collect the results from validation runs

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      Alert Production S17 - 6
    • Team:
      Alert Production

      Description

      Once the validation datasets have been processed, analyze the outputs for completeness. This will include categorizing the failure modes (if any exist ).

      Things to be collected could include:
      1. Pass/Fail
      2. Num ref/num source
      3. Rms scatter and error on that value
      4. Broad categories of failures

        Attachments

          Issue Links

            Activity

            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Started collecting summary statics from data run with both optimistic and pessimistic matchers..

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Started collecting summary statics from data run with both optimistic and pessimistic matchers..
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Created ascii files for each of the processed data from HSC, CFHTLS, and HITS. These data are stored in the directory:

            /project/morriscb/VALIDATE_data

            2 files are stored per rerun: The visits/ccds where both matcher "succeeded" (as in produced a calExp) and where some part of the pipeline failed (i.e. the visti/ccd contains bad data or at least one of the matchers failed).

            These data were created from the output files produced by singleFrameDriver.py by using the Jupyter notebook in "/home/morriscb/Parse and store output logs for matcher.ipynb".

            This script could be more generailized for end to end creation of these logs files given a request from the reviewer.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Created ascii files for each of the processed data from HSC, CFHTLS, and HITS. These data are stored in the directory: /project/morriscb/VALIDATE_data 2 files are stored per rerun: The visits/ccds where both matcher "succeeded" (as in produced a calExp) and where some part of the pipeline failed (i.e. the visti/ccd contains bad data or at least one of the matchers failed). These data were created from the output files produced by singleFrameDriver.py by using the Jupyter notebook in "/home/morriscb/Parse and store output logs for matcher.ipynb". This script could be more generailized for end to end creation of these logs files given a request from the reviewer.
            Hide
            krughoff Simon Krughoff added a comment -

            Can you remind me what you mean by fail? I thought it was something more subtle than just failing to produce a calexp. I.e. the match scatter being too large would indicate failure, but maybe that comes in the analysis step.

            I think we need to categorize the cases where a failure to produce a calexp happened. What were the causes of those failures? I think that needs to be part of this ticket.

            Show
            krughoff Simon Krughoff added a comment - Can you remind me what you mean by fail? I thought it was something more subtle than just failing to produce a calexp. I.e. the match scatter being too large would indicate failure, but maybe that comes in the analysis step. I think we need to categorize the cases where a failure to produce a calexp happened. What were the causes of those failures? I think that needs to be part of this ticket.
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited

            Hey Simon,

            By fail in this context I mean did not produce a calexp for one of 2 cases: 1) Either matcher failed to find any match 2) an earlier pipeline step failed for some reason.

            The number of pipeline failures for each of the different reruns are:
            CFHTLS_u: 2016 visits, 7 pipeline failures
            CFHTLS_i: 2016 visits, 7 pipeline failures
            CFHTLS_z: 2016 visits, 7 pipeline failures
            CFHTLS: 11,700 visits, 2757 pipeline failures
            HITS: 11,040 visits, 162 pipeline failures
            HSC: 3952 visits, 0 pipeline failures.

            A failure to match either catastrophically or with a bad scatter will be investigated in the next ticket DM-10398.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited Hey Simon, By fail in this context I mean did not produce a calexp for one of 2 cases: 1) Either matcher failed to find any match 2) an earlier pipeline step failed for some reason. The number of pipeline failures for each of the different reruns are: CFHTLS_u: 2016 visits, 7 pipeline failures CFHTLS_i: 2016 visits, 7 pipeline failures CFHTLS_z: 2016 visits, 7 pipeline failures CFHTLS: 11,700 visits, 2757 pipeline failures HITS: 11,040 visits, 162 pipeline failures HSC: 3952 visits, 0 pipeline failures. A failure to match either catastrophically or with a bad scatter will be investigated in the next ticket DM-10398 .
            Hide
            krughoff Simon Krughoff added a comment -

            Sounds good. Ship it.

            Show
            krughoff Simon Krughoff added a comment - Sounds good. Ship it.
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Summary data files created successfully.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Summary data files created successfully.

              People

              Assignee:
              cmorrison Chris Morrison [X] (Inactive)
              Reporter:
              krughoff Simon Krughoff
              Reviewers:
              Simon Krughoff
              Watchers:
              Chris Morrison [X] (Inactive), Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.