Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31326

Run faro on gen3_RC2_subset and set it up in Jenkins

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • faro

    Description

      To confirm that gen3_RC2_subset will meet our needs for a CI dataset, run `faro` on those data and check the resulting metrics.

      Make necessary changes to the pipeline in gen3_RC2_subset and set it up to regularly run in Jenkins.

      Attachments

        Issue Links

          Activity

            Oh, so the script (DRP.yaml) that is in gen3_RC2_subset is different from what would be needed in Jenkins? (Sorry if I'm being dense...)

            jcarlin Jeffrey Carlin added a comment - Oh, so the script (DRP.yaml) that is in gen3_RC2_subset is different from what would be needed in Jenkins? (Sorry if I'm being dense...)
            jcarlin Jeffrey Carlin added a comment - - edited

            I added the `faro` steps to the DRP.yaml file in rc2_subset (PR here), and added a shell script to run all the processing in `faro` (PR here).

            I have confirmed that the script runs to completion and generates the expected output data. The next step is to set up a Jenkins job (under the name `verify_drp_metrics`).

            jcarlin Jeffrey Carlin added a comment - - edited I added the `faro` steps to the DRP.yaml file in rc2_subset (PR here ), and added a shell script to run all the processing in `faro` (PR here ). I have confirmed that the script runs to completion and generates the expected output data. The next step is to set up a Jenkins job (under the name `verify_drp_metrics`).

            I'd like you to take a look at some of the comments I have in the faro PR, but it all looks generally OK to me.

            krughoff Simon Krughoff (Inactive) added a comment - I'd like you to take a look at some of the comments I have in the faro PR, but it all looks generally OK to me.

            Thanks for the suggestions – I implemented them, and confirmed that it runs to completion. With just patch 40, it took ~4.5 hours on lsst-devl with 8 cores.

            Successful Jenkins run here.

            jcarlin Jeffrey Carlin added a comment - Thanks for the suggestions – I implemented them, and confirmed that it runs to completion. With just patch 40, it took ~4.5 hours on lsst-devl with 8 cores. Successful Jenkins run here .

            Both PRs have been merged.

            jcarlin Jeffrey Carlin added a comment - Both PRs have been merged.

            People

              jcarlin Jeffrey Carlin
              jcarlin Jeffrey Carlin
              Simon Krughoff (Inactive)
              Jeffrey Carlin, Leanne Guy, Simon Krughoff (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.