Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18599

ap_verify jenkins pipeline does not set PRODUCT env variable

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      When running dispacth_verify.py with the --env jenkins option we expect the PRODUCT environment variable to contain the name of the pipeline, in this case PRODUCT=ap_verify

      https://github.com/lsst/verify/blob/master/python/lsst/verify/metadata/jenkinsci.py#L59

      That works fine for validate_drp but for ap_verify it is not set and the default value unkwon is returned.

      I noticed that in SQuaSH when using that to filter results from different pipelines.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18057 QAWG-REC-37: It must be possible to submit metrics to SQuaSH from arbitrary pipeline execution environments

          • To Do
          is child task of

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18057 QAWG-REC-37: It must be possible to submit metrics to SQuaSH from arbitrary pipeline execution environments

          • To Do
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18103 Revisit SQuaSH InfluxDB data model

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            I am at a loss as to where sqre/validate_drp is obtaining the PRODUCT env var from.

            • It is not present in the docker image dispatch_verify.py is run from before or after sourcing /opt/lsst/software/stack/loadLSST.bash. Eg., lsstsqre/centos:7-stack-lsst_distrib-d_2019_03_25
            • It is not present in the main pipeline script for either sqre/validate_drp or scipipe/ap_verify. Eg., {{$ grep PRODUCT jobs/ap_verify.groovy jobs/validate_drp.groovy pipelines/scipipe/ap_verify.groovy pipelines/sqre/validate_drp.groovy
              }}
            • Both pipelines are using the same utility method to invoke dispatch_verify.py: https://github.com/lsst-sqre/jenkins-dm-jobs/blob/master/pipelines/lib/util.groovy#L1884-L1909

            I could just declare PRODUCT from util.runDispatchVerify() to ensure it is present but continuing to rely on env vars used to be present under a matrix job that was replaced years ago is not likely to become any less brittle.

            How do people feel about passing in either a yaml or json doc as an alternative env type?

            Show
            jhoblitt Joshua Hoblitt added a comment - I am at a loss as to where sqre/validate_drp is obtaining the PRODUCT env var from. It is not present in the docker image dispatch_verify.py is run from before or after sourcing /opt/lsst/software/stack/loadLSST.bash . Eg., lsstsqre/centos:7-stack-lsst_distrib-d_2019_03_25 It is not present in the main pipeline script for either sqre/validate_drp or scipipe/ap_verify . Eg., {{$ grep PRODUCT jobs/ap_verify.groovy jobs/validate_drp.groovy pipelines/scipipe/ap_verify.groovy pipelines/sqre/validate_drp.groovy }} Both pipelines are using the same utility method to invoke dispatch_verify.py : https://github.com/lsst-sqre/jenkins-dm-jobs/blob/master/pipelines/lib/util.groovy#L1884-L1909 I could just declare PRODUCT from util.runDispatchVerify() to ensure it is present but continuing to rely on env vars used to be present under a matrix job that was replaced years ago is not likely to become any less brittle. How do people feel about passing in either a yaml or json doc as an alternative env type?
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Hey Joshua Hoblitt looking more carefully, the PRODUCT env var is not being captured since after valide_drp was converted to a pipeline in Jenkins sometime around April last year!

            Passing in a configuration file defining the execution environment seems a much better option. I could make this change in verify as part of DM-18057.

            Show
            afausti Angelo Fausti added a comment - Hey Joshua Hoblitt looking more carefully, the PRODUCT env var is not being captured since after valide_drp was converted to a pipeline in Jenkins sometime around April last year! Passing in a configuration file defining the execution environment seems a much better option. I could make this change in verify as part of DM-18057 .
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            Angelo Fausti That works for me – should DM-18057 be a blocker of this ticket?

            Show
            jhoblitt Joshua Hoblitt added a comment - Angelo Fausti That works for me – should DM-18057 be a blocker of this ticket?
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            A feature that would be nice is missing yaml/json keys would cause a failure instead of silently filling in a default value.

            Show
            jhoblitt Joshua Hoblitt added a comment - A feature that would be nice is missing yaml/json keys would cause a failure instead of silently filling in a default value.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Seems good to me.

            Show
            afausti Angelo Fausti added a comment - Seems good to me.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti, Joshua Hoblitt, Krzysztof Findeisen, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.