Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28888

Allow SQuaSH uploads from ap_verify Gen 3

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify, jenkins
    • Labels:
      None

      Description

      Alter the scipipe/ap_verify Jenkins job to upload to SQuaSH when run in Gen 3, using the result of DM-21916. This work was previously intended for DM-21888, but the benefit now will be Gen 3 uploads of CI-CosmosPDR2 instead of CI-HiTS2015.

      This work also includes updating the Chronograph dashboards to properly distinguish between Gen 2 and Gen 3 output, but these changes don't lend themselves to easy review.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            The Chronograf queries include a grandfather clause assuming that any uploads without butler_generation (i.e., pre-DM-21916) are Gen 2. I don't know if at some point we want to go back and add that tag to all uploads; I could see such clauses getting out of hand if we keep changing the tag specs (see RFC-758).

            Show
            krzys Krzysztof Findeisen added a comment - - edited The Chronograf queries include a grandfather clause assuming that any uploads without butler_generation (i.e., pre- DM-21916 ) are Gen 2. I don't know if at some point we want to go back and add that tag to all uploads; I could see such clauses getting out of hand if we keep changing the tag specs (see RFC-758 ).
            Hide
            krzys Krzysztof Findeisen added a comment -

            Hi Kian-Tat Lim, would you be willing to review this? It's 80 lines of changes.

            In addition to updating the ap_verify Jenkins job, I've added a function to util that calls the gen3_to_job script added to verify on DM-21916.

            Show
            krzys Krzysztof Findeisen added a comment - Hi Kian-Tat Lim , would you be willing to review this? It's 80 lines of changes. In addition to updating the ap_verify Jenkins job, I've added a function to util that calls the gen3_to_job script added to verify on DM-21916 .
            Hide
            ktl Kian-Tat Lim added a comment -

            Looks good.

            Show
            ktl Kian-Tat Lim added a comment - Looks good.
            Hide
            ktl Kian-Tat Lim added a comment -

            Unfortunately, Jenkins somehow doesn't like the util.joinPath call. I can't see why it should have problems when the later util.runDispatchVerify call appears to work.

            ERROR: Execution failed
            java.lang.NoSuchMethodError: No such DSL method 'joinPath' found among steps [archive, bat, build, catchError, checkout, deleteDir, dir, dockerFingerprintFrom, dockerFingerprintRun, echo, envVarsForTool, error, fileExists, findFiles, getContext, git, input, isUnix, jiraComment, jiraIssueSelector, jiraSearch, junit, library, libraryResource, load, lock, mail, milestone, node, nodesByLabel, parallel, powershell, properties, publishHTML, pwd, readCSV, readFile, readJSON, readManifest, readMavenPom, readProperties, readTrusted, readYaml, resolveScm, retry, script, sh, sha1, sleep, sshagent, stage, stash, step, tee, timeout, tm, tool, touch, unarchive, unstash, unzip, validateDeclarativePipeline, waitUntil, withContext, withCredentials, withDockerContainer, withDockerRegistry, withDockerServer, withEnv, wrap, writeCSV, writeFile, writeJSON, writeMavenPom, writeYaml, ws, zip] or symbols [all, allOf, always, any, anyOf, apiToken, architecture, archiveArtifacts, artifactManager, authorizationMatrix, batchFile, bitbucket, booleanParam, branch, buildButton, buildDiscarder, buildParameter, buildingTag, caseInsensitive, caseSensitive, certificate, changeRequest, changelog, changeset, checkoutToSubdirectory, choice, choiceParam, clock, cloud, command, copyArtifactPermission, copyArtifacts, credentials, cron, crumb, defaultView, demand, disableConcurrentBuilds, disableResume, docker, dockerCert, dockerfile, downloadSettings, downstream, dumb, durabilityHint, envVars, environment, equals, expression, file, fileParam, filePath, fingerprint, frameOptions, freeStyle, freeStyleJob, fromScm, fromSource, git, github, githubAccessToken, githubPush, headRegexFilter, headWildcardFilter, hyperlink, hyperlinkToModels, inheriting, inheritingGlobal, installSource, isRestartedRun, jdk, jdkInstaller, jgit, jgitapache, jnlp, jobDsl, jobName, label, lastCompleted, lastDuration, lastFailure, lastGrantedAuthorities, lastStable, lastSuccess, lastSuccessful, lastWithArtifacts, latestSavedBuild, legacy, legacySCM, list, local, location, logRotator, loggedInUsersCanDoAnything, masterBuild, maven, maven3Mojos, mavenErrors, mavenMojos, mavenWarnings, modernSCM, myView, newContainerPerStage, node, nodeProperties, nonInheriting, nonStoredPasswordParam, none, not, overrideIndexTriggers, paneStatus, parallelsAlwaysFailFast, parameters, password, pattern, permalink, permanent, pipeline-model, pipelineTriggers, plainText, plugin, pollSCM, preserveStashes, projectNamingStrategy, proxy, queueItemAuthenticator, quietPeriod, rateLimitBuilds, remotingCLI, run, runParam, schedule, scmRetryCount, script, scriptApprovalLink, search, security, shell, skipDefaultCheckout, skipStagesAfterUnstable, slave, sourceRegexFilter, sourceWildcardFilter, specific, sshUserPrivateKey, stackTrace, standard, status, string, stringParam, swapSpace, tag, text, textParam, tmpSpace, toolLocation, triggeredBy, unsecured, upstream, url, userSeed, usernameColonPassword, usernamePassword, viewsTabBar, weather, workspace, zfs, zip] or globals [currentBuild, docker, env, params, pipeline, scm]
            	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:199)
            	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
            	at sun.reflect.GeneratedMethodAccessor636634.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:498)
            	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
            	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
            	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
            	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
            	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
            	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
            	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
            	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:158)
            	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
            	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
            	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
            	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:156)
            	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:160)
            	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
            	at WorkflowScript.verifyDataset(WorkflowScript:275)
            

            Show
            ktl Kian-Tat Lim added a comment - Unfortunately, Jenkins somehow doesn't like the util.joinPath call. I can't see why it should have problems when the later util.runDispatchVerify call appears to work. ERROR: Execution failed java.lang.NoSuchMethodError: No such DSL method 'joinPath' found among steps [archive, bat, build, catchError, checkout, deleteDir, dir, dockerFingerprintFrom, dockerFingerprintRun, echo, envVarsForTool, error, fileExists, findFiles, getContext, git, input, isUnix, jiraComment, jiraIssueSelector, jiraSearch, junit, library, libraryResource, load, lock, mail, milestone, node, nodesByLabel, parallel, powershell, properties, publishHTML, pwd, readCSV, readFile, readJSON, readManifest, readMavenPom, readProperties, readTrusted, readYaml, resolveScm, retry, script, sh, sha1, sleep, sshagent, stage, stash, step, tee, timeout, tm, tool, touch, unarchive, unstash, unzip, validateDeclarativePipeline, waitUntil, withContext, withCredentials, withDockerContainer, withDockerRegistry, withDockerServer, withEnv, wrap, writeCSV, writeFile, writeJSON, writeMavenPom, writeYaml, ws, zip] or symbols [all, allOf, always, any, anyOf, apiToken, architecture, archiveArtifacts, artifactManager, authorizationMatrix, batchFile, bitbucket, booleanParam, branch, buildButton, buildDiscarder, buildParameter, buildingTag, caseInsensitive, caseSensitive, certificate, changeRequest, changelog, changeset, checkoutToSubdirectory, choice, choiceParam, clock, cloud, command, copyArtifactPermission, copyArtifacts, credentials, cron, crumb, defaultView, demand, disableConcurrentBuilds, disableResume, docker, dockerCert, dockerfile, downloadSettings, downstream, dumb, durabilityHint, envVars, environment, equals, expression, file, fileParam, filePath, fingerprint, frameOptions, freeStyle, freeStyleJob, fromScm, fromSource, git, github, githubAccessToken, githubPush, headRegexFilter, headWildcardFilter, hyperlink, hyperlinkToModels, inheriting, inheritingGlobal, installSource, isRestartedRun, jdk, jdkInstaller, jgit, jgitapache, jnlp, jobDsl, jobName, label, lastCompleted, lastDuration, lastFailure, lastGrantedAuthorities, lastStable, lastSuccess, lastSuccessful, lastWithArtifacts, latestSavedBuild, legacy, legacySCM, list, local, location, logRotator, loggedInUsersCanDoAnything, masterBuild, maven, maven3Mojos, mavenErrors, mavenMojos, mavenWarnings, modernSCM, myView, newContainerPerStage, node, nodeProperties, nonInheriting, nonStoredPasswordParam, none, not, overrideIndexTriggers, paneStatus, parallelsAlwaysFailFast, parameters, password, pattern, permalink, permanent, pipeline-model, pipelineTriggers, plainText, plugin, pollSCM, preserveStashes, projectNamingStrategy, proxy, queueItemAuthenticator, quietPeriod, rateLimitBuilds, remotingCLI, run, runParam, schedule, scmRetryCount, script, scriptApprovalLink, search, security, shell, skipDefaultCheckout, skipStagesAfterUnstable, slave, sourceRegexFilter, sourceWildcardFilter, specific, sshUserPrivateKey, stackTrace, standard, status, string, stringParam, swapSpace, tag, text, textParam, tmpSpace, toolLocation, triggeredBy, unsecured, upstream, url, userSeed, usernameColonPassword, usernamePassword, viewsTabBar, weather, workspace, zfs, zip] or globals [currentBuild, docker, env, params, pipeline, scm] at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:199) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at sun.reflect.GeneratedMethodAccessor636634.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:158) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:156) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:160) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at WorkflowScript.verifyDataset(WorkflowScript:275)

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Eric Bellm, Ian Sullivan, Kian-Tat Lim, Krzysztof Findeisen, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.