Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Labels:None
-
Story Points:4
-
Epic Link:
-
Sprint:AP S21-3 (February)
-
Team:Alert Production
-
Urgent?:No
Description
Alter the scipipe/ap_verify Jenkins job to upload to SQuaSH when run in Gen 3, using the result of DM-21916. This work was previously intended for DM-21888, but the benefit now will be Gen 3 uploads of CI-CosmosPDR2 instead of CI-HiTS2015.
This work also includes updating the Chronograph dashboards to properly distinguish between Gen 2 and Gen 3 output, but these changes don't lend themselves to easy review.
Attachments
Issue Links
Activity
Hi Kian-Tat Lim, would you be willing to review this? It's 80 lines of changes.
In addition to updating the ap_verify Jenkins job, I've added a function to util that calls the gen3_to_job script added to verify on DM-21916.
Krzysztof Findeisen
added a comment - Hi Kian-Tat Lim , would you be willing to review this? It's 80 lines of changes.
In addition to updating the ap_verify Jenkins job, I've added a function to util that calls the gen3_to_job script added to verify on DM-21916 . Looks good.
Unfortunately, Jenkins somehow doesn't like the util.joinPath call. I can't see why it should have problems when the later util.runDispatchVerify call appears to work.
ERROR: Execution failed
|
java.lang.NoSuchMethodError: No such DSL method 'joinPath' found among steps [archive, bat, build, catchError, checkout, deleteDir, dir, dockerFingerprintFrom, dockerFingerprintRun, echo, envVarsForTool, error, fileExists, findFiles, getContext, git, input, isUnix, jiraComment, jiraIssueSelector, jiraSearch, junit, library, libraryResource, load, lock, mail, milestone, node, nodesByLabel, parallel, powershell, properties, publishHTML, pwd, readCSV, readFile, readJSON, readManifest, readMavenPom, readProperties, readTrusted, readYaml, resolveScm, retry, script, sh, sha1, sleep, sshagent, stage, stash, step, tee, timeout, tm, tool, touch, unarchive, unstash, unzip, validateDeclarativePipeline, waitUntil, withContext, withCredentials, withDockerContainer, withDockerRegistry, withDockerServer, withEnv, wrap, writeCSV, writeFile, writeJSON, writeMavenPom, writeYaml, ws, zip] or symbols [all, allOf, always, any, anyOf, apiToken, architecture, archiveArtifacts, artifactManager, authorizationMatrix, batchFile, bitbucket, booleanParam, branch, buildButton, buildDiscarder, buildParameter, buildingTag, caseInsensitive, caseSensitive, certificate, changeRequest, changelog, changeset, checkoutToSubdirectory, choice, choiceParam, clock, cloud, command, copyArtifactPermission, copyArtifacts, credentials, cron, crumb, defaultView, demand, disableConcurrentBuilds, disableResume, docker, dockerCert, dockerfile, downloadSettings, downstream, dumb, durabilityHint, envVars, environment, equals, expression, file, fileParam, filePath, fingerprint, frameOptions, freeStyle, freeStyleJob, fromScm, fromSource, git, github, githubAccessToken, githubPush, headRegexFilter, headWildcardFilter, hyperlink, hyperlinkToModels, inheriting, inheritingGlobal, installSource, isRestartedRun, jdk, jdkInstaller, jgit, jgitapache, jnlp, jobDsl, jobName, label, lastCompleted, lastDuration, lastFailure, lastGrantedAuthorities, lastStable, lastSuccess, lastSuccessful, lastWithArtifacts, latestSavedBuild, legacy, legacySCM, list, local, location, logRotator, loggedInUsersCanDoAnything, masterBuild, maven, maven3Mojos, mavenErrors, mavenMojos, mavenWarnings, modernSCM, myView, newContainerPerStage, node, nodeProperties, nonInheriting, nonStoredPasswordParam, none, not, overrideIndexTriggers, paneStatus, parallelsAlwaysFailFast, parameters, password, pattern, permalink, permanent, pipeline-model, pipelineTriggers, plainText, plugin, pollSCM, preserveStashes, projectNamingStrategy, proxy, queueItemAuthenticator, quietPeriod, rateLimitBuilds, remotingCLI, run, runParam, schedule, scmRetryCount, script, scriptApprovalLink, search, security, shell, skipDefaultCheckout, skipStagesAfterUnstable, slave, sourceRegexFilter, sourceWildcardFilter, specific, sshUserPrivateKey, stackTrace, standard, status, string, stringParam, swapSpace, tag, text, textParam, tmpSpace, toolLocation, triggeredBy, unsecured, upstream, url, userSeed, usernameColonPassword, usernamePassword, viewsTabBar, weather, workspace, zfs, zip] or globals [currentBuild, docker, env, params, pipeline, scm]
|
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:199)
|
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
|
at sun.reflect.GeneratedMethodAccessor636634.invoke(Unknown Source)
|
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
|
at java.lang.reflect.Method.invoke(Method.java:498)
|
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
|
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
|
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
|
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
|
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
|
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
|
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
|
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:158)
|
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
|
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
|
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
|
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:156)
|
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:160)
|
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
|
at WorkflowScript.verifyDataset(WorkflowScript:275)
|
The Chronograf queries include a grandfather clause assuming that any uploads without butler_generation (i.e., pre-
DM-21916) are Gen 2. I don't know if at some point we want to go back and add that tag to all uploads; I could see such clauses getting out of hand if we keep changing the tag specs (seeRFC-758).