Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33849 Add butler registries YAML file to NCSA, summit, TTS
  3. DM-34147

Set DAF_BUTLER_REPOSITORY_INDEX environment variable in Jenkins

    XMLWordPrintable

    Details

    • Type: Technical task
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: jenkins
    • Labels:
      None

      Description

      It appears that Jenkins may be able to set a global environment variable (Configure System > Global properties > Environment variables > Add) that goes into every build, making this possible.

        Attachments

          Activity

          Hide
          ktl Kian-Tat Lim added a comment - - edited

          While the macOS machines have no access to a shared filesystem, they can't run any tests that would use a non-local Butler anyway.

          Show
          ktl Kian-Tat Lim added a comment - - edited While the macOS machines have no access to a shared filesystem, they can't run any tests that would use a non-local Butler anyway.
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          If it's not too much work I think that would still be highly preferable to not doing it at all. It would mean that tests requiring data/butlers can be decorated with a @unittest.skipIf(sys.platform == "darwin" rather than just a general "skip if no butler possible".

          I'm not really worried about that type of test failing on one operating system, in part because that's a fringe case most of the time anyway, in part because this isn't highly senstivie algorithmic code (which tends to be the significant proportion of cross-platform differences), and finally because the critical place in which commissioning code needs to work is not macOS but nublado/CentOS anyway.

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - If it's not too much work I think that would still be highly preferable to not doing it at all. It would mean that tests requiring data/butlers can be decorated with a @unittest.skipIf(sys.platform == "darwin" rather than just a general "skip if no butler possible". I'm not really worried about that type of test failing on one operating system, in part because that's a fringe case most of the time anyway, in part because this isn't highly senstivie algorithmic code (which tends to be the significant proportion of cross-platform differences), and finally because the critical place in which commissioning code needs to work is not macOS but nublado/CentOS anyway.
          Hide
          ktl Kian-Tat Lim added a comment -

          It's not clear that you should rely on even the CentOS nodes having access to a non-local Butler.  In general, tests should be self-contained, and we may want to run them on cloud nodes or other places.  My understanding is that all tests currently use data from within their own package or from testdata repositories, rather than anything in /repo or elsewhere.

          Show
          ktl Kian-Tat Lim added a comment - It's not clear that you should rely on even the CentOS nodes having access to a non-local Butler.  In general, tests should be self-contained, and we may want to run them on cloud nodes or other places.  My understanding is that all tests currently use data from within their own package or from testdata repositories, rather than anything in /repo or elsewhere.
          Hide
          ktl Kian-Tat Lim added a comment -

          For now, added the environment variable pointing to /project/data-repos.yaml.  It's possible that Jenkins jobs may not see this variable, or that the value is not where Jenkins actually sees the file, or that the pathnames in the file are not where Jenkins sees those, or that Postgres credentials are needed, but we'll cross those bridges when we get to them.

          Show
          ktl Kian-Tat Lim added a comment - For now, added the environment variable pointing to /project/data-repos.yaml .  It's possible that Jenkins jobs may not see this variable, or that the value is not where Jenkins actually sees the file, or that the pathnames in the file are not where Jenkins sees those, or that Postgres credentials are needed, but we'll cross those bridges when we get to them.
          Hide
          ktl Kian-Tat Lim added a comment -

          After Merlin Fisher-Levine reported that this was insufficient, I thought about it some more and looked into Jenkins.  Jenkins jobs are executed within "swarm" containers on the NCSA Kubernetes nodes, and the jobs themselves execute pipelines that generally create new containers and run builds within those.  Neither set of containers mounts the desired Butler repo filesystems, so all that is visible to the Jenkins job is the job-specific workspace.  Moreover, this isolation is desirable from a security and reproducibility standpoint.

          Bottom line: to get data into a Jenkins job, it should be cloned.

          Show
          ktl Kian-Tat Lim added a comment - After Merlin Fisher-Levine reported that this was insufficient, I thought about it some more and looked into Jenkins.  Jenkins jobs are executed within "swarm" containers on the NCSA Kubernetes nodes, and the jobs themselves execute pipelines that generally create new containers and run builds within those.  Neither set of containers mounts the desired Butler repo filesystems, so all that is visible to the Jenkins job is the job-specific workspace.  Moreover, this isolation is desirable from a security and reproducibility standpoint. Bottom line: to get data into a Jenkins job, it should be cloned.

            People

            Assignee:
            ktl Kian-Tat Lim
            Reporter:
            ktl Kian-Tat Lim
            Watchers:
            Kian-Tat Lim, Merlin Fisher-Levine
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.