Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25457

Reevaluate conda mirror, cleanup jenkins jobs

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: conda, jenkins
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      The conda mirror jobs are mostly obsolete now, we should cleanup the mirror and mirroring jobs.

        Attachments

          Issue Links

            Activity

            No builds found.
            bvan Brian Van Klaveren created issue -
            bvan Brian Van Klaveren made changes -
            Field Original Value New Value
            Status To Do [ 10001 ] In Progress [ 3 ]
            bvan Brian Van Klaveren made changes -
            Link This issue relates to DM-24972 [ DM-24972 ]
            Hide
            bvan Brian Van Klaveren added a comment -

            I tried to fix the mirror last night, which we are using at this point ONLY for the installer, since we switched to using conda list --explicit in DM-24972, but fixing the URLs for the mirroring seems to have garbled up index.html in a weird way, and subsequently I had tried to remove the use of conda mirroring last night before the nightly, but I missed a line for the OS X tarballs (which is why the job failed)

            As for history of the mirror, before mid 2017, packages and installer on repo.continuum.io were not served Cloudflare CDN, which is around the time we setup the mirror. Sometime after that, packages for defaults and conda-forge moved to the Cloudflare CDN. Soon after that was the rollout of repo.anaconda.com, corresponding with a renaming of continuum analytics to Anaconda, Inc.

            So I think we shouldn't use (maintain really - we're not using it for the most part) the mirror anymore and we could free it up from S3. The PR to jenkins-dm-jobs reflects this.

            Also, update-cmirror is the only job running in the infra weekly job. That job and associated cron job for it has been removed.

            Show
            bvan Brian Van Klaveren added a comment - I tried to fix the mirror last night, which we are using at this point ONLY for the installer, since we switched to using conda list --explicit in DM-24972 , but fixing the URLs for the mirroring seems to have garbled up index.html in a weird way, and subsequently I had tried to remove the use of conda mirroring last night before the nightly, but I missed a line for the OS X tarballs (which is why the job failed) As for history of the mirror, before mid 2017, packages and installer on repo.continuum.io were not served Cloudflare CDN, which is around the time we setup the mirror. Sometime after that, packages for defaults and conda-forge moved to the Cloudflare CDN. Soon after that was the rollout of repo.anaconda.com, corresponding with a renaming of continuum analytics to Anaconda, Inc. So I think we shouldn't use (maintain really - we're not using it for the most part) the mirror anymore and we could free it up from S3. The PR to jenkins-dm-jobs reflects this. Also, update-cmirror is the only job running in the infra weekly job. That job and associated cron job for it has been removed.
            Hide
            bvan Brian Van Klaveren added a comment -

            The first commit (fc7c606) should be merged tonight. If the rest is controversial, I will split that out.

            Show
            bvan Brian Van Klaveren added a comment - The first commit (fc7c606) should be merged tonight. If the rest is controversial, I will split that out.
            bvan Brian Van Klaveren made changes -
            Reviewers Kian-Tat Lim [ ktl ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            ktl Kian-Tat Lim made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            bvan Brian Van Klaveren made changes -
            Link This issue relates to DM-24981 [ DM-24981 ]
            bvan Brian Van Klaveren made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            bvan Brian Van Klaveren made changes -
            Link This issue relates to DM-25897 [ DM-25897 ]

              People

              Assignee:
              bvan Brian Van Klaveren
              Reporter:
              bvan Brian Van Klaveren
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Brian Van Klaveren, Kian-Tat Lim
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.