Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12273

multiple stack-os-matrix failures: conda cython 0.26 py36_0

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      This error has been seen multiple times this morning and it appears to always been with the combination of centos-6 and py3.

       
      add_anaconda_token: True
      add_pip_as_python_dependency: True
      allow_non_channel_urls: True
      allow_softlinks: True
      always_copy: False
      always_softlink: False
      always_yes: False
      anaconda_upload: None
      auto_update_conda: True
      changeps1: True
      channel_alias: https://conda.anaconda.org
      channel_priority: True
      channels:
        - http://****/pkgs/free
      client_ssl_cert: None
      client_ssl_cert_key: None
      clobber: False
      create_default_packages: []
      custom_channels:
        pkgs/free: https://repo.continuum.io/
        pkgs/r: https://repo.continuum.io/
        pkgs/pro: https://repo.continuum.io/
      custom_multichannels:
        defaults: ["https://repo.continuum.io/pkgs/free", "https://repo.continuum.io/pkgs/r", "https://repo.continuum.io/pkgs/pro"]
        local: []
      default_channels:
        - https://repo.continuum.io/pkgs/free
        - https://repo.continuum.io/pkgs/r
        - https://repo.continuum.io/pkgs/pro
      disallow: []
      envs_dirs:
        - /home/jenkins-slave/workspace/stack-os-matrix@2/centos-6.py3/lsstsw/miniconda/envs
        - /home/jenkins-slave/.conda/envs
      force: False
      json: False
      local_repodata_ttl: 1
      migrated_channel_aliases: []
      offline: False
      path_conflict: clobber
      pinned_packages: []
      pkgs_dirs:
        - /home/jenkins-slave/workspace/stack-os-matrix@2/centos-6.py3/lsstsw/miniconda/pkgs
        - /home/jenkins-slave/.conda/pkgs
      proxy_servers: {}
      quiet: False
      remote_connect_timeout_secs: 9.15
      remote_max_retries: 3
      remote_read_timeout_secs: 60.0
      rollback_enabled: True
      shortcuts: True
      show_channel_urls: None
      ssl_verify: True
      track_features: []
      use_pip: True
      verbosity: 0
      Fetching package metadata .....
       
      PackageNotFoundError: Package missing in current linux-64 channels: 
        - cython 0.26 py36_0
       
      script returned exit code 1
      

      The package is present on our jenkins conda repo mirror:

      http://conda-mirror.lsst.codes.s3.amazonaws.com/pkgs/free/linux-64/cython-0.26-py36_0.tar.bz2

      so it is unlikely to be a network communication glitch. I suspect there may be some sort of conda cached state corruption.

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -
          Show
          jhoblitt Joshua Hoblitt added a comment - I've pushed cleaning up the repodata.json dir(s) between builds. Test #3: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Fupdate-cmirror/detail/update-cmirror/227/pipeline
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Test #3 ended up with 3 copies of the osx-64 repodata.json and no other architectures. After some head scratching, it turns out that there was an accidental hard coded value (osx-64) introduced during the refactoring, causing the osx-64 repo to be mirrored 3 times. This has been fixed and merged.

          Show
          jhoblitt Joshua Hoblitt added a comment - Test #3 ended up with 3 copies of the osx-64 repodata.json and no other architectures. After some head scratching, it turns out that there was an accidental hard coded value ( osx-64 ) introduced during the refactoring, causing the osx-64 repo to be mirrored 3 times. This has been fixed and merged.
          Show
          jhoblitt Joshua Hoblitt added a comment - Test #4: https://ci.lsst.codes/job/sqre/job/infrastructure/job/update-cmirror/228/
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Test #4 looks good. List of artifacts:

          NAME
          SIZE
          pipeline.log
          -
          repodata/linux-64/repodata.json
          5.3 MB
          repodata/noarch/repodata.json
          9.8 KB
          repodata/osx-64/repodata.json
          5.2 MB
          
          

          Show
          jhoblitt Joshua Hoblitt added a comment - Test #4 looks good. List of artifacts: NAME SIZE pipeline.log - repodata/linux- 64 /repodata.json 5.3 MB repodata/noarch/repodata.json 9.8 KB repodata/osx- 64 /repodata.json 5.2 MB
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Summary:

          • The package installation failure seems to have gone away completely after a run of the update-cmirror job
          • aggressive, behind the conda utilties "back", cleanup of cached conda repo metadata was implemented – this may or may not have been necessary to recover from a changed repodata.json. However, it doesn't seem to have any percevable performance impact so these changes are being left enabled for the time being.
          • the update-cmirror job was modified to make a separate download of the upstream repodata.json files, which are archived, before invoking the conda-mirror utlity. This may be useful for determining a root cause if this failure re-occurs.
          Show
          jhoblitt Joshua Hoblitt added a comment - Summary: The package installation failure seems to have gone away completely after a run of the update-cmirror job aggressive, behind the conda utilties "back", cleanup of cached conda repo metadata was implemented – this may or may not have been necessary to recover from a changed repodata.json . However, it doesn't seem to have any percevable performance impact so these changes are being left enabled for the time being. the update-cmirror job was modified to make a separate download of the upstream repodata.json files, which are archived, before invoking the conda-mirror utlity. This may be useful for determining a root cause if this failure re-occurs.

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Joshua Hoblitt, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel