Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15072

Kombu error with LTD Keeper 1.11.0

    XMLWordPrintable

    Details

      Description

      There may be an incompatibility in an implicit dependency found through Celery that is preventing the LTD Keeper workers from booting up:

      Traceback (most recent call last): File "/usr/local/bin/celery", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python3.5/site-packages/celery/__main__.py", line 14, in main _main() File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 326, in main cmd.execute_from_commandline(argv) File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) File "/usr/local/lib/python3.5/site-packages/celery/bin/base.py", line 281, in execute_from_commandline return self.handle_argv(self.prog_name, argv[1:]) File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 480, in handle_argv return self.execute(command, argv) File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 412, in execute ).run_from_argv(self.prog_name, argv[1:], command=argv[0]) File "/usr/local/lib/python3.5/site-packages/celery/bin/worker.py", line 221, in run_from_argv return self(*args, **options) File "/usr/local/lib/python3.5/site-packages/celery/bin/base.py", line 244, in __call__ ret = self.run(*args, **kwargs) File "/usr/local/lib/python3.5/site-packages/celery/bin/worker.py", line 255, in run **kwargs) File "/usr/local/lib/python3.5/site-packages/celery/worker/worker.py", line 99, in __init__ self.setup_instance(**self.prepare_args(**kwargs)) File "/usr/local/lib/python3.5/site-packages/celery/worker/worker.py", line 122, in setup_instance self.should_use_eventloop() if use_eventloop is None File "/usr/local/lib/python3.5/site-packages/celery/worker/worker.py", line 241, in should_use_eventloop self._conninfo.transport.implements.async and File "/usr/local/lib/python3.5/site-packages/kombu/transport/base.py", line 125, in __getattr__ raise AttributeError(key) AttributeError: async
      

      https://console.cloud.google.com/logs/viewer?project=plasma-geode-127520&authuser=1&_ga=2.223555801.-353975650.1530195679&pli=1&minLogLevel=0&expandAll=false&timestamp=2018-07-09T22%3A49%3A09.738000000Z&customFacets&limitCustomFacetWidth=true&dateRangeStart=2018-07-09T21%3A49%3A09.990Z&dateRangeEnd=2018-07-09T22%3A49%3A09.990Z&interval=PT1H&resource=container%2Fcluster_name%2Flsst-docs&scrollTimestamp=2018-07-09T22%3A36%3A18.000000000Z&advancedFilter=resource.type%3D%22container%22%0Aresource.labels.pod_id%3D%22keeper-worker-deployment-5457bf656-zrpmf%22%0Aresource.labels.zone%3D%22us-central1-b%22%0Aresource.labels.project_id%3D%22plasma-geode-127520%22%0Aresource.labels.cluster_name%3D%22lsst-docs%22%0Aresource.labels.container_name%3D%22keeper-worker%22%0Aresource.labels.namespace_id%3D%22ltd-prod%22%0Aresource.labels.instance_id%3D%2244590524507142367%22%0Atimestamp%3D%222018-07-09T22%3A36%3A18.000000000Z%22%0AinsertId%3D%22towqstg1iy3niu%22

      Appears with LTD Keeper 1.11.0.

      The tickets-DM-14122 Docker image deploys ok. But the 1.9.0 image does not.

      1.9.0 was built retroactively much later than the associated ticket was merged (and tickets-DM-14122 was built).

      This means I think there was a change in kombu that we're picking up as a floating implicit dependency of the pinned celery dep.

        Attachments

          Issue Links

            Activity

            Hide
            jsick Jonathan Sick added a comment -

            Celery 4.1.1 was released to fix this issue. http://www.celeryproject.org/news/celery-411-hot-fix-releaesd/

            Show
            jsick Jonathan Sick added a comment - Celery 4.1.1 was released to fix this issue. http://www.celeryproject.org/news/celery-411-hot-fix-releaesd/
            Hide
            jsick Jonathan Sick added a comment -

            Released and deployed as LTD Keeper 1.12.0

            Show
            jsick Jonathan Sick added a comment - Released and deployed as LTD Keeper 1.12.0

              People

              Assignee:
              jsick Jonathan Sick
              Reporter:
              jsick Jonathan Sick
              Watchers:
              Jonathan Sick
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.