Jonathan Sick added a comment - 03/May/18 12:09 PM Changelog: This release includes the celery task queuing system and major internal updates to the application structure and dependencies. DM-14122 . API updates Endpoints that launch asynchronous queue tasks now provide a queue_url field. This is a URL to an endpoint that provides status information on the queued task. For example, after PATCHing an edition with a new build, you can watch the queue_url to see when the rebuild is complete. The queue_urls are provided by the new GET /queue/(id) endpoint. We don't yet provide a way to query the queue in general — you can only get URLs by being the user that triggered the task. Endpoints, especially PATCH /editions/(id), should no longer timeout (500 error) for large documentation projects. The /editions/(id) resource includes a new pending_rebuild field. This field acts as a semaphore and is set to true if there is a pending rebuild task. You can't PATCH the edition's build_url when pending_rebuild is true. If necessary, an operator can PATCH pending_rebuild to false if the Celery task that rebuilds the edition failed. Deployment updates New deployment: keeper-redis. This deployment consists of a single Redis container (official redis:4-alpine image). There is no persistent storage or high-availability at thistime (this was judged a fair trade off since the Celery queue is inherently transient). New service: keeper-redis. This service fronts the keeper-redis deployment. New deployment: keeper-worker-deployment. This deployment mirrors keeper-deployment, except that the run command starts a Celery worker for the LTD Keeper application. This deployment can be scaled up to provide additional workers. The keeper-worker-deployment is not fronted by a service since the Celery workers pull tasks from keeper-redis. Internal updates Dependency updates: Flask 0.12.2 Requests 2.18.4 uwsgi 2.0.17 Flask-SQLAlchemy 2.3.2 PyMySQL 0.8.0 Flask-Migrate 2.1.1 Switched from Flask-Script to flask.cli. The Makefile now fronts most of the Flask commands for convience during development. Run make help to learn more. Application architecture improvements: Moved the Flask application factory out of _ init _.py to keeper.appfactory. Moved the get_auth_token route to the api blueprint. Moved DB connection object to keeper.models.db. Add Product.from_url() and Edition.from_url() methods for consistency with Build.from_url. Logging updates: Now we specifically set up the keeper logger instead of the root logger. This keeps things manageable when turning on debug-level logging. New app configuration for logging level. Debug-level logging is used in the development and testing profiles, while info-level logging is used in production. New celery app factory in keeper.celery. New Celery task queuing infrastructure in keeper.taskrunner. In a request context, application code can add an asynchronous task by calling append_task_to_chain() with a Celery task signature. These task signatures are persisted, within the request context, in flask.g.tasks. Just before a route handler returns it should call launch_task_chain(), which launches the task chain asynchronously. The advantage of this whole-context chain is that it orders asynchronous tasks: editions are rebuilt before the dashboard is created. If a task is known to be fully independent of other tasks it could just be launched immediately. New Celery tasks: keeper.tasks.editionrebuild.rebuild_edition(): copies a build on S3 onto the edition. keeper.tasks.dashboardbuild.build_dashboard(): triggers LTD Dasher. Replace Edition.rebuild() with Edition.set_pending_rebuild to use the new rebuild_edition task.

Jonathan Sick added a comment - 03/May/18 2:21 PM Released as version 1.9.0.