Deploy a Celery Job Queue With Docker – Part 2 Deploy with Docker Swarm on AWS
Feb 13, 2019![](https://kajabi-storefronts-production.kajabi-cdn.com/kajabi-storefronts-production/blogs/18234/images/FGvYDbNxRNOsXP03gRJF_Deploy_a_Celery_Job_Queue_With_Docker_Part_2_Deploy_with_Docker_Swarm_on_AWS_1.jpg)
Overview
In Part 1 of this series we went over the Celery Architecture, how to separate out the components in a docker-compose file, and laid the ground for deployment.
Deploy With AWS CloudFormation
This portion of the blog post assumes you have a ssh key setup. If you don't go to the AWS docs here.
What is CloudFormation?
AWS CloudFormation is an infrastructure design tool that allows users to design their infrastructure by defining file systems, compute requirements, networking, etc. If you have no interest in designing infrastructure, y0u probably don't need to worry. Cloudformation configurations are shareable through templates.
Docker AWS CloudFormation
Getting Started
Docker has come to our rescue here, with a Docker for AWS CloudFormation template. This will, with the click of a few buttons, deploy a docker swarm on AWS for us!!
Click on the page, and scroll down to quick start. Under 'Stable Channel' select 'Deploy Docker Community Edition (CE) for AWS'.
![](https://kajabi-storefronts-production.global.ssl.fastly.net/kajabi-storefronts-production/blogs/18234/images/grbRLvRlSSOy0uimn2fn_Screenshot-2019-02-12-16.23.09.png)
Configuring and Launching Your Swarm Cluster
CloudFormation is some cool stuff. A web UI is generated for you based on the template. In this case the important things to notice are the instance name, number of Manager Nodes, the number of Worker Nodes, and the ssh key you want to use. There are a few other properties, such as daily resource cleanup. Be sure to take note of the SSH Key, as you will need it later.
This is a really small cluster, and only for demo purposes, so I want 1 manager node and 1 worker node.
It's not shown here, but make sure you take note of the name you assign. You will need it later to check out your logs in CloudWatch.
Finish going through the wizard. I basically just press next all the time. Once you've done that you will get to a complete screen, and you want to ensure you submit it. If you don't it will be just like when you write an email and forget to press send.
From this point on we will follow the deployment instructions.
Wait (and wait) for your swarm to start
Once you complete the CloudFormation wizard you will get to the CloudFormation Management Console. If you don't see anything don't panic! Just refresh the page, and you should see a CloudFormation instance come up with the same name you assigned it earlier in the wizard. It will go through several status updates, but after a few minutes you should be able to refresh the page and see a CREATE_COMPLETE Status.
Select your instance, and click on 'Outputs'. You may have to scroll down a bit, but you are looking for that blue url next to the 'Managers' key. Click on it to get to the EC2 Management Console.
Later, you will want to use the DefaultDNSTarget to access the web portion of your application.
The EC2 details for your Swarm manager node should come right up. You are looking for the Public DNS address.
Now, we are going to ssh to our manager instance, and bring up our swarm! My ssh key is named jillianerowe-aws-keypai.pem, and I have it in my ~/.ssh directory (with permissions 400). SSH as user docker at YOUR public IP address.
If you see 'Welcome to Docker' Congratulations! You have your very own compute cluster!
Deploy Your Stack With Docker Swarm
The configuration from docker-compose to docker-swarm is mostly the same, but there are a few key changes you need to note.
These are the same changes that you need to make whether you are deploying to AWS or elsewhere. With the exception of taking a look at the CloudWatch interface in AWS, this process is exactly the same no matter where you are.
Docker Compose to Swarm Mapping
Replace build context with an image
During development we wanted to have an image that updated constantly, but this is not the case for production deployment. That freewheeling madness simply cannot stand. This is production, remember? In fact, it's not even possible to use a build context with docker swarm. You must have an image available from a repository. I replaced the build context with an image I uploaded to Quay.io. If you aren't clear how to go from a development docker image on your machine to an image on quay or dockerhub, check out this post here.
Move your labels from directly under the service to the deploy key
You will notice in the docker-compose.yml file we had all of our labels directly under the service name. Now we want to deploy to docker swarm, and they need to be moved. ( I find this annoying too. )
Change around the networks from bridge to overlay
In the docker-compose configuration I used a bridge network, but for swarm I need an overlay network. I kind of think I could have used an overlay network in the compose configuration as well, but I really hate networking (this seems to be a theme with me), and I need to test it out some more.
Changing localhost to your actual host name
This is kind of aggravating, and I'm really hoping some super smart person has a way around this. Consider this my cry for help! But in the traefik labels you need to ensure that the Host:localhost are changed to Host:My-Actual-Host-Name. This corresponds to the public DNS we saw earlier.
Deployment Configurations
Looking at the docker-compose file, you will see that each service has a deploy configuration. The deploy key in the file is ignored by docker-compose, but if you use docker-swarm you can use this configuration to deploy multiple instances of a single service.