Learn Apache Airflow By Example – 4 Part Series

apacheairflow docker python Dec 02, 2019

Introduction

I've been having a blast the last few months learning ApacheAirflow. It's become an indispensable tool in my getting stuff done toolbox.

Follow me on Twitter @jillianerowe, or send me a message with questions, topic suggestions, ice cream recommendations, or general shenanigans.

Learn Apache Airflow By Example - Part 1 Introduction - Get familiar with the various moving parts that make up the awesomeness that is Airflow.
Learn Airflow By Example - Part 2 Install With Docker - Get your dev environment up and running with a simple docker-compose up -d
Learn Airflow By Example - Part 3 Start Building - Build out a simple DAG, get familiar with the web UI, and learn 3 different ways to trigger your DAGs.
Learn Airflow By Example - Part 4 DAG Patterns - Learn 3 different DAG patterns that can be used as building blocks for more complex analyses.

Get the Source Code

Alternately, if you just want the code you can get the project template by visiting the Dabble of DevOps Store - Docker for Python and Data Science FREE Course.

Bring up the Dev Server

This will take a few minutes to initialize, so if you don't see anything to begin with don't worry!

unzip airflow-template.zip
cd airflow-template
docker-compose up -d
docker-compose logs airflow_webserver

Once you see a message in your logs like 'Airflow starting' or something along those lines, visit http://localhost:8089 in your browser and see the magic!

Looking for More?

If you are looking for more information, or a guide that you can save as an internal SOP, please check out my Apache Airflow Database Configuration Guide. I am bundling all my stacks into a completely FREE Resource Library.

To see the Apache Airflow Docker Dev Stack Template and Guide click here. This is a fully functional stack that includes running Apache Airflow with RabbitMQ and PostgreSQL, along with a ready to run configuration file, and a PDF guide that can serve as internal documentation or an SOP.

Bioinformatics Solutions on AWS Newsletter

Get the first 3 chapters of my book, Bioinformatics Solutions on AWS, as well as weekly updates on the world of Bioinformatics and Cloud Computing, completely free, by filling out the form next to this text.

Bioinformatics Solutions on AWS

If you'd like to learn more about AWS and how it relates to the future of Bioinformatics, sign up here.

We won't send spam. Unsubscribe at any time.