Bioinformatics Solutions on AWS 

This book will give you an idea of the services you need to start from the ground up, from storage and compute infrastructure, to what types of clusters to use for different analysis types. I hope this books gives you the resources you need to design and implement infrastructure on AWS that suits your needs. Of course, I can’t go over every single service offered by AWS. There are way too many of those, and I would never claim to be an expert in each. Instead, this book covers the service options I use most frequently, and lists alternatives that I am aware of.

In this chapter we’ll cover the 2 most common clusters I use on AWS, HPC Clusters built with [AWSParallelCluster] and Elastic Kubernetes Clusters managed with [EKS].

Then, I will once again give the disclaimer that these are not your only cluster solutions. AWS has oodles of solutions all at your AWS console enabled fingertips, and you could plug and play any number of them. These are the solutions I am most fond of and your mileage may vary.

I want to hammer home the point here that no matter the solution, we are spinning up linux servers, and anything you can do with a linux server you can do with an AWS cluster.

Underneath the hood these solutions are very similar. We have nodes that we want to somehow call up programatically so we’re not paying for resources we’re not using. On an HPC cluster this happens when a user submits a job. On a Kubernetes cluster this is usually kicked off by an application, such as when a user logs into the Jupyterhub portal.

We’ve talked a lot about the conceptual idea of a Virtual Lab. Now let’s talk about real projects. As a disclaimer, I am not directly involved with any of these projects, with the exception of one small shameless self promotion. These are simply projects I have used some part of, and I think are fairly widely known and applicable to anyone working in Bioinformatics.

In this chapter we discuss the gnomAD browser, the Human Cell Atlas, and CellProfiler.

Now that we have our clusters it’s time to discuss what we can deploy on them. As always, our aim here is to create an infrastructure that aids in good science in general and great Bioinformatics in particular!

In this chapter we’ll discuss the most common infrastructure components you’ll encounter. Each of these components is used across many AWS services and your understanding of these will be vital in understanding your own infrastructure.

If you are running a High Content Screening Pipeline you probably have a lot of moving pieces. As a non exhaustive list you need to:

• Trigger CellProfiler Analyses, either from a LIMS system, by watching a filesystem, or some other process.

• Keep track of dependencies of CellProfiler Analyses - first run an illumination correction and then your analysis.

• If you have a large dataset and you want to get it analyzed sometime this century you need to split your analysis, run, and then gather the results.

• Once you have results you need to decide on a method of organization. You need to put your data in a database and set up in depth analysis pipelines.

These tasks are much easier to accomplish when you have a system or framework that is built for scientific workflows.

In this we’re going to discuss the BBBC021 dataset and how I would organize and batch the analysis. This book is all about AWS, so we will also discuss how to leverage AWS in order to automatically scale our compute infrastructure while running this analysis.

BBBC021 is a fairly typical HCS analysis. Data comes in. We need to do a correction across the batch for illumination. Then, we’ll process each image individually using a provided Cellprofiler analysis. This isn’t included in the code, but typically once there are results, typically one per image, we would want to do something such as aggregate those results to a database, backup the CSVs, upload to a data lake solution, etc.

Remember how I said no code? Well, I meant it! Instead, what I will do here is to list some of my common haunts for various modules, recipes, and deployment strategies.

Mostly, I am exceedingly opportunistic. I’ll google what I want to do, and see if there is a Terraform or Cloudformation recipe, or a tool built using either of those that I can hammer on. Which ever one most closely matches what I want and that I can most easily understand I’ll use.

The world of data visualization in Bioinformatics gets more and more interesting as time goes on. In this case study we’ll discuss how I would use AWS to develop and deploy a dataset with the [CellxGene].

I want to make a quick side note here about data visualization in general. In this particular example we are going over [CellxGene], but these same principles apply to any data visualization application. [RShiny], for R, and [Dash], for Python, are both excellent platforms to interact with your datasets in real time. This idea of real time data analytics has taken off in the data analytics space, and with that comes the need to scale. The most interesting aspect of these frameworks is that you can deploy a truly hybrid, high availability data visualization platform that can analyze large datasets, on local storage, cloud storage (S3, gcp buckets) in real-time in a browser.