For a long time I’ve maintained a memory aid in the form of a list of useful commands which can be used on the command line for Linux, macOS (OS X), BSD, Solaris, etc., so I thought I’d list them in a sticky blog post in case they come in useful for others. Most of these will run on any Unix-type operating system, though I’ve usually indicated where a command is OS-specific. These can be run manually for admin purposes and also scripted for automation purposes.
My intention here is to show how to set up a simple CI/CD multibranch pipeline for teams looking to explore the continuous integration and continuous delivery end of DevOps. This pipeline provides a starting point which can be changed and expanded depending on particular team requirements.
This article assumes your team is already familiar with git and GitHub, and that you have Jenkins installed and ready to use in a location accessible from GitHub. Installing Jenkins is quite straightforward so I won’t go into that here, as there are plenty of other guides available for that. I also won’t spend time explaining what CI/CD is, because again there’s plenty of info about that out there, and if you’re looking for implementation guides then you probably already know what CI/CD is anyway and just want to get started.
This pipeline uses three branches in the git repository: dev, test and main. The dev branch is for development builds. Upon creation of a pull request and successful merge from dev into the test branch, the test branch will be used to run a simple automated test. Again, after a successful pull request/merge from test to main, the main branch is used for delivery to a staging environment for QA testing. This is quite basic and can be changed and expanded according to team needs, e.g. feature branches for specific areas of code, additional test environments, the addition of deployment to a production environment, etc.
The pull requests and merges are done manually so that code can be reviewed and checked for issues before merging. Apart from that, the rest of the builds, tests, and deliveries/deployments are automated.Continue reading “Creating a CI/CD pipeline with GitHub and Multibranch Pipelines in Jenkins”
I didn’t expect to find myself needing to learn PowerShell for automation purposes, but I must admit I really like it. It seems sort of like an amalgam of Bash, Perl and Python. It’s an unexpectedly impressive creation from Microsoft. I’ve been using PowerShell on macOS but it can also be used easily on Linux, and Windows of course.
I created three simple PowerShell scripts for automated provisioning of Linux EC2 instances within AWS. Running these will provision an Amazon Linux 2 EC2 instance with SSH key pair and Security Group, with a webapp deployed thereon, plus an associated DNS record in Route 53.
You can find these scripts and related config here on my GitHub.Continue reading “AWS Provisioning and Deployment with Linux EC2 instances using PowerShell”
I wrote a suite of Ansible playbooks to provision an ECS (Elastic Container Service) cluster on AWS, running a webapp deployed on Docker containers in the cluster and load balanced from an ALB (Application Load Balancer), with the Docker image for the app pulled from an ECR (Elastic Container Registry) repository.
This is a follow-up to my project/article “How to use Ansible to provision an EC2 instance with an app running in a Docker container” which explains how to get a containerised Docker app running on an regular EC2 instance, using Docker Hub as the image repo. That could work well as a simple Staging environment, but for Production it’s desirable to easily cluster and scale the containers with a load balancer, so I came up with this solution for provisioning/deploying on ECS which is well-suited for this kind of flexibility. (To quote AWS: “Amazon ECS is a fully managed container orchestration service that makes it easy for you to deploy, manage, and scale containerized applications”.) This solution also uses Amazon’s own ECR for Docker images, rather than Docker Hub.Continue reading “How to provision an ECS cluster and deploy a webapp on it with load-balanced Docker containers, using Ansible”
I created this suite of Ansible playbooks to provision a basic AWS (Amazon Web Services) infrastructure on EC2 with a Staging instance, and to deploy a webapp on the Staging instance which runs in a Docker container, pulled from Docker Hub.
Firstly a Docker image is built locally and pushed to a private Docker Hub repository, then the EC2 SSH key and Security Groups are created, then a Staging instance is provisioned. Next, the Docker image is pulled on the Staging instance, then a Docker container is started from the image, with nginx set up on the Staging instance to proxy web requests to the container. Finally, a DNS entry is added for the Staging instance in Route 53.
This is a simple Ansible framework to serve as a basis for building Docker images for your webapp and deploying them as containers on Amazon EC2. It can be expanded in multiple ways, the most obvious being to add an auto-scaled Production environment with Docker containers and a load balancer. (For Ansible playbooks suitable for provisioning an auto-scaled Production environment, check out my previous article and associated files “How to use Ansible for automated AWS provisioning”.) More complex apps could be split across multiple Docker containers for handling front-end and back-end components, so this could also be added as needed.Continue reading “How to use Ansible to provision an EC2 instance with an app running in a Docker container”
I was brought in by a startup to set up their core infrastructure in a way that functioned as needed and could be automated for safe and efficient provisioning and deployment. The key requirement was making RabbitMQ work only with secure certificate-based connections – the AMQPS protocol, rather than AMQP – for security and compliance purposes. This needed to be done within a Kubernetes cluster for storage and shared states via StatefulSets, ease of scaling and deployment, and general flexibility. It was also necessary to set this up on GCP (Google Cloud Platform) as that was already in use by the startup and they didn’t want to consider alternative cloud providers at this stage, so GKE (Google Kubernetes Engine) needed to be used for the Kubernetes cluster.
Getting certificates for use with RabbitMQ within Kubernetes required the setup of cert-manager for certificate management, which in turn needed ingress-nginx to allow incoming connections for Let’s Encrypt verification so that certificates could be issued.
I successfully solved the problems and fulfilled the requirements. It’s still a “work in progress” to some extent. Some of the config is a little “rough and ready” and could be improved with more modularisation and better use of variables and secrets. Also, the initial cluster provisioning is fully automated with Terraform, and the rest is only semi automated currently. So there is room for further improvement.
All the code and documentation is available on my GitHub repository here. Below I will explain the whole process from start to finish.Continue reading “How to automate provisioning and deployment of RabbitMQ with cert-manager on a Kubernetes cluster in GKE within GCP”
I’ve recently produced a series of articles aimed at startups, entrepreneurial solo developers, etc. wanting to take their first steps into Amazon Web Services (AWS) setups for app deployment:
- Choosing and setting up a load balancer in AWS
- Migrating a MySQL database to AWS (with specific focus on RDS)
- Migrating an application to AWS with auto-scaled EC2 instances
I then wanted to move on from discussing manual setup via the GUI interface of the AWS web console, to DevOps-style command-line programmatic setup for automated provisioning of an AWS infrastructure for app deployment, i.e. infrastructure as code (IaC). I have therefore created a suite of Ansible playbooks to provision an entire AWS infrastructure with a Staging instance and an auto-scaled load-balanced Production environment, and to deploy a webapp thereon. The resulting set of Ansible AWS provisioning playbooks and associated files can be found in my GitHub repository, so go ahead and grab it from there if you want to try them out. Keep reading for information on how to set up and use the playbooks (and you can also refer to the README in the repo folder, which contains much of the same information).
With these playbooks, firstly the EC2 SSH key and Security Groups are created, then a Staging instance is provisioned, then the webapp is deployed on Staging from GitHub, then an image is taken from which to provision the Production environment. The Production environment is set up with auto-scaled EC2 instances running behind a load balancer. Finally, DNS entries are added for the Production and Staging environments.Continue reading “How to use Ansible for automated AWS provisioning”
For entrepreneurs, startups, and established companies trying out new projects, there hopefully comes a time when interest in the app increases such that incoming traffic levels start to rise significantly. This will likely necessitate various improvements to the infrastructure running the application so that it’s more robust, reliable and scalable.
The application may previously have been running on a cheap hosted server on a service like DigitalOcean, Linode or OVHcloud, or possibly even a single EC2 (Elastic Compute Cloud) instance on AWS (Amazon Web Services), and the desired solution would now be to move the application to a dynamically auto-scaled EC2 environment so that it can handle the increasing traffic without resource problems and site downtime.
In order to achieve this, it will also be necessary to set up the database on AWS, and the most realistic solution for this is to use Amazon’s RDS (Relational Database Service). I’ve recently covered this process in my article Migrating a MySQL database to AWS (with specific focus on RDS). You’ll also need to set up a load balancer, most likely an ELB (Elastic Load Balancer), to balance the incoming traffic across the auto-scaled EC2 application instances. I’ve recently covered this topic also, in my article Choosing and setting up a load balancer in AWS. So have a read through both of those articles to begin with, and below I’ll cover the rest of the process, i.e. auto-scaling the application instances on EC2.Continue reading “How to migrate an application to AWS with auto-scaled EC2 instances”
My article Security hardening on CentOS 7, Red Hat Enterprise Linux 7 & Amazon Linux was seen by Red Hat guru Karoly Vegh, who helpfully suggested it would be a good idea to automate the process with Ansible.
So here in my GitHub repository are some Ansible playbooks and related support files that I’ve created, which can be used for hardening servers running RHEL 7, CentOS 7 and related Linux distributions such as Amazon Linux 2. With some tweaking these playbooks could be easily modified for use with Ubuntu, Debian and other types of distro.
To use these playbooks, check out the repo and you can run the playbooks in the Ansible_RHEL_CentOS_hardening folder with
ansible-playbook in the usual way. They assume the target servers are defined in your Ansible inventory as centosservers, but of course you can modify that if needed.
If you’ve been running a web application on just one or two servers which contain your web server, application framework and database, there will likely come a time when you need to scale to cope with higher load from more incoming traffic. Whilst the web/application side of things can usually be scaled horizontally across multiple servers without too much trouble, a MySQL database is not so straightforward.
You may be using one or two EC2 instances, or your infrastructure may be hosted somewhere other than AWS. We’ll assume here that you’re migrating the whole app to AWS (if it’s not there already) and that you need to put the database somewhere within AWS as part of the scaling-up process. There are basically two options: install MySQL on one or more EC2 instances and administer it yourself; or use Amazon’s RDS (Relational Database Service) as a simpler way of hosting and managing your database.Continue reading “How to migrate a MySQL database to AWS (with specific focus on RDS)”
When you want to run your web application on more than one EC2 instance for scaling and redundancy purposes, you will probably require some form of load balancer to distribute incoming requests evenly across the instances. There are various possible solutions for this.
One option is to launch another EC2 instance and install a load balancer on it yourself. There are quite a few open source load balancing options, though I would tend to recommend HAProxy as it’s fast, efficient, secure, and very flexible. This option involves setting up your Linux instance and installing the software you need yourself, then configuring your chosen load balancer and installing your SSL certificates, etc. Additionally you would need to estimate the necessary instance size to run the load balancing software without getting overloaded and slowing the site down (bearing in mind that SSL termination can be particularly CPU-intensive), then monitoring it accordingly.
Unless there’s a particular reason to take the approach of installing a load balancer on an EC2 instance, a simpler and more effective option – especially for companies taking their first steps into scaling multiple instances for their application – is likely to be Amazon’s own ELB (Elastic Load Balancer). This doesn’t require an EC2 instance with Linux setup, software installation and configuration, etc. It provides a simple interface with easy SSL termination and it will scale itself automatically as needed, so there is little required in the way of planning and monitoring.Continue reading “How to choose and set up a load balancer in AWS”