Programmable Microservices Infrastructure with Mantl Release 1

Introduction
The proliferation of a containers, public and private clouds and orchestration applications like Kubernetes and Apache Mesos have helped to ratify the use of microservices in production environments. Pini Reznik wrote a great blog summarizing the history here. However, experience shows that more time is being spent deploying, managing and debugging the infrastructure than on the business’s application or service. Mantl is a product which builds upon these orchestration systems and adds infrastructure provisioning, application deployment and a full suite of monitoring applications; for example the ELK stack. The idea is to provide an end-to-end stack of deployment tools to take bare metal infrastructure and turn it into an environment capable of hosting applications of any size.

Today I am proud to announce that Mantl is now Release 1

Key features

• Mantl is able to provision both infrastructure and software as code. This allows users to deploy their services and applications in a programmable, repeatable manner.
• It can deploy to a range of cloud providers: AWS, Azure, CloudStack, DigitalOcean, Google Cloud Platform, etc. Alternatively, private clouds can be provisioned: OpenStack, VMWare, bare metal or any set of systems running CentOS.
• Multi-data centre configurations are supported using a WAN.
• Many services are available to use, out of the box (more added daily): software defined networks, software defined storage, DNS support, databases, monitoring tools, service discovery, logging, security. For example, the ELK stack.
• The user’s entire project stack, from infrastructure to application configuration, can be placed under version control.
• Automated testing. The Mantl project is automatically tested against a range of cloud providers to ensure quality.
• Since Mantl 0.6, it is possible to perform blue/green testing as a part of the upgrade process.

Background
Mantl is capable of installing a huge range of components to enable your application. But Mantl is built upon two key pieces of software. Infrastructure provisioning is provided by Terraform . Terraform is a dedicated infrastructure provisioning tool. It takes a configuration file and connects to various infrastructure providers to allocate resources. The delegation of infrastructure provisioning to Terraform means users are able to utilize a range of infrastructure providers using a tool that is already established within the community.

The second tool is Ansible . Ansible is another provisioning tool which is being used to provision software. YAML based application definitions are fed into Ansible. Ansible then deploys the specified software on the previously provisioned infrastructure.

Architecture

Mantl has three core types of nodes: control, edge and worker.

• Worker nodes –  launch containers that run your microservices and other workloads. These provide the horsepower that run Mantl’s and the users’ applications.

• Control nodes – responsible for the state of the rest of the cluster. If a machine becomes unavailable, it is the control node’s responsibility to inform the rest of the cluster of the loss. On the control nodes there may also be applications that have a similar role and are required by the rest of the cluster. For example Consul, Zookeeper, Mesos, etc.

• Edge nodes – edge services exposed to the internet, whereas the Worker and Control nodes are on a private network. This is to allow public access to a service and is typically running as a proxy to an internal application.

 

Getting Started with Mantl

Mantl’s ethos is to focus a community around an end to end solution, not around an individual component; designed to solve business and team challenges, one glue, improved by the community, tested everywhere. Mantl provides all the components you need to deploy your microservices platform. We have chosen industry standard components and made them work well together, so you don’t have to write any glue code. You can focus on your application instead. Our vision is to enable you to focus on the application code and business agility, not infrastructure APIs. We have put together an easy to follow getting started video.

Mantl components
A component is the term given to a particular piece of software to be installed by Mantl. Typically they are provided out of the box by Mantl by commenting in and out various applications, but users can easily add their own components. Components are then installed and configured by editing the yml configuration files files. A full list of components are available in the documentation. Components are being added all the time.

Some of the most important components include Mesos, Marathon and Consul. Mesos is the backbone of Mantl. It places a layer of abstraction on resources (cpu, ram, ports, etc.) and is able to allocate work over a cluster of machines. Marathon is a simple orchestration tool to start microservices. A service definition would specify the resources required which is then submitted to Mesos for allocation, via Marathon. Marathon then ensures that the service is healthy and restart the service if it ever fails. Consul is a service discovery mechanism. Mesos tasks are automatically given a consul endpoint based upon their name. So to connect to zookeeper, for example, it is as simple as addressing zookeeper.service.consul.

The most important configuration file is the playbook. This defines what will and will not be installed at a high level. For example, the following code defines the software that is installed on a host that has been defined with the role of “control”. On all of the nodes with the role of control, they will also be given the roles of vault, zookeeper, mesos, marathon, chronos and mantlui. These refer to the components that are going to be installed, which are defined elsewhere in the project.
“`
– hosts: role=control
gather_facts: no
vars:
consul_dns_domain: consul
consul_servers_group: role=control
mesos_leaders_group: role=control
mesos_mode: leader
zookeeper_server_group: role=control
roles:
– vault
– zookeeper
– mesos
– marathon
– chronos
– mantlui
“`
There is a sample here

Take the mantlui role as an example. This will call an Ansible role shown below. The role pull’s a docker image, creates a service file (which is a wrapper for the `docker run` command) and starts the service. The result is a fully functioning web application that links to all the other Mantl components.
“`
—
– name: ensure nginx-mantlui docker image is present
sudo: yes
command: /usr/bin/docker pull {{ mantlui_nginx_image }}:{{ mantlui_nginx_image_tag }}
tags:
– mantlui
– bootstrap

– name: configure nginx-mantlui
sudo: yes
template:
src: “{{ item.src }}”
dest: “{{ item.dest }}”
with_items:
– src: nginx-mantlui.service.j2
dest: /usr/lib/systemd/system/nginx-mantlui.service
– src: nginx-mantlui.env.j2
dest: /etc/default/nginx-mantlui.env
notify:
– reload nginx-mantlui
– restart nginx-mantlui
tags:
– mantlui

– name: enable nginx-mantlui
sudo: yes
service:
name: nginx-mantlui
enabled: yes
state: started
notify:
– restart nginx-mantlui
tags:
– mantlui

– include: distributive.yml
“`
There is a sample here.

A definition like this could be described as a little verbose. But this drawback is negligible when compared to the definition’s readability. Users, on their first day, would be able to contribute to a project because the application definitions are so easy to understand. A user’s entire application could be deployed in this way and any new services would only take a simple, version controlled addition to a role.

Security
Security is one of Mantl’s top priorities. A significant investment has been made to ensure that all securable services are secure, everything exists on private networks and components have an audit trail that ends at open source code. Most of all, the process has been simplified, to the point where all security is enabled out of the box. By default, the security module will:

• Use SSH key-based authentication
• Enable ip table rules
• Admin based HTTP authentication on Mesos/Marathon/Consul/etc.
• Optional private docker registry
• Mesos authorization
• Marathon authorization
• Consul authorization, SSL and ACLs
• Zookeeper ACLs
• Vault for secret storage

All a user has to do is run the `security-setup` script and pass the security settings to Ansible: `ansible-playbook –extra-vars=@security.yml your_playbook.yml`. For more information, look here.

Deployment
Software provisioning is controlled by Ansible. After a user has configured their playbook, they are ready to provision the hosts created by Terraform. The scripts will read your Terraform state file to establish the location and credentials to connect to your infrastructure. The provisioning process is as simple as running the command `ansible-playbook –extra-vars=@security.yml your_playbook.yml`. After a few minutes, all of the software will have been provisioned and applications should be running. You can even include tasks to run tests to ensure that applications are running as expected.

Role development
Initial installations of Mantl won’t have any need to provide custom roles. But as soon as a project begins to grow in size, some role development will be necessary.

Roles consist of several folders inside the roles directory.
“`
$ ls -l roles/mesos
total 8
-rw-r–r– 1 phil-mac staff 3629 2 Feb 10:56 README.rst
drwxr-xr-x 3 phil-mac staff 102 2 Feb 10:56 defaults
drwxr-xr-x 5 phil-mac staff 170 2 Feb 10:41 files
drwxr-xr-x 3 phil-mac staff 102 2 Feb 10:56 handlers
drwxr-xr-x 3 phil-mac staff 102 2 Feb 10:41 meta
drwxr-xr-x 6 phil-mac staff 204 2 Feb 10:56 tasks
drwxr-xr-x 12 phil-mac staff 408 2 Feb 10:56 templates
“`
The first file is the README which informs users how to use the role. The other directories are all used by Ansible. The only required directory is `tasks`. Inside here a `main.yml` file defines the steps required to provision the software. Templates can be used to populate custom information in the mustache format. For example if an application has a version number it is possible to provide a default in the `defaults` folder and use the variable in a template with `{{ version_number }}`. This is very useful when creating marathon json files. The `files` folder holds any static content that is referenced by the main task. Handlers are callbacks. Inside here the main task, or other tasks can call a handler to perform an action. Typical handlers are start stop and restart. Finally, inside the `meta` folder it is possible to define role dependencies. For more information visit here.

Summary
Mantl is a product that unifies the provisioning of both hardware and software. But the ease of use and simplicity beguiles the fact that it provides game-changing programmable services. The entire project is able to go under version control. No external dependencies, no unknown requirements. This provides a single view of truth of the service being deployed. Developers are able to test and deploy with the comfort that their test environment is exactly the same as production. It democratises the use of cloud providers. Developers can choose to switch from AWS to GCE to Vagrant to Private cloud, all with a simple configuration change. New features provided by new services are easy to integrate. Simply add a new role, connect your services and deploy. New developers can easily observe the state of the infrastructure and the application as a whole, reducing integration mistakes.

Put simply, Mantl provides a new way of taking all the benefits of a bottom-up approach to development, using microservices, and applies a simple, scalable, programmable top-down architectural overview. Entire business products can be represented through configuration, but still be understandable for everyone to read.

More information
To find out more about Mantl, please visit the Mantl landing page and the GitHub community pages. Comprehensive documentation is available here. Finally, Twitter users can follow progress @usemantl