Twitcher Ansible Playbook

Documentation Status Travis Build GitHub license Join the chat at https://gitter.im/bird-house/birdhouse

Introduction

This Ansible Playbook can completely provision a remote server to run the full stack of Twitcher.

Getting started

Requirements

Target Server

Supported platforms

At the moment, we are testing with CentOS 7 and Ubuntu 18.04.

SSH access; sudo

Beyond the basic platform, the only requirements are that you have ssh access to the remote server with full sudo rights.

For local testing via virtual machine, any machine that supports VirtualBox/Vagrant should be adequate.

Local setup

Note

You will need Ansible only on your client which you use for running the Ansible scripts. The server can be installed remotely.

On your local machine (the one from which you’re controlling the remote server), you will need a recent copy of Ansible (>=2.7). docs.ansible.com has thorough installation instructions.

Warning

Don’t us your OS package manager to install Ansible; you may get an unusably out-of-date version.

You will also nearly certainly want git, both for cloning the playbook and for version-controlling your own work.

To clone the playbook, use the command:

$ git clone https://github.com/bird-house/ansible-twitcher-playbook.git

Quick setup

In the following we give some installation examples.

MacOS

Use brew to install Ansible:

$ brew install git
$ brew install ansible
# check version
$ ansible --version
ansible 2.7.2
Conda

You can use Conda to install Ansible. Conda is available for Linux, MacOS and Windows.

$ conda install -c conda-forge ansible
# check ansible version
$ ansible --version
ansible 2.7.2

If you don’t have Conda installed, the fastest way is to install Miniconda, preferably the Python 3.x version.

Ansible role requirements

We have a several Ansible role dependencies which you may fulfill via Ansible Galaxy with the command:

$ ansible-galaxy -r requirements.yml -p roles install

This should be executed in your playbook directory. Downloaded requirements will be dropped into the roles directory there.

Remote setup

Ansible requires that the target server have a recent Python 2.x on the server. Newer platforms (like Ubuntu Xenial and later) may not have this activated on pristine new machines.

If you get connection errors from Ansible, check the remote machine to make sure Python 2.7 is available. which python2.7 will let you know. If it’s missing, use your package manager to install it.

On Ubuntu Xenial (16.0.4 LTS), sudo apt-get install -y python will do the trick.

Deploy Twitcher

Note

You can safely try the installation using Vagrant or Docker. See Testing.

Prepare

You need a recent Ansible version (>=2.7) on your local client:

$ ansible --version
ansible 2.7.2

If you don’t have Ansible installed, then follow these instructions: Requirements.

Get the Playbook

Clone this playbook from GitHub:

$ git clone https://github.com/bird-house/ansible-twitcher-playbook.git
$ cd ansible-twitcher-playbook

Customize Configuration

Configure your Twitcher installation. See Configuration:

$ cp etc/sample-twitcher-standalone.yml custom.yml
$ vim custom.yml

Running your playbook locally

Warning

If your system has already a Supervisor or a PostgreSQL installation, please remove them manually.

Warning

Make sure your Ansible directory is not world-readable, otherwise the ansible.cfg file will not be read. See Ansible Documentation.

If not already done (see Requirements), fetch required roles/recipes from ansible-galaxy:

$ ansible-galaxy -p roles -r requirements.yml install

Run your playbook locally:

$ ansible-playbook -c local playbook.yml

Note

You can also use the shortcut to run both:

$ make play

For remote deployment follow the instructions in the next section Live host deployment.

Live host deployment

Creating a host file

You’ll need to tell Ansible how to connect to your host. There are multiple ways to do this. The easiest for our purposes is to create a manifest file.

Create a file with a name like myhost.cfg that follows the pattern:

twitcher.demo ansible_ssh_user=twitcher ansible_ssh_host=192.168.1.100 ansible_ssh_port=5555

You may leave off the ansible_ssh_host setting if the hostname is real. However, when doing early provisioning, it’s often not available. ansible_ssh_port is only required if you want to use a non-standard ssh port. ansible_ssh_user should be the login id on the remote machine. That user must have sudo rights.

Running your playbook

$ ansible-playbook --ask-sudo-pass -i myhost.cfg playbook.yml

The --ask-sudo-pass option instructs Ansible to ask for your user password when it uses sudo for provisioning. It’s not required if the remote user has password-less sudo rights.

Configuration

Edit custom.yml

You need to customize the Ansible deployment configuration to install your Twitcher service. Create a custom.yml configuration and overwrite any of the variables found in group_vars/all. There are some prepared sample configurations etc/sample-*.yml for specific deployments. Copy one of those to get started.

Use sqlite Database

You can use a SQLite database with the following settings:

db_install_postgresql: false
db_install_sqlite: true

Use PostgreSQL Database installed by playbook

By default the playbook will install a PostgreSQL database. You can customize the installation. For example you can configure a database user:

db_user: dbuser
db_password: dbuser

Warning

When you change the database user for an existing database the table owners will not be updated.

Use external PostgreSQL Database

If you want to use an existing database you can skip the database installation by setting the variable:

db_install_postgresql: false

You need to configure then the database connection string to your external database:

twitcher_database: "postgresql+psycopg2://user:password@host:5432/twitcher"

Use HTTPS with Nginx

You can enable HTTPS for the Nginx service by setting the variable:

twitcher_enable_https: true

By default it generates a self-signed certificate automatically.

You can also provide your own certificate by setting the following variables:

ssl_certs_local_privkey_path: '/path/to/example.com.key'
ssl_certs_local_cert_path: '/path/to/example.com.pem'

Read the ssl-certs role documentation for details.

Use HTTPS with client certificate validation

When HTTPS is enabled (see above) then optional client certificate validation for ESGF certificates is also activated.

Edit the following variables to change the behaviour:

ssl_certs_enable_verify_client: true
ssl_certs_verify_client: "optional"
ssl_certs_cacert_url: "https://github.com/ESGF/esgf-dist/raw/master/installer/certs/esgf-ca-bundle.crt"

Testing

Test Ansible with Vagrant

Install Vagrant

You need to install Vagrant. See the following links for details:

In short, you can install Vagrant on macOS with Homebrew (and Homebrew Cask):

$ brew cask install virtualbox
$ brew cask install vagrant

You need Ansible locally installed:

$ conda install -c conda-forge ansible
OR
$ brew install ansible # macOS only

Install Ansible roles:

$ ansible-galaxy install -p roles -r requirements.yml --ignore-errors
Run Vagrant

Use Vagrant config:

$ ln -s etc/sample-vagrant.yml custom.yml

Initial setup:

$ vagrant up

Provision with Ansible again:

$ vagrant provision

Login with SSH:

$ vagrant ssh

Run Ansible manually:

$ ansible-playbook -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory playbook.yml

Remove VMs:

$ vagrant destroy -f
Try Twitcher

Run a Twitcher request:

http://192.168.128.100:8000/
Try other OS

Configure Vagrantfile with another Bento Box:

twitcher.vm.box = "bento/ubuntu-18.04"

Alternative: use Vagrant without provisioning

Use Vagrant without provisioning and just to setup a new VM:

$ vagrant destroy -f  # remove previous VM
$ vagrant up --no-provision  # setup new VM
$ vagrant ssh  # ssh into VM

Run the installation manually now:

vagrant> sudo yum install git
vagrant> git clone https://github.com/bird-house/ansible-twitcher-playbook.git
vagrant> cd ansible-twitcher-playbook
vagrant> ./bootstrap.sh
vagrant> ln -s etc/sample-vagrant.yml custom.yml
vagrant> ansible-galaxy install -r requirements.yml
vagrant> ansible-playbook -c local playbook.yml

Test Ansible in a Docker container

Warning

The Nignx and Supervisor services are not automatically started in Docker. You need to do this manually. This will be fixed in a later release.

Start an Ubuntu Docker container with mounted local source:

$ ./run_docker.sh

Update the configuration:

$ ln -s etc/sample-emu.yml custom.yml

Run the Ansible deployment:

$ ./bootstrap.sh
$ make play

Check if application is started (Supervisor):

$ service supervisord status

Check also nginx … might not start automatically in Docker:

$ service nginx status
$ service nginx start # if not already started

Run a Twitcher request:

http://192.168.128.100:8000/

Check log files:

$ supervisorctl tail -f twitcher

Changes

0.1.0 (2020-01-20)

  • Initial release.

Ansible Hints

Show local variables and facts:

$ ansible -m setup -c local localhost

Skip specific tasks for quick tests, for example skip conda tasks:

$ ansible-playbook -c local --skip-tags=conda playbook.yml