Automated Ansible AWX Installation

This article currently covers automated installation of AWX on an Ubuntu-18.04 LXD guest via cloud init.

Installation Notes

The AWX install process process is in the AWX Project repository. A good walk through is also available on youtube. These instructions favor CentOS7 and use ansible to create and install microservices into Docker containers directly or via Openshift or Kubernetes container management platforms.

The instructions favor RedHat Enterprise Linux 7 or the various derived flavors like CentOS7 or Oracle Secure linux that can use the EPEL RPM/yum package repositories.

Installation to apt package managed distributions like Ubuntu is supported under the Tower product and works fine, but documentation favours RHEL7 and it probably receives slightly more testing.

‘’‘The nodejs version’‘’ called out in the INSTALL.md doc is behind what is released under Ubutu-18.04, but the newer version seems to work so far. The python-docker-py package is also not released under Ubuntu 18.04 and is more confusing in the INSTALL.md doc. Python-docker-py is an old python interface to docker that has been replaced by the newer python-docker module. Build works just fine if you install one of either python-docker or python-docker-py which can still be found in pip but it breaks if both are installed. The situation is more confusing since docker-compose pulls in python-docker as a dependency.

The AWX Installation Video walks though the process and is 20 minutes well spent if it’s your first time through.

AWX On LXD via Cloud Init

As it turns out, the entire install works fine in an LXD ubuntu-18.04 guest on ubuntu-18.04 host. The install requires a privileged user. After creating the lxd guest, you can switch it back to unpriviliged with security.nesting=true and things run fine. I believe the extra privileges are required for docker container building due to use of mknod in initial setup, though I stopped short of validating that to 100%.

It is noteworthy that the ubuntu lxd image repo for has a history of being ahead of images: linuxcontainers.org repo in terms of cloud-init support though both appear to have at least some cloud-init support at this point.

Along the way, a manual CentOS7 lxd guest install was validated manaually as well and may have the advantage of being closer to the target developed to since Tower, Ansible, and AWX are RedHat opensource projects, but the ubuntu guest has yet to show flaws.

AWX.yml LXD Profile

Note that the user.user-data string is itself a yaml document. When developing this, it’s useful to use a separate file starting at #cloud-config so you can lint the internal data.

You can also use the yaml2json.py script below to visualize your embedded data structure.

---
config:
  linux.kernel_modules: ip_tables
  security.nesting: "true"
  security.privileged: "true"
  user.user-data: |
    #cloud-config
    output:
      all: '| tee -a /var/log/cloud-init-output.log'
    package_update: true
    package_upgrade: true
    packages:
    - docker.io
    - docker-compose
    - build-essential
    - cloud-utils
    - vim
    - ansible
    - nodejs
    - npm
    - git
    runcmd:
    - set -xe
    - mkdir -p /awx/src
    - mkdir /awx/ca-trust
    - cat > '/awx/ca-trust/cacert.pem' << 'EOF'
    - |-
      -----BEGIN CERTIFICATE-----
      YOUR INTERNAL CA PEM HERE
      -----END CERTIFICATE-----
      EOF
    - chmod 444 '/awx/ca-trust/cacert.pem'
    - chmod 555 '/awx/ca-trust'
    - git clone 'https://github.com/ansible/awx.git' '/awx/src/awx'
    - git clone 'https://github.com/ansible/awx-logos.git' '/awx/src/awx-logos'
    - systemctl start docker
    - systemctl enable docker
    - sed -i 's/postgres_data_dir/# postgres_data_dir/' '/awx/src/awx/installer/inventory'
    - cat >> '/awx/src/awx/installer/inventory' << 'EOF'
    - |-
      postgres_data_dir=/awx/db
      awx_official=true
      project_data_dir=/awx/projects
      ca_trust_dir=/awx/ca-trust
      use_docker_compose=false

      EOF
    - |-
      env HOME=/root USER=root ansible-playbook -vi /awx/src/awx/installer/inventory \
        /awx/src/awx/installer/install.yml
    users:
    - groups:
      - adm
      - audio
      - cdrom
      - dialout
      - dip
      - floppy
      - netdev
      - plugdev
      - sudo
      - video
      - docker
      lock_passwd: true
      name: ubuntu
      shell: /bin/bash
      ssh-authorized-keys:
      - ssh-rsa PUT YOUR SSH KEYS HERE user@asdf
      - ssh-rsa PUT YOUR SSH KEYS HERE user@jkl;
      sudo:
      - ALL=(ALL) NOPASSWD:ALL
description: Installs awx in lxd
devices: {}
name: awx
used_by: []

Cloud Init AWX Guest Creation

Note that the image copy and profile load is persistent on the LXD host only required the first time.

user@mendota$ lxc image copy ubuntu:18.04/amd64 local: --copy-aliases
user@mendota$ lxc profile create awx
user@mendota$ lxc profile edit awx <awx.yml
user@mendota$ lxc launch 18.04/amd64 awx01 -p default -p awx -v ;\
   sleep 10 ;\
   lxc exec monona:awx01 tail -100f /var/log/cloud-init-output.log

This should scroll past a bunch of data with a final success message like the following.

PLAY RECAP *********************************************************************
localhost                  : ok=12   changed=5    unreachable=0    failed=0

Cloud-init v. 18.2 finished at Fri, 15 Jun 2018 15:19:45 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net].  Up 355.00 seconds

At this point you should be able to visit the LXD guest via http://awx01 and see an upgrading spinner. The first run calls a playbook to update the install after which you will get a login screen with default credentials of admin and password.

For extra security you can switch the guest back to unprivileged mode.

user@mendota$ lxc exec awx01 -- shutdown -h now
user@mendota$ lxc config set awx security.privileged false
user@mendota$ lxc start awx01

Since these are containers, you can access the nested namespace pids from the host as well.

user@monona ~ 0 $ lxc list local:awx01 --columns=ns4p
+-------+---------+----------------------+------+
| NAME  |  STATE  |         IPV4         | PID  |
+-------+---------+----------------------+------+
| awx01 | RUNNING | 172.17.0.1 (docker0) | 9442 |
|       |         | 10.0.2.208 (eth0)    |      |
+-------+---------+----------------------+------+
user@monona ~ 0 $ sudo pstree -pTu 9442
systemd(9442,1000000)─┬─accounts-daemon(12267)
                      ├─agetty(12760)
                      ├─atd(12276,1000001)
                      ├─cron(12272)
                      ├─dbus-daemon(12385,1000103)
                      ├─dockerd(12258)─┬─docker-containe(12518)─┬─docker-containe(13248)───memcached(13717,1001000)
                      │                │                        ├─docker-containe(13268)───sh(13836)─┬─epmd(15991,1000100)
                      │                │                        │                                    └─launch.sh(14371)───rabbitmq-server(14401,1000100)───beam.smp(+
                      │                │                        ├─docker-containe(13422)───postgres(13909,1000999)─┬─postgres(15413)
                      │                │                        │                                                  ├─postgres(15414)
                      │                │                        │                                                  ├─postgres(15415)
                      │                │                        │                                                  ├─postgres(15416)
                      │                │                        │                                                  ├─postgres(15417)
                      │                │                        │                                                  ├─postgres(19731)
                      │                │                        │                                                  └─postgres(19796)
                      │                │                        ├─docker-containe(14801)───tini(15212)───bash(15495)───supervisord(18009)─┬─daphne(18356)
                      │                │                        │                                                                         ├─nginx(18357)───nginx(183+
                      │                │                        │                                                                         ├─python(18355)
                      │                │                        │                                                                         └─uwsgi(18359)─┬─uwsgi(183+
                      │                │                        │                                                                                        ├─uwsgi(183+
                      │                │                        │                                                                                        ├─uwsgi(183+
                      │                │                        │                                                                                        ├─uwsgi(183+
                      │                │                        │                                                                                        └─uwsgi(184+
                      │                │                        └─docker-containe(15540)───tini(15695)───bash(15906)───supervisord(19625)─┬─celery(19635)─┬─celery(1+
                      │                │                                                                                                  │               ├─celery(1+
                      │                │                                                                                                  │               ├─celery(1+
                      │                │                                                                                                  │               └─celery(1+
                      │                │                                                                                                  ├─python(19634)
                      │                │                                                                                                  ├─python(19636)
                      │                │                                                                                                  └─python(19637)─┬─python(1+
                      │                │                                                                                                                  ├─python(1+
                      │                │                                                                                                                  ├─python(1+
                      │                │                                                                                                                  └─python(1+
                      │                └─docker-proxy(14675)
                      ├─networkd-dispat(12280)
                      ├─polkitd(12643)
                      ├─rsyslogd(12545,1000102)
                      ├─snapd(12259)
                      ├─sshd(12637)
                      ├─systemd-journal(9981)
                      ├─systemd-logind(12543)
                      ├─systemd-network(11145,1000100)
                      ├─systemd-resolve(11451,1000101)
                      └─systemd-udevd(10427)

yaml2json Utility

The below script can be used to parse through and view interpreted data structures produces by yaml.

yaml2json.py

#!/usr/bin/python
import sys, json, yaml

try:
    if sys.argv[1] == "-":
            y=yaml.load(sys.stdin.read())
    else:
        with open(sys.argv[1]) as i:
            y=yaml.load(i.read())
except:
    print "Usage: \n\t%s [file.yml|-] {key} [key} ...\n"
    print "Dumps yaml content as beuatified json\n"
    print "if argv[1] is -, reads yaml from stdin\n"
    print "if optional key is given, key is extracted and printed to stdout\n"
else:
    i=2
    while len(sys.argv) > i:
        y=y.get(sys.argv[i])
        i += 1
    if isinstance(y,(str,unicode)) is False and isinstance(y,(list,dict)) is True:
        print json.dumps(y,indent=2)
    else:
        print y

Example Usage

user@monona:/tmp$ yaml2json.py
Usage:
        %s [file.yml|-] {key} [key} ...

Dumps yaml content as beuatified json

if argv[1] is -, reads yaml from stdin

if optional key is given, key is extracted and printed to stdout

user@monona:/tmp$ yaml2json.py /tmp/awx.yaml config user.user-data | yaml2json.py -
{
  "users": [
    {
      "ssh-authorized-keys": [
        "ssh-rsa ... me@booger",
        "ssh-rsa ... me@mendota"
      ],
      "shell": "/bin/bash",
      "name": "ubuntu",
      "groups": [
        "adm",
        "audio",
        "cdrom",
        "dialout",
        "dip",
        "floppy",
        "netdev",
        "plugdev",
        "sudo",
        "video",
        "docker"
      ],
      "sudo": [
        "ALL=(ALL) NOPASSWD:ALL"
      ],
      "lock_passwd": true
    }
  ],
  "output": {
    "all": "| tee -a /var/log/cloud-init-output.log"
  },
  "package_update": true,
  "packages": [
    "docker.io",
    "build-essential",
    "cloud-utils",
    "vim",
    "ansible",
    "python-pip",
    "nodejs",
    "npm",
    "git"
  ],
  "package_upgrade": true,
  "runcmd": [
    "set -xe",
    "pip install docker-py",
    "env",
    "mkdir -p /awx/src",
    "mkdir /awx/ca-trust",
    "cat > '/awx/ca-trust/cacert.pem' << 'EOF'",
    "-----BEGIN CERTIFICATE-----\nMIID9TCCAl2gAwIBAgIMWw23QDQvMTzGywMA0GCSqGSIb3DQEBCwUAMBYxFDAS\nBgNVBAMTC2RldmVuZG9yLmlvMB4XDTE4MDUyOTIwMjUzNloXDTIxMDIyMTIwMjUz\nNlowFjEUMBIGA1UEAxMLZGV2ZW5kb3IuaW8wggGiMA0GCSqGSIb3DQEBAQUAA4IB\njwAwggGKAoIBgQDlcLd4os+jXIu8w6mRDI4Tv1XHiFBfAgGCHXiCWgrZZ2NK9hVR\nYsqQ6yh2phJN9TQRU5xCVjiojoivVTfOKbIbYcWsBSam3PSoMYgfejvREgiXT2PP\nW9WyteTSRgDiBYua9PHYmYXMniqhvJ4rP2m9MATuso0yuDs+EXAo0U6wCSLoLnsA\ndpnBiL71xF5kg7/V5sDhgaGnQD21r38oRqJ4UV2BfvdavLNGOtcaCfS1NZpngEHv\nN3FoWkMoA4Vha01K20PY79uWLHlDR3bHAKOs8TKCcH+RvIySM8Vi6/nVSbE6vaz+\nfvNhtNZEVaBRi/Wjt6MUD4JYY8JLi3UiQv1EPNFZi5jaVmO168k+7Au0s93jFZ3n\nr5Je7L7Bb6jQntHECcicTpBpVI7+M/MdlKq88VyUBNWMpVXp2Ieq81Wg2h6MlhBT\nTeH9veHY/B/2apzl08hhqbxb+Or/4LeHEL69fA69AuLfGkGKHLCsUh/71RXjHdfc\n0OE2OEgFEfGz0cMCAwEAAaNDMEEwDwYDVR0TAQH/BAUwAwEB/zAPBgNVHQ8BAf8E\nBQMDBwQAMB0GA1UdDgQWBBTK45tHZnjDCvEipkkIHnnZled98zANBgkqhkiG9w0B\nAQsFAAOCAYEA4GtL+LZ79OjxY0KW05e0sMmdDjMQHVhdfyRe0R6d3A1QeY1gsD6E\nAQ7zOvllE8dcvTeZRKuJ7nK4PECNlBmlD8P/WBK95gHoQh8Ljbr+tWeFAaGTFAAK\nqcldcNM0GHB2TUYPZzpJ/wIv2ry6SUOswffkqqBoWbs//Ho8lPV3YGOAtBcTvJYu\nxmzPioxGPA/9vlC8pcown0KJub8oCVlFq31GbINSe25/ZiLcAp1S/msblzskAbgC\n78FruivGdjN8aWMKY2xzZrrPBYy82hoy4BfHIa8ps61/EGRt4Da73yS8dDDqNz6n\nTEV4woVUxYWUnYmIv0Ghh0IWZAWpAZeuGy6TiNbqPisadscXah+31QV5HbLaits+\nJF2GfiGVkiDJ5qKOH1QYnIbphj89Fr0I4dMa18SpQT1LRk7FXPXxhRJ7vKrLpLua\n9dYm0J/WrpilTBt7o5De1EoElt2oS1nkziFRdQMmcXwVg/Zxk73fRlafudjJoC72\nbJDHwXBjx0KC\n-----END CERTIFICATE-----\nEOF",
    "chmod 444 '/awx/ca-trust/cacert.pem'",
    "chmod 555 '/awx/ca-trust'",
    "git clone 'https://github.com/ansible/awx.git' '/awx/src/awx'",
    "git clone 'https://github.com/ansible/awx-logos.git' '/awx/src/awx-logos'",
    "systemctl start docker",
    "systemctl enable docker",
    "sed -i 's/postgres_data_dir/# postgres_data_dir/' '/awx/src/awx/installer/inventory'",
    "cat >> '/awx/src/awx/installer/inventory' << 'EOF'",
    "postgres_data_dir=/awx/db\nawx_official=true\nproject_data_dir=/awx/projects\nca_trust_dir=/awx/ca-trust\n\nEOF",
    "env HOME=/root USER=root ansible-playbook -vvi /awx/src/awx/installer/inventory /awx/src/awx/installer/install.yml"
  ]
}

Definiitions

Ansible (company)
Ansible the company started the opensource ‘’ansible’’ software and the paide Ansible Tower product attached to it. Ansible has since been aquired by redhat .
AWX Project
Redhat pushed the Ansible Tower source code in the commons under the AWX Project while retaining Tower Product as a supported and curated version of AWX.
Ansible Project
Is the opensource project behind Ansible agentless devops automation software based on python and various remote connectivity options.
Ansible Tower
The productized version of Ansible AWX sold by RedHat. This is a management product build on top of Ansible Engine that layers on additional functionality for ansible including a rest api and web ui. Ansible Tower is reportedly just a curated and supported version of code developed under the AWX Project
Ansible Engine
Refers to a productized and supported version of Ansible provided by redhat and developed under the Ansible Project.
Ansible AWX
The free version of Tower as found in the AWX Project. Reference documentation for usage is found under Ansible Tower with a few exceptions such as AWX install process found in the github repo for the AWX Project
Ansible
Agentless devops automation software based on python and many remote connectivity options with ssh being the protocol supported, though windows remote access, snmp, web services, cloud-ap, and several other options have matured and extend ansible well beyond managing ssh enabled devices.
Docker
Lightweight container platform designed for use in deployment and repeatibility. Original versions of docker share a common ancestor in LXC
LXD
Next generation LXC that included a daemon and rest interface. Lightweight containers like Docker with fuller support for something closer to a virtual machines without the hypervisor overhead. The opensource LXD project was founded and is currently lead by Canonical Ltd.
Kubernetes
A an opensource container management platform developed by google with versions available under both redhat and ubuntu that appear to bundle or preconfigure some addons to round out a cloud platform.
Openshift
A RedHat branded project based on kubernetes.
Cloud-Init
Cloud-init is the defacto multi-distribution package that handles early initialization of a cloud instance.

Todo

Setup a trial instance of Ansible Tower to compare.

Todo

Load up awx with some useful stuff.

Todo

Switching docker_compose=true works fine. It’s a little unclear to me at this time what the difference amounts to.