Ansible Tips and Tricks: Dealing with Unreliable Connections and Services

Red Hat Ansible Automation is widely known to automate and configure Linux and Windows hosts, as well as network automation for routers, switches, firewalls and load balancers. Plus, there are a variety of modules that deal with the cloud and the API around it such as Microsoft Azure, Amazon Web Services (AWS) and Google Compute Engine.  And there are other modules that interact with Software as a Service (SaaS) tools like Slack or ServiceNow. Although the downtime for these APIs is very minimal, it does happen, and it is even more likely that the connection between your Ansible control host (where you are running Ansible from) and the cloud-centric API could be down.

In this blog post, I will cover some tips and tricks for dealing with unreliable connections to cloud-centric APIs and how I build Ansible Playbooks in a reliable manner. As a technical marketing engineer, I consider my customers the Red Hat field teams, and often Solutions Architects are running playbooks from unreliable hotel wireless, coffee shops and sometimes even airplanes! I have to make sure playbooks have some more robustness built in for these odd situations. It is hair-pulling frustrating to get through a 20 task playbook for it to fail on the 19th task because your wireless went out for a couple seconds. This is especially frustrating if you are at the airport just trying to setup a demo or playground to show something to a client.

The Until Loop

Many people that use Ansible are very familiar with the loop construct. A loop (previously known as with_items) is very simple and powerful and allows you to iterate over a list or dictionary in an easy fashion. However, I find that many people are not aware of the until loop. Let us look at how this can work.

The module ec2_vpc_net allows us to create an AWS Virtual Private Cloud.

- name: Create AWS VPC sean-vpc
  ec2_vpc_net:
    name: "sean-vpc”
    cidr_block: "192.168.1.0/16”
    region: "us-east-1”
  register: create_vpc
  until: create_vpc is not failed
  retries: 5

The name, cidr_block and region are module parameters for the ec2_vpc_net module. However the register, until and retries are task level parameters, meaning that you can use these on any module. This task will attempt to create the VPC five times before it gives up and fails.

Let’s step back a minute to see how this works. Each time we run a task there are some common variables that the task returns to let us know how the task performed:

- name: test local playbook
  hosts: localhost
  gather_facts: false

  tasks:
      - name: dumb easy command
        shell: ls -la
        register: task_variable

      - name: debug the var
        debug:
          var: task_variable

When we run this playbook with ansible-playbook test_output.yml we get some standard output (via the debug module) printed to the terminal window (or browser window when using Ansible Tower).

TASK [debug the var] **************************************************************
ok: [localhost] =>
 task_variable:
      changed: true
      cmd: ls -la
      delta: '0:00:00.011018'
      end: '2018-12-07 09:53:14.595811'
      failed: false
<>

One of the key, value pairs we always get returned from any Ansible task is a failed key. If the task completed successfully the task will return a failed: false. If the task failed, the task will return a failed: true. Looking back at the until loop logic for the AWS VPC task:

register: create_vpc
until: create_vpc is not failed
retries: 5

We are registering the result of the task so we can look at the failed key, value pair. The until value is the conditional we are applying. In this case we keep running the task until the create_vpc does not have failed: true. However we don’t want the task to run this for infinity. The default value for “retries” is 3, however I have increased this to 5. The until loop provides significant robustness to the task.

There is also a delay parameter that can be combined with the until loop.  The delay is how much time to wait between retries.  The default value for the delay is 5 seconds.  Check out the documentation for more details and examples of the until loop and the delay parameter.

Changing What A Failure Means

By default, if Ansible fails the playbook will end on that task, for the respective host it was running on. If I had a playbook running on 10 hosts, and it failed on 1 host on task three out of ten, the 7 subsequent tasks would not run for that host. The other hosts would remain unaffected.

With unreliable connections to an outside API we need to think about what is required and not required to define success for a playbook to finish. For example if you had a task spin up a DNS record on AWS’s Route53 service, the DNS can be nice to have, but isn’t required for you to begin using the instance you created. I can use an until loop to make the route53 tasks more reliable, but it might be OK if the Route53 service is down and unusable. I can use the IP address to get some work done done on my instance until I get a more reliable internet connection to re-run the playbook or the Route53 service becomes available again. There are some tasks that are “nice to have” vs. required.

The way to ignore a failed value is to use the ignore_errors parameter which is a task level parameter outlined in the documentation here. I think there is plenty of content in the docs and various blogs about using the ignore_errors so I think it is sufficient to summarize that ignore_errors will show red and report a failed: true key, value pair, but the playbook will continue on.

What happens if we want to combine the until loop with an ignore_errors?

- name: failure test playbook
  hosts: localhost
  gather_facts: false
  tasks:

    - name: purposely fail
      shell: /bin/false
      register: task_register_var
      until: task_register_var is not failed
      retries: 5
      ignore_errors: yes

    - name: debug task_register_var
      debug:
        msg: "{{ task_register_var }}"

We actually get the best of both worlds with an unreliable task. We get robustness with the until loop, combined with an ignore_errors which allows the playbook to complete regardless of that task completing successfully. I find myself using this combination of ignore_errors and until loops in conjunction with services like Let’s Encrypt where it’s not 100% required for me to have an SSL cert to start using the web app (I can rely on a self-signed cert until I can figure out the problem).

The Ansible Playbook outputs like this:

 
TASK [purposely fail] *************************************************************
FAILED - RETRYING: purposely fail (5 retries left).
FAILED - RETRYING: purposely fail (4 retries left).
FAILED - RETRYING: purposely fail (3 retries left).
FAILED - RETRYING: purposely fail (2 retries left).
FAILED - RETRYING: purposely fail (1 retries left).
fatal: [localhost]: FAILED! => changed=true
  attempts: 5
  cmd: /bin/false
  delta: '0:00:00.007936'
  end: '2018-12-07 13:23:13.277624'
  msg: non-zero return code
  rc: 127
  start: '2018-12-07 13:23:13.269688'
  stderr: '/bin/sh: /bin/false: No such file or directory'
  stderr_lines:
  - '/bin/sh: /bin/false: No such file or directory'
  stdout: ''
  stdout_lines: 
...ignoring

TASK [debug task_register_var] ****************************************************
  msg:
	attempts: 5
	changed: true

In the Ansible workshops I am actually using this combination of error handling for Let’s Encrypt to make it easy for Ansible users to troubleshoot the issue.  If there are any tasks that have a failure that can be skipped, I can add it to a variable and print it at the end of the workshop playbook (the playbook responsible for provisioning instances for students to use).

- name: failure test playbook
  hosts: localhost
  gather_facts: false
  vars:
    summary_information: |
      PROVISIONER SUMMARY
      *******************
  tasks:
    - name: ISSUE CERT
      shell: certbot certonly --standalone -d student1.testworkshop.rhdemo.io --email [email protected] --noninteractive --agree-tos
      register: issue_cert
      until: issue_cert is not failed
      retries: 5
      ignore_errors: yes

    - name: set facts for output
      set_fact:
      summary_information: |
        {{summary_information}}
      	- The Lets Encrypt certbot failed, please check https://letsencrypt.status.io/ to make sure the service is running
      when: issue_cert is failed

    - name: print out summary information
      debug:
        msg: "{{summary_information}}"

This prints out a very easy to understand message to the terminal window:

In conclusion, Ansible is extremely flexible at adding some additional logic when it is necessary. The until loop can add robustness and the ignore_errors allows us to determine success criteria. In combination your Ansible Playbooks can be much more user proof, allowing you to have a proactive vs. a reactive approach to troubleshooting issues. Ansible can’t control if an API or service is down, but we can definitely operate more robustly than home made scripts or DIY API implementations. The playbooks provided are extremely human readable and easy for novice users to understand.