As one of our customers pointed out, “job events are not showing in Tower UI”, causing significant performance issues for users trying to view job status updates. To make Red Hat Ansible Tower more approachable in viewing Real-Time job status updates, we’ve applied the following performance improvements. 

 

Performance Improvements

Between the 3.6 and 3.7 releases, there have been significant performance advancements to improve event processing, job running performance and the user interface. This work was done in conjunction with our customers and the Red Hat Scale and Performance team. These include:

• Added notable performance improvements to event processing to drastically speed up stdout ingestion speed.
• Updated Ansible Tower to no longer rely on RabbitMQ for clustering and event distribution. Redis is added as a new dependency for event handling.
• Improved performance in the User Interface for various job views when many simultaneous users are logged into Ansible Tower.
• Improved job run performance and the write speed of stdout for running playbooks and parallel jobs through optimization of the job dependency/scheduling algorithm.
• Fixed event processing for inventories with very large numbers of hosts to prevent Ansible Tower slow down.
• Improved running jobs to no longer block associated inventory updates.

Setup & Performance Tests Covered

To better understand the delay between job run time and job results processing time, as well as running performance benchmarks, we created a HA Tower cluster setup, which is comparable to the average production setup.

Setup

	Node-A	Node-B
Tower Version	Red Hat Ansible Tower v3.6.2	Red Hat Ansible Tower v3.7.1
CPU	X5650 @ 2.67GHz 24 Core	X5650 @ 2.67GHz 24 Core
RAM	98GB RAM	98GB RAM

* VM images are on separate dedicated disks

For proper benchmarking, we needed to have a large number of job events sufficient enough to compare the performance improvements that were made between the versions. To reach that goal, we generated 100k job events by creating an inventory file with 100 fake hosts (with “ansible_connection: local”) and a job that generated 100 events/hosts. Launching 10 jobs concurrently over a given inventory produced 100k events, which was sufficient for our needs.

 

Performance Results

Job duration

The duration of Ansible Tower completing 10 concurrent jobs remains the same on v3.6.2 and v3.7.1,which shows no sudden decrease of performance and thus is sufficient.

Job events processing duration

The duration it takes Ansible Tower to process all of the job results has been improved significantly: a 82.56% speedup of events processing on Ansible Tower v3.7.1 compared to v3.6.2. This is a major improvement and will lead to a drastically improved user experience.

Job events lag

The additional time required to process all the job results after the job finishes has been reduced by 95.77% in Ansible Tower v3.7.1 compared to v3.6.2. Again, a major improvement of performance is clearly visible here, bringing forward a smoother user experience around job events.

 

Takeaways & Where to go Next 

With performance improvements that are made available starting in Ansible Tower v3.7.1, now Ansible Tower can process all the job results in less additional time to the job run time. This helps users to quickly view the job status updates in the Ansible Tower dashboard in a more Real-Time manner, leading to a much smoother and overall more pleasing user experience.

If you’re interested in detailed information on Ansible Tower, then the Ansible Tower Documentation is a must-read. To download and install the latest version, Please visit the Ansible Tower Installation Guide, and to view the release notes of recent Ansible Tower releases, Please visit Release notes 3.7.x and Release notes 3.6.x. If you are interested in more details about the Red Hat Ansible Automation Platform, be sure to check out our e-books.