Image: DALL-E
Artificial intelligence (AI) is revolutionizing how we work and play in exciting ways. At first glance, AI tools, such as ChatGPT, seem to provide all the correct answers. But once we delve deeper and implement the suggestions, it often isn’t as effortless as it appears. This is especially true when generating code.
In this blog, we wanted to put ChatGPT to the test and see how it fares with developing Ansible Playbooks and share our results. We’ll also cover the experience and feedback from developers across domains.
We’ll also provide more information on our upcoming automation AI superpower, Project Wisdom.
First, let’s briefly discuss what ChatGPT is and how it works.
What is ChatGPT?
“We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.”
OpenAI ChatGPT release announcement
ChatGPT is a chatbot developed by OpenAI and built on top of their GPT (Generative Pre-trained Transformer) 3.5 large language model.
Large language models (LLM) are trained on massive amounts of data to predict the next word in a sentence. GPT 3.5 enables ChatGPT to write paragraphs and pages of content, interact conversationally, and provide human-like answers.
ChatGPT uses two techniques to improve its responses: supervised and reinforcement learning.
- Supervised learning teaches models by feeding input data and the desired output.
- Reinforcement learning trains models based on trial and error in interactive environments.
More information on ChatGPT can be found on the OpenAI ChatGPT release blog.
Next, let’s put ChatGPT to the test using a typical scenario we see with our customers.
Generating Ansible content with ChatGPT
We’ll ask ChatGPT to write an Ansible Playbook to create three AWS EC2 instances. We chose this example as it’s an everyday use case with many samples available for ChatGPT to derive a recommendation.
Our test environment consists of:
- Ansible Core version 2.14.1.
- Visual Studio Code and the nifty Ansible VSCode extension.
- Ansible Lint version 6.10.2
We kept the question simple and broad – “Write an Ansible Playbook to create 3 AWS ec2 instances”.
ChatGPT – “Create 3 AWS ec2 instances” code output.
The “good”
ChatGPT did a great job understanding what I asked for. The generated Ansible Playbook included my specific request to create three EC2 instances, as indicated in the count: 3 variable. ChatGPT also kept the variable names consistent. For example, the ec2 variable parameters are referenced in the sequential tasks.
It also provided a concise, easy-to-understand explanation of the variables and playbook.
ChatGPT – “Create 3 AWS ec2 instances” playbook explanation.
The “not so good”
Several times, I asked ChatGPT the same question and received a different answer. This could be acceptable when working individually, but consistent coding standards are crucial when working at scale with multiple team members or departments.
Note
Ansible Lint, now fully supported with the launch of Red Hat Ansible Automation Platform 2.3, provides profile support enabling automators to apply your organization’s coding standards in real-time using the Ansible Visual Studio Code extension. Please visit the Creating Custom Rules for Ansible Lint blog for more information.
I’ve configured my test environment with all the necessary AWS prerequisites. We’re ready to test!
❯ ansible-playbook playbooks/ec2_instances.yml
…output omitted…
ERROR! couldn't resolve module/action 'ec2'. This often indicates a misspelling, missing collection, or incorrect module path.
The error appears to be in '/Users/craig/demos/ansible/public_demos/local_dev/chatgpt-blog/playbooks/cloudops.yml': line 7, column 5, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
tasks:
- name: Launch EC2 instances
^ here
…output omitted…
Ansible-playbook output.
On my initial run, I received the following error:
ERROR! couldn’t resolve module/action ‘ec2’. This often indicates a misspelling, missing collection, or incorrect module path.
The ec2 module was deprecated and removed in the upstream community Ansible package version 4.0.0. This module was replaced with the e2_instance module based on boto3. ChatGPT, unfortunately, recommended a deprecated Ansible module removed in 2021.
Note
Starting in Ansible Core 2.10, we moved to an improved architecture with Ansible Content Collections, the recommended method for developing new Ansible content.
Let’s update the first task with the new amazon.aws.ec2_instance module, and use the Ansible Visual Studio Code extension to lint it for best practices.
…output omitted…
vars:
region: us-east-1
instance_type: t2.micro
image:
id: ami-0dba2cb6798deb6d8
key_name: mykey
security_group: mysecuritygroup
count: 3
tasks:
- name: Create EC2 instances
amazon.aws.ec2_instance:
key_name: "{{ key_name }}"
security_group: "{{ security_group }}"
instance_type: "{{ instance_type }}"
image: "{{ image }}"
region: "{{ region }}"
count: "{{ count }}"
wait: true
vpc_subnet_id: subnet-1234abcd
network:
assign_public_ip: true
register: ec2
…output omitted…
Updated “Create EC2 instances” task.
Let’s review the changes to the first task:
- The image variable was changed to a YAML dictionary containing the id value.
- We used the Fully Qualified Collection Name (FQCN) module name – amazon.aws.ec2_instance
- The network module parameter was changed to a YAML dictionary.
- The boolean module parameters, wait and assign_public_ip, were changed from yes to true.
The second time’s the charm! Let’s re-run the playbook with the updated task.
❯ ansible-playbook playbooks/ec2_instances.yml
...output omitted...
TASK [Create EC2 instances] *******************************************
changed: [localhost]
TASK [Wait for instances to be in running state] ***********************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'public_ip'. 'dict object' has no attribute 'public_ip'nnThe error appears to be in '/home/craig/demos/ansible/public_demos/local_dev/chatgpt-blog/playbooks/ec2_instances_fixed.yml': line 29, column 5, but maynbe elsewhere in the file depending on the exact syntax problem.nnThe offending line appears to be:nn register: ec2n - name: Wait for instances to be in running staten ^ heren"}
...output omitted...
Ansible-playbook output.
This time, the first task ran successfully. However, the second task, [Wait for instances to be in running state], produced the following error:
fatal: [localhost]: FAILED! => {“msg”: “The task includes an option with an undefined variable. The error was: ‘dict object’ has no attribute ‘public_ip.’
Why the error? The amazon.aws.ec2_instance module now saves the output to public_ip_address instead of public_ip. Let’s update the variable name in the task and re-run the playbook.
❯ ansible-playbook playbooks/ec2_instances.yml
...output omitted...
TASK [Wait for instances to be in running state]***********************
msg: 'value of state must be one of: absent, drained, present, started, stopped, got: running'
...output omitted...
Ansible-playbook output.
This time it returned a different error. The [Wait for instances to be in running state] task uses the wait_for Ansible module. Unfortunately, ChatGPT’s suggested playbook uses an incorrect value for the state parameter – running.
I’ve updated the state parameter to present and linted the task using the Ansible Visual Studio Code extension. We’re nearly there! Here’s the updated task:
- name: Wait for instances to be in running state
ansible.builtin.wait_for:
state: present
host: "{{ item.public_ip_address }}"
port: 22
with_items: "{{ ec2.instances }}"
Updated “Wait for instances to be in running state” task.
Let’s review the changes:
- We used the module’s Fully Qualified Collection Name (FQCN) – ansible.builtin.wait_for.
- The state parameter was changed from running to present.
- The item.public_ip variable was changed to item.public_ip_address.
Let’s rerun our updated playbook.
❯ ansible-playbook playbooks/ec2_instances.yml
...output omitted...
TASK [Wait for instances to be in running state] **********************
ok: [localhost] => (item={'ami_launch_index': 0, 'image_id': 'ami-02e0bb36c61bb9715', 'instance_id': 'i-0ee1d103e7734a979'
...output omitted...
TASK [Print instance information] *************************************
ok: [localhost] => (item={'ami_launch_index': 2, 'image_id': 'ami-02e0bb36c61bb9715', 'instance_id': 'i-02ce52702ef9d8143'
...output omitted...
Ansible-playbook output.
It worked! We’ve updated the playbook run using the latest, supported version of Ansible and applied the best practices using Ansible Lint.
Here’s the final playbook version.
---
- name: Create EC2 instances
hosts: localhost
gather_facts: false
vars:
region: us-east-1
instance_type: t2.micro
image:
id: ami-0dba2cb6798deb6d8
key_name: mykey
security_group: mysecuritygroup
count: 3
tasks:
- name: Create EC2 instances
amazon.aws.ec2_instance:
key_name: "{{ key_name }}"
security_group: "{{ security_group }}"
instance_type: "{{ instance_type }}"
image: "{{ image }}"
region: "{{ region }}"
count: "{{ count }}"
wait: true
vpc_subnet_id: subnet-1234abcd
network:
assign_public_ip: true
register: ec2
- name: Wait for instances to be in running state
ansible.builtin.wait_for:
state: present
host: "{{ item.public_ip_address }}"
port: 22
with_items: "{{ ec2.instances }}"
- name: Print instance information
ansible.builtin.debug:
var: item
with_items: "{{ ec2.instances }}"
Final Ansible Playbook version.
Note
It would be possible to use the Ansible Automation Platform compatibility execution environment, available to our subscribed customers, and automation content navigator to run automation content with Ansible Engine 2.9 compatibility. However, we strongly recommend creating new Ansible content using the improved format to future-proof your automation.
Summarizing the experience
ChatGPT’s conversational style enabled us to provide our request in simple, easy-to-understand language, and the Ansible Playbook it generated reduced our time to get started. The playbook, however, needed several modifications to get it working with the latest Ansible version.
ChatGPT’s recommendation also didn’t consider multiple factors, such as best practices, future compatibility, and, more importantly, context.
Context. The missing ingredient
Organizational context is the secret ingredient in scalable, enterprise-grade automation content. At this stage, ChatGPT and similar AI tools can’t provide a recommendation specific to your organization’s environment or products.
The usefulness of the generated content relies heavily on your understanding and approach to the problem you’re trying to solve.
Developer community sentiment
“The scary part was just how confidently incorrect it was.”
AI is already helping new and experienced programmers develop code. However, many developers who’ve tested ChatGPT and similar AI tools share our opinion – It won’t do their job for them quite yet.
In fact, Stack Overflow implemented a temporary ban on the text generated using ChatGPT due to the influx of incorrect answers.
“The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce.”
Posts on forums, such as Hacker News, echo the sentiment. AI reduces development time, but AI-generated code needs to be reviewed, and without a clear understanding of what you want to achieve, it generally won’t provide the answer you’re looking for.
This is where we envision our forthcoming exciting AI project, Project Wisdom, moving beyond generic Ansible content generation to becoming a trusted Ansible content advisor.
The future: Project Wisdom
Project Wisdom, developed in close collaboration with IBM Research, is the future we envision to give you Ansible AI superpowers.
Red Hat is the world’s leading provider of enterprise open source solutions, and we believe in collaborating openly with customers and the community. Project Wisdom builds on this commitment and leverages our experience in creating enterprise products the open source way, trusted by more than 90% of Fortune 500 companies.
We envisage Project Wisdom moving beyond generic recommendations provided by multiple AI-based tools available today to an enterprise-grade solution giving our customers the competitive automation edge.
Attribution
We plan for Project Wisdom to recognize Ansible content contributions by sharing information about the source data used to train its models. If recommendations are similar to the training data used, Project Wisdom will reference the potential source, its author, and the content license.
This acknowledgment aims to foster and strengthen the collaborative relationships that make open source the transformative platform for innovation.
Content with context
We intend for Project Wisdom to generate Ansible content with context. Imagine having Ansible Playbooks instantaneously generated based on your unique environment and automation content. We aim to provide the capability by offering customers private models trained to their specific needs.
Unified experience
We plan to deliver this in a unified experience with Project Wisdom seamlessly integrated into our Ansible developer content tooling.
Project Wisdom is designed with Ansible in mind, and we want it to be your trusted Ansible content advisor.
Please stay tuned for more exciting announcements and sign-up to test Project Wisdom.
Closing words from the ChatGPT expert
We thought it best to ask the prominent expert on ChatGPT to write our closing to this blog: ChatGPT itself.
Where to go next
- Come visit us at Red Hat Summit 2023.
- Missed out on AnsibleFest 2022? Check out the Best of AnsibleFest 2022.
- Self-paced lab exercises – We have interactive, in-browser exercises to help you get started with Ansible Automation Platform.
- Try Ansible Automation Platform free for 60 days.
Originally posted on Ansible Blog
Author: Craig Brandt