How we use AI to write QA test scenarios

[Editor’s note: This is the fifth in a series of blog posts about how builders can inject AI into the software development process.]

In this blog series, we’ve shared how we use AI as a tool to improve our software development process — from research and discovery, to design and development. And our QA team has also been experimenting with AI to accelerate our testing process and improve the quality of our products.

This post shows how to use AI to write test scenarios for a website project. We use OpenAI’s GPT-4 LLM through Imagine AI, an internal generative AI platform created by our parent company WPP.

Some context on the project: 

• We were building a website for a sports event.

• The site was responsive to different browsers, devices, and resolutions. 

• We were testing the login process, including the UI, functionality, and the API. 

Using AI to write QA test scenarios

We knew the AI tool probably wouldn’t produce the perfect output initially. We would likely need to prompt, re-prompt, and continue to optimize our prompts to get the results we wanted. 

To start, we identified this key information to convey in our prompt:

• We are QA professionals

• We use agile methodologies

• This is a website project

• The output is test scenarios that should follow the Gherkin format

• The ticket description is our Jira board

This was the first prompt we used:

As a Quality Assurance professional, I would like you to write test scenarios for a ticket. I am currently working with SCRUM on a web project and the standard for writing tests is the Gherkin model. Please create a full test suit for the following ticket, in Gherkin. This suit should cover for any information that the ticket has and needs to validate that the product is working.

This is the ticket:

<Ticket description>

We gave the AI all of the key elements. Here’s the QA test scenario it produced:

Feature: Login Page Rework

Scenario: User accesses the login page

Given the user is on the login page

Then the login page should match the Figma reference

Scenario: User enters valid credentials and clicks on "Login"

Given the user is on the login page

When the user enters valid credentials

And the user clicks on "Login"

Then the user should be redirected to the home page

Scenario: User clicks on "Forget Password" link

Given the user is on the login page

When the user clicks on the "Forget Password" link

Then the form should change to the "Forget Password" layout

And the layout should have an "Email" field

And the layout should have a "Confirm" CTA

And the layout should have a "Cancel" CTA

Scenario: User clicks on "Set Password" link

Given the user is on the login page for the first time

When the user clicks on the "Set Password" link

Then the form should change to the "Set Password" layout

And the layout should have a "Password" field

And the layout should have a "Confirm Password" field

And the layout should have a "Confirm" CTA

And the layout should have a "Cancel" CTA

Scenario: User enters a password in the "Set Password" layout

Given the user is on the "Set Password" layout

When the user enters a password

Then the password input should have a minimum of 8 characters in length

And the password input should have at least 1 number

And the password input should have at least 1 special character

Scenario: User cancels "Forget Password" or "Set Password" flow

Given the user is on the "Forget Password" or "Set Password" layout

When the user clicks on the "Cancel" CTA

Then the form should change back to the login page layout

Ultimately, the result didn’t meet our needs nor follow some typical QA best practices. 

For example:

• There are multiple behaviors being tested on each test. The AI tested access to the page and visible components in the same scenario. If any of those behaviors failed, we would be unable to tell which just by looking at the test log. 

• There was insufficient test coverage – we needed additional scenarios. 

• The scenario had confusing and ambiguous test lines. For example, it lacked examples of possible inputs and credentials.  

In the same session, we added some new rules for the AI: 

• Test only one behavior for each test written, and separate long tests into two scenarios. 

• Fully consider the user experience in the flow of the application. 

• Create one test specifically for visible components on the screen.

The output was more inline with what we needed and expect for  our QA approach at ArcTouch:

Feature: Login Page Rework

Scenario: Validate login with valid credentials

Given the user is on the Login page

When the user enters valid credentials

And the user clicks on "Login" button

Then the user should be redirected to the dashboard page

Scenario: Access the "Forgot Password" flow

Given the user is on the Login page

When the user clicks on the "Forget Password" link

Then the form should change to the "Forget Password" layout

Scenario: Verify "Forgot Password" layout elements

Given the user is on the "Forgot Password" layout

Then the "Email" field should be present

And the "Confirm" CTA should be present

And the "Cancel" CTA should be present

Scenario: Access the "Set Password" flow for first-time users

Given the user accesses the site for the first time

Then the form should change to the "Set Password" state

Scenario: Verify "Set Password" layout elements

Given the user is on the "Set Password" layout

Then the "Password" field should be present

And the "Confirm Password" field should be present

And the "Confirm" CTA should be present

And the "Cancel" CTA should be present

Scenario: Validate password input rules

Given the user is on the "Set Password" layout

When the user enters a password

Then the password should have a minimum of 8 characters in length

And the password should have at least 1 number

And the password should have at least 1 special character

Scenario: Validate custom fields for texts and titles

Given the user is on the Login page

Then all texts and titles should be custom fields

This clarification drastically improved the results. The tests became much more concise and understandable after adding additional rules based on the previous AI output.  

However, we still needed to make a few more adjustments to finish the job. The AI did not provide enough testing coverage of the specified criteria for the page, nor examples of the inputs needed to validate password rules. Ultimately, we learned that how you present your information makes a dramatic impact on what AI delivers. Adding rules, after the initial prompt, and asking the AI for adjustments helps it understand what needs to be added, removed, or edited. Still, it wasn’t perfect – and we had an idea of what we wanted to try next: 

Using an AI role model prompt for QA Analyst

We next tried using a role model prompt — also known as “role prompting” — to see if the AI generated more useful responses. Instead of telling the AI to simply perform an action, you tell it who is doing the action. In this case, we guide the AI to take on the role of a  “QA Analyst.”

The prompt provides the context of who the AI is modeling, how it thinks about things, how it acts, and what it values. We came up with the following prompt:

You are a world-class QA Analyst with 15 years of experience in writing Test Scenarios in Gherkin. You have a technical background and your audience are engineers with more than 10 years of experience creating top-class award-winning software. Like you, they understand complex technical architecture and terminology and are familiar with technical definitions and modern designs. You follow all the best practices when writing your test scenarios, and you are very careful to make scenarios that can be understood by a full development team of different professionals. I will send you pieces of information about a Login page, requirements, and rules, and you will deliver a full suite of test scenarios that validate that every rule is being covered on the tests. Respond OK if you understood all that.

After our AI tool responded positively to the prompt above, we fed it with the information about the website feature. In this case, the original description was written for human understanding — and not with AI in mind. So, we added some narration about it and the rules that should be followed. We told the AI that the output should be a login page with the functional flow and rules described as you would describe them to a coworker.  

The output we received was very thorough — and provided coverage for every QA test scenario we could think of, and more:

Test Scenario 1: Validate the Login Flow

Given a user is on the Login page

When the user enters valid Email and Password

And clicks on the "Login" button

Then the user should be successfully logged in

Test Scenario 2: Validate the Forgot Password Flow

Given a user is on the Login page

When the user clicks on the "Forgot Your Password" button

Then the user should be navigated to the "Forgot Your Password" state

Given a user is on the "Forgot Your Password" state

When the user clicks on the "Cancel" button

Then the user should be returned to the Login page

Test Scenario 3: Validate the Change Password Flow

Given a user is on the Login page

When the user clicks on the "Change your Password" button

Then the user should be navigated to the "Change your Password" state

Given a user is on the "Change your Password" state

When the user clicks on the "Cancel" button

Then the user should be returned to the Login page

Test Scenario 4: Validate the Set Password Flow

Given a user is setting a new password

When the user starts typing a password

Then all conditions should turn red until the typed passwords attend the corresponding rule

Given a user is setting a new password

When the user types a password with less than 8 characters

Then the password should not be accepted

Given a user is setting a new password

When the user types a password without a special character

Then the password should not be accepted

Given a user is setting a new password

When the user types a password without a number

Then the password should not be accepted

Given a user is setting a new password

When the user types a password that meets all conditions

Then the password should be accepted

Test Scenario 5: Validate Error Message on Invalid Login

Given a user is on the Login page

When the user enters an invalid Email or Password

And clicks on the "Login" button

Then an appropriate error message should be displayed

Test Scenario 6: Validate Input Fields on Login Page

Given a user is on the Login page

When the user views the input fields

Then there should be two input fields: Email and Password

Test Scenario 7: Validate Navigation Buttons on Login Page

Given a user is on the Login page

When the user views the buttons

Then there should be buttons to navigate to "Change your Password" state and "Forgot Your Password" state

Test Scenario 8: Validate Password Visibility Toggle

Given a user is on the Login page

When the user types a password

Then a visibility toggle icon should be present to show/hide the password

Test Scenario 9: Validate Email Format

Given a user is on the Login page

When the user enters an email in incorrect format

And clicks on the "Login" button

Then an appropriate error message should be displayed

Test Scenario 10: Validate Reset Functionality in Forgot Password Flow

Given a user is on the "Forgot Your Password" state

When the user enters a valid registered email

And clicks on the "Reset Password" button

Then a password reset link should be sent to the entered email

Test Scenario 11: Validate Old Password Requirement in Change Password Flow

Given a user is on the "Change your Password" state

When the user enters a new password without entering the old password

And clicks on the "Change Password" button

Then an appropriate error message should be displayed

Test Scenario 12: Validate Screen Reader Compatibility

Given a user is on the Login page

When the user navigates through the page using a screen reader

Then all elements including input fields, buttons, and instructions should be properly read out

Test Scenario 13: Validate Keyboard Navigation

Given a user is on the Login page

When the user navigates through the page using keyboard only

Then the user should be able to access all interactive elements including input fields and buttons

Test Scenario 14: Validate High Contrast Mode

Given a user is on the Login page

When the user enables high contrast mode

Then all elements on the page should be clearly visible and distinguishable

Test Scenario 15: Validate Zoom Functionality

Given a user is on the Login page

When the user zooms in the page up to 200%

Then all elements should remain visible and functional, and text should not overlap or cut off

Test Scenario 16: Validate Error Messages Accessibility

Given a user is on the Login page

When an error message is displayed

Then it should be announced by the screen reader immediately

Test Scenario 17: Validate Alt Text for Non-Text Content

Given a user is on the Login page

When the user navigates to any non-text content (like images, icons)

Then appropriate alt text should be provided for screen readers

Test Scenario 18: Validate Empty Field Submission

Given a user is on the Login page

When the user clicks on the "Login" button without entering any details

Then appropriate error messages should be displayed for both Email and Password fields

Test Scenario 19: Validate Password Masking

Given a user is on the Login page

When the user types in the Password field

Then the entered characters should be masked for security

Test Scenario 20: Validate 'Remember Me' Functionality

Given a user is on the Login page

When the user enters valid credentials and checks the 'Remember Me' option before logging in

Then the user's credentials should be remembered in the next session unless explicitly logged out

Test Scenario 21: Validate Session Timeout

Given a user has logged into the application

When the user remains inactive for the specified session timeout duration

Then the user should be automatically logged out and redirected to the Login page

Test Scenario 22: Validate Concurrent Sessions

Given a user has logged into the application from one device

When the same user tries to log in from another device

Then the user should be informed about the active session and asked to confirm before initiating a new session

Test Scenario 23: Validate Response to Multiple Failed Login Attempts

Given a user is on the Login page

When the user enters incorrect credentials multiple times consecutively

Then the account should be temporarily locked to prevent potential unauthorized access

Test Scenario 24: Validate Cross-Browser Compatibility

Given a user accesses the Login page

When the user is using different browsers (e.g., Chrome, Firefox, Safari, Edge)

Then the Login page should function correctly and consistently across all supported browsers

Test Scenario 25: Validate Responsiveness on Different Devices

Given a user accesses the Login page

When the user is using different devices (e.g., desktop, laptop, tablet, smartphone)

Then the Login page should display and function correctly, adapting to various screen sizes and resolutions

Test Scenario 26: Validate URL Redirection After Successful Login

Given a user is on the Login page

When the user enters valid credentials and clicks on the "Login" button

Then the user should be redirected to the appropriate landing page based on their role and permissions

Test Scenario 27: Validate Password Reset Flow

Given a user has received a password reset link via email

When the user clicks on the link and sets a new password following the password rules

Then the user should be able to log in with the new password successfully

Test Scenario 28: Validate Input Field Validation Messages

Given a user is on the Login page

When the user interacts with input fields (e.g., entering invalid data, leaving fields empty)

Then appropriate validation messages should be displayed near the corresponding input field

Test Scenario 29: Validate Language Support

Given a user is on the Login page

When the user selects a different language from the available options

Then the Login page content should be translated into the selected language

We found that the AI provided test scenarios in batches of 6 to 10 each. Once it had reached a limit for each of the outputs, we prompted it to "Make sure that everything is covered" and to "Please provide more tests based on the information given." Each time, a new batch of scenarios was provided until we reached the very specific test cases shown at the end of the list. 

Simply asking the AI to provide more was a way to force it to gather more information from a single prompt, which eventually provided more complete coverage with a variety of edge cases.

Using AI for generating API test scenarios

Next, we wanted to use the AI to help generate test scenarios for the API that connected the front-end interface with the back-end response.  

We had more success by following an approach similar to the role model prompt. However, when we tried to complete this in a single prompt, informing the AI of what triggered each response and the codes that the API returned, the output was barely coherent.

So, we separated the API prompts into two batches: one for integration with the front end and one for back-end testing in Postman.

First, we described the login page, the components present, and the main functionalities. We described it so the AI would know what components would influence possible API responses, such as input fields and CTA buttons. Then, we asked it to give us API tests that would cover the page:

Feature: Login Page API Integration

Scenario: Validate successful login with API response

Given the user is on the Login page

When the user enters valid credentials

And the user clicks on "Login" button

Then the API should return a successful response

And the user should be redirected to the dashboard page

Scenario: Validate unsuccessful login with API response

Given the user is on the Login page

When the user enters invalid credentials

And the user clicks on "Login" button

Then the API should return an error response

And the user should see an error message

Scenario: Validate "Forgot Password" flow with API response

Given the user is on the "Forgot Password" layout

When the user enters a valid email address

And the user clicks on the "Confirm" CTA

Then the API should return a successful response

And the user should see a confirmation message

Scenario: Validate "Set Password" flow with API response

Given the user is on the "Set Password" layout

When the user enters a valid password

And the user enters a matching confirm password

And the user clicks on the "Confirm" CTA

Then the API should return a successful response

And the user should be redirected to the dashboard page

Scenario: Validate error handling for "Set Password" flow with API response

Given the user is on the "Set Password" layout

When the user enters an invalid password

And the user clicks on the "Confirm" CTA

Then the API should return an error response

And the user should see an error message

Then, we described each one of the APIs used and the possible outcomes in the following prompt:

Considering the ticket above, write isolated tests for API, before the integration with the homepage. These tests will be executed in Postman before the integration with the Front End. They need to be written in Gherkin and should guarantee that the API is functioning correctly and each code is returned correctly. Here are the response codes and expected results for each:

SetPassword API: 200 - Success 403 - Failed 401 - Invalid API Bearer Token 412 - Username not found 417 - Invalid Request Token 500 - Processing Error

Authenticate API: 200 - Success 403 - Invalid Username or Password 401 - Invalid API Bearer Token 412 - Invalid BP User 500 - Processing Error

ChangePassword API: 200 - Success 403 - Invalid Username or Password - Update Failed 401 - Invalid API Bearer Token 500 - Processing Error

IntiateForgotPassword API 200 - Success 412 - Username Not Found 401 - Invalid API Bearer Token 500 - Processing Error

After specifying in great detail the expected results in our prompt, the AI tool responded with the following:

Feature: API Isolated Tests in Postman

Scenario: Validate SetPassword API - Success

Given the user has a valid request for SetPassword API

When the user sends the SetPassword request in Postman

Then the API should return a 200 status code

Scenario: Validate SetPassword API - Failed

Given the user has an invalid request for SetPassword API

When the user sends the SetPassword request in Postman

Then the API should return a 403 status code

Scenario: Validate SetPassword API - Invalid API Bearer Token

Given the user has a SetPassword request with an invalid API Bearer Token

When the user sends the SetPassword request in Postman

Then the API should return a 401 status code

Scenario: Validate SetPassword API - Username not found

Given the user has a SetPassword request with a non-existing username

When the user sends the SetPassword request in Postman

Then the API should return a 412 status code

Scenario: Validate SetPassword API - Invalid Request Token

Given the user has a SetPassword request with an invalid Request Token

When the user sends the SetPassword request in Postman

Then the API should return a 417 status code

Scenario: Validate SetPassword API - Processing Error

Given the user has a SetPassword request that triggers a processing error

When the user sends the SetPassword request in Postman

Then the API should return a 500 status code

Scenario: Validate Authenticate API - Success

Given the user has a valid request for Authenticate API

When the user sends the Authenticate request in Postman

Then the API should return a 200 status code

Scenario: Validate Authenticate API - Invalid Username or Password

Given the user has an Authenticate request with invalid credentials

When the user sends the Authenticate request in Postman

Then the API should return a 403 status code

Scenario: Validate Authenticate API - Invalid API Bearer Token

Given the user has an Authenticate request with an invalid API Bearer Token

When the user sends the Authenticate request in Postman

Then the API should return a 401 status code

Scenario: Validate Authenticate API - Invalid BP User

Given the user has an Authenticate request with an invalid BP User

When the user sends the Authenticate request in Postman

Then the API should return a 412 status code

Scenario: Validate Authenticate API - Processing Error

Given the user has an Authenticate request that triggers a processing error

When the user sends the Authenticate request in Postman

Then the API should return a 500 status code

Scenario: Validate ChangePassword API - Success

Given the user has a valid request for ChangePassword API

When the user sends the ChangePassword request in Postman

Then the API should return a 200 status code

Scenario: Validate ChangePassword API - Invalid Username or Password - Update Failed

Given the user has an invalid request for ChangePassword API

When the user sends the ChangePassword request in Postman

Then the API should return a 403 status code

Scenario: Validate ChangePassword API - Invalid API Bearer Token

Given the user has a ChangePassword request with an invalid API Bearer Token

When the user sends the ChangePassword request in Postman

Then the API should return a 401 status code

Scenario: Validate ChangePassword API - Processing Error

Given the user has a ChangePassword request that triggers a processing error

When the user sends the ChangePassword request in Postman

Then the API should return a 500 status code

Scenario: Validate IntiateForgotPassword API - Success

Given the user has a valid request for IntiateForgotPassword API

When the user sends the IntiateForgotPassword request in Postman

Then the API should return a 200 status code

Scenario: Validate IntiateForgotPassword API - Username Not Found

Given the user has an IntiateForgotPassword request with a non-existing username

When the user sends the IntiateForgotPassword request in Postman

Then the API should return a 412 status code

Scenario: Validate IntiateForgotPassword API - Invalid API Bearer Token

Given the user has an IntiateForgotPassword request with an invalid API Bearer Token

When the user sends the IntiateForgotPassword request in Postman

Then the API should return a 401 status code

Scenario: Validate IntiateForgotPassword API - Processing Error

Given the user has an IntiateForgotPassword request that triggers a processing error

When the user sends the IntiateForgotPassword request in Postman

Then the API should return a 500 status code

Even though the AI tool struggled at first to provide a useful output for API-related tests, separating the front-end integration from back-end testing gave us the results we were looking for. 

The process and result weren’t perfect, but we still were better off using the tool than generating the API test scenarios on our own. 

Takeaways from using AI to write test scenarios

Overall, we found that using AI to write test scenarios and augment our QA testing process was a big help. It definitely improved our efficiency. We estimate that writing these from scratch would take about two days. Using AI to augment that process saved us about half that time.

Most importantly, using AI gave us another resource to ensure we were covering the breadth of tests and corner cases we needed. In other words, using AI helped boost the quality of our testing — the most important contribution any QA tool could offer. 

A few other key takeaways included: 

• There is no perfect prompt, and prompting is a skill you need to learn and develop, like any other.

• No matter how good your prompt is, the AI output is never perfect. QA professionals will always need to review, edit, and take ownership of the written test scenarios generated by any AI tool. 

Last but not least, using AI tools promises to help QA professionals save time in some areas — so you can spend more time and attention on other facets of quality in your project.

At an agency like ArcTouch, it means we use our client’s project budget in smarter ways, by spending less time writing scenarios and more time actually testing those scenarios, to create more lovable and higher quality apps and websites.    

Follow along

To read our series about injecting AI into software development, join our mailing list or follow us on LinkedIn and X. You can also contact us to learn more about how we use AI to help create lovable apps, websites, and connected experiences.