How we use AI to write QA test scenarios
Our QA team members cut their time spent writing scenarios in half using AI
12 min. read - August 1, 2024
[Editor’s note: This is the fifth in a series of blog posts about how builders can inject AI into the software development process.]
In this blog series, we’ve shared how we use AI as a tool to improve our software development process — from research and discovery, to design and development. And our QA team has also been experimenting with AI to accelerate our testing process and improve the quality of our products.
This post shows how to use AI to write test scenarios for a website project. We use OpenAI’s GPT-4 LLM through Imagine AI, an internal generative AI platform created by our parent company WPP.
Some context on the project:
We were building a website for a sports event.
The site was responsive to different browsers, devices, and resolutions.
We were testing the login process, including the UI, functionality, and the API.
Using AI to write QA test scenarios
We knew the AI tool probably wouldn’t produce the perfect output initially. We would likely need to prompt, re-prompt, and continue to optimize our prompts to get the results we wanted.
To start, we identified this key information to convey in our prompt:
We are QA professionals
We use agile methodologies
This is a website project
The output is test scenarios that should follow the Gherkin format
The ticket description is our Jira board
This was the first prompt we used:
As a Quality Assurance professional, I would like you to write test scenarios for a ticket. I am currently working with SCRUM on a web project and the standard for writing tests is the Gherkin model. Please create a full test suit for the following ticket, in Gherkin. This suit should cover for any information that the ticket has and needs to validate that the product is working.
This is the ticket:
<Ticket description>
We gave the AI all of the key elements. Here’s the QA test scenario it produced:
Feature: Login Page Rework
Scenario: User accesses the login pageGiven the user is on the login pageThen the login page should match the Figma reference
Scenario: User enters valid credentials and clicks on "Login"Given the user is on the login pageWhen the user enters valid credentialsAnd the user clicks on "Login"Then the user should be redirected to the home page
Scenario: User clicks on "Forget Password" linkGiven the user is on the login pageWhen the user clicks on the "Forget Password" linkThen the form should change to the "Forget Password" layoutAnd the layout should have an "Email" fieldAnd the layout should have a "Confirm" CTAAnd the layout should have a "Cancel" CTA
Scenario: User clicks on "Set Password" linkGiven the user is on the login page for the first timeWhen the user clicks on the "Set Password" linkThen the form should change to the "Set Password" layoutAnd the layout should have a "Password" fieldAnd the layout should have a "Confirm Password" fieldAnd the layout should have a "Confirm" CTAAnd the layout should have a "Cancel" CTA
Scenario: User enters a password in the "Set Password" layoutGiven the user is on the "Set Password" layoutWhen the user enters a passwordThen the password input should have a minimum of 8 characters in lengthAnd the password input should have at least 1 numberAnd the password input should have at least 1 special character
Scenario: User cancels "Forget Password" or "Set Password" flowGiven the user is on the "Forget Password" or "Set Password" layoutWhen the user clicks on the "Cancel" CTAThen the form should change back to the login page layout
Ultimately, the result didn’t meet our needs nor follow some typical QA best practices.
For example:
There are multiple behaviors being tested on each test. The AI tested access to the page and visible components in the same scenario. If any of those behaviors failed, we would be unable to tell which just by looking at the test log.
There was insufficient test coverage – we needed additional scenarios.
The scenario had confusing and ambiguous test lines. For example, it lacked examples of possible inputs and credentials.
In the same session, we added some new rules for the AI:
Test only one behavior for each test written, and separate long tests into two scenarios.
Fully consider the user experience in the flow of the application.
Create one test specifically for visible components on the screen.
The output was more inline with what we needed and expect for our QA approach at ArcTouch:
Feature: Login Page Rework
Scenario: Validate login with valid credentialsGiven the user is on the Login pageWhen the user enters valid credentialsAnd the user clicks on "Login" buttonThen the user should be redirected to the dashboard page
Scenario: Access the "Forgot Password" flowGiven the user is on the Login pageWhen the user clicks on the "Forget Password" linkThen the form should change to the "Forget Password" layout
Scenario: Verify "Forgot Password" layout elementsGiven the user is on the "Forgot Password" layoutThen the "Email" field should be presentAnd the "Confirm" CTA should be presentAnd the "Cancel" CTA should be present
Scenario: Access the "Set Password" flow for first-time usersGiven the user accesses the site for the first timeThen the form should change to the "Set Password" state
Scenario: Verify "Set Password" layout elementsGiven the user is on the "Set Password" layoutThen the "Password" field should be presentAnd the "Confirm Password" field should be presentAnd the "Confirm" CTA should be presentAnd the "Cancel" CTA should be present
Scenario: Validate password input rulesGiven the user is on the "Set Password" layoutWhen the user enters a passwordThen the password should have a minimum of 8 characters in lengthAnd the password should have at least 1 numberAnd the password should have at least 1 special character
Scenario: Validate custom fields for texts and titlesGiven the user is on the Login pageThen all texts and titles should be custom fields
This clarification drastically improved the results. The tests became much more concise and understandable after adding additional rules based on the previous AI output.
However, we still needed to make a few more adjustments to finish the job. The AI did not provide enough testing coverage of the specified criteria for the page, nor examples of the inputs needed to validate password rules. Ultimately, we learned that how you present your information makes a dramatic impact on what AI delivers. Adding rules, after the initial prompt, and asking the AI for adjustments helps it understand what needs to be added, removed, or edited. Still, it wasn’t perfect – and we had an idea of what we wanted to try next:
Using an AI role model prompt for QA Analyst
We next tried using a role model prompt — also known as “role prompting” — to see if the AI generated more useful responses. Instead of telling the AI to simply perform an action, you tell it who is doing the action. In this case, we guide the AI to take on the role of a “QA Analyst.”
The prompt provides the context of who the AI is modeling, how it thinks about things, how it acts, and what it values. We came up with the following prompt:
You are a world-class QA Analyst with 15 years of experience in writing Test Scenarios in Gherkin. You have a technical background and your audience are engineers with more than 10 years of experience creating top-class award-winning software. Like you, they understand complex technical architecture and terminology and are familiar with technical definitions and modern designs. You follow all the best practices when writing your test scenarios, and you are very careful to make scenarios that can be understood by a full development team of different professionals. I will send you pieces of information about a Login page, requirements, and rules, and you will deliver a full suite of test scenarios that validate that every rule is being covered on the tests. Respond OK if you understood all that.
After our AI tool responded positively to the prompt above, we fed it with the information about the website feature. In this case, the original description was written for human understanding — and not with AI in mind. So, we added some narration about it and the rules that should be followed. We told the AI that the output should be a login page with the functional flow and rules described as you would describe them to a coworker.
The output we received was very thorough — and provided coverage for every QA test scenario we could think of, and more:
Test Scenario 1: Validate the Login FlowGiven a user is on the Login pageWhen the user enters valid Email and PasswordAnd clicks on the "Login" buttonThen the user should be successfully logged in
Test Scenario 2: Validate the Forgot Password FlowGiven a user is on the Login pageWhen the user clicks on the "Forgot Your Password" buttonThen the user should be navigated to the "Forgot Your Password" state
Given a user is on the "Forgot Your Password" stateWhen the user clicks on the "Cancel" buttonThen the user should be returned to the Login page
Test Scenario 3: Validate the Change Password FlowGiven a user is on the Login pageWhen the user clicks on the "Change your Password" buttonThen the user should be navigated to the "Change your Password" state
Given a user is on the "Change your Password" stateWhen the user clicks on the "Cancel" buttonThen the user should be returned to the Login page
Test Scenario 4: Validate the Set Password FlowGiven a user is setting a new passwordWhen the user starts typing a passwordThen all conditions should turn red until the typed passwords attend the corresponding rule
Given a user is setting a new passwordWhen the user types a password with less than 8 charactersThen the password should not be accepted
Given a user is setting a new passwordWhen the user types a password without a special characterThen the password should not be accepted
Given a user is setting a new passwordWhen the user types a password without a numberThen the password should not be accepted
Given a user is setting a new passwordWhen the user types a password that meets all conditionsThen the password should be accepted
Test Scenario 5: Validate Error Message on Invalid LoginGiven a user is on the Login pageWhen the user enters an invalid Email or PasswordAnd clicks on the "Login" buttonThen an appropriate error message should be displayed
Test Scenario 6: Validate Input Fields on Login PageGiven a user is on the Login pageWhen the user views the input fieldsThen there should be two input fields: Email and Password
Test Scenario 7: Validate Navigation Buttons on Login PageGiven a user is on the Login pageWhen the user views the buttonsThen there should be buttons to navigate to "Change your Password" state and "Forgot Your Password" state
Test Scenario 8: Validate Password Visibility ToggleGiven a user is on the Login pageWhen the user types a passwordThen a visibility toggle icon should be present to show/hide the password
Test Scenario 9: Validate Email FormatGiven a user is on the Login pageWhen the user enters an email in incorrect formatAnd clicks on the "Login" buttonThen an appropriate error message should be displayed
Test Scenario 10: Validate Reset Functionality in Forgot Password FlowGiven a user is on the "Forgot Your Password" stateWhen the user enters a valid registered emailAnd clicks on the "Reset Password" buttonThen a password reset link should be sent to the entered email
Test Scenario 11: Validate Old Password Requirement in Change Password FlowGiven a user is on the "Change your Password" stateWhen the user enters a new password without entering the old passwordAnd clicks on the "Change Password" buttonThen an appropriate error message should be displayed
Test Scenario 12: Validate Screen Reader CompatibilityGiven a user is on the Login pageWhen the user navigates through the page using a screen readerThen all elements including input fields, buttons, and instructions should be properly read out
Test Scenario 13: Validate Keyboard NavigationGiven a user is on the Login pageWhen the user navigates through the page using keyboard onlyThen the user should be able to access all interactive elements including input fields and buttons
Test Scenario 14: Validate High Contrast ModeGiven a user is on the Login pageWhen the user enables high contrast modeThen all elements on the page should be clearly visible and distinguishable
Test Scenario 15: Validate Zoom FunctionalityGiven a user is on the Login pageWhen the user zooms in the page up to 200%Then all elements should remain visible and functional, and text should not overlap or cut off
Test Scenario 16: Validate Error Messages AccessibilityGiven a user is on the Login pageWhen an error message is displayedThen it should be announced by the screen reader immediately
Test Scenario 17: Validate Alt Text for Non-Text ContentGiven a user is on the Login pageWhen the user navigates to any non-text content (like images, icons)Then appropriate alt text should be provided for screen readers
Test Scenario 18: Validate Empty Field SubmissionGiven a user is on the Login pageWhen the user clicks on the "Login" button without entering any detailsThen appropriate error messages should be displayed for both Email and Password fields
Test Scenario 19: Validate Password MaskingGiven a user is on the Login pageWhen the user types in the Password fieldThen the entered characters should be masked for security
Test Scenario 20: Validate 'Remember Me' FunctionalityGiven a user is on the Login pageWhen the user enters valid credentials and checks the 'Remember Me' option before logging inThen the user's credentials should be remembered in the next session unless explicitly logged out
Test Scenario 21: Validate Session TimeoutGiven a user has logged into the applicationWhen the user remains inactive for the specified session timeout durationThen the user should be automatically logged out and redirected to the Login page
Test Scenario 22: Validate Concurrent SessionsGiven a user has logged into the application from one deviceWhen the same user tries to log in from another deviceThen the user should be informed about the active session and asked to confirm before initiating a new session
Test Scenario 23: Validate Response to Multiple Failed Login AttemptsGiven a user is on the Login pageWhen the user enters incorrect credentials multiple times consecutivelyThen the account should be temporarily locked to prevent potential unauthorized access
Test Scenario 24: Validate Cross-Browser CompatibilityGiven a user accesses the Login pageWhen the user is using different browsers (e.g., Chrome, Firefox, Safari, Edge)Then the Login page should function correctly and consistently across all supported browsers
Test Scenario 25: Validate Responsiveness on Different DevicesGiven a user accesses the Login pageWhen the user is using different devices (e.g., desktop, laptop, tablet, smartphone)Then the Login page should display and function correctly, adapting to various screen sizes and resolutions
Test Scenario 26: Validate URL Redirection After Successful LoginGiven a user is on the Login pageWhen the user enters valid credentials and clicks on the "Login" buttonThen the user should be redirected to the appropriate landing page based on their role and permissions
Test Scenario 27: Validate Password Reset FlowGiven a user has received a password reset link via emailWhen the user clicks on the link and sets a new password following the password rulesThen the user should be able to log in with the new password successfully
Test Scenario 28: Validate Input Field Validation MessagesGiven a user is on the Login pageWhen the user interacts with input fields (e.g., entering invalid data, leaving fields empty)Then appropriate validation messages should be displayed near the corresponding input field
Test Scenario 29: Validate Language SupportGiven a user is on the Login pageWhen the user selects a different language from the available optionsThen the Login page content should be translated into the selected language
We found that the AI provided test scenarios in batches of 6 to 10 each. Once it had reached a limit for each of the outputs, we prompted it to "Make sure that everything is covered" and to "Please provide more tests based on the information given." Each time, a new batch of scenarios was provided until we reached the very specific test cases shown at the end of the list.
Simply asking the AI to provide more was a way to force it to gather more information from a single prompt, which eventually provided more complete coverage with a variety of edge cases.
Using AI for generating API test scenarios
Next, we wanted to use the AI to help generate test scenarios for the API that connected the front-end interface with the back-end response.
We had more success by following an approach similar to the role model prompt. However, when we tried to complete this in a single prompt, informing the AI of what triggered each response and the codes that the API returned, the output was barely coherent.
So, we separated the API prompts into two batches: one for integration with the front end and one for back-end testing in Postman.
First, we described the login page, the components present, and the main functionalities. We described it so the AI would know what components would influence possible API responses, such as input fields and CTA buttons. Then, we asked it to give us API tests that would cover the page:
Feature: Login Page API Integration
Scenario: Validate successful login with API responseGiven the user is on the Login pageWhen the user enters valid credentialsAnd the user clicks on "Login" buttonThen the API should return a successful responseAnd the user should be redirected to the dashboard page
Scenario: Validate unsuccessful login with API responseGiven the user is on the Login pageWhen the user enters invalid credentialsAnd the user clicks on "Login" buttonThen the API should return an error responseAnd the user should see an error message
Scenario: Validate "Forgot Password" flow with API responseGiven the user is on the "Forgot Password" layoutWhen the user enters a valid email addressAnd the user clicks on the "Confirm" CTAThen the API should return a successful responseAnd the user should see a confirmation message
Scenario: Validate "Set Password" flow with API responseGiven the user is on the "Set Password" layoutWhen the user enters a valid passwordAnd the user enters a matching confirm passwordAnd the user clicks on the "Confirm" CTAThen the API should return a successful responseAnd the user should be redirected to the dashboard page
Scenario: Validate error handling for "Set Password" flow with API responseGiven the user is on the "Set Password" layoutWhen the user enters an invalid passwordAnd the user clicks on the "Confirm" CTAThen the API should return an error responseAnd the user should see an error message
Then, we described each one of the APIs used and the possible outcomes in the following prompt:
Considering the ticket above, write isolated tests for API, before the integration with the homepage. These tests will be executed in Postman before the integration with the Front End. They need to be written in Gherkin and should guarantee that the API is functioning correctly and each code is returned correctly. Here are the response codes and expected results for each:
SetPassword API:200 - Success403 - Failed401 - Invalid API Bearer Token412 - Username not found417 - Invalid Request Token500 - Processing Error
Authenticate API:200 - Success403 - Invalid Username or Password401 - Invalid API Bearer Token412 - Invalid BP User500 - Processing Error
ChangePassword API:200 - Success403 - Invalid Username or Password - Update Failed401 - Invalid API Bearer Token500 - Processing Error
IntiateForgotPassword API200 - Success412 - Username Not Found401 - Invalid API Bearer Token500 - Processing Error
After specifying in great detail the expected results in our prompt, the AI tool responded with the following:
Feature: API Isolated Tests in Postman
Scenario: Validate SetPassword API - SuccessGiven the user has a valid request for SetPassword APIWhen the user sends the SetPassword request in PostmanThen the API should return a 200 status code
Scenario: Validate SetPassword API - FailedGiven the user has an invalid request for SetPassword APIWhen the user sends the SetPassword request in PostmanThen the API should return a 403 status code
Scenario: Validate SetPassword API - Invalid API Bearer TokenGiven the user has a SetPassword request with an invalid API Bearer TokenWhen the user sends the SetPassword request in PostmanThen the API should return a 401 status code
Scenario: Validate SetPassword API - Username not foundGiven the user has a SetPassword request with a non-existing usernameWhen the user sends the SetPassword request in PostmanThen the API should return a 412 status code
Scenario: Validate SetPassword API - Invalid Request TokenGiven the user has a SetPassword request with an invalid Request TokenWhen the user sends the SetPassword request in PostmanThen the API should return a 417 status code
Scenario: Validate SetPassword API - Processing ErrorGiven the user has a SetPassword request that triggers a processing errorWhen the user sends the SetPassword request in PostmanThen the API should return a 500 status code
Scenario: Validate Authenticate API - SuccessGiven the user has a valid request for Authenticate APIWhen the user sends the Authenticate request in PostmanThen the API should return a 200 status code
Scenario: Validate Authenticate API - Invalid Username or PasswordGiven the user has an Authenticate request with invalid credentialsWhen the user sends the Authenticate request in PostmanThen the API should return a 403 status code
Scenario: Validate Authenticate API - Invalid API Bearer TokenGiven the user has an Authenticate request with an invalid API Bearer TokenWhen the user sends the Authenticate request in PostmanThen the API should return a 401 status code
Scenario: Validate Authenticate API - Invalid BP UserGiven the user has an Authenticate request with an invalid BP UserWhen the user sends the Authenticate request in PostmanThen the API should return a 412 status code
Scenario: Validate Authenticate API - Processing ErrorGiven the user has an Authenticate request that triggers a processing errorWhen the user sends the Authenticate request in PostmanThen the API should return a 500 status code
Scenario: Validate ChangePassword API - SuccessGiven the user has a valid request for ChangePassword APIWhen the user sends the ChangePassword request in PostmanThen the API should return a 200 status code
Scenario: Validate ChangePassword API - Invalid Username or Password - Update FailedGiven the user has an invalid request for ChangePassword APIWhen the user sends the ChangePassword request in PostmanThen the API should return a 403 status code
Scenario: Validate ChangePassword API - Invalid API Bearer TokenGiven the user has a ChangePassword request with an invalid API Bearer TokenWhen the user sends the ChangePassword request in PostmanThen the API should return a 401 status code
Scenario: Validate ChangePassword API - Processing ErrorGiven the user has a ChangePassword request that triggers a processing errorWhen the user sends the ChangePassword request in PostmanThen the API should return a 500 status code
Scenario: Validate IntiateForgotPassword API - SuccessGiven the user has a valid request for IntiateForgotPassword APIWhen the user sends the IntiateForgotPassword request in PostmanThen the API should return a 200 status code
Scenario: Validate IntiateForgotPassword API - Username Not FoundGiven the user has an IntiateForgotPassword request with a non-existing usernameWhen the user sends the IntiateForgotPassword request in PostmanThen the API should return a 412 status code
Scenario: Validate IntiateForgotPassword API - Invalid API Bearer TokenGiven the user has an IntiateForgotPassword request with an invalid API Bearer TokenWhen the user sends the IntiateForgotPassword request in PostmanThen the API should return a 401 status code
Scenario: Validate IntiateForgotPassword API - Processing ErrorGiven the user has an IntiateForgotPassword request that triggers a processing errorWhen the user sends the IntiateForgotPassword request in PostmanThen the API should return a 500 status code
Even though the AI tool struggled at first to provide a useful output for API-related tests, separating the front-end integration from back-end testing gave us the results we were looking for.
The process and result weren’t perfect, but we still were better off using the tool than generating the API test scenarios on our own.
Takeaways from using AI to write test scenarios
Overall, we found that using AI to write test scenarios and augment our QA testing process was a big help. It definitely improved our efficiency. We estimate that writing these from scratch would take about two days. Using AI to augment that process saved us about half that time.
Most importantly, using AI gave us another resource to ensure we were covering the breadth of tests and corner cases we needed. In other words, using AI helped boost the quality of our testing — the most important contribution any QA tool could offer.
A few other key takeaways included:
There is no perfect prompt, and prompting is a skill you need to learn and develop, like any other.
No matter how good your prompt is, the AI output is never perfect. QA professionals will always need to review, edit, and take ownership of the written test scenarios generated by any AI tool.
Last but not least, using AI tools promises to help QA professionals save time in some areas — so you can spend more time and attention on other facets of quality in your project.
At an agency like ArcTouch, it means we use our client’s project budget in smarter ways, by spending less time writing scenarios and more time actually testing those scenarios, to create more lovable and higher quality apps and websites.
Follow along
To read our series about injecting AI into software development, join our mailing list or follow us on LinkedIn and X. You can also contact us to learn more about how we use AI to help create lovable apps, websites, and connected experiences.
Article Author:
Subscribe for more insights
Get our newsletter in your inbox.
Related posts
Injecting AI into software development — an ArcTouch blog series
The ArcTouch design and development team shares how to use AI to build lovable apps and websites — better and faster.
5 min. read - April 9, 2024
How AI helped us write user stories for an app
Learn how the ArcTouch development team turns email and Slack messages into AI-generated user stories for an app project.
5 min. read - June 27, 2024
Contact us.
Let's build something lovable. Together.
We help companies of all sizes build lovable apps, websites, and connected experiences.