Testing the AI Chatbots - Pat Walsh Tests

Testing the AI Chatbots – Questions

I tested the AI Chatbots with a wide range of questions and these included:

simple questions with factual answers that can be determined as correct or incorrect An example question was: When was the Battle of Hastings? And a follow up question was: Who died at the Battle of Hastings?
some trickier questions with factual answers to add some complexity An example question was: Who was the Chancellor during Tony Blair’s goverment? And a follow up question was: When did Gordon Brown become Prime Minister?
questions on specific areas of knowledge, such as movies, music, sport, even air fryers! An example question was: Tell me about Paris Olympics 2024 – another question was: What artists have used the AKAI MPC to make music? – and another question was: How long does it take to cook bacon in an air fryer? And some questions on movies and Oscars caused some issues, such a question as: Who has won the Best Director Oscar the most times? And an even trickier question was: How many times has Francis Ford Coppola won the Best Director Oscar?
plus some questions where the wording is garbled to see if it can still get the meaning of the question Such as: Where to oysters eat best London place?
some whimsical questions, that do have a meaning, to see if it can grasp those and what kind of answer it supplies Such as: Does the sea see me or do I see the sea?
some basic garbage questions to check error handling Such as all numbers: 9023840923840928 3948290348092384 230948

For consistency, I used the same set of questions across all the AI Chatbots and marked them accordingly based on the results and also made notes on any bugs or errors I found. See the individual posts for info on the bugs found.

All the posts

I have created a post for each AI Chatbot test in order to fully explore the test results, a post which shows the questions asked, plus a Results post, where I crown the winning AI Chatbot. You can jump straight there or read each test post via the links below.

– Testing the AI Chatbots

– Testing the AI Chatbots: Questions

– Testing the AI Chatbots: Perplexity AI

– Testing the AI Chatbots: OpenAI ChatGPT

– Testing the AI Chatbots: Microsoft Copilot

– Testing the AI Chatbots: Google Gemini

– Testing the AI Chatbots: Results

Note: The heading images used in these posts were created via Bing Image Creator

About my testing services: iOS App Testing / Android App Testing / Website Testing / AI Chatbot Testing

We’ve worked with Pat Walsh on a number of very different projects, from interactive video projects to mobile apps. His work is very thorough and has quickly become an essential and important phase of all the production work we do at Storythings. He’s thorough, diligent, and a pleasure to work with.

Matt Locke, Director, Storythings

Pat Walsh fixed my site within hours of my contacting him – his WordPress website fix service clearly works!

Douglas Green, Douglas Software

Pat has been absolutely incredible at testing each of our projects he works on. He’s very professional, organised and diligent. I look forward to using him again in the future.

Oliver Thurtle, Director of Technology, Drum

Pat Walsh provided Kingston Valdes with a professional and personable service that was excellent value. The site was built and tested quickly, we are delighted with the results, and we have already had positive feedback from some of our clients. If you’re thinking of building a new site or adapting an existing one, I’d certainly recommend speaking with Pat about it.

Kathy Valdes, Kingston Valdes

Couldn’t recommend Pat highly enough. He goes beyond what you’d expect of a QA tester, commenting more broadly on user experience and even design. He’s quick, efficient and thorough – couldn’t ask for any more.

Rob Calvert, Director, Click

We hired Pat to help with QA of a music streaming app. We needed someone with that magic “6th sense” of where the bugs are lurking; the ones which developers tend to miss. Pat more than fulfilled this need, with highly methodical approach and a great eye for detail.

Subsequently Pat has provided QA services for a personal app project of mine, finding a significant issue which had been lurking for some time.

Pat is a real asset to any team – highly competent technically but also easy going and a great guy to work with. Highly recommended!

Andrew Ebling, Head of App Development, Composed

Pat came to Composed to respond to our (increasingly urgent) need of QA on mobile and web and I can testify he more than fulfilled the task. On the Android side, his very structured approach (first exploratory testing, then focus on defining test plans for a variety of scopes) and his clear knowledge of the latest mobile technologies have quickly paid off. He was involved directly with the developers (including myself) and this very dynamic relationship enabled us to annihilate bugs swiftly and efficiently.

David Ferrand, Android Developer, Composed

Pat is an excellent one-stop-shop for testing Apps and Websites. He is a self-starter who just knows what needs to be done and gets on with it, and is responsive to requests for any specific testing requirements. He provides test plans and reports as a matter of course and was a pleasure to work with. I will definitely be working with Pat again if the opportunity arises.

Andrew Birley, PM, Storythings

Pat performs comprehensive website testing for us, across all the main browsers and platforms, including mobile devices. His flexible approach to testing and clear feedback enable us to have confidence that the business end of our website is still working smoothly after system updates and changes.

Steve Liu, Manager, Wicked Lasers

Testing the AI Chatbots – Questions

Testing the AI Chatbots – Questions

Share this to social media...