Device testing in the cloud - I'm now using Kobiton

Device testing in the cloud – I’m now using Kobiton

I’ve recently started using the Kobiton device testing cloud, to enhance the range of iOS and Android devices that I can test apps and websites on.

Over the last few years, I’ve looked into several other device testing cloud platforms from all sorts of companies and each time I haven’t been that impressed, as at least one (if not all) of the required features below have been missing or lacking.

Required Features

1) wide range of devices, including recent devices and OS versions
2) cost-effective pricing (for a freelance tester such as me)
3) access to the devices for both app testing and website testing
4) ease of use and effectiveness of the platform
5) platform allows for both manual and automated testing

Several device testing cloud platforms

Here’s what I’ve found over the years when researching the various device testing cloud platforms I’ve found:

Firebase Test Lab – when I checked the features, this was just for app testing and for automated testing only
Amazon AWS Device Farm – I gave this a trial, but found the device list available for manual testing/access was very limited
Perfecto – the pricing of this (Basic plan starts at $129/month) limits it to larger companies rather than freelance testers like me, plus I wasn’t able to find their device list after much browsing of their website
Sauce Labs – their device list does seem to be up-to-date but again their pricing (Live starts at $119/month, Automated starts at $349/month) limits it to larger companies
Bitbar Cloud – their device list does seem up-to-date but again pricing (Solo plan starts at $99/month) would be an issue for a freelance tester

So how does Kobiton measure up on the Required Features above?

1) wide range of devices – as of the end of April 2018, there were 44 iPhones, 17 iPads, 151 Android phones and 9 Android tablets available for testing and the range is up-to-date, including iPhone X, iPhone 8 / 8 Plus and Samsung Galaxy S8 / S8+
2) cost-effective pricing (for a freelance tester such as me) – the Indie plan costs $10/month and includes 100 minutes of testing and you can add further minutes as required, from $10 per 100 minutes
3) access to the devices for both app testing and website testing – this is available
4) ease of use and effectiveness of the platform – the platform is easy to use and features easy to access
5) platform allows for both manual and automated testing – the platform allows for both types

There’s been a few glitches along the way, but these have mostly been sorted by creating a support ticket and then leaving support to fix the glitch.

Website Testing on Kobiton devices

This is the easiest type of testing to perform, as you just select the required device and wait for it to start up, then you can startup the relevant browser and perform your website testing on the device

App Testing on Kobiton devices

App testing requires some further tasks, as you have to first upload your app to be tested, in .ipa or .apk file format (also, you need to make sure your app is setup properly for ad-hoc distribution)
The uploaded app is then scanned for any security issues such as viruses and malware – when it passes this, the app is then available to test with
To install your app, you startup the required device and then install the app to it

Common Features to using Kobiton devices

On each device, you can manually set the location (using GPS co-ords) and timezone (from a dropdown list) you require, if that’s relevant for your testing
You can take a screenshot at any time and then download that locally, if required
Each device testing session is saved and you can see video of the session
Some device features such as making calls, sending SMS aren’t available

My thoughts on using Kobiton so far…

I was impressed enough with the free trial to move to the Kobiton Indie plan. Since then, I’ve used Kobiton devices in several test projects, along with my own set of test devices.

I think the best way to use a device testing cloud platform such as Kobiton is to use it to complement your own set of test devices, to allow you to test websites or do some smoke testing on a wider range of devices than you have physical access to.

For example, a Kobiton device would be ideal to smoke test an iOS app on a device and iOS version you don’t have access to, to check that the iOS app installs ok, starts up ok, and then do a quick run through the main features to check the stability, display and hopefully that the app is crash-free.

Pat is an excellent one-stop-shop for testing Apps and Websites. He is a self-starter who just knows what needs to be done and gets on with it, and is responsive to requests for any specific testing requirements. He provides test plans and reports as a matter of course and was a pleasure to work with. I will definitely be working with Pat again if the opportunity arises.

Andrew Birley, PM, Storythings

Pat Walsh fixed my site within hours of my contacting him – his WordPress website fix service clearly works!

Douglas Green, Douglas Software

Pat has been absolutely incredible at testing each of our projects he works on. He’s very professional, organised and diligent. I look forward to using him again in the future.

Oliver Thurtle, Director of Technology, Drum

Pat Walsh provided Kingston Valdes with a professional and personable service that was excellent value. The site was built and tested quickly, we are delighted with the results, and we have already had positive feedback from some of our clients. If you’re thinking of building a new site or adapting an existing one, I’d certainly recommend speaking with Pat about it.

Kathy Valdes, Kingston Valdes

Pat came to Composed to respond to our (increasingly urgent) need of QA on mobile and web and I can testify he more than fulfilled the task. On the Android side, his very structured approach (first exploratory testing, then focus on defining test plans for a variety of scopes) and his clear knowledge of the latest mobile technologies have quickly paid off. He was involved directly with the developers (including myself) and this very dynamic relationship enabled us to annihilate bugs swiftly and efficiently.

David Ferrand, Android Developer, Composed

We hired Pat to help with QA of a music streaming app. We needed someone with that magic “6th sense” of where the bugs are lurking; the ones which developers tend to miss. Pat more than fulfilled this need, with highly methodical approach and a great eye for detail.

Subsequently Pat has provided QA services for a personal app project of mine, finding a significant issue which had been lurking for some time.

Pat is a real asset to any team – highly competent technically but also easy going and a great guy to work with. Highly recommended!

Andrew Ebling, Head of App Development, Composed

Pat performs comprehensive website testing for us, across all the main browsers and platforms, including mobile devices. His flexible approach to testing and clear feedback enable us to have confidence that the business end of our website is still working smoothly after system updates and changes.

Steve Liu, Manager, Wicked Lasers

We’ve worked with Pat Walsh on a number of very different projects, from interactive video projects to mobile apps. His work is very thorough and has quickly become an essential and important phase of all the production work we do at Storythings. He’s thorough, diligent, and a pleasure to work with.

Matt Locke, Director, Storythings

Couldn’t recommend Pat highly enough. He goes beyond what you’d expect of a QA tester, commenting more broadly on user experience and even design. He’s quick, efficient and thorough – couldn’t ask for any more.

Rob Calvert, Director, Click

Device testing in the cloud – I’m now using Kobiton