Testing the Untestable: Introducing Subliminal

Just in time for WWDC this week, the iOS team at Inkling is excited to announce a new resource for iOS developers: Subliminal, an open-source Objective-C framework for writing iOS integration tests. If you’re in the San Francisco Bay area for the conference, we invite you to drop by Inkling for a hands-on lab tomorrow evening, June 12th, at 6:30 pm, where you’ll receive one-on-one support for integrating Subliminal into your own app. If you’re not around or just curious, here’s some background on the problem we were solving and how Subliminal could be useful in your own app development.

The Problem

At Inkling, we were recently implementing in-app purchase when we noticed something odd: on iOS 6, the in-app purchase alerts weren’t in the app at all. We had read about Apple using remote view controllers to present system views in different processes, but we didn’t quite understand what that meant until we saw an alert survive a force-quit:

After we figured out what was happening, we were puzzled: how would we test in-app purchase when the alerts weren’t even part of our application? Frameworks like KIF and Frank couldn’t help us, as they rely on being able to directly manipulate views. But, as luck would have it, we were developing a new tool that was just the solution we needed.

Subliminal

Subliminal is Inkling’s new, open-source framework for writing iOS integration tests. Like other similar frameworks, Subliminal lets developers write Objective-C tests that integrate directly into the applications being tested. But Subliminal has a not-so-secret strength: underneath, it’s powered by Apple’s UIAutomation framework.

For example, when it comes to the in-app purchase alert, Subliminal doesn’t need to tap on the alert directly; instead, it can dynamically generate the JavaScript necessary for UIAutomation to perform that action. When a developer writes, in a Subliminal test:

[gist id=5754912]

Subliminal registers the following alert handler with UIAutomation:

[gist id=5754930]

By using UIAutomation, Subliminal can simulate almost any interaction, with the app or the device. Purchase confirmed!

https://vimeo.com/67929775

Testing In-App Purchase with Subliminal from Inkling on Vimeo.

 

Testing Made Easy

Subliminal is much more than just a wrapper around UIAutomation, though. It overcomes several huge problems that come with using UIAutomation directly.

The first difficulty with using UIAutomation alone is that it requires interface elements to be identified by their absolute position within the “element hierarchy,” like this:

[gist id=5754939]

These references are not only difficult to read but are also difficult to write. To refer to any particular element, you have to describe its entire ancestry, while including only the views that UIAutomation deems necessary (images, yes; accessible elements, maybe; private UIWebView subviews, sure!).

Luckily, Subliminal allows developers to identify elements by their properties, independent of their position in the hierarchy. In Subliminal, the above line can be written as:

[gist id=5754942]

When Subliminal needs to manipulate that button, it finds a matching element in the hierarchy and dynamically generates the UIAutomation reference for you. Subliminal abstracts away all the complexity of UIAutomation scripts, letting you focus on writing tests.

Scalable Tests

As we’ve developed Subliminal over the last nine months, we’ve realized that simulated user interaction is only half of integration testing. The other half is establishing the conditions necessary to test that interaction. This context is often bound up in your application logic, which is why it’s a shame that UIAutomation provides no mechanism for manipulating your application directly.

An application is a complete black box to UIAutomation, but not to Subliminal. Subliminal lets tests access and manipulate application state by using “app hooks”: Objective-C methods defined by the application, made available to the tests.

For instance, before running the tests, our application delegate registers a “download manager” singleton as being able to download a book:

[gist id=5754948]

Then, tests that involve interactions with particular content can simply call
“downloadBookWithId:” from their set-up methods:

[gist id=5754953]

App hooks have helped us to keep our tests independent: only one test need evaluate the download UI, while the others can use the programmatic interface. App hooks have also let us re-use our application’s code without making our tests dependent on our application’s structure.

A Framework for Objective-C Developers

It can be difficult to write integration tests–they’ve got to cover the whole application. That’s why we built Subliminal to let you simulate user interaction and exercise the app directly, using the most reliable and familiar tools. And we’ve only scratched the surface of what Subliminal can do. Among other things, it includes native support for continuous integration–Subliminal eventests itself using Travis!

You can find out more details, and learn how to write your first test in under 10 minutes, at theGitHub project homepage. And, as we mentioned above, we’d love for you to join us at Inkling HQ for a hands-on lab tomorrow, June 12th, for one-on-one support integrating Subliminal into your own app. We’ve found Subliminal really useful, and we hope you do, too!