Thinking of full-featured iOS automation

The purpose of this article is to share my experience of iOS automation as wide as possible. So that’s why I’m writing it in English. The other reason – I still can’t find the answers I need (on the Internet). So I’m happy to post MY research. The main question here is how to set up a continuously integrated iOS automation. Because I think that ‘true’ automation is supposed to require as little manual effort as possible. Bugs in code have to be discovered and fixed ASAP. So in my opinion it’s reasonable to run the test suite after each build.

I would not talk here about iOS automation basics. There are lots of good articles about that. So I’d better point the most distinguishing of them.
O’REILLY article
Alex Vollmer & tuneup_js library
Apple’s UIAutomation documentation
Changes to UIAutomation in iOS 5

We assume that we selected Instruments app + UIAutomation framework as our tool. I worked with this tool for some months and here what I can say about the advantages:

• Native Apple support
• Works with iOS simulator as well as with real device
• Simulation of user interactions (works from UI perspective)
• Extensible API, so we can build the whole testing framework
• We don’t need to incorporate any additional code in our app
• Can be used in concert with other instruments such as memory leak checking

The only disadvantage is that the tool is a bit buggy and raw (in my opinion). But almost every software is so And one more thing. The tool is not supposed to work in continuous integration. But I think I solved that. And I want to share my solution with you.

At the beginning I had an iOS app that is used to get access to online backup system. It’s build process is automated using Bamboo. There were some unit tests written in Xcode. The idea was to start working on functional regression testing. So I started with very simple framework built on top of tuneup_js library and error handling code provided on Apple’s site. Other prerequisites for starting the automation are:

• You have to build debug version of your app. For instance, if you are testing on iOS Simulator, you need an app file usually located here <build-dir>/build/Debug-iphonesimulator/<appname>.app  where <build-dir> is your main build dir and <appname> is the name of your app.
• You have to specify accessibility labels for controls (buttons, fields etc) in your app using Interface Builder in Xcode (please read details in Alex Vollmer article) and set AccessibilityEnabled and ApplicationAccessibilityEnabled settings of iOS Simulator to TRUE (if you are testing on simulator) (please see below how to do it).

OK, so I wrote an easy test that logs in to account, makes some assertions and quits:

test("Test 01", function(target, app) {

//login screen init
loginWindow = app.mainWindow();
navBar = loginWindow.navigationBar();
mainView = loginWindow.scrollViews()[0];
loginField = mainView.textFields()["username"];
passField = mainView.secureTextFields()["password"];
loginButton = mainView.buttons()["loginbutton"];

//login screen actions
assertEquals("Log In", navBar.name());
loginField.setValue("login");
passField.setValue("password");
loginButton.tap();
loginButton.waitForInvalid();

//my computers screen init
mainWindow = app.mainWindow();
navBar = mainWindow.navigationBar();
settingsButton = navBar.leftButton();
mainTable = mainWindow.tableViews()[0];

//my computers screen actions
assertEquals("My Computers", navBar.name());
settingsButton.tap();
settingsButton.waitForInvalid();

//settings screen init
settingsWindow = app.mainWindow();
navBar = settingsWindow.navigationBar();
settingsTable = settingsWindow.tableViews()[0];
logOutCell = settingsTable.cells()["Log Out"];

//settings screen actions
assertEquals("Settings", navBar.name());
logOutCell.tap();
target.delay(1);

});

Let’s think – what will we need if we run this test after each build? Of course we need gathering and handling of test results! So that we will have the statistics: which error appeared/fixed in which build. When this script is running via command line version of Instruments we have the messages in console like:

2012-01-24 19:23:37 +0200 Start: Test 01
2012-01-24 19:23:38 +0200 Debug: target.frontMostApp().mainWindow().scrollViews()[0].textFields()["username"].tap()
2012-01-24 19:23:38 +0200 Debug: target.frontMostApp().mainWindow().scrollViews()[0].secureTextFields()["password"].tap()
2012-01-24 19:23:39 +0200 Debug: target.frontMostApp().mainWindow().scrollViews()[0].buttons()["loginbutton"].tap()
2012-01-24 19:23:49 +0200 Error: <My Computers> but received <Log In>

But Bamboo assumes to work with structured results, e.g. in JUnit format. And UIAutomation can’t produce such results. No problem! I wrote a Java solution that converts the output of Instruments to JUnit-compatible xml (the usage is explained below). So we will have the xml like:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="1" hostname="Mac OS X" id="0" name="RunTestSuite" package="com.ios.tests" tests="1" time="12" timestamp="2012-01-24T19:23:37">
<testcase classname="com.ios.tests" name="Test 01" time="12">
<failure message="Expected &lt;My Computers&gt; but received &lt;Log In&gt;" type="Failure">
target.frontMostApp().mainWindow().scrollViews()[0].textFields()["username"].tap()
target.frontMostApp().mainWindow().scrollViews()[0].secureTextFields()["password"].tap()
target.frontMostApp().mainWindow().scrollViews()[0].buttons()["loginbutton"].tap()
</failure>
</testcase>
</testsuite>

We also will need a .trace file that is used as parameter when launching tests from command line. I didn’t find a way to omit that  That is a binary file created by Instruments app, so you’ll have to create you own. All you need is to open Instruments, select a proper target (the debug version of your app or some physical device) and save. But the issue here is that you’ll not be able to change the path to your target in any way  other than re-saving the .trace file. We are doing the continuos integration, you remember? So the trick is to put the target file on your local machine in a folder that exactly equals to folder on build machine. And only after that select it in Instruments and save. Other issue with .trace file is that it is actually a folder, so I had problems when tried to save it to SVN and checkout again. The workaround is to zip this file and unzip during the automation testing process.

So here is my bash script that does all the magic:

#!/bin/sh

#ensure xcode4 location is specified in /etc/path-to-xcode4.2
#by this we assume that xcode4 is installed on current machine
XCODE="`cat /etc/path-to-xcode4.2`"; if [ "$XCODE" == "" ]; then
echo "No suitable Xcode found"; exit; fi

#alias for xcodebuild (of xcode4) with our project
xcodebuild=("$XCODE"/usr/bin/xcodebuild -project "appname.xcodeproj")

#alias for instruments (of xcode4)
instruments=("$XCODE"/usr/bin/instruments)

#ensure we have iphonesimulatorsdk >= 5.0
SIMULATOR_SDK="`$xcodebuild -showsdks | perl -ne 'next unless m/(-sdk iphonesimulator(\d+\.\d+))/; next unless $2 >= 5.0; print "$1\n"; last'`"
if [ "$SIMULATOR_SDK" == "" ]; then
echo "error: iphonesimulatorsdk5.0 or later is not available." >&2
exit 1
fi

#build needed appname.app using iphonesimulatorsdk5.0 (not included in regular build)
./Tools/with-keychain CodeSigning.keychain 'password' "$xcodebuild" -target appname -configuration Debug $SIMULATOR_SDK

#location of needed appname.app file
appnameapp=(./build/Debug-iphonesimulator/appname.app)

#setting Accessibility defaults to TRUE
defaults write ~/Library/Application\ Support/iPhone\ Simulator/5.0/Library/Preferences/com.apple.Accessibility AccessibilityEnabled -bool TRUE
defaults write ~/Library/Application\ Support/iPhone\ Simulator/5.0/Library/Preferences/com.apple.Accessibility ApplicationAccessibilityEnabled -bool TRUE

#unzip tests.trace
chmod 777 ./UIAutomation/tests.trace.zip
unzip -o ./UIAutomation/tests.trace.zip tests.trace/* -d ./UIAutomation/

#run the test
"$instruments" -D ./UIAutomation/tests.trace "$appnameapp" > ./UIAutomation/res.txt || exit 1

#convert res.txt to junit format
java -jar ./UIAutomation/TxtToJunit.jar ./UIAutomation/res.txt > ./UIAutomation/result.xml

#quit iOS Simulator
osascript -e 'tell application "iPhone Simulator" to quit'

I need to comment on the next things:

• I assume Xcode 4.2 or later is installed on build machine. In my case I was forced to work with exact version of Xcode (Xcodes 4 and 3 were installed side-by-side for some purposes) so that’s why path to Xcode 4.2 is specified in file /etc/path-to-xcode4.2 and is reading from that file. If it can’t be read, we exit the test (lines 5-6)
• I also need iphonesimulatorsdk5.0 or later so I check for it’s presence in system (lines 15-19)
• During the automation test process we are building the needed appname.app file (debug version) and we are using our CodeSigning.keychain file with all necessary Code Signing info (line 22)
• We are also setting the accessibility settings for iOS Simulator (lines 28-29)
• As you can see, in line 36 we are specifying the exact path to our app despite it’s specified already in tests.trace file. This is a “feature”. I can’t simplify this for now.

So this bash script is saved in run_automation.sh file (in the main build dir) and is launched after each build of app. For that I’ve added 2 tasks on Bamboo (to existing build process): one for launching of ‘run_automation.sh’ and another is task of ‘JUnit Parser’ type (and I specified the folder where result.xml file will appear after the whole process).

Another epic issue I encountered (that almost killed my hope to success) was access to ‘console’. So I found that in order to launch Instruments (with iOS Simulator) on remote machine you need your build user to be logged to console there. Let me explain. Usually (in Mac OS X) every remote user connects to some terminal (TTY) like s001, s002 etc. But there is also a possibility to connect to console (remotely). For instance, via VNC. If you go this way, use short-term trick – switch to ‘Login Window’ back (using Fast User Switching) and quit VNC client. So in this case your user remains logged to console. You can discuss other ways with your IT administrator.

So after solving all these blockers I’ve got working automation. You can login to Bamboo interface and view all the runs or manually trigger a new run.

But why I name this article “Thinking of full-featured iOS automation”? Because I’m thinking of further steps. In my vision it could be:

• Creating of data repository: accounts, pages, objects to verify etc (for instance using JavaScript objects)
• Building the test framework on top of UIAutomation API
• Writing the tests, grouping them in suites

If you have any questions, please contact me.

•