Automated cross platform testing of iOS and Android hybrid Ionic/Cordova/Angular apps using Cucumber, Protractor, Chai and Appium

 ionic-logoangular-logochai-logocucumber-logocordova-logoappium-logo   protractor-logo

Important Note

This article was written before Xcode 8 came out. Using Xcode 8 will cause the code in this tutorial to break. The information on how to update this tutorial for iOS testing is here. I am currently fixing the problems with using Appium Beta (1.6.0) with Android and will follow up with an updated article on automation testing for Ionic 2/Angular 2. If you are still on Xcode 7 and using the current release version of Appium (1.5.3) and Ionic 1 then read on.

Getting Set Up

I’m going to assume certain things from the outset such as you are on a Mac and you have Xcode, Node and Android Studio installed.

I have written this post because I was not able to find all the information in one place for achieving cross platform testing for Ionic apps using Cucumber (cucumber.js). Ionic is built on AngularJS making it really easy to knock together mobile apps quickly with a well supported community and Cucumber (when used correctly) makes it possible to engage all your stakeholders, testers and developers upfront in order to specify your project in plain text so everyone knows what to build, what to test, what to automate and what product is wanted.

You can follow the tutorial and write the code or download all the files from github
https://github.com/flashquartermaster/myappname
If you get the files from github the just run:

from the root of the app to install the testing modules.

So, we need to install some stuff to get started, first up fire up the Terminal and install Cordova and Ionic. You can read about installing Cordova here and Ionic here. (Ionic serves as a wrapper for Cordova much like PhoneGap)

Check that your environment is set up correctly by running

Navigate to where you want to put your new app and create it, you can use

but I’m going to use

Which gives you options to create a few templates such as a tabbed interface or a sidebar (more here). For this setup, I’ll use the blank template.

and you should see you project files.

You can test your new app in a browser from the app directory that you are currently in by typing

Important Note! Do not run ‘ionic serve’ and ‘appium’ simultaneously. Appium will not be able to create a new session as both will be pointing at localhost:8100.

But we don’t want to test in the browser we want to test (at least initially) on iOS and Android emulators as behaviours can differ from the browser environment.

You may at this point want to upgrade to the latest builds of cordova-ios and cordova-android. You can do this by running (at the time of writing)

and

Then remove the default iOS platform install with

Then rebuild it with

Then add the Android platform

Building A Simple Login App For Testing

Ok, so I’m going to use a typical use case of having a login page that we are going to test so let’s build it quickly. This is just a quicky so I’m not going to do anything fancy with the architecture. Also, this is not a tutorial about Behaviour Driven Development otherwise we would be doing the specifications first to drive out the UI and application interfaces before making anything function then stepping in and doing unit tests to then start writing our production code. (If you want to learn more about BDD have a look at this video) This tutorial is really all about getting the automated tests set up and running.

Edit the index.html file and remove

and replace it with

and if you want to update the name from ‘Ionic Blank Starter’ to something else

Open myappname/www/js/app.js and add

and change the angular.module definition in app.js from

to

This gives us a login state and a home page state. Now create a directory called ‘templates’ in your www directory and create two html files, login.html and home.html

Your home.html should look like this:

and your login.html should look like this

Create a new file controllers.js in the js directory and include the following code

Add the controller.js file to your index.html by adding

just after

You can now test your app with

and check that when you hit the login button you get sent to the home page.

This just runs in the browser, if you want to learn more about the preparing, building, emulating and running commands for Android and iOS look for Ionic here and Cordova here.

Setting Up Your Testing Environment

The next thing we need to do is set up our testing environment so we have a few things to install. I’m doing the installs locally in the project because I had problems when some items were installed globally and linked to the project using:

So from your myappname directory:

We use:

to make sure that this testing environment is saved as a development dependency in your app’s package.json.

Here are links to the various packages we are installing so you know what they are, why they are there and how to use them in anger. This tutorial will keep their usage pretty basic:

  • appium: Test automation framework for native and hybrid apps using the WebDriver protocol
  • appium-doctor: Tool to verify the appium installation and ensure you are set up correctly for iOS and Android testing
  • authorize-ios: For Authorising the use of the iOS Simulator for automated testing
  • protractor: An end-to-end test framework for AngularJS applications
  • chai: An assertion framework
  • chai-as-promised: Extends Chai for asserting facts from javascript promises
  • cucumber: Behaviour Drive Development framework for creating executable specifications that are documentation and automated tests
  • protractor-cucumber-framework(Scroll down the page): Module for working with cucumber within protractor since the two projects were decoupled
  • wd: WebDriver/Selenium client
  • wd-bridge: A bridge between the wd driver and other selenium clients

The first thing we need to do is make sure that everything is set up correctly for appium. We do this from the app root directory with the command:

Hopefully, you will see something like:

If not correct any errors that appear. I had to set a couple of environment variables the first time I ran it but there is plenty of info online about how to do this.

Next, we need to authorise the use of the iOS simulator for automated testing so run:

Right, we’re ready to rock!

Creating Your Tests

First set up the directory structure:

  • Create a directory in /myappname called ‘tests’
  • Create a directory in /myappname/tests called ‘features’
  • Create a directory in /myappname/tests/features called ‘step_definitions’
  • Create another directory in /myappname/tests/features called ‘support’

Now set up the files you will need:

  • Inside the /myappname/tests directory create a javascript file ‘android-conf.js’
  • Inside the /myappname/tests directory create a javascript file ‘ios-conf.js’. These two will be protractor config files
  • Inside the /myappname/tests/features directory create a new file called ‘login.feature’
  • Inside the /myappname/tests/features/step_definitions directory create a new file called ‘login_page_steps.js’
  • Inside the /myappname/tests/features/support directory create a new file called ‘world.js’

Firstly we use the Gherkin language by specifying what our product will do in plain text, in a way that everyone can understand. So add some text to the login.feature file:

To understand the terminology of Features, Scenarios and the like look at Cucumber’s Gherkin Reference.

Next, set up the Cucumber ‘world’ in world.js

What we have done here is used Chai and Chai As Promised to replace the Protractor expectations which will break when they are used within Cucumber.

Now that we have the world you can do an initial set up of your step definition file. So edit login_page_steps.js and add the following:

Setting Up Protractor for iOS Testing

Next, we need to set up our Protractor config file.

There is plenty more you can do with the config file but let’s just examine this one so you know what it all means.

The ‘seleniumAddress’ is the http location of appium.

The ‘specs’ option shows Protractor where to find your feature files and is wild-carded to find any that you put in the features directory.

The ‘framework’ and ‘frameworkPath’ set up is to tell Protractor to use the Cucumber framework for testing.

The ‘cucumberOpts’ tell Cucumber to use the ‘step_definitions’ and ‘support’ directories to find files relating to running the tests in the feature files. It also tells Cucumber it to format its progress to the Terminal in a very readable way (For other ways to get Cucumber to output its results look here). This is a basic setup and there are other options like running specific tags in your feature files.

The ‘capabilities’ are the configuration for appium. The ‘platformName’, ‘platformVersion’ and ‘deviceName’ specify which simulator you will use. The ‘browserName’ is blank because we will be testing the app rather than the app in the simulators browser. ‘autoWebview’ specifies that we will be using an app that is a hybrid app hence a web view rather than a native app. We specify ‘fullReset’ to true in order to stop appium running the simulator to do some setup, destroying it and recreating it again to test the app. Then we set the ‘app’ to a variable that specifies the location of the app once it has been built. In my set up this is:
‘/Users/myUserName/path/to/my/app directory/myappname/platforms/ios/build/emulator/myappname.app’.

The ‘baseUrl’ is set to empty. Normally it would be set to the url ‘http://10.0.2.2:8000’ this would allow you to navigate around your app as if it were quite simple http (although you might need to set browser.resetUrl = ‘http://’ in the onPrepare function to avoid getting file paths back when querying the current url with the web driver). However, I ran into problems doing this on Android so to fix this and make it completely cross platform I chose to navigate using the file:// protocol.

The Android issues revolved around the necessity to install the Chrome browser in the Android emulators to run appium tests. All the apk’s for the Chrome browser I found would not install with adb, that was because the emulators I created in Android Studio were for the x86 architecture and the apk’s were made for the arm architecture. You can create or change your emulators to run the arm architecture but unfortunately, they run 10 times slower than the x86 emulators on OSX (10x slower is the official figure, personally I found it to be so slow as to invalidate the rationale behind automation testing). I also found it to be incredibly difficult to get the actual Chrome install to take and had to resort to trying to install the Play Store on the emulator in order to try to get Chrome that way. I then encountered great difficulty being able to set the write permissions for the emulator’s file system with the adb shell in order to install the Play Store. The best way I found to get around this was to use Genymotion (more about that later) and simply install an x86/arm bridge (search for ARM Translation v1.1.zip installation instructions) and the Play Store with a simple drag and drop flashable zip file (downloadable here). However, despite these instructions saying that you need to do this, you do not, as this tutorial will show

The ‘onPrepare’ function sets up wd, protractor and the wd-bridge. It also sets up the use of the file:// protocol. The location of the app through appium involves a dynamically created directory structure, as a result we need to figure out where the app is by querying ‘window.location’ and reconstructing the path using the ‘origin’ (which is ‘file://’) and ‘pathname’ (which is the path to your index.html) and then telling Protractor to use the file protocol by setting the ‘resetUrl’ to file:// otherwise it will use http://. The request to ‘window.location’ is an asynchronous call so we need to make a deferred promise because onPrepare has no callback function. Finally, we set ‘browser.ignoreSynchronization = true’ this stops an error where Angular is not found on the index page when navigating using the file:// protocol.

First Couple of Test Runs

Build your app from the apps root directory by running:

And check myappname/platforms/ios/build/emulator/ to see if myappname.app is in there.

Then open a new terminal window and navigate to your apps root directory and start up appium by running:

And you should see:

Start another Terminal window at the root of the app and change directory to be in the tests directory. Then run protractor:

You should see your app running in the simulator that you have chosen and Cucumber will print out the scenarios and their outcomes.

At this point, Cucumber does something really useful. It prints out some code snippets in the Terminal for you to add to your steps file. These code snippets match the Gherkin language of Give, When, Then and has a regular expression to match the natural language in the feature file. E.g.

So, just copy and paste the code snippets into the login_page_steps.js so that it looks like this:

Now when you run

You will get a different result and the steps will be listed as ‘pending’ rather than ‘undefined’ and where a step is pending then the following steps will be skipped, so instead of

you get

Top!

Automating The First Scenario

At this point, we want to start actually do some navigating and checking the results. Let’s look at our first scenario ‘Showing the login page’. Editing the login_page_steps.js change

to

browser.get( ” ) gets the index.html page and because of the line $urlRouterProvider.otherwise(‘/login’) in the app.js we should automatically navigate to the login page.
callback() tells Cucumber we are done with this step.

Next is an assertion ‘Then I should see the login screen’ so we use Chai’s assertion framework to analyse where which page we are on.

becomes

This says to look at the relative url and if it meets the expectation of being on the login page (as specified in the ‘login’ state set up in the config part of app.js).
Then it calls back to Cucumber all is ok via the ‘notify’ keyword.
browser.getLocationAbsUrl() returns a promise so we use Chai As Promised’s ‘eventually’ keyword to manage the asynchronous nature and keep polling until the assertion is true or times out.

When we run Protractor now

the first scenario’s steps should have gone green and the ‘Logging In’ scenario should still be pending. Cucumber should say:

Automating the Login Scenario

Now it’s time to fill in some text boxes and pushing a button. Let’s look at our second scenario ‘Logging in’. Editing the login_page_steps.js change

we put some code into this that is very similar to what we have just done

Then look at When assertion

Here we will need to grab the two text boxes, fill in some info and then click the button. To find the email/username text field we can use Protractor’s ability to find an element by its id (There are plenty of other ways to find things on a page this cheatsheet is a good place to start and the Protractor api can be found here

and then we want to send it some text

next get the password text field and fill that in too

Then we need to grab a reference to the button and click it. Note in the html the doLogin() function is triggered by ng-click. I originally used on-tap events but Protractor’s click() function did not work. I hope to be able to write some more on automating gestures at a later date.

Finally, we need to callback() but not before putting the browser driver to sleep. This is not necessary in this example but when the login call is asynchronous for instance when using Ionic.Auth.login() then I have found that the expectations are not met unless the driver is sent to sleep for a few seconds. It is here commented for reference, if you uncomment it the test will fail because the default timeout is 5 seconds.

You can change the default timeout by adding this.setDefaultTimeout(10 * 1000); to your file or by adding an environment file to the support directory. Because the support directory is required by Cucumber in the Protractor config file in order to load world.js you can add the file env.js with the contents

the final method should look like:

Lastly, we need to sort out the Then assertion ‘I successfully log in and see the home page’. It should check that we are on the home page but we can also look at some the elements on the page to make sure they exist or have certain text. Here are a few things we can do:

when we run Protractor now everything should have gone green and we should see:

iOS Job Done! Have a cup of tea.

Setting Up Protractor for Android Testing

Right, time to edit the android-conf.js in the tests directory. It should look like this

Looks pretty familiar, doesn’t it? In fact, it’s almost identical to the iOS config with a couple of notable differences. The app location is a path to the android debug apk. Naturally, the ‘platformName’ and ‘platformVersion’ in the appium capabilities are for Android and we don’t have a ‘deviceName’ yet. Also, note that we have switched off ‘fullReset’ because we don’t have the problem with the emulator firing up twice.

Let’s fill in those blanks. So, from the root of the app:

When it is done you will get the location of the apk printed to the console. Mine is in the format ‘/Users/username/path/to/myappname/platforms/android/build/outputs/apk/android-debug.apk’. Replace the contents of the appLocation variable with this path.

You can automatically start up an Android simulator by adding the ‘avd’ capability to your protractor config with the name of the Android Virtual Device

but as I mentioned earlier they are slow so I’ve been using Genymotion because it is really fast, much faster than the iOS simulator.

Before installing Genymotion, install VirtualBox then install Genymotion. You can get a free version for personal use here.

Once you’ve got Genymotion installed add a new virtual device.

You will need to get the virtual device’s name, so navigate to your Android sdk, if you don’t know where it is just

Go into the ‘platform-tools’ directory of the Android sdk and run

Copy the url of the device and put it in the ‘deviceName’ of the appium capabilities in your android-conf.js. It should look something like ‘192.168.56.101:5555’.

Run Protractor from your ‘tests’ directory as you did for iOS:

After watching the iOS simulator you’ll notice these tests really fly in Genymotion.

Cross platform hybrid app automation testing, Job Done!

3 thoughts on “Automated cross platform testing of iOS and Android hybrid Ionic/Cordova/Angular apps using Cucumber, Protractor, Chai and Appium

  • HI, this post is great and works fine for IOS in my mac with Sierra and Xcode updated to the latest version Version 8.2.1 (8C1002) but for Android i am not able to run the test using an Android emulator. Here you are the error i am getting: Do you know how to fix that?

    MyUser$ protractor tests/android-conf.js
    [17:29:04] I/hosted – Using the selenium server at http://localhost:4723/wd/hub
    [17:29:04] I/launcher – Running 1 instances of WebDriver
    [17:29:23] E/launcher – An unknown server-side error occurred while processing the command. Original error: unknown error: Chrome version must be >= 53.0.2785.0
    (Driver info: chromedriver=2.25.426935 (820a95b0b81d33e42712f9198c215f703412e1a1),platform=Mac OS X 10.12.2 x86_64)
    [17:29:23] E/launcher – WebDriverError: An unknown server-side error occurred while processing the command. Original error: unknown error: Chrome version must be >= 53.0.2785.0
    (Driver info: chromedriver=2.25.426935 (820a95b0b81d33e42712f9198c215f703412e1a1),platform=Mac OS X 10.12.2 x86_64)
    at WebDriverError (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/error.js:27:5)
    at Object.checkLegacyResponse (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/error.js:639:15)
    at parseHttpResponse (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/http/index.js:538:13)
    at client_.send.then.response (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/http/index.js:472:11)
    at ManagedPromise.invokeCallback_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:1379:14)
    at TaskQueue.execute_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2913:14)
    at TaskQueue.executeNext_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2896:21)
    at asyncRun (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2820:25)
    at /usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:639:7
    at process._tickCallback (internal/process/next_tick.js:103:7)
    From: Task: WebDriver.createSession()
    at Function.createSession (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver.js:329:24)
    at Builder.build (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/builder.js:458:24)
    at Hosted.DriverProvider.getNewDriver (/usr/local/lib/node_modules/protractor/built/driverProviders/driverProvider.js:37:33)
    at Runner.createBrowser (/usr/local/lib/node_modules/protractor/built/runner.js:198:43)
    at /usr/local/lib/node_modules/protractor/built/runner.js:277:30
    at _fulfilled (/usr/local/lib/node_modules/protractor/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/usr/local/lib/node_modules/protractor/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/usr/local/lib/node_modules/protractor/node_modules/q/q.js:796:13)
    at /usr/local/lib/node_modules/protractor/node_modules/q/q.js:556:49
    at runSingle (/usr/local/lib/node_modules/protractor/node_modules/q/q.js:137:13)
    [17:29:23] E/launcher – Process exited with error code 199

Leave a Reply