Ryan Grenz

Introduction

As BDD frameworks go, Behat has got to be one of the best.. of any language!

When it first caught my eye at the start of this year, I noticed that it had been really well thought out programmatically, had bags of potential as a Cucumber-alternative, and already suited me greatly as a PHP developer. However, it was a beta product, lacking in documentation, and a little rough around the edges.
It was for this reason I wasn’t able to fully convince my team at BSkyB that we should start using it (we voted for JBehave/Selenium – something which we are now regretting!). Instead, I set about experimenting with it in my spare time to bring it back to the table as a viable option again later (very soon in fact!).

Anyway 6 months on, Behat’s author Konstantin Kudryashov has done an amazing job of progressing the framework, and has succeeded in adding some powerful new features.
Mink is one of them: A decoupled emulated browser abstraction layer which, by default, ties in Goutte client (headless PHP-based emulated browser, not meant for javascript websites), or Sahi (a brilliantly strong, but lesser known Selenium competitor) for using real browsers for testing.
You can also add or create any other PHP drivers you want too, using the neat interfaces Konstantin has created.

Hmm.. Only real browsers with Sahi you say? Well, today that is not strictly true.

I recently discovered PhantomJS and was very impressed with what you can do with it. PhantomJS is a full-stack headless WebKit browser, with full Javascript support and, as of v1.1.0, network proxy support. Bingo! I immediately set about trying to get this working with Behat, using Mink/Sahi as the test broker.

OK, enough introduction. Lets see some action!

Context

For me personally, I write and execute my Behat tests in OSX, but my Sahi and PhantomJS instance are on a Windows XP machine somewhere else on the network.
As the application we are developing at BSkyB needs to work primarily in Firefox 4 and Google Chrome, I wanted to experiment to see what differences were present between tests run with PhantomJS and Google Chrome itself.

Criteria like:

• Do the tests pass or fail consistently between Phantom and Chrome?
• Is the execution time of tests less with PhantomJS compared to Chrome?
• Is the browser rendering identical between Phantom and Chrome?

In theory, the answer to the above 3 questions should be yes.

As a side note, I should also point out that we use PhantomJS as a test-runner for our QUnit based javascript unit tests already (and possibly some Jasmine BDD tests to test frontend behaviours of some of our JS components in isolation in future), so was keen to see if it would be viable to reuse PhantomJS in a different context to run our full BDD tests too.

Setting up Sahi and PhantomJS

Because Sahi operates browsers through a network proxy, Sahi and any browser instances can be spread onto separate machines on your network, running on any combination of OS. However, for Sahi to setup and teardown browser instances, Sahi and the browsers currently need to be resident on the same machine.
(I haven’t tried it, but you should be able to use Sahi SIDs you define yourself, for eg. CHROME-OSX for Chrome on OSX, and CHROME-XP for Chrome on WinXP, and have the browsers permanently running on each separate machine for Sahi to use. If anyone has tried this successfully, please let me know.)

On that note, I downloaded and installed PhantomJS 1.1.0 at C:\phantomjs, and Sahi 3.5 at C:\sahi, both on my WinXP machine.

Now we need to add a special script which PhantomJS uses to setup the browser testing environment.
I create a new file at C:\phantomjs\phantom-sahi.js, with the following javascript inside:

if (phantom.state.length === 0) {
    if (phantom.args.length === 0) {
        console.log('Usage: sahi.js ');
        phantom.exit();
    } else {
        var address = unescape(phantom.args[0]);
        phantom.state = "sahi script running";
        console.log('Loading ' + address);
        phantom.open(address);
    }
} else {
    if (phantom.loadStatus == 'success') {
        console.log('Page title is ' + document.title);
    } else {
        console.log('FAIL to load the address');
    }
}

Next, I add PhantomJS as an available browser to Sahi, and ensure it uses the phantom-sahi.js file I just created to instantiate the browser and open the Sahi setup URL.

To do this, edit the C:\sahi\userdata\config\browser_types.xml file and add the following piece of XML inside the browserTypes block:

	phantomjs
	PhantomJS
	safari.png
C:\phantomjs\phantomjs.exe
--proxy=localhost:9999 C:\phantomjs\phantom-sahi.js
phantomjs.exe
	100
	true

Notice the use of PhantomJS’s network proxy here

Behat Configuration

Back over on my OSX box, I needed to configure behat.yml to point Sahi commands at my WinXP box (192.168.56.102), and set the browser to PhantomJS.

default:
    environment:
        parameters:
            start_url:    http://google.com
            browser:      phantomjs
            sahi:
              host:				192.168.56.102
#              sid:				PHANTOMTEST

NOTE: the Sahi SID there is an optional setting (hence the commenting out). I use it so that I can do console level debugging from PhantomJS directly. More about this later.

Next, create a feature file like this:

Feature: Do a Google search
  In order to find pages about Behat
  As a user
  I want to be able to use google.com to locate search results

    @javascript
    Scenario: I search for Behat
     Given I fill in "Behat github" for "q"
      When I press "Google Search"
      Then I should see "Trying to use Mink"

All the steps have been written now, so lets run it :

⚡ behat
Feature: Do a Google search
  In order to find pages about Behat
  As a user
  I want to be able to use google.com to locate search results

  @javascript
  Scenario: I search for Behat             # features/test/simple.feature:7
    Given I fill in "Behat github" for "q" # /usr/local/Cellar/php/5.3.6/lib/php/mink/src/Behat/Mink/Integration/steps/mink_steps.php:35
    When I press "Google Search"           # /usr/local/Cellar/php/5.3.6/lib/php/mink/src/Behat/Mink/Integration/steps/mink_steps.php:23
    Then I should see "Trying to use Mink" # /usr/local/Cellar/php/5.3.6/lib/php/mink/src/Behat/Mink/Integration/steps/mink_steps.php:62

1 scenario (1 passed)
3 steps (3 passed)
0m1.634s

There we have it, Behat can now run tests through Sahi to PhantomJS.

PhantomJS screenshots when Behat steps fail

Debugging failed steps is kind of tricky. Sure, you can just rerun the same test suite in Google Chrome to get a better idea.
However I intend to execute these tests in Hudson, and would be nice to generate Behat HTML reports, with a screenshot of the failure somehow attached.

I thought hard about how Behat and Phantom could play together to make Phantom call phantom.render(…) to generate me a screengrab,

I could think of only 2 ideas:

• simply get Behat to send a Sahi _eval command to call phantom.render() in the browser (Yes, currently the ‘phantom’ object is available in the browser session, but I think it will soon be removed for security)
• or get Phantom to dynamically create a hidden DOM element in the page being tested, and get Behat to tell Sahi to put some content inside it. Then, register an element.onchange event to take the screenshot

Option 2 is the one I ran with.
By the way, I did try and use Object.watch to monitor a javascript property change instead of using physical page elements. Unfortunately this is only directly implemented by Mozilla.

First, I modified by C:\phantomjs\phantom-sahi.js file to the following:

if (phantom.state.length === 0) {
    if (phantom.args.length === 0) {
        console.log('Usage: sahi.js ');
        phantom.exit();
    } else {
        var address = unescape(phantom.args[0]);
        phantom.state = "sahi script running";
        console.log('Loading ' + address);
        phantom.viewportSize = {width: 1024, height: 768};
        phantom.open(address);
    }
} else {
    if (phantom.loadStatus == 'success') {
        console.log('Page title is ' + document.title);

	// Insert Behat state register
	behatStateElement = document.createElement('input');
	behatStateElement.setAttribute('type', 'text');
	behatStateElement.setAttribute('style', 'display:none');
	behatStateElement.setAttribute('id', 'BEHAT_STATE');
	document.body.appendChild(behatStateElement);

	// Now add event listener to changes on it
	behatState = document.getElementById('BEHAT_STATE');
	behatState.onchange = function() {
		phantom.render(behatState.value);
	}

    } else {
        console.log('FAIL to load the address');
    }
}

As you can see, the modifications I made are to dynamically insert an input[type='text'] element with id=’BEHAT_STATE’ to the bottom of the body tag.
An onchange event is registered on that element so that when a Behat step fails, Sahi will change the innerHTML of the element and trigger the event in PhantomJS.
Once the event is triggered, PhantomJS simply renders the screenshot to a file.
You’ll also notice I need to set the viewport width and height to something. You need to do this otherwise phantom.render won’t do anything.

Next, I created a Behat afterStep hook, to catch any step failures and get Sahi to work some magic:

$hooks->afterStep('', function($event) {

    $environment = $event->getEnvironment();

    if ($environment->getParameter('browser') == 'phantomjs' && $event->getResult() == StepEvent::FAILED) {

        $environment->getClient()->findById('BEHAT_STATE')->setValue('failshot.png');
    }
});

To test this, lets first purposely break our scenario as follows:

Feature: Do a Google search
  In order to find pages about Behat
  As a user
  I want to be able to use google.com to locate search results

    @javascript
    Scenario: I search for Behat
     Given I fill in "Behat github" for "q"
      When I press "Google Search"
      Then I should see "something that isnt there"

Lets see this in action now:

⚡ behat
Feature: Do a Google search
  In order to find pages about Behat
  As a user
  I want to be able to use google.com to locate search results

  @javascript
  Scenario: I search for Behat                    # features/test/simple.feature:7
    Given I fill in "Behat github" for "q"        # /usr/local/Cellar/php/5.3.6/lib/php/mink/src/Behat/Mink/Integration/steps/mink_steps.php:35
    When I press "Google Search"                  # /usr/local/Cellar/php/5.3.6/lib/php/mink/src/Behat/Mink/Integration/steps/mink_steps.php:23
    Then I should see "something that isnt there" # /usr/local/Cellar/php/5.3.6/lib/php/mink/src/Behat/Mink/Integration/steps/mink_steps.php:62
      Failed asserting that  matches PCRE pattern "/something that isnt there/".

1 scenario (1 failed)
3 steps (2 passed, 1 failed)
0m1.634s

BOOM!

And over on our PhantomJS/Sahi server, you’ll see a freshly squeezed screenshot generated:

To Do

What if the Sahi/PhantomJS server is not the same machine as the one the Behat tests are executed on?

This makes attaching the screenshot to the HTML report quite difficult since they are not shared on the same machine.
I think the easiest thing for now is to run a webserver on the same box as Phantom and Sahi, and modify the HTML report formatter to include the failshot.
The filename should be more reflective of the Sahi SID (eg. failshot-3d477f12.png) to ensure each report references the right failshot.
I don’t like the idea of the screenshots being on a separate machine to where the reports are though.

Some more thinking to be done here yet. I’ll update this post when I have more information.