DocuSign Dev Blog

Brought to you by the development teams at DocuSign

Jeeves, an Open Source Webdriver

Our previous post covered a high-level overview of the tech stack behind the New DocuSign Experience (or as it’s still called internally, Martini). Today, we’re excited to announce the release of our open source webdriver, Jeeves, a tool that helps ease the pain of writing automated browser UI tests. This tool is the cornerstone of our test framework and drives nearly all of our UI automation for the New DocuSign Experience. This post explains the reasons why we built it, and how it differs from other currently available tools.

Why did we build Jeeves?

To fully understand why we built Jeeves, it will help to explain the problems we faced before it was built. With any large-scale enterprise software product, automated testing is an absolute must. This is especially true if the goal is to rapidly iterate based on feedback and release often. Initially our automated test suite used the official selenium-webdriver module. We experienced flakiness with its promises implementation, and our thin abtraction layer over it was rife with explicit waits, inconsistent interfaces, and unexplained timeouts which translated to an unreliable test suite. This led to extremely low productivity for our QA engineers while they spent time trying to discern if a failure was real or just the result of some unexpected timeout. Since there were only two QA engineers on the Martini team at that time, all the time we wasted chasing down intermittent test failures due to problems with the framework meant that we were struggling to keep up with our automation coverage goals.

Frustrated by the problems with the existing test framework architecture, I wanted to design something that would not only solve our problems with test reliability, but also reduce complexity and make it easier for non-QA engineers to write and debug tests. To improve our test reliability, we needed to start with a foundation that is maintained regularly and works across different browsers. The obvious choice to fill this need was wd, which implements the Selenium JSON Wire Protocol while still supporting PhantomJS. We also needed a solution that would work with our existing tests (all written in the Node.js callback style), and we wanted something that would simplify the introduction to our test framework for new team members.

How is Jeeves different?

With the amazing wd module at the core of Jeeves, the logical question you might ask is, “Why not just wd instead of building a wrapper?” While this is a valid concern, wd alone did not strike at the heart of the secondary problem I was trying to solve: to make writing UI tests quick & easy. While wd is extremely flexible in terms of the browser & element interactions it provides, it still lacks some nice-to-have features which can somewhat deter or hinder the average feature developer from effectively writing tests with it.

Jeeves answers this by offering some helpful additions which include:

  • Detailed logging on all method calls, and optionally displayed logging from wd itself.
    • Extra logging allows for quicker debugging from the console output, even while the tests are still running!
  • Simple abstraction for writing a named series of steps
  • Hassle-free screenshots! Jeeves.takeScreenshot handles saving the raw image buffer to a configurable directory.
    • Saving screenshots from image buffers is not hard, but it shouldn’t clutter your test code
  • Jeeves also implements two methods for element drag & drop even though this feature is not directly supported by wd or Selenium JSON Wire Protocol
  • Jeeves provides two ways for interacting with elements.
    • You can use the traditional method of finding the element first, and then calling a separate method to interact with it.
      • E.g.:
        it 'should click the div', (done) ->
          Jeeves.getElementByCss 'div.clickable', (err, elem) ->
            Jeeves.clickElement elem, (err) ->
              done err
    • Or you can append one of the convenience suffixes to make the method do the element finding for you.
      • Such as:
        • ByClassName
        • ByCssSelector
        • ById
        • ByName
        • ByLinkText
        • ByPartialLinkText
        • ByTagName
        • ByXPath
        • ByCss
      • E.g.:
        it 'should click the div', (done) ->
          Jeeves.clickElementByCss 'div.clickable', (err) ->
            done err
  • UI tests are not synchronous, so we need to wait for elements, URL changes, and the occasional attribute.
    • Jeeves extends wd‘s waitFor pattern to wait for more than just elements:
      • Jeeves.waitForUrlToChange
      • Jeeves.waitForVisibleElementBy...
      • Jeeves.waitForElementToHideBy...
      • Jeeves.waitForAttributeBy...
      • Jeeves.waitForAnyTextBy...
      • Jeeves.waitForSpecificTextBy...
      • Jeeves.waitForAndGetTextBy...
      • Jeeves.waitForAndClickElementBy...

These enhancements which Jeeves adds on top of the reliable wd core have proven to be powerful tools for maintaining a large suite of automated browser UI tests. Check out our examples to see how you can get started.

comments powered by Disqus