Yak Shaving Failing Integration Tests with React and Rails

react on railsNovember 26, 2016by Justin Gordon

This Thanksgiving weekend, I decided to a major refresh of the react-webpack-rails-tutorial, including:

Update all Ruby and NPM libraries to the latest, including updating the linters to the latest and some big updates to the react-bootstrap API
Get CI working on both Travis and Codeship.
Yarn, Webpack V2, a react-native client, and Action Cable.

To save others the pain in getting the integration tests working again, here’s my tips. I’m assuming you’re using Capybara and Ruby on Rails.

Use the latest versions, so long as they are supported. Since I recently upgraded to Mac OS Sierra, I generally needed the latest versions. It’s worth noting that selenium-webdriver 3.x is not yet supported by Capybara.

Here’s a few general tips:

Update your JavaScript code to clear out any warnings and errors from the console.
When you have a problem try to narrow it down and make sure it’s reproducible. Does the failing tests pass on it’s own, without the test suite? Maybe there is some order dependency of your specs? One thing to definitely do is to use the random order option for rspec. The order is not totally random. After each run, you’ll see the “seed” and you can use that re-run the specs in exactly the same order. config.order = :random
Consider using the rspec-retry gem. This will allow you to automatically retry any failing tests. If you do use this, when doing local development, you can set your retry count to 1, so you don’t see the same failure N times! export RSPEC_RETRY_RETRY_COUNT=1
Try your best to tweak one thing at a time, such as a spec running configuration, and assess if that causes more or less failures. Yes, this is super tedious when tracking down failures on a CI environment.
Take care that your setup of database cleaner matches up with the current docs.
On the react-webpack-rails-tutorial, we’ve got both Travis and Codeship setups. You’re free to look at those and ask me any questions. The travis setup is in the .travis.yml file. You can see our Codeship setup at the bottom of this article. The only driver that seems to reliably pass every time is selenium. I had high hopes that poltergeist or selenium_chrome would pass, but the wouldn’t. Locally, on my Mac, all 3 main drivers pass.
When testing locally, I prefer poltergeist because of how it will fail if there are any warnings or errors in the logs.

Configure Your Integration Tests to Support Multiple Drivers

If you test setup allows you to switch drivers via an environment variable, that can help identify if a test failure is due to a real issue, or an issue with one of the test drivers.

If you look at our spec/rails_helper.rb, you’ll see support for:

PhantomJS via the poltergeist gem, with options for skipping animations and ignoring errors.
Selenium, both Chrome and Firefox.
Webkit.

Of these, my favorites are PhantomJS and Selenium Chrome. Webkit appears to poorly support the modern JavaScript and React in my React on Rails apps.

PhantomJS

Why It Rocks!

PhantomJS fails a test for any error or warning in the JavaScript logs. Plus you can usually see log issue that caused the test to fail.
PhantomJS, being headless, is fast, especially if animations are turned off. To turn off animations, see this file and this line.

Why It’s Frustrating!

PhantomJS seems riddled with some race conditions that cause it to crash. Some of the times that PhantomJS crashed, there was a real issue in the test. However, some to the times, there was no issue at all. I reported this crashing in this Poltergeist Github Issue. The issues seem especially related to screens with animations, using ReactCSSTransitionGroup and animations regarding opacity. The errors you’ll see include: Capybara::Poltergeist::DeadClient, Errno::EPIPE, and Errno::ECONNRESET.

Workaround to the Crashes

With much fiddling, I developed the code to switch from PhantomJS to Selenium Chrome when PhantomJS crashes. If you’ve got issues with PhantomJS crashing, try adding this file, poltergeist.rb, to your spec/support directory. Here’s a gist of this where you can comment.

  
          # See in use here: 
        
          # https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/spec/support/poltergeist.rb
        
          # This file supports 2 strategies:
        
          # 1. switch_to_selenium: switch drivers
        
          # 2. restart_poltergeist
        
          RESTART_PHANTOMJS = ENV["RESTART_PHANTOMJS"] &&
        
                              %w(TRUE YES).include?(ENV["RESTART_PHANTOMJS"].upcase)
        
          puts "RESTART_PHANTOMJS = #{RESTART_PHANTOMJS}"
        
          CAPYBARA_TIMEOUT_RETRIES = 5
        
          # HACK: workaround for Capybara Poltergeist StatusFailErrors, simply retries
        
          # based on https://gist.github.com/afn/c04ccfe71d648763b306
        
          RSpec.configure do |config|
        
            config.around(:each, type: :feature) do |ex|
        
              example = RSpec.current_example
        
              use_selenium = false
        
              original_driver = Capybara.default_driver
        
              CAPYBARA_TIMEOUT_RETRIES.times do
        
                example.instance_variable_set("@exception", nil)
        
                # Private method in rspec:
        
                # rspec-core-3.5.4/lib/rspec/core/memoized_helpers.rb:139
        
                __init_memoized
        
                if use_selenium
        
                  puts "Switching to selenium from #{Capybara.current_driver}"
        
                  Capybara.current_driver = js_selenium_driver
        
                  Capybara.javascript_driver = js_selenium_driver
        
                end
        
                ex.run
        
                example_ex = example.exception
        
                break unless example_ex
        
                is_multiple_exception = example_ex.is_a?(RSpec::Core::MultipleExceptionError)
        
                break unless example_ex.is_a?(Capybara::Poltergeist::StatusFailError) ||
        
                             example_ex.is_a?(Capybara::Poltergeist::DeadClient) ||
        
                             is_multiple_exception
        
                if is_multiple_exception
        
                  m_exceptions = example_ex.all_exceptions
        
                  idx = m_exceptions.find_index do |exception|
        
                    exception.is_a?(Capybara::Poltergeist::StatusFailError) ||
        
                      exception.is_a?(Capybara::Poltergeist::DeadClient) ||
        
                      exception.class < SystemCallError
        
                  end
        
                  break unless idx
        
                end
        
                puts "\n"
        
                puts "=" * 80
        
                puts "Exception caught! #{example_ex.ai}"
        
                puts "when running example:\n  #{example.full_description}"
        
                puts "  at #{example.location} with driver #{Capybara.current_driver}."
        
                if RESTART_PHANTOMJS
        
                  PhantomJSRestart.call
        
                else
        
                  use_selenium = true
        
                end
        
                puts "=" * 80
        
              end
        
              Capybara.current_driver = original_driver
        
              Capybara.javascript_driver = original_driver
        
              Capybara.use_default_driver
        
            end
        
          end
        
          # Rather than using switching to use selenium, we could have restarted Phantomjs
        
          module PhantomJSRestart
        
            def self.call
        
              puts "Restarting phantomjs: iterating through capybara sessions..."
        
              session_pool = Capybara.send("session_pool")
        
              session_pool.each do |mode, session|
        
                msg = "  => #{mode} -- "
        
                driver = session.driver
        
                if driver.is_a?(Capybara::Poltergeist::Driver)
        
                  msg += "restarting"
        
                  driver.restart
        
                else
        
                  msg += "not poltergeist: #{driver.class}"
        
                end
        
                puts msg
        
              end
        
            end
        
          end

view raw poltergeist.rb hosted with ❤ by GitHub

Dependency Updates

You will want to ensure you have the current versions of both PhantomJS and the Chrome Driver.

Phantomjs

Update to 2.1.1 with:

brew update && brew upgrade phantomjs

Your CI system should use these steps, from .travis.yml:

**before_install**:

  - mkdir $PWD/travis-phantomjs

  - curl -sSL https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2 -o $PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2

  - tar -xvf $PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2 -C $PWD/travis-phantomjs

  - export PATH=$PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64/bin:$PATH

On CodeShip, you will need to set the PHANTOMJS_VERSION environment variable to 2.1.1.

Chrome Driver

If you’re using the Rubygem chromedriver-helper, then you should run the command chromedriver-update. You can run that command after you run bundle in your CI setup. However, you might have installed via brew, so you should update via brew update && brew upgrade chromedriver.

Codeship Setup

Setup Commands

export PHANTOMJS_HOST=”https://s3.amazonaws.com/codeship-packages"

export PHANTOMJS_VERSION=2.1.1

\curl -sSL https://raw.githubusercontent.com/codeship/scripts/master/packages/phantomjs.sh | bash -s

# We support all major ruby versions: 1.9.3, 2.0.0, 2.1.x, 2.2.x and JRuby

rvm use 2.3.1

bundle config build.nokogiri — use-system-libraries

bundle install

chromedriver-update

nvm install stable && nvm alias default stable

npm install npm@latest -g

npm install

cd client && npm run build:client && npm run build:server

cd ..

export RAILS_ENV=test

bundle exec rake db:schema:load

Test Commands

COVERALLS_REPO_TOKEN=YynC2SEiNITamhqXRCpzDzlmeuoVqdFgp DRIVER=selenium_chrome bundle exec rake

bundle exec rake lint

Closing Remark

Could your team use some help with topics like this and others covered by ShakaCode's blog and open source? We specialize in optimizing Rails applications, especially those with advanced JavaScript frontends, like React. We can also help you optimize your CI processes with lower costs and faster, more reliable tests. Scraping web data and lowering infrastructure costs are two other areas of specialization. Feel free to reach out to ShakaCode's CEO, Justin Gordon, at justin@shakacode.com or schedule an appointment to discuss how ShakaCode can help your project!

Justin Gordon

Share this article

Are you looking for a software development partner who can

develop modern, high-performance web apps and sites?

See what we've done

	# See in use here:
	# https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/spec/support/poltergeist.rb

	# This file supports 2 strategies:
	# 1. switch_to_selenium: switch drivers
	# 2. restart_poltergeist

	RESTART_PHANTOMJS = ENV["RESTART_PHANTOMJS"] &&
	%w(TRUE YES).include?(ENV["RESTART_PHANTOMJS"].upcase)
	puts "RESTART_PHANTOMJS = #{RESTART_PHANTOMJS}"

	CAPYBARA_TIMEOUT_RETRIES = 5

	# HACK: workaround for Capybara Poltergeist StatusFailErrors, simply retries
	# based on https://gist.github.com/afn/c04ccfe71d648763b306
	RSpec.configure do \|config\|
	config.around(:each, type: :feature) do \|ex\|
	example = RSpec.current_example
	use_selenium = false
	original_driver = Capybara.default_driver
	CAPYBARA_TIMEOUT_RETRIES.times do
	example.instance_variable_set("@exception", nil)

	# Private method in rspec:
	# rspec-core-3.5.4/lib/rspec/core/memoized_helpers.rb:139
	__init_memoized

	if use_selenium
	puts "Switching to selenium from #{Capybara.current_driver}"
	Capybara.current_driver = js_selenium_driver
	Capybara.javascript_driver = js_selenium_driver
	end

	ex.run

	example_ex = example.exception

	break unless example_ex

	is_multiple_exception = example_ex.is_a?(RSpec::Core::MultipleExceptionError)

	break unless example_ex.is_a?(Capybara::Poltergeist::StatusFailError) \|\|
	example_ex.is_a?(Capybara::Poltergeist::DeadClient) \|\|
	is_multiple_exception

	if is_multiple_exception
	m_exceptions = example_ex.all_exceptions

	idx = m_exceptions.find_index do \|exception\|
	exception.is_a?(Capybara::Poltergeist::StatusFailError) \|\|
	exception.is_a?(Capybara::Poltergeist::DeadClient) \|\|
	exception.class < SystemCallError
	end

	break unless idx
	end

	puts "\n"
	puts "=" * 80
	puts "Exception caught! #{example_ex.ai}"
	puts "when running example:\n #{example.full_description}"
	puts " at #{example.location} with driver #{Capybara.current_driver}."

	if RESTART_PHANTOMJS
	PhantomJSRestart.call
	else
	use_selenium = true
	end
	puts "=" * 80
	end
	Capybara.current_driver = original_driver
	Capybara.javascript_driver = original_driver
	Capybara.use_default_driver
	end
	end

	# Rather than using switching to use selenium, we could have restarted Phantomjs
	module PhantomJSRestart
	def self.call
	puts "Restarting phantomjs: iterating through capybara sessions..."
	session_pool = Capybara.send("session_pool")
	session_pool.each do \|mode, session\|
	msg = " => #{mode} -- "
	driver = session.driver
	if driver.is_a?(Capybara::Poltergeist::Driver)
	msg += "restarting"
	driver.restart
	else
	msg += "not poltergeist: #{driver.class}"
	end
	puts msg
	end
	end
	end