This Thanksgiving weekend, I decided to a major refresh of the react-webpack-rails-tutorial, including:
- Update all Ruby and NPM libraries to the latest, including updating the linters to the latest and some big updates to the react-bootstrap API
- Get CI working on both Travis and Codeship.
- Yarn, Webpack V2, a react-native client, and Action Cable.
To save others the pain in getting the integration tests working again, here’s my tips. I’m assuming you’re using Capybara and Ruby on Rails.
Use the latest versions, so long as they are supported. Since I recently upgraded to Mac OS Sierra, I generally needed the latest versions. It’s worth noting that selenium-webdriver 3.x is not yet supported by Capybara.
Here’s a few general tips:
- When you have a problem try to narrow it down and make sure it’s reproducible. Does the failing tests pass on it’s own, without the test suite? Maybe there is some order dependency of your specs? One thing to definitely do is to use the random order option for rspec. The order is not totally random. After each run, you’ll see the “seed” and you can use that re-run the specs in exactly the same order. config.order = :random
- Consider using the rspec-retry gem. This will allow you to automatically retry any failing tests. If you do use this, when doing local development, you can set your retry count to 1, so you don’t see the same failure N times! export RSPEC_RETRY_RETRY_COUNT=1
- Try your best to tweak one thing at a time, such as a spec running configuration, and assess if that causes more or less failures. Yes, this is super tedious when tracking down failures on a CI environment.
- Take care that your setup of database cleaner matches up with the current docs.
- On the react-webpack-rails-tutorial, we’ve got both Travis and Codeship setups. You’re free to look at those and ask me any questions. The travis setup is in the .travis.yml file. You can see our Codeship setup at the bottom of this article. The only driver that seems to reliably pass every time is selenium. I had high hopes that poltergeist or selenium_chrome would pass, but the wouldn’t. Locally, on my Mac, all 3 main drivers pass.
- When testing locally, I prefer poltergeist because of how it will fail if there are any warnings or errors in the logs.
Configure Your Integration Tests to Support Multiple Drivers
If you test setup allows you to switch drivers via an environment variable, that can help identify if a test failure is due to a real issue, or an issue with one of the test drivers.
If you look at our spec/rails_helper.rb, you’ll see support for:
- PhantomJS via the poltergeist gem, with options for skipping animations and ignoring errors.
- Selenium, both Chrome and Firefox.
Why It Rocks!
- PhantomJS, being headless, is fast, especially if animations are turned off. To turn off animations, see this file and this line.
Why It’s Frustrating!
PhantomJS seems riddled with some race conditions that cause it to crash. Some of the times that PhantomJS crashed, there was a real issue in the test. However, some to the times, there was no issue at all. I reported this crashing in this Poltergeist Github Issue. The issues seem especially related to screens with animations, using ReactCSSTransitionGroup and animations regarding opacity. The errors you’ll see include: Capybara::Poltergeist::DeadClient, Errno::EPIPE, and Errno::ECONNRESET.
Workaround to the Crashes
With much fiddling, I developed the code to switch from PhantomJS to Selenium Chrome when PhantomJS crashes. If you’ve got issues with PhantomJS crashing, try adding this file, poltergeist.rb, to your spec/support directory. Here’s a gist of this where you can comment.
You will want to ensure you have the current versions of both PhantomJS and the Chrome Driver.
Update to 2.1.1 with:
brew update && brew upgrade phantomjs
Your CI system should use these steps, from .travis.yml:
- mkdir $PWD/travis-phantomjs
- curl -sSL https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2 -o $PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
- tar -xvf $PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2 -C $PWD/travis-phantomjs
- export PATH=$PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64/bin:$PATH
On CodeShip, you will need to set the PHANTOMJS_VERSION environment variable to 2.1.1.
If you’re using the Rubygem chromedriver-helper, then you should run the command chromedriver-update. You can run that command after you run bundle in your CI setup. However, you might have installed via brew, so you should update via brew update && brew upgrade chromedriver.
\curl -sSL https://raw.githubusercontent.com/codeship/scripts/master/packages/phantomjs.sh | bash -s
# We support all major ruby versions: 1.9.3, 2.0.0, 2.1.x, 2.2.x and JRuby
rvm use 2.3.1
bundle config build.nokogiri — use-system-libraries
nvm install stable && nvm alias default stable
npm install npm@latest -g
cd client && npm run build:client && npm run build:server
bundle exec rake db:schema:load
COVERALLS_REPO_TOKEN=YynC2SEiNITamhqXRCpzDzlmeuoVqdFgp DRIVER=selenium_chrome bundle exec rake
bundle exec rake lint