Optimized, Parallelized CircleCI Configuration for ReactOnRails

At ShakaCode, our internal app, FriendsAndGuests, tends to be our guinea pig for the bleeding-edge use cases of ReactOnRails. Just getting CI to work was no exception. There are a lot of workarounds and “gotchas,” but this should hopefully save you the pain I went through. We chose to use CircleCI as our CI provider and CodeCov as our code coverage provider. This guide assumes you’ve read through the basic documentation for both.

Parallelization

When your app starts to scale, your test suite is inevitably going to get slower. Your first line of defense should always be to stay judicious with what and how you test (see Martin Fowler’s Test Pyramid). Still, this slowdown is inevitable with growth. CircleCI has a great feature called parallelization that allows you to easily configure your tests to run across multiple containers. Parallelizing your build can greatly increase your speed.

If your testing frameworks can be configured to use JUnit reporters (as RSpec, Jest, and ESlint can), CircleCI will read those reports, figure out the median time to test each file, and then properly distribute those tests across the containers. Distributing the files in this way ensures that one container doesn’t end up with only fast unit tests and the other with only slow feature specs. You want the containers to finish their suites around the same time—you’re only as fast as your slowest container.

As a bonus, if any tests fail, they show up in an easy-to-read format right at the top of the page. That means no more searching through endless lines of noise to find your failing test.

If your tool does not have a JUnit reporter but still allows for passing files via the command line in a space-separated format (by the way, scss-lint, slim-lint, and rubocop all let you do this), then you can still use parallelization! The only difference here is that CircleCI won’t be able to tell how long each file takes to be run through the tool, so it falls back to using file size and you may not get as well-calibrated division of the files across the containers.

Here’s a list of tools we currently use where we can’t use parallelization:

• Brakeman
• flow
• Bundle-audit

CodeCov Flags

One of the cool new features coming out of CodeCov is the use of flags. They allow you to easily split up the coverage of different test suites.

I’ve got guys on my team who only do backend and likewise guys who only do frontend, and they don’t care about each others’ code coverage, so why combine the two? Flags allow you to do that.

circle.yml

There’s a lot going on here, make sure you’re up to speed with the basics of how CircleCI works and what different options mean. Also, watch out for those pwd: client lines, that means we’re running all of the commands in the “present working directory” of the client folder.

machine:

  environment:

    RAILS_ENV: test

    RACK_ENV: test

    YARN_VERSION: 0.24.6

    PATH: "${PATH}:${HOME}/.yarn/bin"

  node:

    version: 7.7.3

dependencies:

  pre:

    # Install Yarn

    - |

      if [[ ! -e ~/.yarn/bin/yarn || $(yarn --version) != "${YARN_VERSION}" ]]; then

        echo "Download and install Yarn."

        curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version $YARN_VERSION

      else

        echo "The correct version of Yarn is already installed."

      fi

  cache_directories:

    - "~/.yarn"

    - "~/.cache/yarn"

  override:

    - yarn install --no-progress --no-emoji

    - bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3

  post:

    - yarn run build:all:rspec:

        pwd: client

    - mkdir -p $CIRCLE_TEST_REPORTS/jest

    # HACK: junit-reporter fails if file doesn't already exist

    - touch $CIRCLE_TEST_REPORTS/jest/test-results.xml

test:

  override:

    - bundle exec rspec -r rspec_junit_formatter --format progress --format RspecJunitFormatter -o $CIRCLE_TEST_REPORTS/rspec/junit.xml:

        parallel: true

        environment:

          CODECOV_FLAG: backend # tags all Ruby tests

        files:

          - spec/**/*_spec.rb

  post:

    # HACK: for why maxWorkers is needed, see https://github.com/facebook/jest/issues/535#issuecomment-171445481

    - $(yarn bin)/jest --testResultsProcessor jest-junit-reporter --coverage --maxWorkers 3:

        pwd: client

        environment:

          NODE_ENV: test

          TEST_REPORT_PATH: $CIRCLE_TEST_REPORTS/jest # used by jest-junit-reporter

        files:

          - 'app/**/*.spec.js'

          - 'app/**/*.spec.jsx'

    - $(yarn bin)/codecov -p .. --F frontend:

        pwd: client

        parallel: true

    # Other stuff such as rubocop, eslint, flow, scss-lint, brakeman, bundle-audit

general:

  artifacts:

    - tmp/capybara # great for debugging feature test screenshots

    - log/test.log # sometimes needed for hard-to-track bugs

    - tmp/brakeman-report.html # if you're using brakeman

yarn

Since we’re using yarn instead of npm, we need to use override during the dependencies step to avoid installing via npm. That means we now need to manually run bundle check and bundle install as well.

We deviate a bit from the docs regarding yarn because sometimes we want to install newer versions of yarn than what comes with the CircleCI container. We use a simple shell script and cache technique that CircleCI itself used to recommend before they added built-in yarn support. In the post step, we build our webpack bundles ahead of time.

RSpec

Although CircleCI can automatically detect we are using Ruby and configure itself correctly, we have to override this and run RSpec ourselves because we want to specify the custom environment variable CODECOV_FLAG so that CodeCov will know that these are backend tests.

Jest

Next, we run Jest. We’re not using our normal test script from the package.json because we need to do some special things for CI. Jest has a weird issue on CircleCI where it can go over memory if you don’t limit the number of workers. To get around this, we can specify that the maxWorkers is 3. While we certainly could run Jest in parallel, there seems to be some type of strange issue with how the code coverage results get merged together across containers, so opt not to do it here.

Jest JUnit Reporter

In order to get JUnit output (again, this allows for those nice little errors at the top of the page), we need to specify using jest-junit-reporter via the --testResultsProcesser flag. The test results file must be put in a place where CircleCI can find it, however, so we need to use the special TEST_REPORT_PATH that the jest-junit-reporter package will pick up (docs). Unfortunately, there seems to be a bug where if the file does not already exist, the process fails, so we need to put create a blank file at that location first using the touch command.

Jest Coverage

Then, we tell Jest to collect coverage with the --coverage flag. After the tests run, we can use the codecov package to upload the results. We must take care to tell it that the root of the project is actually a folder level up using -p .., otherwise CodeCov will get confused and think that your Rails app folder is the same folder as the client/app folder! While we’re at it, we pass the -F frontend flag to tag our coverage as frontend.

Note: Don’t use --root or --flags as the CLI help recommends; these don’t actually work. Use the short versions I listed above. Also, see comment in the “Jest” section about parallelization needing to be turned off due to coverage report merge problems.

Codecov.yml

coverage:

  status:

    project: off

    patch:

      frontend:

        flags: frontend

      backend:

        flags: backend

flags:

  backend:

    paths:

      - app

      - lib

      - db

  frontend:

    paths:

      - client

Here’s how we set CodeCov to give us separate frontend and backend coverage status (I prefer to only see the patch status so I turned project off). Note that the partial line coverage feature seems to cause “file not found in report” errors in CodeCov when trying to view your files, so I don’t enable it here.

client/package.json

// ...

"devDependencies": {

  // ...

  "codecov": xxx,

  "identity-obj-proxy": xxx,

  "jest": xxx,

  "jest-junit-reporter": xxx,

},

"jest": {

    "collectCoverageFrom": [

      "app/**/*.{js,jsx}",

      "!**/types/**"

    ],

    "coverageReporters": [

      "lcov"

    ],

    "moduleNameMapper": {

      "\\.(css|scss)$": "identity-obj-proxy"

    },

    "resetModules": true,

    "resetMocks": true,

    "roots": [

      "<rootDir>/app/"

    ]

  }

}

Most of this is self-explanatory if you’ve read the documentation. We are using the identity-obj-proxy package to fake our css and Sass files during Jest testing since Webpack usually handles that. You may need to make a file mock for other types of imports. Babel support is automatically included in Jest’s core, so no need to configure anything there.

Note: You may also need to set up any aliases you’ve made with Webpack, but my colleague Alex Fedoseev set us up with babel-plugin-module-resolver and has eliminated our need to do that!

Additionally, the Jest coverage output can get kind of spammy because it spits the entire coverage file to STDOUT, so we override that with coverageReporters so that it uses lcov only (this is the type that’s needed by CodeCov). No more spammy output!

spec/rails_helper.rb

if ENV["CI"]

  require "simplecov"

  require "codecov"

SimpleCov.formatter = SimpleCov::Formatter::Codecov

  SimpleCov.start("rails")

end

unless ENV["CI"]

  ReactOnRails::TestHelper.configure_rspec_to_compile_assets(config)

end

Here we startup SimpleCov with CodeCov as the formatter. In addition, we use ReactOnRails’s test helper in development to make sure we don’t forget to run our tests against stale Webpack bundles. However, we already made sure to build the bundles in our circle.yml config, so we can skip this step when in CI.

Gemfile

# ...

group :test do

  gem "codecov", require: false

  gem "rspec_junit_formatter"

end

Self-explanatory, and that’s it!

bonus: flow

If you use Flow, it can take a while to initialize the server. You can tell CircleCI to start up the server in the background at the beginning of your test suite. Assuming you’ve got enough jobs going on (so that you avoid a race condition)your server finishes starting up, then when you start your Flow check, it should be pretty much instantaneous.

test:

  pre:

    - $(yarn bin)/flow start: # start up flow server in background so it's ready to go

        pwd: client

        background: true