JetBlack

A black-box testing utility for command line tools and gems. Written in Ruby, with RSpec in mind. Features:

• Each session takes place within a unique temporary directory, outside the project
• Synchronously run commands then write assertions on:
  • The stdout / stderr content
  • The exit status of the process
• Exercise interactive command line interfaces
• Manipulate files in the temporary directory:
  • Create files
  • Create executable files
  • Append content to files
  • Copy fixture files from your project
• Modify the environment without changing the parent test process:
  • Override environment variables
  • Escape the current Bundler context
  • Adjust $PATH to include your executable / Subject Under Test
• RSpec matchers (optional)

The temporary directory is discarded after each spec. This means you can write & modify files and run commands (like git init) without worrying about tidying up after or impacting your actual project.

Setup

group :test do
  gem "jet_black"
end

RSpec setup

If you're using RSpec, you can load matchers with the following require (optional):

# spec/spec_helper.rb

require "jet_black/rspec"

Any specs you write in the spec/black_box folder will then have an inferred :black_box meta type, and the matchers will be available in those examples.

Manual RSpec setup

Alternatively you can manually include the matchers:

# spec/cli/example_spec.rb

require "jet_black"
require "jet_black/rspec/matchers"

RSpec.describe "my command line tool" do
  include JetBlack::RSpec::Matchers
end

Usage

Running commands

require "jet_black"

session = JetBlack::Session.new
result = session.run("echo foo")

result.stdout # => "foo\n"
result.stderr # => ""
result.exit_status # => 0

Providing stdin data:

session = JetBlack::Session.new
session.run("./hello-world", stdin: "Alice")

Running interactive commands

session = JetBlack::Session.new

result = session.run_interactive("./hello-world") do |terminal|
  terminal.expect("What's your name?", reply: "Alice")
  terminal.expect("What's your location?", reply: "Wonderland")
end

expect(result.exit_status).to eq 0
expect(result.stdout).to eq <<~TXT
  What's your name?
  Alice
  What's your location?
  Wonderland
  Hello Alice in Wonderland
TXT

If you don't want to wait for a process to finish, you can end the interactive session early:

session = JetBlack::Session.new

result = session.run_interactive("./long-cli-flow") do |terminal|
  terminal.expect("Question 1", reply: "Y")
  terminal.end_session(signal: "INT")
end

File manipulation

session = JetBlack::Session.new

session.create_file "file.txt", <<~TXT
  The quick brown fox
  jumps over the lazy dog
TXT

session.create_executable "hello-world.sh", <<~SH
  #!/bin/sh
  echo "Hello world"
SH

session.append_to_file "file.txt", <<~TXT
  shiny
  new
  lines
TXT

# Subdirectories are created for you:
session.create_file "deeper/underground/jamiroquai.txt", <<~TXT
  I'm going deeper underground, hey ha
  There's too much panic in this town
TXT

Copying fixture files

It's ideal to create pertinent files inline within a spec, to provide context for the reader, but sometimes it's better to copy across a large or non-human-readable file.

1. Create a fixture directory in your project, such as spec/fixtures/black_box.

2. Configure the fixture path in spec/support/jet_black.rb:

   require "jet_black"
   
   JetBlack.configure do |config|
     config.fixture_directory = File.expand_path("../fixtures/black_box", __dir__)
   end

3. Copy fixtures across into a session's temporary directory:

   session = JetBlack::Session.new
   session.copy_fixture("src-config.json", "config.json")
   
   # Destination subdirectories are created for you:
   session.copy_fixture("src-config.json", "config/config.json")

Environment variable overrides

session = JetBlack::Session.new
result = session.run("printf $FOO", env: { FOO: "bar" })

result.stdout # => "bar"

Provide a nil value to unset an environment variable.

Clean Bundler environment

If your project's test suite is invoked with Bundler (e.g. bundle exec rspec) but you want to run commands like bundle install and bundle exec with a different Gemfile in a given spec, you can configure the session or individual commands to run with a clean Bundler environment.

Per command:

session = JetBlack::Session.new
session.run("bundle install", options: { clean_bundler_env: true })

Per session:

session = JetBlack::Session.new(options: { clean_bundler_env: true })
session.run("bundle install")
session.run("bundle exec rake")

$PATH prefix

Given the root of your project contains a bin directory containing my_awesome_bin.

Configure the path_prefix to the directory containing your executable(s):

# spec/support/jet_black.rb

require "jet_black"

JetBlack.configure do |config|
  config.path_prefix = File.expand_path("../../bin", __dir__)
end

Then the $PATH of each session will include the configured directory, and your executable should be invokable:

JetBlack::Session.new.run("my_awesome_bin")

RSpec matchers

Given the RSpec setup is configured, you'll have access to the following matchers:

• have_stdout which accepts a string or regular expression
• have_stderr which accepts a string or regular expression
• have_no_stdout which asserts the stdout is empty
• have_no_stderr which asserts the stderr is empty

And the following predicate matchers:

• be_a_success / be_success asserts the exit status was zero
• be_a_failure / be_failure asserts the exit status was not zero

Example assertions

# spec/black_box/cli_spec.rb

RSpec.describe "my command line tool" do
  let(:session) { JetBlack::Session.new }

  it "does the work" do
    expect(session.run("my_tool --good")).
      to be_a_success.and have_stdout(/It worked/)
  end

  it "explodes with incorrect arguments" do
    expect(session.run("my_tool --bad")).
      to be_a_failure.and have_stderr("Oh no!")
  end
end

However these assertions can be made with built-in matchers too:

RSpec.describe "my command line tool" do
  let(:session) { JetBlack::Session.new }

  it "does the work" do
    result = session.run("my_tool --good")

    expect(result.stdout).to match(/It worked/)
    expect(result.exit_status).to eq 0
  end

  it "explodes with incorrect arguments" do
    result = session.run("my_tool --bad")

    expect(result.stderr).to match("Oh no!")
    expect(result.exit_status).to eq 1
  end
end

More examples

• JetBlack's own higher-level tests
• A more complex scenario testing a gem in a fresh Rails app. Shows how to:
  • Include the gem-under-test via the Rails app's Gemfile
  • Use a clean Bundler environment to use the Gemfile of the new Rails app (instead of the Bundler context of the gem's test suite)