This is Part 3 in a multi-part series to detail the creation of a “simple” project combining Ruby, MongoDB, RSpec, Sinatra, and Capybara in preperation for a larger-scale side project set to begin January 2013. For more in this series, see the Pokephile category. Part 4 of this series details moving from a development environment to a production environment using Heroku. The code for this side-project is located on Github, and the final product can be found here.

Heroku is a hosting service for different types of web applications. The best thing about Heroku is it’s free, you get a decent subdomain for your application, and there’s no spam email. Go ahead and sign up if you don’t already have an account.

Now we need the Heroku Toolbelt. I’ll illustrate for Ubuntu 12.10, but there’s also documentation for installing on any OS.

1

$ wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh

The toolbelt installs some Heroku-specific applications in addition to ensuring you have Foreman and Git on your system. Now it’s time to tell the Heroku Toolbelt who we are.

1
2
3
4
5
6
7
8

$ heroku login
Enter your Heroku credentials.
Email: larry@example.com
Password:
Could not find an existing public key.
Would you like to generate one? [Yn]
Generating new SSH public key.
Uploading ssh public key /home/larry/.ssh/id_rsa.pub

In order for Heroku to figure out what ruby gems are needed to run your application, you need to specify a Gemfile. It’s wise to specify a specific version of Ruby, and it’s also a good idea to keep the gems' versions close to the state you developed with. For the Pokephile application created in this series, this is my Gemfile:

1
2
3
4
5
6
7
8
9
10
11
12
13

source :rubygems

ruby '1.9.3'

gem 'sinatra', '~>1.3.2'
gem 'haml', '~>3.1.6'
gem 'mongoid', '~>3.0.14'

group :development, :test do
  gem 'capybara', '~>2.0.1'
  gem 'rspec', '~>2.11.0'
  gem 'nokogiri', '~>1.5.5'
end

First I specify a source for my gems: ‘rubygems’ defaults to “http://rubygems.org” and hasn’t failed me yet. Next I specify that I want to use Ruby 1.9.3, a necessity because Mongoid 3.x doesn’t work correctly with 1.9.2. The versions of the first three gems were chosen by typing the following in the command line to determine which version I had installed on my machine:

1
2
3
4
5
6

$ gem query | grep 'sinatra\|haml\|mongoid'
haml (3.1.7, 3.1.6)
mongoid (3.0.17, 3.0.15, 3.0.14)
sinatra (1.3.3, 1.3.2)
sinatra-contrib (1.3.2)
sinatra-reloader (1.0)

The ‘~>’ operator tells Bundler to use greater-than-equal but stop before next highest version. So, for Sinatra ‘~>1.3.2’ means that Bundler will accept anything greater-than-or-equal-to ‘1.3.2’ and less than ‘1.4.0.’ I tend to rely on the ‘~>’ operator so I can be sure no APIs are changed in my gems.

The next block is a conditional checking in which environment the gems are being installed. This defaults to :development if none is specified. I put the gems used for testing in this block since they’re not needed to run the application, but a developer/tester would need these to run the tests.

For this Gemfile to be meaningful, we need to use a program called Bundler to “bundle” the gems and their dependencies in a Gemfile.lock file.

1
2
3
4
5
6
7

$ pwd
project/
$ sudo apt-get install bundler
...
$ bundle install
...
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Running bundler will create a Gemfile.lock file and make sure your system has the specified gems. If some gems are missing or need new versions to be installed, bundler will ask the user for their password to get the required gems.

The next step for setting things up is to set up Git. If your application is already using Git, you only need to commit all files to verify that the Gemfile and Gemfile.lock make it into the repository.

1
2
3
4

$ cd project/
$ git init
$ git add .
$ git commit -m "Initial commit."

Now we create a Heroku project and give it a meaningful name. If you can’t think of a meaningful name, use ‘heroku create’ and Heroku will come up with something for you.

1
2
3
4

$ heroku create meaningful-name
Creating meaningful-name... done, stack is cedar
http://meaningful-name.herokuapp.com/ | git@heroku.com:meaningful-name.git
Git remote heroku added

We’re almost there. Because we included accessing Mongo databases in our application, we have to take care of that on the web. The easiest way to do that is using a Heroku Add-on. At this time, there are two major Heroku Add-ons for Mongo databases: MongoLab and MongoHQ. Both services have a starter service for $0/month, which is pretty awesome in my opinion. I flipped a coin and picked MongoLab for this application. Adding the add-on to our project:

1

$ heroku addons:add mongolab:starter

If you haven’t already, Heroku will ask you to “verify your account” before continuing. This means that you have to put in some credit card information. Note that you will not be charged, I guess Heroku just wants some indication that you might eventually pay for something. After you put in your credit card information, you may need to run the above command again.

Now that MongoLab is set up on the server-side, we need to tell Mongoid how to connect to that server. The following command will give you the environment variable needed to connect to the server:

1

$ heroku config | grep MONGOLAB_URI

Now we update our mongoid.yml file to use that string:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

development:
  sessions:
    default:
      database: dev
      hosts:
        - localhost
  options:

test:
  sessions:
    default:
      database: test
      hosts:
        - localhost

production:
  sessions:
    default:
      uri: <%= ENV['MONGOLAB_URI'] %>
      options:
        skip_version_check: true
        safe: true

Because of the way my application works, I want to prepopulate the database with some Pokemon.

1
2
3
4
5
6
7
8
9
10
11
12
13

$ export MONGOLAB_URI=`heroku config | grep MONGOLAB_URI | cut -c 15-`
$ pwd
project/
$ cd tools/populate
$ irb
>> require './populater'
true
>> require 'mongoid'
true
>> Mongoid.load! '../../mongoid.yml', :production
{"sessions"=>{"default"=>{"uri"=>nil, "options"=>{"skip_version_check"=>true, "safe"=>true}}}}
>> Populater.new.add_pokemon 1000
nil

With the production database populated, we need to set an environment variable in our production application defining the environment.

1

$ heroku config:add MONGOID_ENV=production

Now that we’ve made changes to the mongoid.yml file, we should commit again and push to Heroku.

1
2
3

$ git commit -a -m "Updating mongoid.yml file for production"
...
$ git push heroku master

And that’s it! Check your Heroku URL to make sure everything looks okay and call it a day, or make some upgrades as I did for my personal version of this project.

This is Part 3 in a multi-part series to detail the creation of a “simple” project combining Ruby, MongoDB, RSpec, Sinatra, and Capybara in preperation for a larger-scale side project set to begin January 2013. For more in this series, see the Pokephile category. Part 3 of this series describes using Capybara to test a Sinatra application. The code for this side-project is located on Github, and the final product can be found here.

Now that the database is populated with data and I’ve switched over to a simpler interface with Mongo, I can actually start creating a UI. For simplicity’s sake, I like to use Sinatra on small projects. Sinatra makes it easy to create a web application with minimal effort. With an emphasis on testing this series, I want to be sure to throughly test my UI and any application integration. Cucumber is a brilliant DSL which allows a programmer to describe in plain English how an application should be behaving. The Capybara gem is a Rack app that simulates running your application and performing basic user tasks, such as clicking a button, following a link, or, on a lower level, looking at your HTML source. Install Capybara like so:

1

sudo gem install capybara

Here’s what I want my application to do:

• User is on home page.
• User enters name of Pokemon and presses ‘Search’.
• User is redirected to search page.
• User can see some information about Pokemon.
• User can repeat the search process.

Error scenario:

• User enters garbage data.
• User is redirected to search page.
• User sees error message.
• User can repeat search process.

So I’ll begin by writing my features. Cucumber syntax is meant to be readable by non-technical persons, so the “code” may look a bit odd. All Cucumber feature files are written something like this:

1
2
3
4
5
6
7
8

Feature: Viewer visits the Home Page
  In order to read the page
  As a viewer
  I want to see the home page of my app

Scenario: View the home page
  Given I am on the home page
 Then I should see "Zowee, mama!" on the page

The “Feature” lines are not used in testing; they are simply to give some relevance to the file’s feature and to attempt to prevent scope creep in the file. The “Scenario” lines are what’s important. The first line is the test name, the “Given” line is the pre-condition, and the final line is what should be observed.

In reality, I have only one main feature: “Search.” One can argue that I also have a feature of “seeing” my home page and my search page, but those are both trivial cases, so for the purpose of this blog post, I’ll skip such tests. Both my success and error case revolves around searching, and there’s not really much else to do in the app. I have to create a directory for the cukes, and that directory is called “features.” I create such a directory in my “project/tools/test” directory and create a new file called “search.feature.” Now I’ll write the feature:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

Feature: Viewer vists the page
 In order to search the page
 As a visitor
 I want to search for Pokemon.

Scenario: Find correct Pokemon from home page
  Given I am on the home page
 When I type "Bulbasaur" in the search bar
 And I click "Search"
 Then I should be on the "search" page
 And I should see "#001 - Bulbasaur"
 And I should see an image with url "http://img.pokemondb.net/artwork/bulbasaur.jpg"
 And I should see "Types: Grass, Poison"

Scenario: Show error text from home page
  Given I am on the home page
 When I type "Johnny Bravo" in the search bar
 And I click "Search"
 Then I should be on the "search" page
 And I should see "Lol! Could not find a Pokemon named 'Johnny Bravo.' Try something else!"

Scenario: Find correct Pokemon from search page
  Given I am on the search page
 When I type "Bulbasaur" in the search bar
 And I click "Search"
 Then I should see "#001 - Bulbasaur"
 And I should see an image with url "http://img.pokemondb.net/artwork/bulbasaur.jpg"
 And I should see "Types: Grass, Poison"

Scenario: Show error text from search page
  Given I am on the search page
 When I type "Johnny Bravo" in the search bar
 And I click "Search"
 Then I should be on the "search" page
 And I should see "Lol! Could not find a Pokemon named 'Johnny Bravo.' Try something else!"

Now I’ve overlooked a lot of tests that one would normally write while doing this, such as verifying that the search bar and search buttons exist and are enabled, but I’d like to keep it simple for now and just stick to testing my search feature. What do these tests do? The first scenario starts on the home page, enters data in the search box, presses the search button, and then verifies that all expected Pokemon data is visible. When writing cukes, I can append statements with an “And” statement as seen above. Run Cucumber:

1

$ cucumber

Cucumber doesn’t exactly give us errors, but it also doesn’t give us success. Fortunately, what it did give us was sample code for all of the steps we need to write. So, let’s perform some copy/paste magic and create a steps file:

1
2

$ mkdir -p features/step_definitions
$ touch features/step_definitions/search_steps.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Given /^I am on the home page$/ do
  pending # express the regexp above with the code you wish you had
end

When /^I type "(.*?)" in the search bar$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

When /^I click "(.*?)"$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

Then /^I should be on the "(.*?)" page$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

Then /^I should see "(.*?)"$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

Then /^I should see an image with url "(.*?)"$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

Given /^I am on the search page$/ do
  pending # express the regexp above with the code you wish you had
end

If I futilely run “cucumber” again now, my tests still don’t pass because I haven’t actually implemented my steps. This is where Capybara comes in. I found that a Capybara cheat sheet is quite helpful while writing out my steps. The syntax I’m going to use is similar to RSpec, except that it includes some Capybara methods. The first two test steps I want to deal with are the “Given” steps.

1
2
3
4
5
6
7
8

Given /^I am on the home page$/ do
  visit '/'
end

Given /^I am on the search page$/ do
  visit '/'
  click_button "Search"
end

All these statements are just Regular Expressions, as indicated by the /^$/. The regex acts as a sort of method name that Cucumber finds to run the steps. Given I am on the home page is trivial: just ‘visit’ the index. Given I am on the search page will first require me to click the “search” button. This is valid because my spec above says this is how to get to the search page. Now can do the ‘When’ statements.

1
2
3
4
5
6
7

When /^I type "(.*?)" in the search bar$/ do |arg1|
  fill_in "pokemon-input", :with => arg1
end

When /^I click "(.*?)"$/ do |arg1|
  click_button arg1
end

When I click just needs to click a button/link/whatever on the screen. The “(.*?)” is a regular expression that will match anything in quotes and assign it to the variable ‘arg1.’ So I can give any button description and Capybara will try to click a button with the given content. When I type in the search bar takes the regex arg1 and uses the “fill_in” method to fill in a text input with id “pokemon-input.” The rest of the steps are all about what should be observed after performing the Given/When steps.

1
2
3
4
5
6
7
8
9
10
11

Then /^I should be on the "(.*?)" page$/ do |arg1|
  current_path.should == "/#{arg1}"
end

Then /^I should see "(.*?)"$/ do |arg1|
  page.should have_content(arg1)
end

Then /^I should see an image with url "(.*?)"$/ do |arg1|
  find(:xpath, "//img[@src='#{arg1}']").should_not be_nil
end

Now I have all of the necessary steps defined! So, I’ll run Cucumber and… Actual errors! None of my four tests made it past the Given step, so I see the output ‘(4 failed, 19 skipped).’ The only way to fix these errors is to finally start writing a web application. So I’ll move back out to the root of my project directory and create a file for my application called ‘app.rb’ and give it the most basic information to run. And, if you haven’t already, install Sinatra.

1

$ sudo gem install sinatra

1
2
3
4
5
6
7
8

require 'sinatra'
require 'haml'

class Application < Sinatra::Base
  get '/' do
      haml :index
  end
end

The Application class inherits from the Sinatra::Base class. This allows me to define a ‘get’ operation to perform actions and load a web page. ‘get \’\‘ do’ signifies the first page a user sees when they go to my web application, commonly known as a home or index page. I plan to use HAML to create my page, so I make a call to haml followed by the name of my HAML document as a symbol. We need to define an ‘index.haml’ page and stick it in a directory called ‘views’ for Sinatra to find it.

1

$ sudo gem install haml

1
2
3
4
5
6

!!!
%html
  %head
      %title Pokemon App
  %body
      LOL HAI.

Tough work. If you’re not familiar with HAML, it’s a markup language that is “compiled” into an HTML page. The main difference between HAML and HTML is that HAML parses white space to figure out where closing tags should be placed. So now I want to run my application. I want to run it using ‘rackup,’ so I’d like to define a ‘config.ru’ file in the root of my project directory to do all the work for me.

1
2
3

require './app'

run Application.new

Now we run ‘rackup’ from the root of my project directory and see the fruits of my labor. Open up a web browser and enter ‘localhost:9292’ in the address bar. You should see a very simple web page with the content of “LOL HAI” and a title of “Pokemon App.” If you view the source, you’ll see the HTML the HAML was compiled into. Just beautiful, isn’t it? Now if I switch back to my test directory and run Cucumber, what happens? The same result. That’s because I need to tell Capybara what to load before trying to run the tests. I do this by defining an “env.rb” file in a “support” directory of the features directory.

1
2
3
4
5
6

require 'capybara'
require 'capybara/cucumber'

require_relative "../../../../app"

Capybara.app = Application

All I do is require my “app.rb” file which is seemingly forever away and then set the Capybara.app variable to my Application class. Now I run Cucumber and… ‘(4 failed, 17 skipped, 2 passed)’ Two steps passed! Yippee! Now if only the rest passed as well. Looking at my ‘search.feature’ file, I can see that the first ‘When’ step is about typing into the search bar. So my first design decision is what kind of search bar I want. I’ve opted for the fun way out: using the Twitter Bootstrap’s typeahead. The typeahead has functionality to give suggestions while the user types, and the best news is this is already coded for us. Adding the code for my search bar and a search button:

1
2
3
4
5
6
7
8
9
10
11
12
13

!!!
%html
  %head
      %title Pokemon App
      %link(rel="stylesheet" href="http://twitter.github.com/bootstrap/assets/css/bootstrap.css")
  %body
      %script{:type => "text/javascript", :src  => "http://code.jquery.com/jquery.min.js"}
      %script{:type => "text/javascript", :src  => "http://twitter.github.com/bootstrap/assets/js/bootstrap-typeahead.js"}
      #search{:style => "position: absolute; width: 100%; text-align: center; top: 10%;"}
          %form{:action => "search", :method => "POST", :id => "pokemon-search"}
              %input{:type => "text", :class => "input-large", :id => "pokemon-input", :name => "pokemon", "data-provide"=>"typeahead", "data-items"=>"10", "autocomplete"=>"off", :autofocus => "", :placeholder => "Find a Pokémon...", "data-source" => Pokemon.only(:name).map {|x| x.name}}
              %button{:type=>"submit", :class => "btn btn-small", :style => "margin-bottom: 10px; font-weight: bold;"}
                  Search

In the \<head> tag, I include a link to the Bootstrap stylesheet. In the \<body> tag, I include a link to the JQuery and Bootstrap Typeahead JavaScript files remotely so I don’t have to keep track of them. I then add a \<div> tag called “search” and center it on the page. Inside the div tag I create a form whose action sends a POST signal to the “search” action. Inside the form is first the typeahead, then a small submit button. The important parameters in the typeahead are “data-items” and “data-source;” “data-items” tells the JavaScript function how many items to suggest at a time, and “data-source” is an array of data for the JavaScript to search. Notice that my “data-source” uses the Pokemon class created previously, so I need to be able to set up a Mongoid connection to access that data. I’ll make this connection in my “config.ru” file:

1
2
3
4
5
6
7
8
9
10

require './app'
require 'mongoid'

class Application
  configure do
      Mongoid.load! 'mongoid.yml'
  end
end

run Application.new

I have chosen to extend the Application class in my “config.ru” file to prevent interference with my test setups later. Taking a look at the application would be a good idea, but if I run “rackup” now Mongoid will complain about environment setup. By default, “Mongoid.load!” will try to load the “development” settings, so I need to include a “development” setup in my “mongoid.yml.” For now, it’s going to be identical to my “test” environment setup except for the database name:

1
2
3
4
5
6
7
8
9
10
11
12
13

test:
  sessions:
    default:
      database: test
      hosts:
        - localhost

development:
  sessions:
    default:
      database: dev
      hosts:
        - localhost

And ensuring some Pokemon are in the “dev” database:

1
2
3
4
5
6
7
8
9
10
11
12

$ irb
>> Dir.pwd
=> "project/tools/populate"
>> require 'mongoid'
=> true
>> Mongoid.load! '../../mongoid.yml'
=> {"sessions"=>{"default"=>{"database"=>"dev", "hosts"=>["localhost"]}}}
>> require './populater'
=> true
>> Populater.new.add_pokemon 152
=> nil
>>

And requiring the Pokemon model in app.rb:

1
2
3
4
5
6
7
8
9
10

require 'sinatra'
require 'haml'

require_relative 'pokemon'

class Application < Sinatra::Base
  get '/' do
      haml :index
  end
end

Finally, run ‘rackup’ from the root of the ‘project’ directory and load up the web application at ‘localhost:9292.’ There’s now a typeahead and a search button in the top center of the page, and typing in the ‘Search’ bar shows up to 10 suggestion Pokemon. Now I’ll return to my Cucumber tests. I need to add a line in the ‘env.rb’ file to set up the Mongoid environment and ensure there are Pokemon in the collection.

1
2
3
4
5
6
7
8
9
10
11
12

require 'capybara'
require 'capybara/cucumber'
require 'mongoid'

require_relative '../../../populate/populater'
require_relative "../../../../app"

Mongoid.load! '../../mongoid.yml', :test

Populater.new.add_pokemon(10)

Capybara.app = Application

Now I can run Cucumber and see some jovial results: ‘(4 failed, 9 skipped, 10 passed).’ I now have more steps passing than failing! The root cause of the failures is that there currently is no “search” page; let’s fix that:

1
2
3
4
5
6
7

...
class Application < Sinatra::Base
  ...
  post '/search' do
      haml :search
  end
end

I want my search page to have a search bar just like my index page. If I want them to be identical, I want to only have to change that code once. When writing HAML, I can create a ‘layout.haml’ file to act as a base page for my application and move all the text from ‘index.haml.’ I’ll add a ‘=yield’ statement where I want the information from ‘index.haml’ and ‘search.haml’ to be placed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

!!!
%html
  %head
      %title Pokemon App
      %link(rel="stylesheet" href="http://twitter.github.com/bootstrap/assets/css/bootstrap.css")
  %body
      %script{:type => "text/javascript", :src  => "http://code.jquery.com/jquery.min.js"}
      %script{:type => "text/javascript", :src  => "http://twitter.github.com/bootstrap/assets/js/bootstrap-typeahead.js"}
      #search{:style => "position: absolute; width: 100%; text-align: center; top: 10%;"}
          %form{:action => "search", :method => "POST", :id => "pokemon-search"}
              %input{:type => "text", :class => "input-large", :id => "pokemon-input", :name => "pokemon", "data-provide"=>"typeahead", "data-items"=>"10", "autocomplete"=>"off", :autofocus => "", :placeholder => "Find a Pokémon...", "data-source" => Pokemon.only(:name).map {|x| x.name}}
              %button{:type=>"submit", :class => "btn btn-small", :style => "margin-bottom: 10px; font-weight: bold;"}
                  Search
      =yield

1
2

#search-text{:style => "position: absolute; width: 100%; text-align: center; top: 2%; font-weight: bold;"}
  Begin typing to search for your Pokemon!

At this point, I’ll create a ‘search.haml’ file in the ‘views’ directory, but leave it empty. Running Cucumber now, I get ‘(4 failed, 4 skipped, 15 passed).’ Pretty close! All I fail now is actually seeing the desired information on the page. First I want to get access to the Pokemon searched for: I can do that by parsing the params passed to us in app.rb:

1
2
3
4
5
6
7
8
9

...

class Application < Sinatra::Base
  ...
  post '/search' do
      @pokemon = Pokemon.where(name: params[:pokemon]).first
      haml :search
  end
end

So now on my search page:

1
2
3
4
5
6
7
8

#search-text{:style => "position: absolute; width: 100%; text-align: center; top: 2%; font-weight: bold;"}
  Search for another Pokemon.
#search-results{:style => "position: absolute; width: 100%; text-align: center; top: 20%; font-weight: bold;"}
  %img{:src => @pokemon.image, :height => "250px"}
  %br
  = "##{@pokemon.number} - #{@pokemon.name}"
  %br
  = "Types: #{@pokemon.types.first}#{@pokemon.types.count < 2 ? ' ' : ', ' + @pokemon.types.last}"

I reference the class variable “@pokemon” and access its data. To output Ruby-formatted strings, I use an “=” sign. Running Cucumber, I now see ‘(2 failed, 21 passed).’ 2 of my 4 tests are passing! A trivial amount of investigation reveals that I didn’t deal with the situation where the user types in garbage data. That can mostly be done in the HAML file, but I also want a way to get the bad text the user gave me so they can see what was wrong.

1
2
3
4
5
6
7
8
9
10
11

#search-text{:style => "position: absolute; width: 100%; text-align: center; top: 2%; font-weight: bold;"}
  Search for another Pokemon.
#search-results{:style => "position: absolute; width: 100%; text-align: center; top: 20%; font-weight: bold;"}
  - unless @pokemon.nil?
      %img{:src => @pokemon.image, :height => "250px"}
      %br
      = "##{@pokemon.number} - #{@pokemon.name}"
      %br
      = "Types: #{@pokemon.types.first}#{@pokemon.types.count < 2 ? ' ' : ', ' + @pokemon.types.last}"
  - else
      = "Lol! Could not find a Pokemon named '#{@name}.' Try something else!"

1
2
3
4
5
6
7
8
9

...
class Application < Sinatra::Base
  ...
  post '/search' do
      @name = params[:pokemon]
      @pokemon = Pokemon.where(name: @name).first
      haml :search
  end
end

Ruby code with no output is preceded with a ‘-’ and because HAML is space-sensitive, there’s no need to include ‘end’ statements. Now I run Cucumber and… 4 scenarios/23 steps passed! I have a functional web application! Of course, the page itself is somewhat bland, some of the styles could be put in a stylesheet and reused, and the only tests I’ve written are for super-high-level functionality. Those are all problems someone with infinite time would deal with, so I’ll just leave the page is is for now.

The next blog post in this series will be about deploying this application to the web using Heroku.

This is Part 2 in a multi-part series to detail the creation of a “simple” project combining Ruby, MongoDB, RSpec, Sinatra, and Capybara in preperation for a larger-scale side project set to begin January 2013. For more in this series, see the Pokephile category. Part 2 details refactoring code using the MongoDB Ruby driver to use Mongoid. The code for this side-project is located on Github.

What I’ve Done

In a previous post, I described creating a class that would populate a database with data scraped from the internet. I used the MongoDB Ruby driver to accomplish this. However, using the driver can be laborious and there are simpler ways. In this post, I’m going to refactor the Populater class to use Mongoid.

Mongoid

Mongoid (pronounced mann-goyd) is an Object-Document Wrapper for Ruby. Using mongoid abstracts some of the database operations that must be performed when using the MongoDB Ruby driver. It comes in handy when using models in an MVC application. To install the Mongoid gem:

1

sudo gem install mongoid

Refactoring

In populater.rb, we only inserted one structure of document into our “pokemons” collection. That makes this a great opportunity to use Mongoid. We remember that there were four fields in our document: number (string), name (string), image link (string), and types (array). Knowing this, we can create a model for this data:

1
2
3
4
5
6
7
8
9
10

require 'mongoid'

class Pokemon
  include Mongoid::Document

  field :number, type: String
  field :name, type: String
  field :types, type: Array
  field :image, type: String
end

And that’s it for our model. Although we specified the types in this case, it’s not necessary if we want a looser definition of our model. Here’s how we change our implementation file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

#require 'mongo' #deleted
require_relative '../../pokemon' #added
require 'nokogiri'
require 'open-uri'

class Populater
  #def initialize(db_name) #removed
  def initialize #added
      #@col = Mongo::Connection.new.db(db_name)["pokemons"] #deleted
      #@col.remove #deleted
      Pokemon.delete_all #added
      @data = Nokogiri::HTML(open("http://pokemon.wikia.com/wiki/List_of_Pok%C3%A9mon"))
  end

  def add_pokemon(num_to_add)
    @data.xpath("//table[@class='wikitable sortable']/tr").each do |row|
        break if num_to_add <= 0
        dex_num = row.at_xpath('td/text()').to_s.strip
        next if dex_num.nil? || dex_num.empty?
        dex_name = row.at_xpath('td[2]/a/text()').to_s.strip

        unless dex_num == "000"
            type_1 = row.at_xpath('td[4]/a/span/text()').to_s.strip
            type_2 = row.at_xpath('td[5]/a/span/text()').to_s.strip || row.at_xpath('td[5]/text()').to_s.strip
            image_link = "http://img.pokemondb.net/artwork/#{dex_name.downcase}.jpg"
        else
            type_1 = row.at_xpath('td[4]/text()').to_s.strip
            type_2 = row.at_xpath('td[5]/text()').to_s.strip
            image_link = "images/missingo.png"
        end

        types = Array.new
        types << type_1 unless type_1.nil? || type_1.empty?
        types << type_2 unless type_2.nil? || type_2.empty?

        #@col.insert({:number => dex_num, :name => dex_name, :types => types, :image => image_link}) #deleted
        Pokemon.create {:number => dex_num, :name => dex_name, :types => types, :image => image_link} #added

        num_to_add -= 1
    end
  end
end

That one’s easy. We deleted four lines and added 3. However, now you can see that the Populater does not have to deal with connecting to the database, it only has to know what model it wants to modify. So we’ve removed some complexity from this file by no longer requiring the database name on initialization. However, that means that someone else has to be in charge of setting up the initial connection. In the overlying project, we want that someone else to be a controller. In our tests, we want that someone else to be our test file. So let’s do it. We’re going to start by adding a config section in our before:all block.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

require_relative '../../populate/populater'
#require 'mongo' #removed
require 'mongoid' #added
require_relative '../../../pokemon'

describe Populater do
  before:all do
      Mongoid.configure do |config| #added
          config.connect_to 'test' #added
      end # added
      #@col = Mongo::Connection.new.db('test')["pokemons"] # removed
  end
  ...
end

In doing this, we’ve set up any of our document models to use the ‘test’ database. Now we go through each test and replace the Mongo Ruby Driver syntax with Mongoid syntax, which is similar to Ruby’s Array syntax.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

...

describe Populater do
  ...
  before:each do
      #@populater = Populater.new('test') #removed
      @populater = Populater.new #added
  end

  describe "#new" do
      it "does not throw when creating instance" do
          #expect {Populater.new('test')}.to_not raise_error #removed
          expect {Populater.new}.to_not raise_error #added
      end

      it "takes one param and returns a Populater instance" do
          @populater.should be_an_instance_of Populater
      end

      it "empties pokemon collection" do
          #@col.insert({:test => "hi there"}) #removed
          #@col.find.count.should_not eql 0 #removed
          #Populater.new('test') #removed
          #@col.find.count.should eql 0 #removed
          Pokemon.create #added
          Pokemon.count.should eql 1 #added
          Populater.new #added
          Pokemon.count.should eql 0 #added
      end
  end
  ...
end

The ‘new’ tests are straightforward. We remove the usage of an input parameter to the Populater initializer. The only significant change we make is to the “empties pokemon collection” test. Here we replace the Mongo Ruby Driver syntax of inserting into a collection with Mongoid syntax of creating a Pokemon document. The ‘create’ method inserts a document into the collection with the given values, or defaults if none are given. We also see that we can remove the “find” syntax completely and just use a “count” method on the document type.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

...
describe Populater do
  ...
  describe "#add_pokemon" do
      it "adds 0 pokemon given 0" do
          @populater.add_pokemon 0
          #@col.find.count.should eql 0 #removed
          Pokemon.count.should eql 0 #added
      end
      ...
      it "adds pokemon with a number" do
          @populater.add_pokemon 1
          #@col.find.count.should eql 1 #removed
          #@col.find.first['number'].should_not be_nil #removed
          Pokemon.count.should eql 1 #added
          Pokemon.first['number'].should_not be_nil #added
      end
    ...
  end
end

The tests for adding 0, 1, and 2 documents to the collection are all very similar. The only change is to replace the Mongo Ruby Driver “find.count” syntax with the Mongoid “count.” The “adds pokemon with a ____” tests all undergo the same changes. I replace the “.find.first” statement with a simple “.first” to get the same meaning. So our Populater has been converted to use Mongoid instead of the Mongo Ruby Driver. Bully for us.

There’s one more change that would be nice to make before we hang up our hats. Configuring Mongoid using the .config syntax is okay, but it would be a lot nicer to keep all of our configuration in a file. We can create such a file called “mongoid.yml” and put some configuration information in it:

1
2
3
4
5
6

test:
  sessions:
    default:
      database: test
      hosts:
        - localhost

This syntax is valid in Mongoid 3.x. This is a very simply configuration for our test environment. Now we can go back into our test file and change the ‘before:all’ block:

1
2
3
4
5
6
7
8
9
10

...
describe Populater do
  before:all do
      #Mongoid.configure do |config| #removed
      #  config.connect_to 'test' #removed
      #end # removed
      Mongoid.load! '../../../mongoid.yml', 'test' #added
  end
  ...
end

The second parameter can be a string or a symbol. Now there’s only one file to modify the environment configurations, and we’re better off for it.

This is Part 1 in a multi-part series to detail the creation of a “simple” project combining Ruby, MongoDB, RSpec, Sinatra, and Capybara in preperation for a larger-scale side project set to begin January 2013. For more in this series, see the Pokephile category. Part 1 details getting started with MongoDB and creating a collection using data scraped off the web using Nokogiri. The code for this side-project is located on Github.

A little background

NoSQL is a database service used when working with a large amount of data that doesn’t fit a relational model (read: wikipedia). It allows for mass storage without the overhead of SQL relations. There are many types of schemaless database services (here’s a list), but in particular I’ve been looking into what’s called “Document Store.”

Documents can be any number of key-value fields with a unique id. Document Store services usually encode data in a simple format such as XML, YAML, JSON, or BSON for storage purposes. MongoDB is a document store service which uses BSON to store documents. In Mongo, we connect to a specific database and then we can look through “collections,” which are more-or-less equivalent to “tables” in relational databases.

What about MongoDB and the Ruby driver?

The first step is to get MongoDB working on your machine. Install MongoDB for your system – on Ubuntu 12.10 I do this:

1

sudo apt-get install mongodb mongodb-dev mongodb-clients mongodb-server

Then we start up the daemon:

1

sudo service mongodb start

What’s the concept?

The concept here is that we are going to have a database populated with Pokemon. The user types a Pokemon’s name into a search field and submits the form, which brings up an image of the Pokemon and some useful information.

Getting started

Since I would like to focus on MongoDB, we can start by populating our database with Pokemon. If you’re not familiar with Pokemon, there are lots of them (~650 at the date of this blog post). For my purposes, I may want to only add the first ~150 Pokemon, or I may want to add every Pokemon imaginable. I want it to be easy to add more if any new ones are added. So I’m going to start this project by creating a Populater, and we’re going to use TDD to help us create it.

If you don’t have RSpec installed, it’s as easy as opening up a shell and:

1

$ sudo gem install rspec mongo

I’m going to put the Populater in a tools directory, and I’m going to put my spec files in a test/spec directory. The directory structure I want to use is as follows:

1
2
3
4
5

project
--tools
----populate
----test
------spec

In the ‘tools/test/spec’ directory, I create ‘populater_spec.rb.’ We’ll write our first test:

1
2
3
4
5
6
7

describe Populater do
  describe "#new" do
      it "does not throw when creating instance" do
          expect {Populater.new}.to_not raise_error
      end
  end
end

The syntax for RSpec is mostly pseudo-English, so it’s fairly straightforward to follow. The first ‘describe’ block says that we are describing the Populater class. The second ‘describe’ block says that we are describing the ‘new’ method of the ‘Populater’ class. The inner-most block is our test. We want to make sure that no exception is thrown when we create a new Populater. To run this test, open a terminal and type:

1
2
3

$ pwd
~/project/tools/test
$ rspec populater_spec.rb

We get a big fat compile error, obviously due to the fact that there’s no such thing as a ‘Populater’ class. So create the file ‘populater.rb’ in ‘project/tools/populate’ and create the class:

1
2

class Populater
end

And include the ‘Populater’ class in our spec file:

1
2
3
4
5
6
7
8
9

require_relative '../../populate/populater'

describe Populater do
  describe "#new" do
      it "does not throw when creating instance" do
          expect {Populater.new}.to_not raise_error
      end
  end
end

Now run rspec. Hooray, we’re passing all our tests! Let’s add another test and some let’s have RSpec do a little work before each test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

require_relative '../../populate/populater'

describe Populater do
  before:each do
      @populater = Populater.new
  end

  describe "#new" do
      it "does not throw when creating instance" do
          expect {Populater.new}.to_not raise_error
      end

      it "takes no params and returns a Populater instance" do
          @populater.should be_an_instance_of Populater
      end
  end
end

The ‘before:each’ syntax tells RSpec to perform this action before running each test. This way, we don’t have to type out ‘Populater.new’ in each test. When we run RSpec, this test passes. Now let’s actually do something meaningful in our new call. We want the Populater to empty all Pokemon from our database as it begins. In order to do this, we need to also tell the Populater what database to use, so we’ll refactor slightly to pass in the name of our database to the Populater.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

require_relative '../../populate/populater'
require 'mongo'

describe Populater do
  before:all do
      @col = Mongo::Connection.new.db('test')["pokemons"]
  end
  before:each do
      @populater = Populater.new('test')
  end

  describe "#new" do
      it "does not throw when creating instance" do
          expect {Populater.new('test')}.to_not raise_error
      end

      it "takes one param and returns a Populater instance" do
          @populater.should be_an_instance_of Populater
      end

      it "empties pokemon collection" do
          @col.insert({:test => "hi there"})
          @col.find.count.should_not eql 0
          Populater.new('test')
          @col.find.count.should eql 0
      end
  end
end

Similar to the ‘before:each’ syntax, the ‘before:all’ syntax runs the statement once. Here we want to get a handle to the ‘pokemons’ collection from our ‘test’ database. In our test, we run a ‘find’ with no arguments on the ‘pokemons’ collection to query everything in that collection. We also have an ‘insert’ statement where we insert an arbitrary document into our collection. You’ll note later that this garbage document looks nothing like the Pokemon documents we insert, which is just another reason to love document-store databases. We run RSpec and we fail the test. Let’s open up ‘populater.rb’ and fix this.

1
2
3
4
5
6
7
8

require 'mongo'

class Populater
  def initialize(db_name)
      @col = Mongo::Connection.new.db(db_name)["pokemons"]
      @col.remove
  end
end

Test fixed. We connect to the same database and access the same collection and remove all the old data on intialize. So now we actually want to add Pokemon to the collection. We’ll pick up a new ‘describe’ block for an ‘add_pokemon’ method. We’ll then test that calling it with 0 adds no Pokemon to the collection.

1
2
3
4
5
6
7
8

...
describe "#add_pokemon" do
  it "adds 0 pokemon given 0" do
      @populater.add_pokemon 0
      @col.find.count.should eql 0
  end
end
...

When we run our tests, we get a NoMethodError and fail. We create a trivial fix in populater.rb

1
2
3
4
5
6

class Populater
  ...

  def add_pokemon(num)
  end
end

And we pass the test, having added 0 Pokemon to our database. Let’s do it with 1 now.

1
2
3
4
5
6
7
8
9

...
describe "#add_pokemon" do
  ...
  it "adds 1 pokemon given 1" do
      @populater.add_pokemon 1
      @col.find.count.should eql 1
  end
end
...

We fail. Another trivial fix:

1
2
3
4
5
6
7
8
9

class Populater
  ...

  def add_pokemon(num)
      (0...num).each do |x|
          @col.insert({number: x})
      end
  end
end

We pass again. We’ll also pass when checking for multiple Pokemon:

1
2
3
4
5
6
7
8
9

...
describe "#add_pokemon" do
  ...
  it "adds 2 pokemon given 2" do
      @populater.add_pokemon 2
      @col.find.count.should eql 2
  end
end
...

But we’re missing substance. There’s only garbage being shoved in our database. Our TDD methodology breaks down slightly here because we want our database to have dynamic information scraped from a website, and I don’t want to hard code any data nor do I want to scrape the same website in my tests and my implementation. So we’re going to do a little bit of behind-the-scenes stuff and test that the fields we want are simply not nil. I want each Pokemon to have a number, name, an array of types, and a link to an image:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

...
describe "#add_pokemon" do
  ...
  it "adds pokemon with a number" do
      @populater.add_pokemon 1
      @col.find.count.should eql 1
      @col.find.first['number'].should_not be_nil
  end
  it "adds pokemon with a name" do
      @populater.add_pokemon 1
      @col.find.count.should eql 1
      @col.find.first['name'].should_not be_nil
  end
  it "adds pokemon with array of types" do
      @populater.add_pokemon 1
      @col.find.count.should eql 1
      types = @col.find.first['types']
      types.should_not be_nil
      types.should be_an_instance_of Array
      types.should have_at_least(1).items
      types.should have_at_most(2).items
  end
  it "adds pokemon with image link" do
      @populater.add_pokemon 1
      @col.find.count.should eql 1
      image = @col.find.first['image']
      image.should_not be_nil
      image.should_not be_empty
  end
end
...

There are many websites where you can get this kind of data for each Pokemon, but I chose the Pokemon Wiki for its consistency. In the initializer of the Populater, I open up the URL using Nokogiri so I can access the sweet, creamy data contained within. In my add_pokemon method, I extract this data I want based on the way the table is set up on the website. To continue, we need to install the Nokogiri gem:

1

sudo gem install nokogiri

And now we add the logic to add_pokemon:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

require 'mongo'
require 'nokogiri'
require 'open-uri'

class Populater
  def initialize(db_name)
      @col = Mongo::Connection.new.db(db_name)["pokemons"]
      @col.remove
      @data = Nokogiri::HTML(open("http://pokemon.wikia.com/wiki/List_of_Pok%C3%A9mon"))
  end

  def add_pokemon(num_to_add)
      @data.xpath("//table[@class='wikitable sortable']/tr").each do |row|
          break if num_to_add <= 0
          dex_num = row.at_xpath('td/text()').to_s.strip
          next if dex_num.nil? || dex_num.empty?
          dex_name = row.at_xpath('td[2]/a/text()').to_s.strip

          unless dex_num == "000"
              type_1 = row.at_xpath('td[4]/a/span/text()').to_s.strip
              type_2 = row.at_xpath('td[5]/a/span/text()').to_s.strip || row.at_xpath('td[5]/text()').to_s.strip
              image_link = "http://img.pokemondb.net/artwork/#{dex_name.downcase}.jpg"
          else
              type_1 = row.at_xpath('td[4]/text()').to_s.strip
              type_2 = row.at_xpath('td[5]/text()').to_s.strip
              image_link = "images/missingo.png"
          end

          types = Array.new
          types << type_1 unless type_1.nil? || type_1.empty?
          types << type_2 unless type_2.nil? || type_2.empty?

          @col.insert({:number => dex_num, :name => dex_name, :types => types, :image => image_link})

          num_to_add -= 1
      end
  end
end

I’ll admit The add_pokemon method is now quite a bit more daunting to interpret. Here’s the breakdown of what’s going on: Nokogiri finds us the table tag with class of ‘wikitable sortable’ and we iterate over that. There are two breaking conditions of our loop: we hit the max number of Pokemon as given, or we can’t find anymore Pokemon in the table. So we check that we haven’t hit our max. Then we find the Pokemon’s number in the table after we manually parse the HTML. In the case of this table, the first row is all garbage, so we continue to the next row if we are on the first row. We then grab the name from the table, which is luckily always in the same place. The branch is for the special case of Pokemon #000 (Missingo), which is set up slightly differently in the table for some reason. We create an empty array and shove our types in it, but we have to be careful because not all Pokemon have two types. We then create a document in the braces and insert it into the collection. The final step is to decrement the loop counter.

Tests pass. We now have a working Populater! Now we can either write a script or open up the irb and populate as necessary and we know that the Populater is functional:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ irb
>> Dir.pwd
=> "project/tools/populate"
>> require 'mongo'
=> true
>> col = Mongo::Connection.new.db('dev')["pokemons"]
=> ...
>> col.find.count
=> 0
>> require './populater'
=> true
>> Populater.new('dev').add_pokemon 152
=> nil
>> col.find.count
152
>>

If you want to further familiarize yourself with the MongoDB Ruby driver, you should check out the MongoDB Koans. Unfortunately, the original MongoDB Koans have not been updated in a while, and so my more recent installations of Ruby and the MongoDB driver didn’t work. I found a set of updated koans which worked with my install of Ruby 1.9.3. However, the updated version also had a couple of annoying issues with deprecations, so I created my own fork on GitHub with the fixes.

Who Moved My Cheese?: An Amazing Way to Deal with Change in Your Work and in Your Life
Spencer Johnson

The Gist

Who Moved My Cheese? is a book about being prepared to lose your job. The book was written during times of rapid economic growth caused by the dot-com bubble, which resulted in many companies emerging out of the primordial ooze and then falling back down into the tarpit after an extremely short time on the market. The author uses a cute analogy about cheese to convey that success is fleeting, and each day could be the day a company will shut down.

My Feelings

This is another book I read during summer 2012 and wrote a mediocre write-up regarding. This write-up is fresh and in my new write-up style.

The authors of this book want to help people realize that their job is a gift, not a privilage. Just because a company is doing well this quarter does not imply that it will still be in business next quarter. During the time period this book was written in, this was especially true. The theory of companies during the late ‘90s was to spend as much money as possible to gather as many customers as possible disregarding profits in the short-term (read: Get Large or Get Lost). Of course, this kind of mentality did not fit all businesses, and many companies which looked like they would be the future leaders of the American economy were quickly forgotten by the mid-2000s (see: Pets.com).

The authors want employees to stop fearing change, and to accept it as a normal part of life. Sometimes companies falter, and oftentimes there’s nothing an individual can do about it. In doing so, sometimes old friends are left behind while an individual moves on in search of bigger and better things. Sometimes it takes a long time to find those bigger and better things, but every time it’s worth it due to the thrill of confronting your fears and exploring something new.

The use of a “maze” with two tiny people and two anthropomorphic mice is strange and feels purposefully forced. The authors wanted to make a memorable anecdote, and they succeeded with their strange analogy. Having said that, several times I was confused by the names of the four characters as they were very similar and nonsensical. Because of this, I had to reread some sections after referencing the first page of the book to determine who was who.

The primary author appended “MD” to his name on the front cover. Similarly, the “secondary” author appended a “PhD” to his name on the front cover. This bothered me as it is an obvious ploy to make people value the opinions held within the book more highly because they are written by “doctors.”

Who Would Like This

This is a book that tries to prepare people for the worst and to teach them not to just give up when the future looks grim. It could be an interesting book for anyone, especially after several years of an economic boom caused by a new technology. A person who is very content in their current position at a company could also benefit from this book, as it may give them some incentive to seek out change.