Enforcing spec coverage with CruiseControl, RCov, and RSpec
June 2nd, 2007 | 6 comments
Now that Alex has gotten CruiseControl.rb setup setup at work, I wanted to use it to enforce good test coverage - and I wanted the output to look good doing it. Turns out this isn't that hard, but requires a few things:
- CruiseControl.rb up and running
- RSpec and the Rails RSpec plugin
- RCov (gem install rcov)
Make sure RCov is installed on whatever machine CruiseControl is installed on (and RSpec, unless it's in your rails/vendor directory)
At this point all you need is a custom cruise task (CruiseControl will run rake cruise instead of the default task if a cruise task exists). Mine looks like this:
lib/tasks/cruise.rake
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
rspec_base = File.expand_path("#{RAILS_ROOT}/vendor/plugins/rspec/lib") $LOAD_PATH.unshift(rspec_base) if File.exist?(rspec_base) require 'spec/rake/spectask' require 'spec/rake/verify_rcov' RCov::VerifyTask.new(:verify_rcov) { |t| t.threshold = 100.0 } desc "Task for cruise Control" task :cruise do RAILS_ENV = ENV['RAILS_ENV'] = 'test' # Without this, it will drop your production database. CruiseControl::invoke_rake_task 'db:reset' CruiseControl::invoke_rake_task 'cruise_coverage' CruiseControl::invoke_rake_task 'verify_rcov' end desc "Run specs and rcov" Spec::Rake::SpecTask.new(:cruise_coverage) do |t| t.spec_opts = ['--options', "#{RAILS_ROOT}/spec/spec_rcov.opts"] t.spec_files = FileList['spec/**/*_spec.rb'] t.rcov = true t.rcov_opts = ['--exclude', 'spec,/usr/lib/ruby', '--rails', '--text-report'] end |
This will invoke RSpec telling it to use RCov with a text report. If RSpec fails (broken tests) your build will fail. If it passes it will call the verify_rcov task which will parse the RCov output and fail unless the coverage is equal to or greater the percentage you set in the RCov::VerifyTask.
If you do not set both ENV['RAILS_ENV'] and RAILS_ENV to test, CruiseControl will run in production mode and drop your production database. CruiseControl is awesome like that.
Manually running the specs I like the colored progress output, but for viewing in CruiseControl the colors get mangled, and it's nice to see the specdoc format - so I made a seperate .opts file for RSpec in spec/spec_rcov.opts. If you don't care, change the spec_rcov.opts above to spec.opts and ignore this part. The spec_rcov.opts I used is:
1 2 3 4 5 |
--format specdoc --loadby mtime --reverse |
And that's it! You'll get a readable output, and nobody will be able to commit testless code or broken tests without breaking the build.
The one with RSpec is in rspec/rake_tasks/verify_rcov.rake - Take a look at that, as it might suite your needs!
Update: updated to use the verify_rcov that comes with RSpec 1.02+. Thanks Chad!
Thanks for this, rake db:reset is not in my standard list of rake tasks, so I've opted for rake db:test:clone.
db:reset will be in rails 2
Is there a way for the rcov output to be visible on the cruisecontrol site? This setup didn't work for me. Going to give it another try. Said it couldn't find the coverage dir, so I'll create one. site doesn't have access to that dir on cruisecontrol. I thought there was an artifact directory for that purpose.
This is what I get when I run this scrip. Any ideas why this might happen?
I do have the coverage directory. Index.html isn't there, its supposed to be generated.
/var/www/cruise/projects/feedback/work root$ ruby1.8 -e require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run rake aborted! No such file or directory - coverage/index.html (See full trace by running task with --trace) (in /var/www/cruise/projects/feedback/work) [CruiseControl] Invoking Rake task "cruise" [CruiseControl] Invoking Rake task "db:test:clone" [CruiseControl] Invoking Rake task "cruise_coverage" [CruiseControl] Invoking Rake task "verify_rcov" dir : /var/www/cruise/projects/feedback/work command : ruby1.8 -e "require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run" executed command : echo /var/www/cruise/projects/feedback/work root$ ruby1.8 -e "require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run" >> /var/www/cruise/projects/feedback/build-91/build.log && ruby1.8 -e "require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run" >> /var/www/cruise/projects/feedback/build-91/build.log 2>&1 exitstatus: 1 STDERR TAIL START STDERR TAIL END
This is working. Key tip - don't check in the coverage directory. put that in your ignore list. I also added to ignore more files in the exclude.
The target coverage seems to be a little odd. In the % coverage totals it shows 28.4% but then for the verify it says coverage must be at least 90% but was 44.2%.
Also it looks terrible in IE, but great in Firefox...
I've found one good way to output the coverage information to the CruiseControl directory (so it shows up under "artifacts"):
1. create {RAILS_ROOT}/lib/build_settings.rb
2. in that file, put
module BuildSettings
attr_reader :coverage_dir
def coverage_dir
@coverage_dir ||= ( ENV['CC_BUILD_ARTIFACTS'] || 'coverage' )
end
end
3. In your rake file, do: require "#{RAILS_ROOT}/lib/build_settings" at the top
4. In each namespace in your rake file (including the base one), do: include BuildSettings
5. use t.rcov_dir = coverage_dir inside your SpecTask