UPDATE: using in-dyno postgres puts Heroku CI on par with running Rspec locally!
Experimenting with using Heroku CI for a Rails app, we were a little surprised just how slow Rspec specs run.
As you can see however the runtime cost is about the same since the largest dyno performance-l costs twice as much as the default dyno used by Heroku CI.
Unfortunately, we cannot test using in-dyno Postgres since Heroku does not seem to offer a way to specify the Postgres version for in-dyno instances.
| Mac OS X | Heroku CI "M" dyno | Heroku CI "L" dyno | |
|---|---|---|---|
| Dyno cost/hour | $0.34 | $0.68 | |
| Run Time | 5:53 | 59:00 | 31:43 |
| Heroku cost | $ 0.34 | $0.36 |
Next Step: If we can get parallel tests to run, it will be interesting to see if the overall runtime (and cost) remains constant so we can shorten the test to say 2 minutes by running 16 dynos.
If so that may be GREAT news for using Heroku CI vs Travis CI or Circle CI, whose fixed monthly cost is HUGE for even a few concurrent instances.