#50 ✓resolved
Wincent Colaiuta

REQUEST: Separate performance specs from other specs

Reported by Wincent Colaiuta | March 1st, 2008 @ 09:58 AM | in 0.7.1

I've been trying the Thin 0.7.0 specs out on RHEL 5.1 and I'm seeing what I would consider to be a bunch of spurious failures. For example:

12)
'Thin::Server on TCP socket should handle POST in less then 0.007 RubySecond' FAILED
took <0.008859357696567 RubySeconds>, should take less than 0.007 RubySeconds.
/home/wcolaiuta/work/thin/thin.git/spec/server/tcp_spec.rb:44:

So the code is working, but its going slower than the specs say it should. Is this a bug in Thin, or is it that the granularity and cross-machine stability of "RubySeconds" isn't fine enough? (I suspect the latter, but the point is this kind of failure is probably spurious and is reducing the value of the spec suite as a means of confirming the correctness of the code.)

That's just one example; out of 17 failures, 4 of them are benchmark_unit specs which miss out by an insignificant margin.

I think there's a clear need to separate the specs into two different classes: those that confirm the correct behaviour of the code, and those that check the performance (speed) of the code.

Basically, as an end user I want to be able to run the spec suit and see that everything is working correctly. Checking that everything is working fast is a separate concern and I want to test it separately.

And if you might want to increase the margins in your RubySeconds specifications, but I think that's a less important issue than keeping them isolated.

Now I am going to investigate the other 13 failures (on your development platform you're seeing a 100% pass rate right? or am I running into known issues here?).

Comments and changes to this ticket

  • macournoyer

    macournoyer March 1st, 2008 @ 01:46 PM

    • Milestone set to 0.8.0
    • State changed from “new” to “open”

    Sure all specs pass on my machine, I'm a big TDD/BDD fan ;)

    benchmark_unit is supposed to be machine independent but I guess it's not 100% correct.

    I like the idea of isolating the perf specs, I'll try to do that.

    Can you paste the output of your failures?

  • macournoyer

    macournoyer March 1st, 2008 @ 02:09 PM

    hey Wincent if you would like to help w/ this I can give you a github invite so you can fork the Thin repo. It's much better then patch ping-pong.

    Just send me your email.

  • macournoyer

    macournoyer March 2nd, 2008 @ 12:56 AM

    • Milestone changed from 0.8.0 to 0.7.1

    I moved all perf specs under spec/perf and exclude them from rake spec. Run through rake spec:perf now.

    thx for the suggestion Wincent!

  • macournoyer

    macournoyer March 2nd, 2008 @ 12:56 AM

    • State changed from “open” to “resolved”

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

People watching this ticket

Tags

Pages