#50 ✓resolved
Wincent Colaiuta

REQUEST: Separate performance specs from other specs

Reported by Wincent Colaiuta | March 1st, 2008 @ 09:58 AM | in 0.7.1

I've been trying the Thin 0.7.0 specs out on RHEL 5.1 and I'm seeing what I would consider to be a bunch of spurious failures. For example:

12)
'Thin::Server on TCP socket should handle POST in less then 0.007 RubySecond' FAILED
took <0.008859357696567 RubySeconds>, should take less than 0.007 RubySeconds.
/home/wcolaiuta/work/thin/thin.git/spec/server/tcp_spec.rb:44:

So the code is working, but its going slower than the specs say it should. Is this a bug in Thin, or is it that the granularity and cross-machine stability of "RubySeconds" isn't fine enough? (I suspect the latter, but the point is this kind of failure is probably spurious and is reducing the value of the spec suite as a means of confirming the correctness of the code.)

That's just one example; out of 17 failures, 4 of them are benchmark_unit specs which miss out by an insignificant margin.

I think there's a clear need to separate the specs into two different classes: those that confirm the correct behaviour of the code, and those that check the performance (speed) of the code.

Basically, as an end user I want to be able to run the spec suit and see that everything is working correctly. Checking that everything is working fast is a separate concern and I want to test it separately.

And if you might want to increase the margins in your RubySeconds specifications, but I think that's a less important issue than keeping them isolated.

Now I am going to investigate the other 13 failures (on your development platform you're seeing a 100% pass rate right? or am I running into known issues here?).

Comments and changes to this ticket

  • macournoyer

    macournoyer March 1st, 2008 @ 01:46 PM

    • Milestone set to 0.8.0
    • State changed from “new” to “open”

    Sure all specs pass on my machine, I'm a big TDD/BDD fan ;)

    benchmark_unit is supposed to be machine independent but I guess it's not 100% correct.

    I like the idea of isolating the perf specs, I'll try to do that.

    Can you paste the output of your failures?

  • Wincent Colaiuta

    Wincent Colaiuta March 1st, 2008 @ 02:08 PM

    Yes; here are the timing-related failures:

    2)
    'Thin::Response should be fast' FAILED
    took <0.000141749723145072 RubySeconds>, should take less than 0.00011 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/response_spec.rb:63:
    
    11)
    'Thin::Server on TCP socket should handle GET in less then 0.0045 RubySecond' FAILED
    took <0.00973499188750674 RubySeconds>, should take less than 0.0045 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/server/tcp_spec.rb:40:
    
    12)
    'Thin::Server on TCP socket should handle POST in less then 0.007 RubySecond' FAILED
    took <0.008859357696567 RubySeconds>, should take less than 0.007 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/server/tcp_spec.rb:44:
    
    16)
    'Thin::Request performance should be faster then 0.0002 RubySeconds' FAILED
    took <0.000255813953488372 RubySeconds>, should take less than 0.0002 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/request/perf_spec.rb:21:
    

    And here are the other failing cases (haven't looked into them very closely yet):

    1)
    'Thin::Connection should return remote_address' FAILED
    expected: "127.0.0.1",
         got: nil (using ==)
    /home/wcolaiuta/work/thin/thin.git/spec/connection_spec.rb:55:
    
    3)
    NoMethodError in 'Thin::Logging should output debug when set to true'
    protected method `debug' called for #<TestLogging:0xb7c7cfec>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:16:
    
    4)
    NoMethodError in 'Thin::Logging should output trace when set to true'
    protected method `trace' called for #<TestLogging:0xb7c7b4e4>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:22:
    
    5)
    NoMethodError in 'Thin::Logging should not output when silenced'
    protected method `log' called for #<TestLogging:0xb7c79810>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:28:
    
    6)
    NoMethodError in 'Thin::Logging should not output when silenced [deprecated]'
    protected method `log' called for #<TestLogging:0xb7c78078>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:36:
    
    7)
    'Daemonizing should send kill signal if timeout' FAILED
    expected false, got true
    /home/wcolaiuta/work/thin/thin.git/spec/daemonizing_spec.rb:112:
    
    8)
    NoMethodError in 'Thin::Server app builder should work with Rack url mapper'
    You have a nil object when you didn't expect it!
    You might have expected an instance of Array.
    The error occurred while evaluating nil.+
    /home/wcolaiuta/work/thin/thin.git/spec/server/builder_spec.rb:36:
    
    9)
    'Thin::Server on UNIX domain socket should retreive remote address' FAILED
    expected "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 519\r\nConnection: close\r\nServer: thin 0.7.0 codename Spherical Cow\r\n\r\n{\"rack.url_scheme\"=>\"http\", \"PATH_INFO\"=>\"/\", \"rack.run_once\"=>false, \"rack.input\"=>#<StringIO:0xb7df6e04>, \"SCRIPT_NAME\"=>\"\", \"SERVER_PROTOCOL\"=>\"HTTP/1.1\", \"rack.errors\"=>#<IO:0xb7ef501c>, \"REMOTE_ADDR\"=>\"V\", \"REQUEST_PATH\"=>\"/\", \"SERVER_SOFTWARE\"=>\"thin 0.7.0 codename Spherical Cow\", \"HTTP_VERSION\"=>\"HTTP/1.1\", \"rack.multithread\"=>false, \"rack.version\"=>[0, 2], \"REQUEST_URI\"=>\"/\", \"rack.multiprocess\"=>false, \"QUERY_STRING\"=>\"\", \"GATEWAY_INTERFACE\"=>\"CGI/1.2\", \"HTTP_CONNECTION\"=>\"close\", \"REQUEST_METHOD\"=>\"GET\"}" to include "\"REMOTE_ADDR\"=>\"\""
    /home/wcolaiuta/work/thin/thin.git/spec/server/unix_socket_spec.rb:15:
    
    10)
    Errno::ENOENT in 'Thin::Server on UNIX domain socket should handle GET in less then 0.002 RubySecond'
    No such file or directory - /tmp/thin_test.sock
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:154:in `initialize'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:154:in `new'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:154:in `send_data'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:166:in `get'
    /home/wcolaiuta/work/thin/thin.git/spec/server/unix_socket_spec.rb:19:
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:29:in `times'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:29:in `matches?'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:28:in `matches?'
    /home/wcolaiuta/work/thin/thin.git/spec/server/unix_socket_spec.rb:19:
    
    13)
    Errno::ECONNREFUSED in 'Thin::Server on Swiftiply should GET from Net::HTTP'
    Connection refused - connect(2)
    /home/wcolaiuta/work/thin/thin.git/spec/server/swiftiply_spec.rb:16:
    
    14)
    Errno::ECONNREFUSED in 'Thin::Server on Swiftiply should POST from Net::HTTP'
    Connection refused - connect(2)
    /home/wcolaiuta/work/thin/thin.git/spec/server/swiftiply_spec.rb:20:
    
    15)
    'Thin::UnixConnection should return remote_address' FAILED
    expected: "127.0.0.1",
         got: nil (using ==)
    /home/wcolaiuta/work/thin/thin.git/spec/connectors/unix_server_spec.rb:41:
    
    17)
    'Thin::Request legacy Mongrel tests should raise error on large mangled field values' FAILED
    expected Thin::InvalidRequest but nothing was raised
    /home/wcolaiuta/work/thin/thin.git/spec/request/mongrel_spec.rb:11:
    

    May be something wrong with my local setup, so I am going to try and reproduce on another machine.

  • macournoyer

    macournoyer March 1st, 2008 @ 02:09 PM

    hey Wincent if you would like to help w/ this I can give you a github invite so you can fork the Thin repo. It's much better then patch ping-pong.

    Just send me your email.

  • Wincent Colaiuta

    Wincent Colaiuta March 1st, 2008 @ 01:49 PM

    Sure, it's win at wincent dot com.

  • Wincent Colaiuta

    Wincent Colaiuta March 1st, 2008 @ 02:14 PM

    Ok, on the subject of the failing specs, I get some of the failures when I try on another machine. Will open a separate ticket seeing as it's really a separate issue.

  • Wincent Colaiuta

    Wincent Colaiuta March 1st, 2008 @ 02:01 PM

    Ok, opened up ticket #51 for tracking down the spec failures.

  • macournoyer

    macournoyer March 2nd, 2008 @ 12:56 AM

    • Milestone changed from 0.8.0 to 0.7.1

    I moved all perf specs under spec/perf and exclude them from rake spec. Run through rake spec:perf now.

    thx for the suggestion Wincent!

  • macournoyer

    macournoyer March 2nd, 2008 @ 12:56 AM

    • State changed from “open” to “resolved”

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

People watching this ticket

Tags

Referenced by

Pages