Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

REQUEST: Separate performance specs from other specs

#50

I've been trying the Thin 0.7.0 specs out on RHEL 5.1 and I'm seeing what I would consider to be a bunch of spurious failures. For example:

12)
'Thin::Server on TCP socket should handle POST in less then 0.007 RubySecond' FAILED
took <0.008859357696567 RubySeconds>, should take less than 0.007 RubySeconds.
/home/wcolaiuta/work/thin/thin.git/spec/server/tcp_spec.rb:44:

So the code is working, but its going slower than the specs say it should. Is this a bug in Thin, or is it that the granularity and cross-machine stability of "RubySeconds" isn't fine enough? (I suspect the latter, but the point is this kind of failure is probably spurious and is reducing the value of the spec suite as a means of confirming the correctness of the code.)

That's just one example; out of 17 failures, 4 of them are benchmark_unit specs which miss out by an insignificant margin.

I think there's a clear need to separate the specs into two different classes: those that confirm the correct behaviour of the code, and those that check the performance (speed) of the code.

Basically, as an end user I want to be able to run the spec suit and see that everything is working correctly. Checking that everything is working fast is a separate concern and I want to test it separately.

And if you might want to increase the margins in your RubySeconds specifications, but I think that's a less important issue than keeping them isolated.

Now I am going to investigate the other 13 failures (on your development platform you're seeing a 100% pass rate right? or am I running into known issues here?).

Reported by Wincent Colaiuta · March 1st, 2008 @ 09:58 AM

State: open
Milestone: 0.8.0
Assigned to: macournoyer macournoyer

Activity

  1. macournoyer
    macournoyer
    • Milestone set to 0.8.0
    • State changed from new to open

    Sure all specs pass on my machine, I'm a big TDD/BDD fan ;)

    benchmark_unit is supposed to be machine independent but I guess it's not 100% correct.

    I like the idea of isolating the perf specs, I'll try to do that.

    Can you paste the output of your failures?

    March 1st, 2008 @ 01:46 PM

  2. Wincent Colaiuta
    Wincent Colaiuta

    Yes; here are the timing-related failures:

    2)
    'Thin::Response should be fast' FAILED
    took <0.000141749723145072 RubySeconds>, should take less than 0.00011 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/response_spec.rb:63:
    
    11)
    'Thin::Server on TCP socket should handle GET in less then 0.0045 RubySecond' FAILED
    took <0.00973499188750674 RubySeconds>, should take less than 0.0045 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/server/tcp_spec.rb:40:
    
    12)
    'Thin::Server on TCP socket should handle POST in less then 0.007 RubySecond' FAILED
    took <0.008859357696567 RubySeconds>, should take less than 0.007 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/server/tcp_spec.rb:44:
    
    16)
    'Thin::Request performance should be faster then 0.0002 RubySeconds' FAILED
    took <0.000255813953488372 RubySeconds>, should take less than 0.0002 RubySeconds.
    /home/wcolaiuta/work/thin/thin.git/spec/request/perf_spec.rb:21:
    

    And here are the other failing cases (haven't looked into them very closely yet):

    1)
    'Thin::Connection should return remote_address' FAILED
    expected: "127.0.0.1",
         got: nil (using ==)
    /home/wcolaiuta/work/thin/thin.git/spec/connection_spec.rb:55:
    
    3)
    NoMethodError in 'Thin::Logging should output debug when set to true'
    protected method `debug' called for #<TestLogging:0xb7c7cfec>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:16:
    
    4)
    NoMethodError in 'Thin::Logging should output trace when set to true'
    protected method `trace' called for #<TestLogging:0xb7c7b4e4>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:22:
    
    5)
    NoMethodError in 'Thin::Logging should not output when silenced'
    protected method `log' called for #<TestLogging:0xb7c79810>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:28:
    
    6)
    NoMethodError in 'Thin::Logging should not output when silenced [deprecated]'
    protected method `log' called for #<TestLogging:0xb7c78078>
    /home/wcolaiuta/work/thin/thin.git/spec/logging_spec.rb:36:
    
    7)
    'Daemonizing should send kill signal if timeout' FAILED
    expected false, got true
    /home/wcolaiuta/work/thin/thin.git/spec/daemonizing_spec.rb:112:
    
    8)
    NoMethodError in 'Thin::Server app builder should work with Rack url mapper'
    You have a nil object when you didn't expect it!
    You might have expected an instance of Array.
    The error occurred while evaluating nil.+
    /home/wcolaiuta/work/thin/thin.git/spec/server/builder_spec.rb:36:
    
    9)
    'Thin::Server on UNIX domain socket should retreive remote address' FAILED
    expected "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 519\r\nConnection: close\r\nServer: thin 0.7.0 codename Spherical Cow\r\n\r\n{\"rack.url_scheme\"=>\"http\", \"PATH_INFO\"=>\"/\", \"rack.run_once\"=>false, \"rack.input\"=>#<StringIO:0xb7df6e04>, \"SCRIPT_NAME\"=>\"\", \"SERVER_PROTOCOL\"=>\"HTTP/1.1\", \"rack.errors\"=>#<IO:0xb7ef501c>, \"REMOTE_ADDR\"=>\"V\", \"REQUEST_PATH\"=>\"/\", \"SERVER_SOFTWARE\"=>\"thin 0.7.0 codename Spherical Cow\", \"HTTP_VERSION\"=>\"HTTP/1.1\", \"rack.multithread\"=>false, \"rack.version\"=>[0, 2], \"REQUEST_URI\"=>\"/\", \"rack.multiprocess\"=>false, \"QUERY_STRING\"=>\"\", \"GATEWAY_INTERFACE\"=>\"CGI/1.2\", \"HTTP_CONNECTION\"=>\"close\", \"REQUEST_METHOD\"=>\"GET\"}" to include "\"REMOTE_ADDR\"=>\"\""
    /home/wcolaiuta/work/thin/thin.git/spec/server/unix_socket_spec.rb:15:
    
    10)
    Errno::ENOENT in 'Thin::Server on UNIX domain socket should handle GET in less then 0.002 RubySecond'
    No such file or directory - /tmp/thin_test.sock
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:154:in `initialize'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:154:in `new'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:154:in `send_data'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:166:in `get'
    /home/wcolaiuta/work/thin/thin.git/spec/server/unix_socket_spec.rb:19:
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:29:in `times'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:29:in `matches?'
    /home/wcolaiuta/work/thin/thin.git/spec/controllers/../spec_helper.rb:28:in `matches?'
    /home/wcolaiuta/work/thin/thin.git/spec/server/unix_socket_spec.rb:19:
    
    13)
    Errno::ECONNREFUSED in 'Thin::Server on Swiftiply should GET from Net::HTTP'
    Connection refused - connect(2)
    /home/wcolaiuta/work/thin/thin.git/spec/server/swiftiply_spec.rb:16:
    
    14)
    Errno::ECONNREFUSED in 'Thin::Server on Swiftiply should POST from Net::HTTP'
    Connection refused - connect(2)
    /home/wcolaiuta/work/thin/thin.git/spec/server/swiftiply_spec.rb:20:
    
    15)
    'Thin::UnixConnection should return remote_address' FAILED
    expected: "127.0.0.1",
         got: nil (using ==)
    /home/wcolaiuta/work/thin/thin.git/spec/connectors/unix_server_spec.rb:41:
    
    17)
    'Thin::Request legacy Mongrel tests should raise error on large mangled field values' FAILED
    expected Thin::InvalidRequest but nothing was raised
    /home/wcolaiuta/work/thin/thin.git/spec/request/mongrel_spec.rb:11:
    

    May be something wrong with my local setup, so I am going to try and reproduce on another machine.

    March 1st, 2008 @ 02:08 PM

  3. macournoyer
    macournoyer

    hey Wincent if you would like to help w/ this I can give you a github invite so you can fork the Thin repo. It's much better then patch ping-pong.

    Just send me your email.

    March 1st, 2008 @ 02:09 PM

  4. Wincent Colaiuta
    Wincent Colaiuta

    Sure, it's win at wincent dot com.

    March 1st, 2008 @ 01:49 PM

  5. Wincent Colaiuta
    Wincent Colaiuta

    Ok, on the subject of the failing specs, I get some of the failures when I try on another machine. Will open a separate ticket seeing as it's really a separate issue.

    March 1st, 2008 @ 02:14 PM

  6. Wincent Colaiuta
    Wincent Colaiuta

    Ok, opened up ticket #51 0.7.0 (and master) spec failures for tracking down the spec failures.

    March 1st, 2008 @ 02:01 PM

  7. macournoyer
    macournoyer
    • Milestone changed from 0.8.0 to 0.7.1

    I moved all perf specs under spec/perf and exclude them from rake spec. Run through rake spec:perf now.

    thx for the suggestion Wincent!

    March 2nd, 2008 @ 12:56 AM

  8. macournoyer

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.