Note that this is a synthetic microbenchmark, not a real world benchmark. In the real world, other effects often swamp the kinds of things measured here.
Poller_bench sets up an array of socketpairs, has a Poller monitor the read end of each socketpair, and measures how long it takes to execute the following snippet of code with various styles of Poller:
for (k=0; k<num_active; k++)
write(fdpairs[k * spacing + spacing/2][1], "a", 1);
poller.waitAndDispatchEvents(0);
where spacing = num_pipes / num_active.
poller.waitAndDispatchEvents() calls
poll() or
ioctl(m_dpfd, DP_POLL, &dopoll),
as appropriate, then calls an event handler for each ready socket.
The event handler for this benchmark just executes
read(event->fd, buf, 1);
cd /dev mknod poll u 15 125 chmod 666 /dev/pollwhere 15 is MISC_MAJOR and 125 is DEVPOLL_MINOR from the kernel sources; your MISC_MAJOR may differ, be sure to check /usr/src/linux/include/linux/major.h for the definition of MISC_MAJOR on your system.
cd /usr/include/asm ln -s ../linux/devpoll.h
Also, a line near the end of /usr/include/sys/poll_impl.h may need to be moved to get it to compile when included from C++ programs.
On Linux, if you want kernel profile results, boot with argument 'profile=2' to enable the kernel's builtin profiler.
Run the shell script Poller_bench.sh as follows:
su sh Poller_bench.sh
The script raises file descriptor limits, then runs the command
./Poller_bench 5 1 spd 100 1000 10000
It should be run on an idle machine, with no email client, web browser, or X server running. The Pentium III machine at my disposal was running a single sshd; the Solaris machine was running two sshd's and an idle XWindow server, so it wasn't quite as idle.
On a 167MHz sun4u Sparc Ultra-1 running SunOS 5.7 (Solaris 7) Generic_106541-11:
pipes 100 1000 10000
select 151 - -
poll 470 676 3742
/dev/poll 61 70 92
165133 microseconds to open each of 10000 socketpairs
29646 microseconds to close each of 10000 socketpairs
On a 4X400Mhz Enterprise 450 running Solaris 8 (results contributed by Doug Lea):
pipes 100 1000 10000
select 60 - -
poll 273 388 1559
/dev/poll 27 28 34
116586 microseconds to open each of 10000 socketpairs
19235 microseconds to close each of 10000 socketpairs
(The machine wasn't idle, but at most one CPU was doing other stuff
during test, and the test seemed to occupy only one CPU.)
On an idle 650 MHz dual Pentium III running Red Hat Linux 6.2
with kernel 2.2.14smp plus the /dev/poll patch plus Dave Miller's
patch to speed up close():
pipes 100 1000 10000
select 28 - -
poll 23 890 11333
/dev/poll 19 146 4264
(Time to open or close socketpairs was not recorded, but was under 14 microseconds.)
On the same machine as above, but with kernel 2.4.0-test10-pre4 smp:
pipes 100 1000 10000
select 52 - -
poll 49 1184 14660
26 microseconds to open each of 10000 socketpairs
14 microseconds to close each of 10000 socketpairs
(Note: the /dev/poll patch does not apply cleanly to recent 2.4.0-test kernels, I believe,
and I did not try it.)
On a single processor 600Mhz Pentium-III with 512MB of memory, running FreeBSD 4.x-STABLE
(results contributed by Jonathan Lemon):
pipes 100 1000 10000 30000
select 54 - - -
poll 50 552 11559 35178
kqueue 8 8 8 8
(Note: Jonathan also varied the number of active
pipes, and found that kqueue's time scaled linearly with
that number, whereas poll's time scaled linearly with
number of total pipes.)
The test was also run with pipes instead of socketpairs (results not shown); the performance on Solaris was about the same, but the /dev/poll driver on Linux did not perform well with pipes. According to Niels Provos,
The hinting code which causes a considerable speed up for /dev/poll only applies to network sockets. If there are any serious applications that make uses of pipes in a manner that would benefit from /dev/poll then the pipe code needs to return hints too.
2.4.0-test10-pre4 was slower than 2.2.14 in all cases tested.
I should show results for pipes as well as socketpairs.
The Linux 2.2.14 /dev/poll driver printed messages to the console when sockets were closed; this should probably be disabled for production.
The 2.2.14 Linux /dev/poll driver was about six times faster than poll() for 1000 fds, but fell down to only 2.7 times faster at 10000 fds. The Solaris /dev/poll driver was about seven times faster than poll() at 100 fds, and increased to 40 times faster at 10000 fds.
Under Linux 2.2.14, when the number of idle sockets was increased from 100 to 10000, the time to check for active sockets with poll() and /dev/poll increased by a factor of 493 and 224, respectively. This is terribly, horribly bad scaling behavior.
Under Linux 2.4.0-test10-pre4, when the number of idle sockets was increased from 100 to 10000, the time to check for active sockets with poll() increased by a factor of 300. This is terribly, horribly bad scaling behavior.
There seems to be a scalability problem in poll() under both Linux 2.2.14 and 2.4.0-test10-pre4 and in /dev/poll under Linux 2.2.14.
poll() is stuck with an interface that dictates O(n) behavior on total pipes; still, Linux's implementation could be improved. The design of the current Linux /dev/poll patch is O(n) in total pipes, in spite of the fact that its interface allows it to be O(1) in total pipes and O(n) only in active pipes.
See also the recent discussions on linux-kernel.
If you run the above test on a Linux system booted with 'profile=2', Poller_bench will output one kernel profiling data file per test condition. Poller_bench.sh does a gross analysis using 'readprofile | sort -rn | head > bench%d%c.top' to find the kernel functions with the highest CPU usage, where %d is the number of socketpairs, and %c is p for poll, d for /dev/poll, etc.
'more bench10000*.top' shows the results for 10000 socketpairs. On 2.2.14, it shows:
::::::::::::::
bench10000d.dat.top
::::::::::::::
901 total 0.0008
833 dp_poll 1.4875
27 do_bottom_half 0.1688
7 __get_request_wait 0.0139
4 startup_32 0.0244
3 unix_poll 0.0203
::::::::::::::
bench10000p.dat.top
::::::::::::::
584 total 0.0005
236 unix_poll 1.5946
162 sock_poll 4.5000
148 do_poll 0.6727
24 sys_poll 0.0659
7 __generic_copy_from_user 0.1167
This seems to indicate that /dev/poll spends nearly all of its time in dp_poll(),
and poll spends a fair bit of time in three routines: unix_poll, sock_poll, and do_poll.
On 2.4.0-test10-pre4 smp, 'more bench10000*.top' shows:
::::::::::::::
2.4/bench10000p.dat.top
::::::::::::::
1507 total 0.0011
748 default_idle 14.3846
253 unix_poll 1.9167
209 fget 2.4881
195 sock_poll 5.4167
29 sys_poll 0.0342
29 fput 0.1272
29 do_pollfd 0.1648
It seems curious that the idle routine should show up so much,
but it's probably just the second CPU doing nothing.
Poller_bench.sh will also try to do a fine analysis of dp_poll() using the 'profile' tool (source included), which is a variant of readprofile that shows hotspots within kernel functions. Looking at its output for the run on 2.2.14, the three four-byte regions that take up the most CPU time in dp_poll() in the 10000 socketpair case are
c01d9158 39.135654% 326 c01d9174 11.404561% 95 c01d91a0 27.250900% 227Looking at the output of 'objdump -d /usr/src/linux/vmlinux', that region corresponds to the object code: