Hello this is another set of benchmarks of the hinsightd webserver. Again hinsightd v0.10 is a multithreaded server and the current branch under active development while hinsightd v0.9 is a singlethreaded server with more features but also more bugs.

What we test here is how fast the server responds to increases in concurrent connections. While this is not a perfect metric for servers, a webserver that can respond to more requests in a given time period is not only more likely to resist a DDoS attack but you can also add more processing per request before the delays become noticeable. Please consult a full list of features before deciding which webserver to use for your own site.

index.html

Testing a small 102 bytes file. This tests how fast a server can open a connection, parse headers, and close the connection, nothing that impressive. But it should be noted the multithreaded branch is the fastest server tested.

test	hinsightd/0.9.19	hinsightd/0.10.1	nginx/1.25.3	lighttpd/1.4.72	Apache/2.4.58	caddy/2.7.5
index.html -c 50	93850.01	127296.1	117020.65	56887.03	40069.56	37654.71
index.html -c 100	95571.23	121204.78	114876.51	56396.17	41065.91	36727.71
index.html -c 250	90076.29	106621.17	106724.72	53473.93	38698.64	35235.71
index.html -c 500	78003.12	91496.33	91003.4	50046.54	35879.2	33626.33
index.html -c 750	71083.31	83508.7	82136.87	50651.63	35209.66	32187.05
index.html -c 1000	66276.52	77152.16	76078.6	48633.87	33906.01	31376.68

jquery.js

For a more realistic test, the current jquery library, a 89.7kB minified js file. Unfortunately my server doesn't support sendfile so to make the test 'fair' I've disabled it across the board. I actually believe without sendfile this is a more realistic benchmark because in the web development world of today you rarely have unencrypted or uncompressed connections. This should test how fast a server is able to get data from the disk and push it to the http client with minimal changes.

In previous benchmarks (and in the 0.9 branch) the lack of TCP CORK flag really slowed down everything, adding support for it vastly improved performance but it's still under other servers. Further research and optimization needed.

test	hinsightd/0.9.19	hinsightd/0.10.1	nginx/1.25.3	lighttpd/1.4.72	Apache/2.4.58	caddy/2.7.5
jquery.js -c 50	2103.32	28276.54	36029.15	18746.64	26397.06	28822.85
jquery.js -c 100	3288.06	27483.85	32395.48	18618.75	27594.5	28424.11
jquery.js -c 250	5974.85	25550.88	31045.68	18116.5	24566.16	27583.62
jquery.js -c 500	10033.39	23537.39	30233.58	17595.81	23997.39	26233.43
jquery.js -c 750	10200.92	21585.31	29622.17	17745.63	23632.95	25614.69
jquery.js -c 1000	9777.69	21598.88	28680.33	17556.98	23047.9	24684.84

index.html via ssl

While I would like to benchmark SSL connections, previous tests proved that without a secondary machine this is impossible, testing SSL connections is very CPU intensive so without another machine it just tests how fast the benchmark software can create connections on a CPU bottlenecked system, not very useful data.

If you find these tests useful a more comprehensive set of tests and benchmarks is include with the source code in the external/tests directory.

Notes:

1. tests were ran on a 4 core machine with no hyperthreading. hinsightd 0.10.1 and nginx have 4 workers/threads enabled the rest are left at the default.
2. tests are run 6 times for each concurrency level and only the highest result is taken into account.
3. both hinsightd branches are using the current HEAD.
4. nginx config
5. lighttd config
6. the apache2 config is too complicated to post, but it's the distribution default.
7. caddy config

This post is about concurrency tests for the hinsightd server. Hinsightd v0.9 is a purely single threaded/single process webserver while the new v0.10 rewrite is a multithreaded webserver using mostly the same code as hinsightd v0.9. Currently the rewrite is still a work in progress so it doesn't have all the features of the previous version.

Edit: a flaw was detected in the testing methodology so the results are fairly useless.

index.html

Testing against a small file. This test basically tests how fast a server can open a connection, parse headers, and close the connection. I have to say I'm surprised of the results in my tests my development branch is always faster than nginx but for these tests I had to disable access logging in nginx, I did not expect such a large different. In any case time for optimizations.

jquery.js

This tests against a medium sized file. My own server seems to have some persistent bandwidth issues that I haven't pinpointed yet. The apache2 line is so bad because apache keeps throwing errors, usually it gets fixed by increasing the number of iterations but for medium sized file this takes too long for me to test.

index.html via ssl

This test is the one that surprises me the most. I wanted to implement threads in the server because it should logically increase ssl throughput but this test looks exactly the opposite of that. I have to investigate further, I don't understand why threading would decrease performance for ssl processing unless openssl is single threaded but as far as I know this is not the case.

Edit: This benchmark is basically useless because I tested on a single machine so both the benchmark software and the server were fighting for the same resources, this is true for the tests above too but if the cpu is not bottlenecked the effect is slightly less. I will retest some day when I get access to another linux machine.

test	hinsightd/0.10.1	hinsightd/0.9.19	nginx/1.25.2	lighttpd/1.4.71	Apache/2.4.57	caddy/2.7.4
index.html -c 50	88235.55	89009.15	111221.1	57542.03	41061.69	37059.68
index.html -c 100	113072.29	90904.96	109440.32	57628.58	41711.33	36194.77
index.html -c 250	93982.31	86229.94	103420.1	54405.19	38868.61	35264.54
index.html -c 500	82594.8	75005.81	90039.8	50410.34	38918.99	33320.12
index.html -c 750	77411.36	68771.53	81939.0	50894.73	38114.99	32053.03
index.html -c 1000	72974.06	64592.39	75759.87	49168.32	37654.15	31623.95
jquery.js -c 50	2534.11	2205.48	30930.45	17948.91	26170.89	28317.06
jquery.js -c 100	10683.82	3621.08	28285.42	17870.61	25567.6	28060.88
jquery.js -c 250	13657.51	6018.14	27123.35	17457.67	24235.43	27269.8
jquery.js -c 500	13657.47	8494.7	26590.51	17128.92	23576.01	25962.63
jquery.js -c 750	13221.6	8376.68	25870.81	17267.22	6619.22	25533.33
jquery.js -c 1000	12419.48	8177.4	24944.75	16978.42	648.32	25040.0
ssl_index.html -c 50	29414.62	33393.0	33263.59	1725.03	14528.04	16864.54
ssl_index.html -c 100	21528.99	26042.68	23523.49	1726.26	5700.76	12044.7
ssl_index.html -c 250	11090.64	14406.29	12238.76	1395.36	5073.35	5912.12
ssl_index.html -c 500	6121.92	9036.61	6503.25	1055.53	3271.95	2937.05
ssl_index.html -c 750	4135.2	6644.76	4363.63	1274.92	3496.11	1950.87
ssl_index.html -c 1000	3081.8	5260.24	3370.99	987.59	5125.96	1460.06

Notes:

1. tests were ran on a 4 core machine with no hyperthreading with 4 threads enabled in: hinsightd 0.10.1 and nginx.
2. tests are run 5 times for each concurrency level and only the highest result is taken into account
3. nginx config
4. lighttd config
5. the apache2 config is complicated and has many files so I will not post it here, it's also mostly the distribution default
6. caddy config
7. due to testing on a single machine the ssl benchmark is basically useless

New benchmarks are here, after several optimization patches and some unexplained performance improvements, here we are.

Optimization takes a lot of time and depends on trial and error plus modern magic, old code is rarely in a good shape, frequently breaks at the lightest change, insufficient documentation also hampers optimization efforts, and that dreaded feeling of "what does this do again ?" is encountered way too often. But the biggest problem by far is motivation, it's very hard to justify spending 90% time to increase performance by 10%; despite all of this sometimes magic prevails and the stars align to improve performance. Though I have to be honest I wish I knew why moving 3 lines of code to another function increases performance by 15k req/s.

So on to the tests, spoiler alert we managed to beat Lighttpd.

for 250 concurrency ab -k -c 250 -n 10000 http://localhost:<port>/

server and version	failed requests	requests/sec (mean)
hinsightd/0.9.17	0	63035.01
nginx/1.23.2	0	26673.64
lighttpd/1.4.67	0	53693.29
Apache/2.4.54	414	37474.10
Caddy/2.6.2	0	35412.02

for 500 concurrency ab -k -c 500 -n 10000 http://localhost:<port>/

server and version	failed requests	requests/sec (mean)
hinsightd/0.9.17	0	54984.63
nginx/1.23.2	0	26172.73
lighttpd/1.4.67	215	1613.59
Apache/2.4.54	1221	34305.55
Caddy/2.6.2	0	33995.57

Notes:

Please note I am a server developer not a server administrator and might not know of all the ways to increase performance of other servers while I know how to get every last bit of performance out of my own. So please take these kind of tests with a drop of salt.

Due to rapid io uring development and possible variance, it's needs to be noted we are using linux kernel version 6.0.9.

I wish I had the time to investigate why Lighttpd starts going very slowly when overloaded.

Performance usually comes with tradeoffs. These can take many forms, from skipping security checks to only optimizing a small part of the program that is being benchmarked. If you let performance guide a design other aspects of a program can suffer. Or in other words it doesn't matter how fast a broken program runs.

Despite optimization not being a priority some benchmarks are so interesting that are worth sharing. The condensing patch referenced below condenses multiple network write request into a single one.

for 500 concurrency ab -k -c 500 -n 10000 http://localhost:<port>/

server and version	requests/sec (mean)
before patches	22790.46
-O2 alone	23167.02
After the condensing patch	26393.86
After patch + -O2	30257.64
nginx/1.21.6	26683.67
lighttpd/1.4.64	fails at 500 concurrency
Apache/2.4.54	fails at 500 concurrency

bonus for 100 concurrency due to other server giving errors ab -k -c 100 -n 10000 http://localhost:<port>/

server and version	requests/sec (mean)
after patch + -O2	32136.88
nginx/1.21.6	26788.39
lighttpd/1.4.64	56971.62
Apache/2.4.54	fails at 100 concurrency

Switching from -Os to -O3 has a small performance impact, but reducing write requests adds around 4k req/s (about 20% faster). Looking at the other servers defaults I have to say I'm confused, I thought nginx would be faster, it does handle concurrency better than either apache or lighttpd though. Also it seems lighttpd is really fast at static requests.

While testing consumes time and I can't say I'm a big fan of optimization, free performance with minimal tradeoffs is best performance.

Now on to figuring out why file transfers are so slow.

They say brevity is the soul of wit. This is expecially true for programming where the more you write the farther from efficiency you get.

Even knowing that I always struggle to make managable git commits. I have no problem writing small code, but whenever version control is involved the desire to fix x and y and z gets the better of me.

Sure smaller commits are easier to review, but what about the cognitive load of fighting your urge to write long commits ? It might sound like a joke but is it really ?