Benchmarks

[simone-sanfratello]

Hi,

those are the benchmarks I ran on my laptop, executing "hello world" servers very close to what is in this repo "benchmarks" folder.

Laptop: Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz 12 GB RAM

deno 1.1.3

fastify 3.0.2

fastro 0.13.11

node 12.1.8.1

The results are totally different to https://deno.land/x/fastro#benchmarks.

How do you get that?

Thank you

[ynwd]

https://github.com/fastrodev/fastro/blob/master/benchmarks/readme.md
https://github.com/fastrodev/fastro/blob/v0.13.11/benchmarks/benchmark.js

[trivikr]

The numbers run on my machine match with that in README:

System information

$ sw_vers                           
ProductName:	Mac OS X
ProductVersion:	10.14.6
BuildVersion:	18G3020

$ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz

Deno 1.1.3

$ deno --version
deno 1.1.3
v8 8.5.216
typescript 3.9.2

$ deno run --allow-net deno_http.ts &
[1] 86853
$ deno_http listen on 3005

$ autocannon -c100 localhost:3005 
Running 10s test @ http://localhost:3005
100 connections

┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬───────────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%   │ Avg     │ Stdev   │ Max       │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼───────────┤
│ Latency │ 3 ms │ 4 ms │ 8 ms  │ 10 ms │ 4.44 ms │ 3.83 ms │ 318.86 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴───────────┘
┌───────────┬────────┬────────┬─────────┬─────────┬──────────┬─────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%     │ 97.5%   │ Avg      │ Stdev   │ Min    │
├───────────┼────────┼────────┼─────────┼─────────┼──────────┼─────────┼────────┤
│ Req/Sec   │ 14351  │ 14351  │ 20111   │ 22575   │ 20200.41 │ 2112.53 │ 14347  │
├───────────┼────────┼────────┼─────────┼─────────┼──────────┼─────────┼────────┤
│ Bytes/Sec │ 732 kB │ 732 kB │ 1.03 MB │ 1.15 MB │ 1.03 MB  │ 108 kB  │ 732 kB │
└───────────┴────────┴────────┴─────────┴─────────┴──────────┴─────────┴────────┘

Req/Bytes counts sampled once per second.

202k requests in 10.07s, 10.3 MB read

$ npm run kill

> benchmark@1.0.0 kill /Users/trivikr/workspace/fastro/benchmarks
> node benchmark.js kill

$ 
[1]  + terminated  deno run --allow-net deno_http.ts

Node 14.5.0

$ node -v
v14.5.0

$ node node_http &
[1] 86992
$ node listening on: 3006

$ autocannon -c100 localhost:3006 
Running 10s test @ http://localhost:3006
100 connections

┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬───────────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%   │ Avg     │ Stdev   │ Max       │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼───────────┤
│ Latency │ 2 ms │ 5 ms │ 10 ms │ 14 ms │ 5.14 ms │ 4.03 ms │ 253.45 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴───────────┘
┌───────────┬────────┬────────┬─────────┬─────────┬──────────┬─────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%     │ 97.5%   │ Avg      │ Stdev   │ Min    │
├───────────┼────────┼────────┼─────────┼─────────┼──────────┼─────────┼────────┤
│ Req/Sec   │ 8871   │ 8871   │ 18911   │ 19567   │ 17876.41 │ 3046.34 │ 8868   │
├───────────┼────────┼────────┼─────────┼─────────┼──────────┼─────────┼────────┤
│ Bytes/Sec │ 931 kB │ 931 kB │ 1.99 MB │ 2.05 MB │ 1.88 MB  │ 320 kB  │ 931 kB │
└───────────┴────────┴────────┴─────────┴─────────┴──────────┴─────────┴────────┘

Req/Bytes counts sampled once per second.

179k requests in 10.06s, 18.8 MB read

$ npm run kill

> benchmark@1.0.0 kill /Users/trivikr/workspace/fastro/benchmarks
> node benchmark.js kill

$
[1]  + terminated  node node_http

[trivikr]

Verified that benchmarks shared in fastro are incorrect, as date and connection headers are not sent.

Details in #8

[simone-sanfratello]

Thank you @trivikr

[ynwd]

#8

Thank you @trivikr

I take the hello-world example from here:
https://deno.land/#getting-started

[trivikr]

It's visible from responses on #6 and #8 that fastro maintainers are not interested in sharing the real benchmarks and want to just boast about fake benchmarks. It won't help in the long run, and gives a bad name for Deno community.

Good news is that there are some serious players among Deno web frameworks (like drash) who take the responsibility drashland/drash#322

[ynwd]

@trivikr this benchmark is only to compare the speed of displaying 'hello-world'.

btw, I already update and move the benchmark to https://github.com/fastrodev/fastrodev.github.io/blob/master/benchmarks/deno_http.ts

the speed becomes a little slower.