Skip to content

S
slash
Unverified Commit c4670945 authored by Nils Goroll's avatar Nils Goroll
Browse files
Options

doc: Add an updated performance report

parent 63b003ec
Hide whitespace changes
Inline Side-by-side
Showing with 1035 additions and 8 deletions
README.rst
View file @ c4670945
......@@ -254,6 +254,8 @@ xxhash are being used.
Benchmark
---------
.. _wrk: https://github.com/wg/wrk
On a test system
* HPE ProLiant DL380 Gen10 Plus
......@@ -264,12 +266,12 @@ On a test system
configured with
* 26TB storage on a RAID-0 raw device over the SSDs
* 300GB RAM cache using 1GB huge pages
* 32GB RAM cache using 1GB huge pages
under
* Ubuntu 20.04.4 LTS (focal)
* kernel 5.15.0-50-generic
* Ubuntu 20.04.6 LTS (focal)
* kernel 5.15.0-94-generic
and benchmarked with
......@@ -277,7 +279,7 @@ and benchmarked with
* consequently, objsize_hint = 2MB
* cURL requests to localhost over http
* `wrk`_ requests to localhost over http
* the tuning parameters::
......@@ -285,12 +287,12 @@ and benchmarked with
hash_obj=xxh3_64,
hash_log=xxh3_64,
mem_reserve_chunks=512,
discard=true
discard_immediate=2MB
fellow typically outperforms the following minimum performance
metrics:
* 40 GiB/s for memory cache hits
* 50 - 60 GiB/s for memory cache hits
* 14 GiB/s for cache misses on an empty storage
......@@ -298,8 +300,12 @@ metrics:
discard requests
These figures are in Gibibytes per second as reported by varnishstat
in ``MAIN.s_resp_bodybytes`` and are not to be confused with figures
in Gigabits per second.
in `wrk`_ and are not to be confused with figures in Gigabits per
second.
.. _dev-03.rst: doc/perf/dev-03.rst
See `dev-03.rst`_ for details.
ACKNOWLEDGEMENTS
================
......
doc/perf/dev-03.rst 0 → 100644
View file @ c4670945
===============================================
SLASH/fellow PERFORMANCE TEST dev-03 2024-02-08
===============================================
This is a report about a SLASH/fellow performance test on a fairly
high end system with NVMe flash storage and 512GB DRAM, of which 32GB
were used as SLASH/fellow memory cache.
The test uses object sizes similar to those commonly used for HLS
video streaming (random sizes 1 Byte - 4MB, ~ 2MB average).
The amount of used RAM was kept deliberately low in order to be able
to report performance of working sets larger than the fellow memory
cache within reasonable test durations.
*Note* that for production use, where the goal is to keep all of the
working set in memory, using more RAM is (almost) always better.
Results
=======
The table below gives an overview of the measured transfer rate
against ``varnishd`` listening on the loopback interface over http (no
TLS) with different working *set size*\ s, before and after a
restart. *empty* is for a first test run where the cache is initially
empty, *restarted* is for a second test run where the objects from the
first run are read from disk after restarting ``varnishd``.
Set size `>` demarks the first test with a working set size of
10000000000 objects, which will only ever trigger cache misses.
Note that the numbers are in net Giga\ *bytes* per second as reported
by `wrk`_. To convert to gross network throughput in Gigabits per
second, they need to be multiplied by a factor of 8-10.
======== ============ ============
set size empty restarted
======== ============ ============
2GB 62.84 GB/s 57.19 GB/s
4GB 53.25 GB/s 59.76 GB/s
8GB 49.37 GB/s 57.55 GB/s
16GB 51.99 GB/s 52.50 GB/s
32GB 45.68 GB/s 50.43 GB/s
64GB 27.41 GB/s 30.83 GB/s
128GB 19.79 GB/s 19.54 GB/s
256GB 17.73 GB/s 16.69 GB/s
512GB 16.99 GB/s 15.23 GB/s
1TB 16.71 GB/s 15.03 GB/s
2TB 16.38 GB/s 14.43 GB/s
> 14.25 GB/s 13.95 GB/s
======== ============ ============
Full output of the test script
==============================
prep: clearing cache
--------------------
::
Present bans:
1707409784.643409 0 - obj.status != 0
...
Present bans:
1707409784.643409 0 - obj.status != 0
empty cache, all misses
-----------------------
::
Running 10s test @ http://localhost:8081/10000000000
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 141.36ms 77.95ms 1.10s 88.76%
Req/Sec 75.49 25.31 272.00 65.08%
74014 requests in 10.13s, 144.27GB read
Requests/sec: 7309.73
Transfer/sec: 14.25GB
1024 objects (working set ~ 2GB)
--------------------------------------------------
::
Running 1s test @ http://localhost:8081/1024
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.83ms 22.67ms 289.48ms 93.62%
Req/Sec 308.37 78.83 0.96k 77.36%
36409 requests in 1.10s, 69.23GB read
Requests/sec: 33045.68
Transfer/sec: 62.84GB
2048 objects (working set ~ 4GB)
--------------------------------------------------
::
Running 2s test @ http://localhost:8081/2048
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 27.82ms 32.25ms 327.00ms 90.91%
Req/Sec 272.79 86.50 1.13k 74.34%
57688 requests in 2.10s, 111.85GB read
Requests/sec: 27463.14
Transfer/sec: 53.25GB
4096 objects (working set ~ 8GB)
--------------------------------------------------
::
Running 4s test @ http://localhost:8081/4096
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 32.44ms 39.31ms 404.54ms 89.04%
Req/Sec 254.16 82.24 0.93k 72.53%
104196 requests in 4.10s, 203.92GB read
Requests/sec: 25407.92
Transfer/sec: 49.73GB
8192 objects (working set ~ 16GB)
--------------------------------------------------
::
Running 8s test @ http://localhost:8081/8192
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 68.92ms 123.39ms 1.45s 88.77%
Req/Sec 266.55 115.02 1.03k 74.07%
215175 requests in 8.10s, 421.17GB read
Requests/sec: 26562.08
Transfer/sec: 51.99GB
16384 objects (working set ~ 32GB)
--------------------------------------------------
::
Running 16s test @ http://localhost:8081/16384
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 66.48ms 77.43ms 847.50ms 85.07%
Req/Sec 234.92 112.20 810.00 67.91%
376585 requests in 16.10s, 735.36GB read
Requests/sec: 23392.24
Transfer/sec: 45.68GB
32768 objects (working set ~ 64GB)
--------------------------------------------------
::
Running 32s test @ http://localhost:8081/32768
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 84.15ms 81.71ms 1.11s 72.37%
Req/Sec 142.93 61.16 393.00 67.53%
448060 requests in 32.10s, 0.86TB read
Requests/sec: 13958.75
Transfer/sec: 27.41GB
65536 objects (working set ~ 128GB)
--------------------------------------------------
::
Running 1m test @ http://localhost:8081/65536
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 112.17ms 96.43ms 1.03s 66.83%
Req/Sec 105.79 52.14 440.00 63.60%
658011 requests in 1.09m, 1.26TB read
Requests/sec: 10107.51
Transfer/sec: 19.79GB
131072 objects (working set ~ 256GB)
--------------------------------------------------
::
Running 2m test @ http://localhost:8081/131072
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 120.59ms 97.33ms 1.73s 72.39%
Req/Sec 94.81 47.17 316.00 66.63%
1190491 requests in 2.19m, 2.27TB read
Requests/sec: 9080.70
Transfer/sec: 17.73GB
262144 objects (working set ~ 512GB)
--------------------------------------------------
::
Running 4m test @ http://localhost:8081/262144
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 126.66ms 106.20ms 1.99s 77.20%
Req/Sec 90.80 46.05 450.00 63.55%
2281390 requests in 4.37m, 4.35TB read
Socket errors: connect 0, read 0, write 0, timeout 17
Requests/sec: 8704.25
Transfer/sec: 16.99GB
524288 objects (working set ~ 1024GB)
--------------------------------------------------
::
Running 9m test @ http://localhost:8081/524288
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 125.96ms 95.79ms 1.71s 75.94%
Req/Sec 89.05 43.89 525.00 65.60%
4475935 requests in 8.73m, 8.55TB read
Requests/sec: 8540.24
Transfer/sec: 16.71GB
1048576 objects (working set ~ 2048GB)
--------------------------------------------------
::
Running 17m test @ http://localhost:8081/1048576
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 130.25ms 106.56ms 1.96s 79.82%
Req/Sec 87.80 44.61 727.00 66.23%
8787846 requests in 17.47m, 16.76TB read
Socket errors: connect 0, read 0, write 0, timeout 2
Requests/sec: 8384.55
Transfer/sec: 16.38GB
restart
-------
::
Running 10s test @ http://localhost:8081/10000000000
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 161.77ms 146.43ms 1.25s 92.27%
Req/Sec 76.00 27.92 180.00 71.72%
72295 requests in 10.10s, 140.96GB read
Requests/sec: 7155.24
Transfer/sec: 13.95GB
1024 objects (working set ~ 2GB) after restart
-------------------------------------------------------------------
::
Running 1s test @ http://localhost:8081/1024
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 50.86ms 73.05ms 600.43ms 85.23%
Req/Sec 293.02 143.39 830.00 72.09%
32949 requests in 1.10s, 63.07GB read
Requests/sec: 29876.83
Transfer/sec: 57.19GB
2048 objects (working set ~ 4GB) after restart
-------------------------------------------------------------------
::
Running 2s test @ http://localhost:8081/2048
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 49.08ms 86.17ms 753.01ms 88.92%
Req/Sec 306.04 103.27 1.11k 77.73%
64817 requests in 2.10s, 125.59GB read
Requests/sec: 30840.80
Transfer/sec: 59.76GB
4096 objects (working set ~ 8GB) after restart
-------------------------------------------------------------------
::
Running 4s test @ http://localhost:8081/4096
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 44.45ms 84.96ms 917.34ms 91.05%
Req/Sec 294.62 83.95 0.90k 76.46%
120824 requests in 4.10s, 235.92GB read
Requests/sec: 29471.44
Transfer/sec: 57.55GB
8192 objects (working set ~ 16GB) after restart
-------------------------------------------------------------------
::
Running 8s test @ http://localhost:8081/8192
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 58.27ms 118.64ms 1.27s 91.20%
Req/Sec 269.79 93.60 1.09k 73.47%
217370 requests in 8.10s, 425.31GB read
Requests/sec: 26833.96
Transfer/sec: 52.50GB
16384 objects (working set ~ 32GB) after restart
-------------------------------------------------------------------
::
Running 16s test @ http://localhost:8081/16384
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 63.02ms 72.98ms 347.36ms 83.48%
Req/Sec 259.04 112.07 1.01k 70.11%
415709 requests in 16.10s, 811.88GB read
Requests/sec: 25820.65
Transfer/sec: 50.43GB
32768 objects (working set ~ 64GB) after restart
-------------------------------------------------------------------
::
Running 32s test @ http://localhost:8081/32768
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 72.91ms 68.41ms 533.65ms 62.97%
Req/Sec 160.81 61.82 424.00 72.56%
504557 requests in 32.10s, 0.97TB read
Requests/sec: 15717.99
Transfer/sec: 30.83GB
65536 objects (working set ~ 128GB) after restart
-------------------------------------------------------------------
::
Running 1m test @ http://localhost:8081/65536
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 109.26ms 86.58ms 767.05ms 64.31%
Req/Sec 104.89 54.63 292.00 59.21%
649283 requests in 1.08m, 1.24TB read
Requests/sec: 9973.95
Transfer/sec: 19.54GB
131072 objects (working set ~ 256GB) after restart
-------------------------------------------------------------------
::
Running 2m test @ http://localhost:8081/131072
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 124.46ms 91.55ms 759.16ms 69.71%
Req/Sec 90.25 49.81 270.00 59.86%
1120997 requests in 2.18m, 2.14TB read
Requests/sec: 8550.87
Transfer/sec: 16.69GB
262144 objects (working set ~ 512GB) after restart
-------------------------------------------------------------------
::
Running 4m test @ http://localhost:8081/262144
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 135.80ms 97.39ms 1.06s 72.55%
Req/Sec 83.03 47.79 250.00 59.70%
2046475 requests in 4.37m, 3.90TB read
Requests/sec: 7807.97
Transfer/sec: 15.23GB
524288 objects (working set ~ 1024GB) after restart
-------------------------------------------------------------------
::
Running 9m test @ http://localhost:8081/524288
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 135.57ms 90.77ms 835.85ms 71.20%
Req/Sec 81.26 46.11 250.00 58.33%
4027469 requests in 8.73m, 7.69TB read
Requests/sec: 7684.59
Transfer/sec: 15.03GB
1048576 objects (working set ~ 2048GB) after restart
-------------------------------------------------------------------
::
Running 17m test @ http://localhost:8081/1048576
100 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 142.05ms 97.44ms 899.92ms 73.12%
Req/Sec 78.65 46.44 260.00 57.65%
7744866 requests in 17.47m, 14.77TB read
Requests/sec: 7389.45
Transfer/sec: 14.43GB
System load during test
=======================
During the test, the system was typically more than 50% idle, which is
an indication that the test is I/O bound::
$ vmstat 60
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 69464 202410736 50912 7709452 0 0 10757 3149 3 1 1 3 96 0 0
223 7 69464 192664416 51296 9724232 0 0 4919 2513731 314741 107234 8 6 86 1 0
0 0 69464 191686400 51384 9722648 0 0 9400968 2317697 371648 335905 11 62 26 1 0
4 1 69464 191398560 51472 9722012 0 0 14482218 2253856 148776 273507 9 36 53 2 0
1 0 69464 191155888 51568 9721696 0 0 13731747 4386963 160242 288917 14 37 47 2 0
1 0 69464 191127328 51664 9721700 0 0 15596486 104986 112777 225355 3 28 67 2 0
49 2 69464 190893488 51760 9721516 0 0 11456120 6893480 174569 322040 21 37 39 3 0
140 0 69460 191604944 51856 9721552 0 0 16940128 1856907 146230 266669 7 34 57 2 0
1 1 69460 191851152 51952 9721556 0 0 15846608 197853 118554 228458 3 27 67 2 0
161 0 69460 192602560 52040 9721560 0 0 15617619 34962 110662 219406 3 26 69 2 0
349 22 69460 191145472 52128 9721500 0 0 11464849 3825948 154921 255680 13 28 57 2 0
9 0 69460 194706704 52224 9721536 0 0 10762823 9767782 177420 350682 28 44 26 2 0
140 0 69460 194933472 52320 9721536 0 0 17323713 3018133 155549 283714 9 37 51 2 0
195 0 69460 195315664 52416 9721536 0 0 17074078 886169 128156 249735 5 31 62 2 0
1 0 69460 195446016 52512 9721540 0 0 16301019 310213 113480 228364 3 28 67 2 0
1 0 69460 195797584 52600 9721548 0 0 15787320 118627 109553 217403 3 26 69 2 0
183 1 69460 196596080 52688 9721548 0 0 16168332 50680 109715 220010 3 27 68 2 0
0 1 69460 197314048 52784 9721552 0 0 15344664 22341 103398 210420 3 25 70 2 0
2 1 69460 197726352 52872 9721552 0 0 15048547 11594 106499 210265 3 25 70 2 0
122 3 69460 191424720 52960 9721200 0 0 8355773 7740166 155856 283934 24 32 41 2 0
52 0 69460 192555280 53056 9721200 0 0 8130211 12345205 176456 371017 36 46 15 4 0
25 10 69460 193987600 53152 9721200 0 0 14001051 7109300 169061 312055 20 43 35 1 0
1 0 69456 194631600 53248 9721200 0 0 16980765 3887962 157294 284129 11 40 47 2 0
108 0 69456 195083008 53344 9721204 0 0 17339439 2039765 137748 266830 7 34 56 2 0
5 0 69456 195574816 53440 9721208 0 0 16986540 1118933 124968 252264 5 31 62 3 0
59 0 69456 196240432 53536 9721216 0 0 16443208 657238 116955 236182 4 29 65 2 0
0 0 69456 196791536 53632 9721216 0 0 16215713 389106 113128 229558 3 28 66 3 0
0 0 69448 197199312 53728 9721220 0 0 15636864 237601 108900 219082 3 26 68 2 0
150 0 69448 197080448 53824 9721224 0 0 15655835 159539 107972 212760 3 26 68 2 0
0 1 69448 197839232 53912 9721228 0 0 15894937 98273 109535 216729 3 26 68 2 0
0 0 69448 198089536 54000 9721232 0 0 15212431 64716 104308 212601 3 25 70 2 0
1 0 69448 198243360 54088 9721168 0 0 15002059 43079 100363 207347 3 24 71 2 0
0 0 69448 198409984 54176 9721168 0 0 15446733 30559 104785 211869 3 25 70 2 0
0 0 69448 198381728 54264 9721172 0 0 15404809 21132 105374 210776 3 24 70 2 0
5 32 69448 198384496 54352 9721172 0 0 15821002 15101 102391 212307 3 25 70 2 0
138 1 69448 198431200 54440 9721176 0 0 15598402 12377 104204 211207 3 24 71 2 0
1 0 69448 198583408 54536 9721176 0 0 14495220 8934 96281 199489 2 23 73 2 0
101 0 69444 191447712 55700 9720884 0 0 2981196 2554744 223879 295601 10 44 45 1 0
3 1 69444 191509024 55764 9721112 0 0 17330795 5351 142737 308914 4 44 51 1 0
1 0 69444 191541488 55844 9721176 0 0 15775044 4910 118275 233402 3 30 65 2 0
19 31 69444 191491008 55916 9721184 0 0 15932532 4486 109474 226725 3 28 67 2 0
1 0 69444 191543312 55996 9721252 0 0 15761865 4865 110525 223501 3 28 67 2 0
0 0 67624 191511616 56292 9724244 26 0 15082416 6423 101589 208644 3 26 70 2 0
0 0 67624 191944960 56372 9724244 0 0 15317962 4562 102098 214086 3 26 69 2 0
22 4 67624 192789696 56444 9724252 0 0 15170384 4332 103927 208423 3 25 70 2 0
140 1 67624 193643824 56516 9724260 0 0 15060120 5734 103666 212408 3 25 70 2 0
0 0 67616 194664128 56604 9724328 0 0 15262156 6906 107187 211481 3 25 70 2 0
163 0 67608 195470176 56692 9724332 0 0 15612739 4633 101540 214699 3 26 69 2 0
0 1 67608 196387904 56772 9724336 0 0 15247511 5143 101167 215174 3 25 70 2 0
3 1 67608 196962464 56844 9724336 0 0 15091545 4524 101300 210783 3 25 70 2 0
1 1 67608 197226432 56916 9724336 0 0 15680795 4463 102286 216023 3 26 69 2 0
1 0 67608 197509072 56980 9724340 0 0 15749257 5769 106821 214870 3 26 69 2 0
50 163 64260 197431664 60208 9805496 72 0 15256303 4531 102914 212934 3 25 70 2 0
1 0 64260 197514800 60296 9805564 0 0 15179336 4565 98408 211279 3 25 70 2 0
13 5 64260 197439936 60368 9805596 0 0 15307416 4651 101366 208875 3 25 70 2 0
4 0 64260 197495152 60448 9805600 0 0 15320412 8094 103271 207944 3 25 70 2 0
206 2 64260 197046592 60536 9805600 0 0 14524664 6463 95388 199545 2 23 72 2 0
92 2 64260 197376816 60632 9805600 0 0 14932795 5901 100394 206855 3 24 71 2 0
108 1 64260 197303952 60720 9805600 0 0 15324613 7192 103941 209523 3 25 70 2 0
0 0 64260 197536832 60800 9805600 0 0 14983817 4976 99700 204073 3 24 71 2 0
7 0 64260 197481856 60880 9805600 0 0 15126241 4756 101454 207305 3 24 71 2 0
37 65 64260 197541152 60952 9805600 0 0 14818391 4755 95401 206983 2 23 72 2 0
4 1 64240 197538688 61048 9805604 0 0 14911826 4705 98213 203294 2 24 72 2 0
1 0 64240 197541760 61120 9805604 0 0 14941184 3269 96913 204241 2 23 72 2 0
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 1 64240 197465056 61192 9805604 0 0 15119574 4408 100326 208374 2 24 71 2 0
1 0 64240 197554672 61280 9805608 0 0 15167651 5987 103529 206401 3 24 71 2 0
1 0 64240 197556960 61360 9805608 0 0 15040158 4481 101229 206013 2 24 71 2 0
1 0 64240 197505984 61448 9805608 0 0 15061703 4469 96975 205275 2 24 72 2 0
1 0 64240 197546096 61520 9805608 0 0 14924359 4400 98635 204376 2 23 72 2 0
1 0 64240 197555296 61600 9805608 0 0 14665006 4360 96295 201570 2 23 72 2 0
2 1 64240 197486528 61664 9805608 0 0 14814272 4354 96405 205111 2 23 72 2 0
1 0 64236 197564416 61736 9805608 0 0 15022690 4398 99067 205219 2 24 72 2 0
0 0 64228 197757632 61784 9805608 0 0 6468547 3277 44011 88976 1 10 88 1 0
0 0 64228 197763264 61848 9805608 0 0 0 10 457 602 0 0 100 0 0
Full output of the varnishd start script
========================================
::
$ ./start_varnish.sh
++ id -u
+ sudo chown 1001 /dev/mapper/slash-nvmeraid0
+ pkill varnishd
+ true
+ pkill varnishlog
+ true
+ ulimit -c unlimited
+ ulimit -l unlimited
+ ulimit -n 1048576
+ make -j 20 check
Making check in src
make[1]: Entering directory '/home/slink/src/slash/src'
make check-am
make[2]: Entering directory '/home/slink/src/slash/src'
if test "x." != "x." ; then \
s=`cd . && pwd`; \
for f in `cd $s && find . -type f -name \*.rst`; do \
d=`dirname $f`; \
test -d $d || mkdir -p $d; \
test -f $f || ln -s $s/$f $f; \
done \
fi
make check-TESTS check-local
make[3]: Entering directory '/home/slink/src/slash/src'
make[4]: Entering directory '/home/slink/src/slash/src'
PASS: bitf_segmentation_test
PASS: vtc/buddy_edgecases.vtc
PASS: vtc/buddy_c00062.vtc
PASS: vtc/buddy_global.vtc
PASS: vtc/fellow_vcl.vtc
PASS: vtc/buddy_vcl.vtc
PASS: vtc/buddy_c00075.vtc
PASS: vtc/vut_slashmap.vtc
PASS: vtc/loadmasters.vtc
PASS: vtc/fellow_c00062.vtc
PASS: vtc/fellow_cl.vtc
PASS: vtc/buddy_c00097.vtc
PASS: vtc/buddy_t02005.vtc
PASS: vtc/fellow_c00093.vtc
PASS: buddy_test_when
PASS: buddy_test
PASS: buddy_test_witness
PASS: vtc/fellow_coverage.vtc
PASS: vtc/fellow_t02005.vtc
PASS: vtc/fellow_global.vtc
PASS: vtc/fellow_b00062.vtc
PASS: fellow_log_test_ndebug
PASS: vtc/fellow_global_shared.vtc
PASS: fellow_log_test
PASS: fellow_cache_test
============================================================================
Testsuite summary for slash trunk-1.0.0-rc1
============================================================================
# TOTAL: 25
# PASS: 25
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
make[4]: Leaving directory '/home/slink/src/slash/src'
make[3]: Leaving directory '/home/slink/src/slash/src'
make[2]: Leaving directory '/home/slink/src/slash/src'
make[1]: Leaving directory '/home/slink/src/slash/src'
make[1]: Entering directory '/home/slink/src/slash'
make[1]: Leaving directory '/home/slink/src/slash'
+ make -j 20 install
Making install in src
make[1]: Entering directory '/home/slink/src/slash/src'
make install-am
make[2]: Entering directory '/home/slink/src/slash/src'
if test "x." != "x." ; then \
s=`cd . && pwd`; \
for f in `cd $s && find . -type f -name \*.rst`; do \
d=`dirname $f`; \
test -d $d || mkdir -p $d; \
test -f $f || ln -s $s/$f $f; \
done \
fi
make[3]: Entering directory '/home/slink/src/slash/src'
if test "x." != "x." ; then \
s=`cd . && pwd`; \
for f in `cd $s && find . -type f -name \*.rst`; do \
d=`dirname $f`; \
test -d $d || mkdir -p $d; \
test -f $f || ln -s $s/$f $f; \
done \
fi
/bin/mkdir -p '/tmp/bin'
/bin/mkdir -p '/tmp/share/doc/slash'
/bin/mkdir -p '/tmp/lib/varnish/vmods'
/usr/bin/install -c -m 644 vmod_slash.vcc '/tmp/share/doc/slash'
/bin/bash ../libtool --mode=install /usr/bin/install -c libvmod_slash.la '/tmp/lib/varnish/vmods'
/bin/bash ../libtool --mode=install /usr/bin/install -c slashmap buddy_test buddy_test_when bitf_segmentation_test '/tmp/bin'
/bin/mkdir -p '/tmp/share/man/man1'
/bin/mkdir -p '/tmp/share/man/man3'
/bin/mkdir -p '/tmp/share/man/man7'
/usr/bin/install -c -m 644 slashmap.1 '/tmp/share/man/man1'
/usr/bin/install -c -m 644 slash-counters.7 '/tmp/share/man/man7'
/usr/bin/install -c -m 644 vmod_slash.3 '/tmp/share/man/man3'
libtool: install: /usr/bin/install -c .libs/libvmod_slash.so /tmp/lib/varnish/vmods/libvmod_slash.so
libtool: install: /usr/bin/install -c slashmap /tmp/bin/slashmap
libtool: install: /usr/bin/install -c .libs/libvmod_slash.lai /tmp/lib/varnish/vmods/libvmod_slash.la
libtool: install: /usr/bin/install -c buddy_test /tmp/bin/buddy_test
libtool: install: /usr/bin/install -c buddy_test_when /tmp/bin/buddy_test_when
libtool: install: /usr/bin/install -c bitf_segmentation_test /tmp/bin/bitf_segmentation_test
libtool: finish: PATH="/tmp/sbin:/tmp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin" ldconfig -n /tmp/lib/varnish/vmods
----------------------------------------------------------------------
Libraries have been installed in:
/tmp/lib/varnish/vmods
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the 'LD_RUN_PATH' environment variable
during linking
- use the '-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to '/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[3]: Leaving directory '/home/slink/src/slash/src'
make[2]: Leaving directory '/home/slink/src/slash/src'
make[1]: Leaving directory '/home/slink/src/slash/src'
make[1]: Entering directory '/home/slink/src/slash'
make[2]: Entering directory '/home/slink/src/slash'
make[2]: Nothing to be done for 'install-exec-am'.
/bin/mkdir -p '/tmp/share/doc/slash'
/usr/bin/install -c -m 644 README.rst COPYING LICENSES/BSD-2-Clause LICENSES/LGPL-2.1 '/tmp/share/doc/slash'
make[2]: Leaving directory '/home/slink/src/slash'
make[1]: Leaving directory '/home/slink/src/slash'
+ vlogopt=(-q 'vxid == 0' -g raw -i Storage -n /tmp/t -t off)
+ typeset -ra vlogopt
+ pids=()
+ typeset -a pids
+ pids+=($!)
+ /tmp/bin/varnishlog -d -q 'vxid == 0' -g raw -i Storage -n /tmp/t -t off
+ pids+=($!)
+ args=(-a 127.0.0.1:8081 -f $PWD/fellow_test_allcache2mb.vcl -n /tmp/t -E /tmp/lib/varnish/vmods/libvmod_slash.so -sfellow=fellow,/dev/mapper/slash-nvmeraid0,26TB,32GB,2MB -p 'auto_restart=off' -p 'startup_timeout=3600' -p 'thread_pool_watchdog=600')
+ typeset -ra args
+ /tmp/bin/varnishlog -q 'vxid == 0' -g raw -i Storage -n /tmp/t -t off
+ /tmp/sbin/varnishd -a 127.0.0.1:8081 -f /home/slink/src/slash/fellow_test_allcache2mb.vcl -n /tmp/t -E /tmp/lib/varnish/vmods/libvmod_slash.so -sfellow=fellow,/dev/mapper/slash-nvmeraid0,26TB,32GB,2MB -p auto_restart=off -p startup_timeout=3600 -p thread_pool_watchdog=600 -F
EEE </tmp/lib/varnish/vmods/libvmod_slash.so>
eee </tmp/lib/varnish/vmods/libvmod_slash.so>
ee2 vext_cache/libvmod_slash.so,ewoockee.so
Warnings:
Message from VCC-compiler:
FOUND VMOD in VEXT ../vext_cache/libvmod_slash.so,ewoockee.so
GOOD VMOD slash in VEXT ../vext_cache/libvmod_slash.so,ewoockee.so
IMPORT slash from VEXT
Debug: Version: varnish-trunk revision f0dc9964e953fbcc11869957f11382098be0a9ab
Debug: Platform: Linux,5.15.0-94-generic,x86_64,-jnone,-sfellow,-sdefault,-Elibvmod_slash.so,-hcritbit
Debug: Child (292840) Started
0 Storage - fellow fellow: loading...
0 Storage - ... done: 0.079294s
0 Storage - fellow: rewrite (unknown) obj_add: 0 obj_chg: 0 obj_del_alloced: 0 obj_del_free: 0 obj_del_thin: 0 reg_add: 0 reg_del_alloced: 0 reg_del_free: 0 ban_add_imm: 0 ban_add_reg: 0 ban_exp_imm: 0 ban_exp_reg: 0
0 Storage - fellow fellow: 0 resurrected in 0.079841s (0.000000/s), 0 already expired
Child launched OK
Info: Child (292840) said Child starts
Info: Child (292840) said Loaded -E vext_cache/libvmod_slash.so,ewoockee.so
Info: Child (292840) said mmap(34359738368, MAP_HUGETLB | MAP_HUGE_1GB) succeeded
Info: Child (292840) said fellow: metadata (bitmap) memory: 136351480 bytes
Info: Child (292840) said fellow: /dev/mapper/slash-nvmeraid0: empty block 0, will initialize
Info: Child (292840) said fellow: hdr objsize 0, objsz_hint 2097152
Info: Child (292840) said io_uring ctx 0x7f5d89d02780 registering 32 buffers
Info: Child (292840) said io_uring ctx 0x7f5d89d028c0 registering 32 buffers
Info: Child (292840) said fellow: metadata (bitmap) memory: 1772530408 bytes
Info: Child (292840) said io_uring ctx 0x7f5d89d02a00 registering 32 buffers
Info: Child (292840) said fellow: region[0 = active] 2097152/2097152
Info: Child (292840) said fellow: region[1 = empty] 4194304/2097152
Info: Child (292840) said fellow: region[2 = pend] 6291456/2097152
Info: Child (292840) said io_uring ctx 0x7f5d89d02b40 registering 32 buffers
0 Storage - fellow: take new region[2 = empty] 6291456/2097152 -> 4294967296/3988180992
0 Storage - bitfs 1 segments
0 Storage - fellow: blkdiscard works, enabling async
0 Storage - ... done: 4.460371s
0 Storage - fellow: rewrite (unknown) obj_add: 0 obj_chg: 0 obj_del_alloced: 0 obj_del_free: 0 obj_del_thin: 0 reg_add: 0 reg_del_alloced: 0 reg_del_free: 0 ban_add_imm: 0 ban_add_reg: 0 ban_exp_imm: 1 ban_exp_reg: 0
0 Storage - fellow: take new region[0 = empty] 2097152/2097152 -> 8589934592/3988180992
0 Storage - bitfs 1 segments
0 Storage - ... done: 0.121003s
0 Storage - fellow: rewrite (unknown) obj_add: 0 obj_chg: 0 obj_del_alloced: 0 obj_del_free: 0 obj_del_thin: 0 reg_add: 0 reg_del_alloced: 0 reg_del_free: 0 ban_add_imm: 0 ban_add_reg: 0 ban_exp_imm: 1 ban_exp_reg: 0
0 Storage - fellow: take new region[1 = empty] 4194304/2097152 -> 12884901888/3988180992
0 Storage - bitfs 1 segments
0 Storage - ... done: 0.118580s
Debug: Stopping Child
0 Storage - fellow: rewrite (unknown) obj_add: 2170660 obj_chg: 525624 obj_del_alloced: 0 obj_del_free: 63 obj_del_thin: 46 reg_add: 2172944 reg_del_alloced: 0 reg_del_free: 63 ban_add_imm: 1 ban_add_reg: 0 ban_exp_imm: 4 ban_exp_reg: 0
0 Storage - bitfs 1 segments
0 Storage - ... done: 1.088788s
0 Storage - bitfs 1 segments
0 Storage - ... done: 1.670546s
Info: Child (292840) said Child dies
Info: Child (292840) ended
Debug: Child cleanup complete
Debug: Child (296832) Started
0 Storage - fellow fellow: loading...
0 Storage - bitfs 1 segments
0 Storage - fellow fellow: resurrected 100
0 Storage - fellow fellow: resurrected 323000
0 Storage - fellow fellow: resurrected 602000
0 Storage - fellow fellow: resurrected 833000
0 Storage - fellow fellow: resurrected 1065000
0 Storage - fellow fellow: resurrected 1293000
0 Storage - fellow fellow: resurrected 1546000
0 Storage - fellow fellow: resurrected 1788000
0 Storage - fellow fellow: resurrected 2039000
0 Storage - ... done: 12.805422s
Child launched OK
0 Storage - fellow fellow: system init until cache load t1 = 4.234890
0 Storage - fellow fellow: 2170546 resurrected in 8.608215s (252148.801654/s), 5 already expired
Info: Child (296832) said Child starts
Info: Child (296832) said Loaded -E vext_cache/libvmod_slash.so,ewoockee.so
Info: Child (296832) said mmap(34359738368, MAP_HUGETLB | MAP_HUGE_1GB) succeeded
Info: Child (296832) said fellow: metadata (bitmap) memory: 136351480 bytes
Info: Child (296832) said fellow: /dev/mapper/slash-nvmeraid0: hdr 9 generation 890
Info: Child (296832) said fellow: hdr objsize 2103412, objsz_hint 2097152
Info: Child (296832) said io_uring ctx 0x7f5d89d12780 registering 32 buffers
Info: Child (296832) said io_uring ctx 0x7f5d89d128c0 registering 32 buffers
Info: Child (296832) said fellow: metadata (bitmap) memory: 1772530408 bytes
Info: Child (296832) said io_uring ctx 0x7f5d89d12a00 registering 32 buffers
Info: Child (296832) said fellow: region[0 = empty] 8589934592/3988180992
Info: Child (296832) said fellow: region[1 = pend] 12884901888/3988180992
Info: Child (296832) said fellow: region[2 = active] 4294967296/3988180992
Info: Child (296832) said io_uring ctx 0x7f5d89d12b40 registering 32 buffers
0 Storage - fellow: blkdiscard works, enabling async
Setup Information
=================
system
------
::
# inxi -Fxz
System: Kernel: 5.15.0-94-generic x86_64 bits: 64 compiler: N/A Console: tty 0 Distro: Ubuntu 20.04.6 LTS (Focal Fossa)
Machine: Type: Server Mobo: HPE model: ProLiant DL380 Gen10 Plus serial: <filter> UEFI: HPE v: U46 date: 11/03/2021
CPU: Topology: 2x 24-Core model: Intel Xeon Gold 6336Y bits: 64 type: MT MCP SMP arch: N/A L2 cache: 72.0 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 461812
Speed: 800 MHz min/max: 800/3600 MHz Core speeds (MHz): 1: 800 2: 801 3: 801 4: 801 5: 801 6: 801 7: 801 8: 800
9: 801 10: 800 11: 896 12: 801 13: 801 14: 801 15: 801 16: 801 17: 800 18: 801 19: 800 20: 801 21: 801 22: 801
23: 801 24: 1074 25: 801 26: 801 27: 801 28: 801 29: 801 30: 801 31: 801 32: 800 33: 801 34: 801 35: 801 36: 801
37: 801 38: 801 39: 800 40: 801 41: 801 42: 801 43: 801 44: 800 45: 801 46: 801 47: 800 48: 801 49: 801 50: 801
51: 801 52: 800 53: 800 54: 801 55: 801 56: 801 57: 801 58: 801 59: 801 60: 801 61: 801 62: 801 63: 801 64: 801
65: 801 66: 801 67: 801 68: 801 69: 801 70: 898 71: 801 72: 801 73: 801 74: 801 75: 801 76: 801 77: 801 78: 801
79: 801 80: 801 81: 801 82: 801 83: 801 84: 801 85: 801 86: 801 87: 801 88: 801 89: 800 90: 801 91: 801 92: 801
93: 1056 94: 800 95: 801 96: 801
...
Drives: Local Storage: total: 54.73 TiB used: 7.82 GiB (0.0%)
ID-1: /dev/nvme0n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
ID-2: /dev/nvme1n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
ID-3: /dev/nvme2n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
ID-4: /dev/nvme3n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
ID-5: /dev/nvme4n1 model: HPE NS204i-p Gen10+ Boot Controller size: 447.07 GiB
ID-6: /dev/nvme5n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
ID-7: /dev/nvme6n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
ID-8: /dev/nvme7n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
ID-9: /dev/nvme8n1 model: MZXLR3T8HBLS-000H3 size: 3.49 TiB
lvm setup
---------
This script configures a RAID0 setup. For production use, we recommend RAID10.
::
devs=(
/dev/nvme0n1
/dev/nvme1n1
/dev/nvme2n1
/dev/nvme3n1
/dev/nvme5n1
/dev/nvme6n1
/dev/nvme7n1
/dev/nvme8n1
)
set -eux
pvcreate "${devs[@]}" || true
if ! vgs slash 2>/dev/null ; then
vgcreate slash "${devs[@]}"
fi
lvcreate -i "${#devs[@]}" -I 4M \
-l 95%FREE \
-n nvmeraid0 slash
versions
--------
* liburing: 5a1b0c9b358aa8edfdfddc7f10a4821697073df9
* varnish-cache: f0dc9964e953fbcc11869957f11382098be0a9ab
* libvmod-blobsynth: 288d5706329d8d1d527bfa2b2bef738688dfa6d9
* slash: 63b003ec461f756ef34bc310ae286030317b731d
VCL
---
The VCL has been written to generate random numerical URLs between 1
and a maximum number from the request in order do avoid complications
in the test client.
Cache objects are generated with a random length between 1 Byte and
4MB, resulting in an average of 2MB.
In order to measure the actual stevedore efficiency and not some
backend's performance, the VCL itself generates cache objects: A 4MB
blob with random data is used, from which random lengths of data are
taken.
This shell command was used to create a 4MB file with random data as
base64::
dd if=/dev/urandom bs=$((4*1024*1024)) count=1 | base64 -w0 >4mb.b64
The file contents were then literally pasted in the initializer of the
``data`` blob, which is left out here for clarity::
vcl 4.1;
import slash;
import std;
import blob;
import blobsynth;
backend none none;
sub vcl_init {
slash.tune_fellow(storage.fellow,
chunk_bytes = 2MB,
hash_obj=xxh3_64,
hash_log=xxh3_64,
mem_reserve_chunks=512,
discard_immediate=2MB
);
slash.as_transient(storage.fellow);
new data = blob.blob(BASE64, "*** READ INFO ABOVE ***");
}
sub vcl_recv {
# url specifies number of objects to hit
set req.url = "/" + std.integer(real=std.random(1,std.real(regsub(req.url, "^/", ""))));
return (hash);
}
sub vcl_backend_fetch {
set bereq.backend = none;
}
sub vcl_backend_error {
set beresp.storage = storage.fellow;
set beresp.status = 200;
set beresp.ttl = 1y;
set beresp.grace = 0s;
blobsynth.synthetic(
blob.sub(
data.get(),
std.bytes(real=std.random(0,4194304))));
return (deliver);
}
sub vcl_backend_response {
set beresp.storage = storage.fellow;
set beresp.http.storage = beresp.storage;
}
Varnish-Cache Startup
---------------------
SLASH/fellow is configured to use 26TB NVME storage and 32GB RAM in
huge pages of 1GB::
#!/bin/bash
set -eux
# grub:
# GRUB_CMDLINE_LINUX="hugepagesz=1GB hugepages=300"
##[[ $(cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages) -eq 300 ]]
sudo chown $(id -u) /dev/mapper/slash-nvmeraid0
pkill varnishd || true
pkill varnishlog || true
ulimit -c unlimited
ulimit -l unlimited
ulimit -n $((1024*1024))
make -j 20 check
make -j 20 install
typeset -ra vlogopt=(
-q 'vxid == 0'
-g raw
-i Storage
-n /tmp/t
-t off
)
typeset -a pids=()
/tmp/bin/varnishlog -d "${vlogopt[@]}" 2>/dev/null &
pids+=($!)
/tmp/bin/varnishlog "${vlogopt[@]}" 2>/dev/null &
pids+=($!)
typeset -ra args=(
-a 127.0.0.1:8081
-f $PWD/fellow_test_allcache2mb.vcl
-n /tmp/t
-E /tmp/lib/varnish/vmods/libvmod_slash.so
-sfellow=fellow,/dev/mapper/slash-nvmeraid0,26TB,32GB,2MB
-p 'auto_restart=off'
-p 'startup_timeout=3600'
-p 'thread_pool_watchdog=600'
)
/tmp/sbin/varnishd \
"${args[@]}" -F
Test script
-----------
.. _wrk: https://github.com/wg/wrk
As the test client, `wrk`_ is used from a shell script. First, the
cache is cleared, then a test with only cache misses is run, followed
by the tests with differing working set sizes. Then the cache is
restarted (such that fellow reads all objects from disk) and the tests
re-run::
#!/bin/bash
set -eu
typeset -ra va=(varnishadm -n /tmp/t)
typeset -ra wrk=(../wrk/wrk -t 100 -c 1000)
echo prep: clearing cache
echo --------------------
echo
echo ::
echo
(
"${va[@]}" param.set ban_lurker_age 1
"${va[@]}" ban obj.status != 0
"${va[@]}" ban.list
echo ...
while [[ $("${va[@]}" ban.list | awk '{ sum += $2 } END { print sum }') -gt 0 ]] ; do
sleep 1
done
"${va[@]}" ban.list
) | sed 's:^: :'
echo
echo empty cache, all misses
echo -----------------------
echo
echo ::
echo
"${wrk[@]}" -H 'Host: miss' http://localhost:8081/10000000000 2>&1 |
sed 's:^: :'
echo
for e in {10..20} ; do
n=$((2**e))
echo $n objects "(working set ~ $((2 * n / 1024))GB)"
echo --------------------------------------------------
echo
echo ::
echo
"${wrk[@]}" -d $((n / 1000))s -H "Host: ${e}" http://localhost:8081/$n |
sed 's:^: :'
echo
done
echo restart
echo -------
echo
echo ::
echo
(
"${va[@]}" stop
"${va[@]}" -t 3600 start
"${wrk[@]}" -H 'Host: miss' http://localhost:8081/10000000000 2>&1
) | sed 's:^: :'
echo
for e in {10..20} ; do
n=$((2**e))
echo $n objects "(working set ~ $((2 * n / 1024))GB)" after restart
echo -------------------------------------------------------------------
echo
echo ::
echo
"${wrk[@]}" -d $((n / 1000))s -H "Host: ${e}" http://localhost:8081/$n |
sed 's:^: :'
echo
done
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment