uplex-varnish/varnishevent
7
0
Fork 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions

New output buffering #4

Open
slink wants to merge 2 commits from new_output_buffering into master
pull from: new_output_buffering
merge into: uplex-varnish:master
Conversation 10 Commits 2 Files changed 2 +257 -94
slink commented 2018-03-27 16:13:06 +00:00 (Migrated from code.uplex.de)
Copy link

Using varnishevent to output udp datagrams as with ...

#!/bin/bash
varnishevent ... >/dev/udp/$HOST/$PORT

... I ran into an issue with partial strings being received. The reason is that each UDP datagram is independent and data contained within cannot be continued in subsequent datagrams (and even if it was, that could get lost). So, for use with UDP, any datagram must contain complete records, which, in the case of varnishevent, are lines, and, consequently, records must not span write() system calls.

Initially I thought that setting output.bufsiz lower than the effective UDP unfragmented maximum packet size (output.bufsiz=1428) would help, but it turns out that, as varnishevent was based on stdio buffered output, that didn't help: except when setlinebuf() / setvbuf(..., _IOLBF, ...) was used, stdio (at least for glibc) always fills the buffer completely before issuing a system write(). (see https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/genops.c;h=2fec221b99729718d27cef453db2d3e0c2c56233;hb=HEAD#l379 https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/iofwrite.c;h=800341b7da546e5b7fd2005c5536f4c90037f50d;hb=HEAD#l39 )

As line buffering would lead to a significantly higher system call rate, I bit the bullet and implemented simple output buffering for varnishevent which never splits any records.

Using varnishevent to output udp datagrams as with ... ``` #!/bin/bash varnishevent ... >/dev/udp/$HOST/$PORT ``` ... I ran into an issue with partial strings being received. The reason is that each UDP datagram is independent and data contained within cannot be continued in subsequent datagrams (and even if it was, that could get lost). So, for use with UDP, any datagram must contain complete records, which, in the case of varnishevent, are lines, and, consequently, records must not span `write()` system calls. Initially I thought that setting `output.bufsiz` lower than the effective UDP unfragmented maximum packet size (`output.bufsiz=1428`) would help, but it turns out that, as `varnishevent` was based on stdio buffered output, that didn't help: except when `setlinebuf()` / `setvbuf(..., _IOLBF, ...)` was used, stdio (at least for glibc) always fills the buffer completely before issuing a system `write()`. (see https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/genops.c;h=2fec221b99729718d27cef453db2d3e0c2c56233;hb=HEAD#l379 https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/iofwrite.c;h=800341b7da546e5b7fd2005c5536f4c90037f50d;hb=HEAD#l39 ) As line buffering would lead to a significantly higher system call rate, I bit the bullet and implemented simple output buffering for varnishevent which never splits any records.
slink commented 2018-03-27 16:13:37 +00:00 (Migrated from code.uplex.de)
Copy link

Reassigned to @geoff

Reassigned to @geoff
slink commented 2019-05-30 18:37:46 +00:00 (Migrated from code.uplex.de)
Copy link

added 2 commits

  • 21e14e69 - 1 commit from branch master
  • e0e3eb7d - Merge branch 'master' into new_output_buffering

Compare with previous version

added 2 commits <ul><li>21e14e69 - 1 commit from branch <code>master</code></li><li>e0e3eb7d - Merge branch &#39;master&#39; into new_output_buffering</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=10&start_sha=b70c9d105b79155574e178d360199ada2a037698)
slink commented 2019-05-30 18:37:46 +00:00 (Migrated from code.uplex.de)
Copy link

added 2 commits

  • a6ac776e - 1 commit from branch master
  • 08a624ee - Merge branch 'master' into new_output_buffering

Compare with previous version

added 2 commits <ul><li>a6ac776e - 1 commit from branch <code>master</code></li><li>08a624ee - Merge branch &#39;master&#39; into new_output_buffering</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=10&start_sha=e0e3eb7dde1e8c81fbb1f757a86c699ea95a88f6)
slink commented 2019-05-30 18:40:00 +00:00 (Migrated from code.uplex.de)
Copy link

added 4 commits

  • 08a624ee...a00794a4 - 3 commits from branch master
  • a2c4b91e - Merge branch 'master' into new_output_buffering

Compare with previous version

added 4 commits <ul><li>08a624ee...a00794a4 - 3 commits from branch <code>master</code></li><li>a2c4b91e - Merge branch &#39;master&#39; into new_output_buffering</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=11&start_sha=08a624eec08b1b7220627348b68a0c17cba4fd3f)
slink commented 2021-06-29 13:45:09 +00:00 (Migrated from code.uplex.de)
Copy link

added 3 commits

  • 5b00968b - 1 commit from branch master
  • 2fbceeb8 - add a lightweight output buffering implementation
  • d040d901 - use our own output buffer

Compare with previous version

added 3 commits <ul><li>5b00968b - 1 commit from branch <code>master</code></li><li>2fbceeb8 - add a lightweight output buffering implementation</li><li>d040d901 - use our own output buffer</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=73&start_sha=a2c4b91ef1d6540d8404b5dd2adea43e6a126a40)
slink commented 2021-10-12 07:34:49 +00:00 (Migrated from code.uplex.de)
Copy link

added 3 commits

  • d7dc0556 - 1 commit from branch master
  • 1f44b3f8 - add a lightweight output buffering implementation
  • 02674327 - use our own output buffer

Compare with previous version

added 3 commits <ul><li>d7dc0556 - 1 commit from branch <code>master</code></li><li>1f44b3f8 - add a lightweight output buffering implementation</li><li>02674327 - use our own output buffer</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=77&start_sha=d040d90172bd525615a48482d98c3cba5666b6e5)
slink commented 2021-10-12 07:43:33 +00:00 (Migrated from code.uplex.de)
Copy link

added 3 commits

  • eb95c8bd - 1 commit from branch master
  • f103b4f6 - add a lightweight output buffering implementation
  • 2832c5b6 - use our own output buffer

Compare with previous version

added 3 commits <ul><li>eb95c8bd - 1 commit from branch <code>master</code></li><li>f103b4f6 - add a lightweight output buffering implementation</li><li>2832c5b6 - use our own output buffer</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=78&start_sha=02674327a99b2052c0c0f1046701352d2d920d68)
slink commented 2021-10-28 15:37:51 +00:00 (Migrated from code.uplex.de)
Copy link

added 3 commits

  • 6892f91c - 1 commit from branch master
  • a131ea98 - add a lightweight output buffering implementation
  • 3bb6b086 - use our own output buffer

Compare with previous version

added 3 commits <ul><li>6892f91c - 1 commit from branch <code>master</code></li><li>a131ea98 - add a lightweight output buffering implementation</li><li>3bb6b086 - use our own output buffer</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=79&start_sha=2832c5b676939f1a9d5853bd37dc0ac1c31cd5c1)
slink commented 2022-12-01 15:27:16 +00:00 (Migrated from code.uplex.de)
Copy link

added 1 commit

Compare with previous version

added 1 commit <ul><li>f5a911ee - Standardize LICENSE</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=86&start_sha=3bb6b0862224ef8d9f7000fabce0b7bab36d1c3b)
slink commented 2023-06-27 09:03:24 +00:00 (Migrated from code.uplex.de)
Copy link

added 4 commits

  • f5a911ee...23b1d54d - 2 commits from branch master
  • a894808b - add a lightweight output buffering implementation
  • 6cbceb1b - use our own output buffer

Compare with previous version

added 4 commits <ul><li>f5a911ee...23b1d54d - 2 commits from branch <code>master</code></li><li>a894808b - add a lightweight output buffering implementation</li><li>6cbceb1b - use our own output buffer</li></ul> [Compare with previous version](https://code.uplex.de/uplex-varnish/varnishevent/merge_requests/1/diffs?diff_id=92&start_sha=f5a911ee64f0515554b78a75ff5124411dc9be05)
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin new_output_buffering:new_output_buffering
git switch new_output_buffering

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff new_output_buffering
git switch new_output_buffering
git rebase master
git switch master
git merge --ff-only new_output_buffering
git switch new_output_buffering
git rebase master
git switch master
git merge --no-ff new_output_buffering
git switch master
git merge --squash new_output_buffering
git switch master
git merge --ff-only new_output_buffering
git switch master
git merge new_output_buffering
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
uplex-varnish/varnishevent!4
No description provided.