1. 30 Aug, 2021 9 commits
    • Martin Blix Grydeland's avatar
      Make f00007.vtc stable · 96b196b5
      Martin Blix Grydeland authored
      96b196b5
    • Martin Blix Grydeland's avatar
      Redo H/2 tx data handling · 91143910
      Martin Blix Grydeland authored
      This implements stream data handling using a buffer between the H/2
      session thread and each stream thread. This is needed to avoid head of
      line blocking on the session socket when a data frame is received for a
      stream thread that is not yet ready to receive it.
      
      The buffer used will have to be as large as the send window the peer
      expects at the time the stream is opened. This will typically be 65535
      unless the h2_initial_window_size parameter has been changed.
      
      Stream window updates will then be issued only once data is removed from
      the buffer by the request body being consumed from the request handling
      thread, limited in size to what space is then available in the buffer.
      91143910
    • Martin Blix Grydeland's avatar
      Set up 'wrk->vsl' pointer for H/2 session thread · 7ab694ac
      Martin Blix Grydeland authored
      The H/2 session thread does have a VSL buffer already set up, but the
      'wrk->vsl' pointer was not set. This caused issue for e.g. LRU_NukeOne()
      as it wants to log. Set the buffer for the duration that the worker is
      dedicated as an H/2 session thread.
      7ab694ac
    • Martin Blix Grydeland's avatar
      64c2b664
    • Martin Blix Grydeland's avatar
      Use SML_AllocBuf also for -sfile · e10b61da
      Martin Blix Grydeland authored
      e10b61da
    • Martin Blix Grydeland's avatar
      Use SML_AllocBuf with -smalloc · 3142d4a8
      Martin Blix Grydeland authored
      3142d4a8
    • Martin Blix Grydeland's avatar
      9588cfe7
    • Martin Blix Grydeland's avatar
      STV temp buffer API · 19cc1e4c
      Martin Blix Grydeland authored
      This is an API for getting an arbitrary buffer through the
      stevedores. The stevedore in question may then deploy LRU nuking or other
      measures to control resource usage.
      19cc1e4c
    • Dridi Boukelmoune's avatar
      http1: Missing workspace release · 66e2ea31
      Dridi Boukelmoune authored
      Working on the workspace sanitizer (ancestor of the workspace emulator)
      final rollbacks were needed to unwind allocations. There was however a
      branch where error handling was missing a workspace release, and it was
      fine before the introduction of the final rollbacks.
      
      To avoid turning the workspace emulator into a DoS vector the rollbacks
      are now only enforced for emulator builds. The specific "insufficient
      workspace" log is amended to ensure future changes to the session
      workspace footprint don't accidentally remove test coverage for that
      branch. The same could be done for other "insufficient workspace" logs
      in the PROXY protocol parsing.
      
      Refs 0632b846 (req: Prevent early rollback)
      Refs ce71896a (sess: Plug conceptual leak)
      Refs 246b1eb1 (busyobj: Plug conceptual leak)
      Refs 5b4f0f1a (htc: Defer workspace rollbacks for request tasks)
      Refs #3644
      
      Spotted by Alf's single process fuzzing setup that we should eventually
      revisit.
      
      Refs #3152
      66e2ea31
  2. 28 Aug, 2021 5 commits
  3. 27 Aug, 2021 7 commits
    • Dridi Boukelmoune's avatar
      vrt: Make sure VRT_Rollback() has a snapshot · 8e36cb70
      Dridi Boukelmoune authored
      Otherwise we would entirely unravel the task workspace.
      8e36cb70
    • Dridi Boukelmoune's avatar
      travis: Workspace emulator coverage · f237ed15
      Dridi Boukelmoune authored
      f237ed15
    • Dridi Boukelmoune's avatar
      circleci: Workspace emulator coverage · 5770abc3
      Dridi Boukelmoune authored
      5770abc3
    • Dridi Boukelmoune's avatar
      ws_emu: Introduce the workspace emulator · cfa0983d
      Dridi Boukelmoune authored
      The goal of the workspace emulator is to replicate the regular workspace
      behavior with individual allocations and make it work transparently.
      
      It's the successor of the workspace sanitizer from #3320 with notable
      differences:
      
      - enabled at configure time instead of run time
      - in a separate source file instead of mixed in
      - using sparse allocations instead of built-in red zones
      
      This means that the workspace emulator can be combined with regular
      sanitizer, in particular asan and lsan. If available, asan's public
      interface is used to mitigate the possible overflow of a reservation
      after some of it was released.
      
      Even without sanitizers, the fact that we integrate with jemalloc by
      default and enable its abort and junk options in varnishtest is enough
      to detect a use-after-free in some cases.
      
      With sanitizers though, the workspace emulator can observe #3550.
      
      One drawback is that the logic is split in two files, and some functions
      are identical in the two files. It might be possible to split cache_ws.c
      into something like cache_ws_alloc.c and cache_ws_util.c for example.
      
      Closes #3320
      Refs #3550
      Refs #3600
      cfa0983d
    • Dridi Boukelmoune's avatar
      ws: New WS_ReqPipeline() · 17ca56af
      Dridi Boukelmoune authored
      When a session has data pipelined we perform a dirty dance to move it at
      the beginning of the workspace. The rollbacks used to occur between
      HTC_RxPipeline() and HTC_RxInit() calls until it was centralized in the
      latter.
      
      With a dedicated WS_ReqPipeline() operation we can capture the semantics
      of initializing an existing connection for its next task with or without
      data fetched from the previous task.
      
      While conceptually there is still a use-after-free since the pipelined
      data may belong to the same workspace, it is fine if that happens within
      the bounds of an atomic workspace operation.
      17ca56af
    • Dridi Boukelmoune's avatar
      htc: Defer workspace rollbacks for request tasks · 5b4f0f1a
      Dridi Boukelmoune authored
      When we take on a new request on a connection from which something was
      already received, we need to pipeline it and we do so at the beginning
      of the request workspace. There's a high probability that the pipeline
      is coming from the same workspace, which is a form of use-after-free
      only made safe by the workspace implementation details.
      
      To avoid the conceptual use-after-free, we defer req workspace rollbacks
      and perform them during the next HTC_RxInit() call before the pipelining
      operation.
      
      Because HTTP/1 works directly on the session, a worker can safely switch
      back and forth between sess and req tasks. This means that unless the
      session goes idle the same workspace is used from one client request
      to the next, hence the rollback previously happening in Req_Cleanup().
      
      With h2 however there is a disconnect between the session and streams.
      The connection is received in req0's workspace, and then copied into
      a stream's req workspace via the pipelining scheme. Rollbacks can be
      deferred as well, but they need to happen otherwise the session will
      soon overflow. Independent HTC_RxInit() calls happen for req0 in the
      h2 session thread, and for h2 streams in the regular request task code
      path.
      
      PROXY Protocol parsing may result in receiving more than the proxy
      preamble itself and pipelining will happen, whether it is via a req
      for HTTP/1 or req0 for h2.
      
      On the other end of the spectrum when Varnish acts as a client it
      only sends one HTTP/1 request at a time for a given connection, so
      we never expect pipelining to occur in fetch task.
      5b4f0f1a
    • Dridi Boukelmoune's avatar
      vrt: Fix use-after-release · 2bb355c7
      Dridi Boukelmoune authored
      2bb355c7
  4. 26 Aug, 2021 1 commit
  5. 25 Aug, 2021 18 commits