Merge branch 'video-streaming-hls'

NEWS

+Overview of Changes in audiowmark-0.5.0:
+
+* support HTTP Live Streaming for audio/video streaming
+* fix floating point wav input
+* improve command line option handling (ArgParser)
+* support seeking on internal watermark state
+
 Overview of Changes in audiowmark-0.4.2:
 
 * compile fixes for g++-9 and clang++-10

README.adoc

@@ -235,6 +235,8 @@ Use watermarking key from file <filename> (see <<key>>).
 --strength <s>::
 Set the watermarking strength (see <<strength>>).
 
+Videos can be watermarked on-the-fly using <<hls>>.
+
 == Output as Stream
 
 Usually, an input file is read, watermarked and an output file is written.
@@ -342,6 +344,149 @@ same. The watermarker has been designed and tested for stereo files,
 so the number of channels should really be `2`. This is also the
 default.
 
+[[hls]]
+== HTTP Live Streaming
+
+=== Introduction for HLS
+
+HTTP Live Streaming (HLS) is a protocol to deliver audio or video streams via
+HTTP.  One example for using HLS in practice would be: a user watches a video
+in a web browser with a player like `hls.js`. The user is free to
+play/pause/seek the video as he wants. `audiowmark` can watermark the audio
+content while it is being transmitted to the user.
+
+HLS splits the contents of each stream into small segments. For the watermarker
+this means that if the user seeks to a position far ahead in the stream, the
+server needs to start sending segments from where the new play position is, but
+everything in between can be ignored.
+
+Another important property of HLS is that it allows separate segments for the
+video and audio stream of a video. Since we watermark only the audio track of a
+video, the video segments can be sent as they are (and different users can get
+the same video segments). What is watermarked are the audio segments only, so
+here instead of sending the original audio segments to the user, the audio
+segments are watermarked individually for each user, and then transmitted.
+
+Everything necessary to watermark HLS audio segments is available within
+`audiowmark`. The server side support which is necessary to send the right
+watermarked segment to the right user is not included.
+
+[[hls-requirements]]
+=== HLS Requirements
+
+HLS support requires some headers/libraries from ffmpeg:
+
+* libavcodec
+* libavformat
+* libavutil
+* libswresample
+
+To enable these as dependencies and build `audiowmark` with HLS support, use the
+`--with-ffmpeg` configure option:
+
+[subs=+quotes]
+....
+*$ ./configure --with-ffmpeg*
+....
+
+In addition to the libraries, `audiowmark` also uses the two command line
+programs from ffmpeg, so they need to be installed:
+
+* ffmpeg
+* ffprobe
+
+=== Preparing HLS segments
+
+The first step for preparing content for streaming with HLS would be splitting
+a video into segments. For this documentation, we use a very simple example
+using ffmpeg. No matter what the original codec was, at this point we force
+transcoding to AAC with our target bit rate, because during delivery the stream
+will be in AAC format.
+
+[subs=+quotes]
+....
+*$ ffmpeg -i video.mp4 -f hls -master_pl_name replay.m3u8 -c:a aac -ab 192k \
+  -var_stream_map "a:0,agroup:aud v:0,agroup:aud" \
+  -hls_playlist_type vod -hls_list_size 0 -hls_time 10 vs%v/out.m3u8*
+....
+
+This splits the `video.mp4` file into an audio stream of segments in the `vs0`
+directory and a video stream of segments in the `vs1` directory. Each segment
+is approximately 10 seconds long, and a master playlist is written to
+`replay.m3u8`.
+
+Now we can add the relevant audio context to each audio ts segment. This is
+necessary so that when the segment is watermarked in order to be transmitted to
+the user, `audiowmark` will have enough context available before and after the
+segment to create a watermark which sounds correct over segment boundaries.
+
+[subs=+quotes]
+....
+*$ audiowmark hls-prepare vs0 vs0prep out.m3u8 video.mp4*
+AAC Bitrate:  195641 (detected)
+Segments:     18
+Time:         2:53
+....
+
+This steps reads the audio playlist `vs0/out.m3u8` and writes all segments
+contained in this audio playlist to a new directory `vs0prep` which
+contains the audio segments prepared for watermarking.
+
+The last argument in this command line is `video.mp4` again. All audio
+that is watermarked is taken from this audio master. It could also be
+supplied in `wav` format. This makes a difference if you use lossy
+compression as target format (for instance AAC), but your original
+video has an audio stream with higher quality (i.e. lossless).
+
+=== Watermarking HLS segments
+
+So with all preparations made, what would the server have to do to send a
+watermarked version of the 6th audio segment `vs0prep/out5.ts`?
+
+[subs=+quotes]
+....
+*$ audiowmark hls-add vs0prep/out5.ts send5.ts 0123456789abcdef0011223344556677*
+Message:      0123456789abcdef0011223344556677
+Strength:     10
+
+Time:         0:15
+Sample Rate:  44100
+Channels:     2
+Data Blocks:  0
+AAC Bitrate:  195641
+....
+
+So instead of sending out5.ts (which has no watermark) to the user, we would
+send send5.ts, which is watermarked.
+
+In a real-world use case, it is likely that the server would supply the input
+segment on stdin and send the output segment as written to stdout, like this
+
+[subs=+quotes]
+....
+*$ [...] | audiowmark hls-add - - 0123456789abcdef0011223344556677 | [...]*
+[...]
+....
+
+The usual parameters are supported in `audiowmark hls-add`, like
+
+--key <filename>::
+Use watermarking key from file <filename> (see <<key>>).
+
+--strength <s>::
+Set the watermarking strength (see <<strength>>).
+
+The AAC bitrate for the output segment can be set using:
+
+--bit-rate <bit_rate>::
+Set the AAC bit-rate for the generated watermarked segment.
+
+The rules for the AAC bit-rate of the newly encoded watermarked segment are:
+
+* if the --bit-rate option is used during `hls-add`, this bit-rate will be used
+* otherwise, if the `--bit-rate` option is used during `hls-prepare`, this bit-rate will be used
+* otherwise, the bit-rate of the input material is detected during `hls-prepare`
+
 == Dependencies
 
 If you compile from source, `audiowmark` needs the following libraries:
@@ -352,6 +497,9 @@ If you compile from source, `audiowmark` needs the following libraries:
 * libzita-resampler
 * libmpg123
 
+If you want to build with HTTP Live Streaming support, see also
+<<hls-requirements>>.
+
 == Building fftw
 
 `audiowmark` needs the single prevision variant of fftw3.

configure.ac

-AC_INIT([audiowmark], [0.4.2])
+AC_INIT([audiowmark], [0.5.0])
 AC_CONFIG_SRCDIR([src/audiowmark.cc])
 AC_CONFIG_AUX_DIR([build-aux])
 AC_CONFIG_MACRO_DIR([m4])
@@ -39,6 +39,12 @@ AC_DEFUN([AC_ZITA_REQUIREMENTS],
     )
 ])
 
+dnl ffmpeg stuff
+AC_DEFUN([AC_FFMPEG_REQUIREMENTS],
+[
+  PKG_CHECK_MODULES(FFMPEG, libavcodec libavformat libavutil libswresample)
+])
+
 dnl FFTW3
 AC_DEFUN([AC_FFTW_CHECK],
 [
@@ -72,6 +78,18 @@ AC_ZITA_REQUIREMENTS
 AC_FFTW_CHECK
 AM_PATH_LIBGCRYPT
 
+dnl -------------------- ffmpeg is optional ----------------------------
+AC_ARG_WITH([ffmpeg], [AS_HELP_STRING([--with-ffmpeg], [build against ffmpeg libraries])], [], [with_ffmpeg=no])
+if test "x$with_ffmpeg" != "xno"; then
+  AC_FFMPEG_REQUIREMENTS
+  HAVE_FFMPEG=1
+else
+  HAVE_FFMPEG=0
+fi
+AC_DEFINE_UNQUOTED(HAVE_FFMPEG, $HAVE_FFMPEG, [whether ffmpeg libs are available])
+AM_CONDITIONAL([COND_WITH_FFMPEG], [test "x$with_ffmpeg" != "xno"])
+dnl -------------------------------------------------------------------------
+
 # need c++14 mode
 AX_CXX_COMPILE_STDCXX_14(ext)
 
@@ -82,3 +100,11 @@ AC_LANG_POP([C++])
 
 AC_CONFIG_FILES([Makefile src/Makefile])
 AC_OUTPUT
+
+# Output summary message
+
+echo
+echo "---------------------------------------------------------------------------"
+echo "$PACKAGE_NAME $PACKAGE_VERSION"
+echo "---------------------------------------------------------------------------"
+echo " * use ffmpeg libs: $with_ffmpeg (required for HLS)"

src/.gitignore

@@ -8,3 +8,5 @@ testmp3
 testrandom
 teststream
 testlimiter
+testhls
+testmpegts

src/Makefile.am

@@ -5,13 +5,14 @@ COMMON_SRC = utils.hh utils.cc convcode.hh convcode.cc random.hh random.cc wavda
 	     audiostream.cc audiostream.hh sfinputstream.cc sfinputstream.hh stdoutwavoutputstream.cc stdoutwavoutputstream.hh \
 	     sfoutputstream.cc sfoutputstream.hh rawinputstream.cc rawinputstream.hh rawoutputstream.cc rawoutputstream.hh \
 	     rawconverter.cc rawconverter.hh mp3inputstream.cc mp3inputstream.hh wmcommon.cc wmcommon.hh fft.cc fft.hh \
-	     limiter.cc limiter.hh shortcode.cc shortcode.hh
-COMMON_LIBS = $(SNDFILE_LIBS) $(FFTW_LIBS) $(LIBGCRYPT_LIBS) $(LIBMPG123_LIBS)
+	     limiter.cc limiter.hh shortcode.cc shortcode.hh mpegts.cc mpegts.hh hls.cc hls.hh audiobuffer.hh \
+	     wmget.cc wmadd.cc
+COMMON_LIBS = $(SNDFILE_LIBS) $(FFTW_LIBS) $(LIBGCRYPT_LIBS) $(LIBMPG123_LIBS) $(FFMPEG_LIBS)
 
-audiowmark_SOURCES = audiowmark.cc wmget.cc wmadd.cc $(COMMON_SRC)
+audiowmark_SOURCES = audiowmark.cc $(COMMON_SRC)
 audiowmark_LDFLAGS = $(COMMON_LIBS)
 
-noinst_PROGRAMS = testconvcode testrandom testmp3 teststream testlimiter testshortcode
+noinst_PROGRAMS = testconvcode testrandom testmp3 teststream testlimiter testshortcode testmpegts
 
 testconvcode_SOURCES = testconvcode.cc $(COMMON_SRC)
 testconvcode_LDFLAGS = $(COMMON_LIBS)
@@ -30,3 +31,14 @@ testlimiter_LDFLAGS = $(COMMON_LIBS)
 
 testshortcode_SOURCES = testshortcode.cc $(COMMON_SRC)
 testshortcode_LDFLAGS = $(COMMON_LIBS)
+
+testmpegts_SOURCES = testmpegts.cc $(COMMON_SRC)
+testmpegts_LDFLAGS = $(COMMON_LIBS)
+
+if COND_WITH_FFMPEG
+COMMON_SRC += hlsoutputstream.cc hlsoutputstream.hh
+
+noinst_PROGRAMS += testhls
+testhls_SOURCES = testhls.cc $(COMMON_SRC)
+testhls_LDFLAGS = $(COMMON_LIBS)
+endif

src/audiobuffer.hh

+/*
+ * Copyright (C) 2018-2020 Stefan Westerfeld
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef AUDIOWMARK_AUDIO_BUFFER_HH
+#define AUDIOWMARK_AUDIO_BUFFER_HH
+
+#include <assert.h>
+
+class AudioBuffer
+{
+  const int           n_channels = 0;
+  std::vector<float>  buffer;
+
+public:
+  AudioBuffer (int n_channels) :
+    n_channels (n_channels)
+  {
+  }
+  void
+  write_frames (const std::vector<float>& samples)
+  {
+    buffer.insert (buffer.end(), samples.begin(), samples.end());
+  }
+  std::vector<float>
+  read_frames (size_t frames)
+  {
+    assert (frames * n_channels <= buffer.size());
+    const auto begin = buffer.begin();
+    const auto end   = begin + frames * n_channels;
+    std::vector<float> result (begin, end);
+    buffer.erase (begin, end);
+    return result;
+  }
+  size_t
+  can_read_frames() const
+  {
+    return buffer.size() / n_channels;
+  }
+};
+
+#endif /* AUDIOWMARK_AUDIO_BUFFER_HH */

src/audiostream.cc

@@ -89,7 +89,7 @@ AudioOutputStream::create (const string& filename, int n_channels, int sample_ra
     {
       SFOutputStream *sfostream = new SFOutputStream();
       out_stream.reset (sfostream);
-      err = sfostream->open (filename, n_channels, sample_rate, bit_depth, n_frames);
+      err = sfostream->open (filename, n_channels, sample_rate, bit_depth);
       if (err)
         return nullptr;
     }

src/hls.hh

+/*
+ * Copyright (C) 2018-2020 Stefan Westerfeld
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef AUDIOWMARK_HLS_HH
+#define AUDIOWMARK_HLS_HH
+
+#include <string>
+
+int hls_add (const std::string& infile, const std::string& outfile, const std::string& bits);
+int hls_prepare (const std::string& in_dir, const std::string& out_dir, const std::string& filename, const std::string& audio_master);
+
+Error ff_decode (const std::string& filename, WavData& out_wav_data);
+
+#endif /* AUDIOWMARK_MPEGTS_HH */

src/hlsoutputstream.hh

+/*
+ * Copyright (C) 2018-2020 Stefan Westerfeld
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef AUDIOWMARK_HLS_OUTPUT_STREAM_HH
+#define AUDIOWMARK_HLS_OUTPUT_STREAM_HH
+
+#include "audiostream.hh"
+#include "audiobuffer.hh"
+
+#include <assert.h>
+
+extern "C" {
+#include <libavformat/avformat.h>
+#include <libavutil/opt.h>
+#include <libswresample/swresample.h>
+#include <libavutil/avassert.h>
+#include <libavutil/timestamp.h>
+}
+
+class HLSOutputStream : public AudioOutputStream {
+  AVStream         *m_st = nullptr;
+  AVCodecContext   *m_enc = nullptr;
+  AVFormatContext  *m_fmt_ctx = nullptr;
+
+  /* pts of the next frame that will be generated */
+  int64_t           m_next_pts = 0;
+  int               m_samples_count = 0;
+  int               m_start_pos = 0;
+
+  AVFrame          *m_frame = nullptr;
+  AVFrame          *m_tmp_frame = nullptr;
+
+  size_t            m_cut_aac_frames = 0;
+  size_t            m_keep_aac_frames = 0;
+
+  SwrContext       *m_swr_ctx = nullptr;
+
+  int               m_bit_depth = 0;
+  int               m_sample_rate = 0;
+  int               m_n_channels = 0;
+  AudioBuffer       m_audio_buffer;
+  size_t            m_delete_input_start = 0;
+  int               m_bit_rate = 0;
+  std::string       m_channel_layout;
+
+  enum class State {
+    NEW,
+    OPEN,
+    CLOSED
+  };
+  State             m_state = State::NEW;
+
+  Error add_stream (AVCodec **codec, enum AVCodecID codec_id);
+  Error open_audio (AVCodec *codec, AVDictionary *opt_arg);
+  AVFrame *get_audio_frame();
+  enum class EncResult {
+    OK,
+    ERROR,
+    DONE
+  };
+  EncResult write_audio_frame (Error& err);
+  void close_stream();
+  AVFrame *alloc_audio_frame (AVSampleFormat sample_fmt, uint64_t channel_layout, int sample_rate, int nb_samples, Error& err);
+
+  int write_frame (const AVRational *time_base, AVStream *st, AVPacket *pkt);
+public:
+  HLSOutputStream (int n_channels, int sample_rate, int bit_depth);
+  ~HLSOutputStream();
+
+  void set_bit_rate (int bit_rate);
+  void set_channel_layout (const std::string& channel_layout);
+
+  Error open (const std::string& output_filename, size_t cut_aac_frames, size_t keep_aac_frames, double pts_start, size_t delete_input_start);
+  int bit_depth() const override;
+  int sample_rate() const override;
+  int n_channels() const override;
+  Error write_frames (const std::vector<float>& frames) override;
+  Error close() override;
+};
+
+#endif /* AUDIOWMARK_HLS_OUTPUT_STREAM_HH */
+

src/limiter.cc

@@ -66,6 +66,27 @@ Limiter::process (const vector<float>& samples)
   return out;
 }
 
+size_t
+Limiter::skip (size_t zeros)
+{
+  assert (block_size >= 1);
+
+  size_t buffer_size = buffer.size();
+  buffer_size += zeros * n_channels;
+
+  /* need at least two complete blocks in buffer to produce output */
+  const size_t buffered_blocks = buffer_size / n_channels / block_size;
+  if (buffered_blocks < 2)
+    {
+      buffer.resize (buffer_size);
+      return 0;
+    }
+
+  const size_t blocks_todo = buffered_blocks - 1;
+  buffer.resize (buffer_size - blocks_todo * block_size * n_channels);
+  return blocks_todo * block_size;
+}
+
 float
 Limiter::block_max (const float *in)
 {

src/limiter.hh

@@ -42,6 +42,7 @@ public:
   void set_ceiling (float ceiling);
 
   std::vector<float> process (const std::vector<float>& samples);
+  size_t             skip (size_t zeros);
   std::vector<float> flush();
 };
 

src/mp3inputstream.hh

@@ -56,4 +56,3 @@ public:
 };
 
 #endif /* AUDIOWMARK_MP3_INPUT_STREAM_HH */
-

src/mpegts.cc

+/*
+ * Copyright (C) 2018-2020 Stefan Westerfeld
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <array>
+#include <regex>
+
+#include <inttypes.h>
+#include "utils.hh"
+#include "mpegts.hh"
+
+using std::string;
+using std::vector;
+using std::map;
+using std::regex;
+
+class TSPacket
+{
+public:
+  enum class ID { awmk_file, awmk_data, unknown };
+
+private:
+  std::array<unsigned char, 188> m_data;
+
+  std::array<unsigned char, 12>
+  get_id_bytes (ID type)
+  {
+    if (type == ID::awmk_file)
+     return { 'G', 0x1F, 0xFF, 0x10, 'A', 'W', 'M', 'K', 'f', 'i', 'l', 'e' };
+    if (type == ID::awmk_data)
+      return { 'G', 0x1F, 0xFF, 0x10, 'A', 'W', 'M', 'K', 'd', 'a', 't', 'a' };
+    return {0,};
+  }
+public:
+  bool
+  read (FILE *file, Error& err)
+  {
+    size_t bytes_read = fread (m_data.data(), 1, m_data.size(), file);
+    if (bytes_read == 0) /* probably eof */
+      return false;
+
+    if (bytes_read == m_data.size()) /* successful read */
+      {
+        if (m_data[0] == 'G')
+          return true;
+
+        err = Error ("bad packet sync while reading transport (.ts) packet");
+        return false;
+      }
+
+    err = Error ("short read while reading transport stream (.ts) packet");
+    return false;
+  }
+  Error
+  write (FILE *file)
+  {
+    size_t bytes_written = fwrite (m_data.data(), 1, m_data.size(), file);
+    if (bytes_written != m_data.size())
+      return Error ("short write while writing transport stream (.ts) packet");
+
+    return Error::Code::NONE;
+  }
+  void
+  clear (ID type)
+  {
+    std::fill (m_data.begin(), m_data.end(), 0);
+    auto id = get_id_bytes (type);
+    std::copy (id.begin(), id.end(), m_data.begin());
+  }
+  unsigned char&
+  operator[] (size_t n)
+  {
+    return m_data[n];
+  }
+  bool
+  id_eq (size_t offset, unsigned char a, unsigned char b, unsigned char c, unsigned char d)
+  {
+    return m_data[offset] == a && m_data[offset + 1] == b && m_data[offset + 2] == c && m_data[offset + 3] == d;
+  }
+  ID
+  get_id()
+  {
+    if (id_eq (0, 'G', 0x1F, 0xFF, 0x10) && id_eq (4, 'A', 'W', 'M', 'K'))
+      {
+        if (id_eq (8, 'f', 'i', 'l', 'e'))
+          return ID::awmk_file;
+        if (id_eq (8, 'd', 'a', 't', 'a'))
+          return ID::awmk_data;
+      }
+    return ID::unknown;
+  }
+  constexpr size_t
+  size()
+  {
+    return m_data.size(); // is constant
+  }
+  const std::array<unsigned char, 188>&
+  data()
+  {
+    return m_data;
+  }
+};
+
+Error
+TSWriter::append_file (const string& name, const string& filename)
+{
+  vector<unsigned char> data;
+  FILE *datafile = fopen (filename.c_str(), "r");
+  ScopedFile datafile_s (datafile);
+  if (!datafile)
+    return Error ("unable to open data file");
+  int c;
+  while ((c = fgetc (datafile)) >= 0)
+    data.push_back (c);
+
+  entries.push_back ({name, data});
+  return Error::Code::NONE;
+}
+
+void
+TSWriter::append_vars (const string& name, const map<string, string>& vars)
+{
+  vector<unsigned char> data;
+  for (auto kv : vars)
+    {
+      for (auto k : kv.first)
+        data.push_back (k);
+      data.push_back ('=');
+      for (auto v : kv.second)
+        data.push_back (v);
+      data.push_back (0);
+    }
+
+  entries.push_back ({name, data});
+}
+
+void
+TSWriter::append_data (const string& name, const vector<unsigned char>& data)
+{
+  entries.push_back ({name, data});
+}
+
+Error
+TSWriter::process (const string& inname, const string& outname)
+{
+  FILE *infile = fopen (inname.c_str(), "r");
+  FILE *outfile = fopen (outname.c_str(), "w");
+  ScopedFile infile_s (infile);
+  ScopedFile outfile_s (outfile);
+
+  if (!infile)
+    {
+      error ("audiowmark: unable to open %s for reading\n", inname.c_str());
+      return Error (strerror (errno));
+    }
+
+  if (!outfile)
+    {
+      error ("audiowmark: unable to open %s for writing\n", outname.c_str());
+      return Error (strerror (errno));
+    }
+
+  while (!feof (infile))
+    {
+      TSPacket p;
+      Error err;
+      bool read_ok = p.read (infile, err);
+      if (!read_ok)
+        {
+          if (err)
+            return err;
+        }
+      else
+        {
+          err = p.write (outfile);
+          if (err)
+            return err;
+        }
+    }
+
+  for (auto entry : entries)
+    {
+      string header = string_printf ("%zd:%s", entry.data.size(), entry.name.c_str()) + '\0';
+      vector<unsigned char> data = entry.data;
+      for (size_t i = 0; i < header.size(); i++)
+        data.insert (data.begin() + i, header[i]);
+
+      TSPacket p_file;
+      p_file.clear (TSPacket::ID::awmk_file);
+      size_t data_pos = 0;
+      int pos = 12;
+      while (data_pos < data.size())
+        {
+          p_file[pos++] = data[data_pos];
+          if (pos == 188)
+            {
+              Error err = p_file.write (outfile);
+              if (err)
+                return err;
+
+              p_file.clear (TSPacket::ID::awmk_data);
+              pos = 12;
+            }
+          data_pos++;
+        }
+
+      if (pos != 12)
+        {
+          Error err = p_file.write (outfile);
+          if (err)
+            return err;
+        }
+    }
+
+  return Error::Code::NONE;
+}
+
+bool
+TSReader::parse_header (Header& header, vector<unsigned char>& data)
+{
+  for (size_t i = 0; i < data.size(); i++)
+    {
+      if (data[i] == 0) // header is terminated with one single 0 byte
+        {
+          string s = (const char *) (&data[0]);
+
+          static const regex header_re ("([0-9]*):(.*)");
+          std::smatch sm;
+          if (regex_match (s, sm, header_re))
+            {
+              header.data_size = atoi (sm[1].str().c_str());
+              header.filename = sm[2];
+
+              // erase header including null termination
+              data.erase (data.begin(), data.begin() + i + 1);
+              return true;
+            }
+        }
+    }
+  return false;
+}
+
+Error
+TSReader::load (const string& inname)
+{
+  if (inname == "-")
+    {
+      return load (stdin);
+    }
+  else
+    {
+      FILE *infile = fopen (inname.c_str(), "r");
+      ScopedFile infile_s (infile);
+      if (!infile)
+        return Error (string_printf ("error opening input .ts '%s'", inname.c_str()));
+      return load (infile);
+    }
+}
+
+Error
+TSReader::load (FILE *infile)
+{
+  vector<unsigned char> awmk_stream;
+  Header header;
+  bool header_valid = false;
+  Error err;
+  while (!feof (infile))
+    {
+      TSPacket p;
+      bool read_ok = p.read (infile, err);
+      if (!read_ok)
+        {
+          if (err)
+            return err;
+        }
+      else
+        {
+          TSPacket::ID id = p.get_id();
+          if (id == TSPacket::ID::awmk_file)
+            {
+              /* new stream start, clear old contents */
+              header_valid = false;
+              awmk_stream.clear();
+            }
+          if (id == TSPacket::ID::awmk_file || id == TSPacket::ID::awmk_data)
+            {
+              awmk_stream.insert (awmk_stream.end(), p.data().begin() + 12, p.data().end());
+
+              if (!header_valid)
+                {
+                  if (parse_header (header, awmk_stream))
+                    {
+                      awmk_stream.reserve (header.data_size + p.size());
+                      header_valid = true;
+                    }
+                }
+              // done? do we have enough bytes for the complete entry?
+              if (header_valid && awmk_stream.size() >= header.data_size)
+                {
+                  awmk_stream.resize (header.data_size);
+
+                  m_entries.push_back ({ header.filename, std::move (awmk_stream)});
+
+                  header_valid = false;
+                  awmk_stream.clear();
+                }
+            }
+        }
+    }
+  return Error::Code::NONE;
+}
+
+const vector<TSReader::Entry>&
+TSReader::entries()
+{
+  return m_entries;
+}
+
+const TSReader::Entry *
+TSReader::find (const string& name) const
+{
+  for (const auto& entry : m_entries)
+    if (entry.filename == name)
+      return &entry;
+
+  return nullptr;
+}
+
+map<string, string>
+TSReader::parse_vars (const string& name)
+{
+  map<string, string> vars;
+
+  auto entry = find (name);
+  if (!entry)
+    return vars;
+
+  enum { KEY, VALUE } mode = KEY;
+  string s;
+  string key;
+  for (auto c : entry->data)
+    {
+      if (c == '=' && mode == KEY)
+        {
+          key = s;
+          s.clear();
+          mode = VALUE;
+        }
+      else if (c == '\0' && mode == VALUE)
+        {
+          vars[key] = s;
+          s.clear();
+          mode = KEY;
+        }
+      else
+        {
+          s += c;
+        }
+    }
+  return vars;
+}

src/mpegts.hh

+/*
+ * Copyright (C) 2018-2020 Stefan Westerfeld
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef AUDIOWMARK_MPEGTS_HH
+#define AUDIOWMARK_MPEGTS_HH
+
+#include <map>
+
+class TSReader
+{
+public:
+  struct Entry
+  {
+    std::string                filename;
+    std::vector<unsigned char> data;
+  };
+private:
+  struct Header
+  {
+    std::string filename;
+    size_t      data_size = 0;
+  };
+  std::vector<Entry> m_entries;
+  bool parse_header (Header& header, std::vector<unsigned char>& data);
+  Error load (FILE *infile);
+public:
+  Error load (const std::string& inname);
+  const std::vector<Entry>& entries();
+  const Entry *find (const std::string& filename) const;
+  std::map<std::string, std::string> parse_vars (const std::string& name);
+};
+
+class TSWriter
+{
+  struct Entry
+  {
+    std::string                name;
+    std::vector<unsigned char> data;
+  };
+  std::vector<Entry> entries;
+public:
+  Error append_file (const std::string& name, const std::string& filename);
+  void  append_vars (const std::string& name, const std::map<std::string, std::string>& vars);
+  void  append_data (const std::string& name, const std::vector<unsigned char>& data);
+  Error process (const std::string& in_name, const std::string& out_name);
+};
+
+int pcr (const std::string& filename, const std::string& outname, double time_offset_ms);
+
+#endif /* AUDIOWMARK_MPEGTS_HH */

src/sfinputstream.cc

@@ -18,6 +18,7 @@
 #include "sfinputstream.hh"
 
 #include <assert.h>
+#include <string.h>
 
 using std::string;
 using std::vector;
@@ -29,12 +30,21 @@ SFInputStream::~SFInputStream()
 
 Error
 SFInputStream::open (const string& filename)
+{
+  return open ([&] (SF_INFO *sfinfo) {
+    return sf_open (filename.c_str(), SFM_READ, sfinfo);
+  });
+}
+
+
+Error
+SFInputStream::open (std::function<SNDFILE* (SF_INFO *)> open_func)
 {
   assert (m_state == State::NEW);
 
   SF_INFO sfinfo = { 0, };
 
-  m_sndfile = sf_open (filename.c_str(), SFM_READ, &sfinfo);
+  m_sndfile = open_func (&sfinfo);
 
   int error = sf_error (m_sndfile);
   if (error)
@@ -67,13 +77,18 @@ SFInputStream::open (const string& filename)
           m_bit_depth = 24;
           break;
 
-      case SF_FORMAT_FLOAT:
       case SF_FORMAT_PCM_32:
           m_bit_depth = 32;
           break;
 
+      case SF_FORMAT_FLOAT:
+          m_bit_depth = 32;
+          m_read_float_data = true;
+          break;
+
       case SF_FORMAT_DOUBLE:
           m_bit_depth = 64;
+          m_read_float_data = true;
           break;
 
       default:
@@ -101,24 +116,39 @@ SFInputStream::read_frames (vector<float>& samples, size_t count)
 {
   assert (m_state == State::OPEN);
 
-  vector<int> isamples (count * m_n_channels);
-  sf_count_t r_count = sf_readf_int (m_sndfile, &isamples[0], count);
-
-  if (sf_error (m_sndfile))
-    return Error (sf_strerror (m_sndfile));
-
-  /* reading a wav file and saving it again with the libsndfile float API will
-   * change some values due to normalization issues:
-   *   http://www.mega-nerd.com/libsndfile/FAQ.html#Q010
-   *
-   * to avoid the problem, we use the int API and do the conversion beween int
-   * and float manually - the important part is that the normalization factors
-   * used during read and write are identical
-   */
-  samples.resize (r_count * m_n_channels);
-  const double norm = 1.0 / 0x80000000LL;
-  for (size_t i = 0; i < samples.size(); i++)
-    samples[i] = isamples[i] * norm;
+  if (m_read_float_data) /* float or double input */
+    {
+      samples.resize (count * m_n_channels);
+
+      sf_count_t r_count = sf_readf_float (m_sndfile, &samples[0], count);
+
+      if (sf_error (m_sndfile))
+        return Error (sf_strerror (m_sndfile));
+
+      samples.resize (r_count * m_n_channels);
+    }
+  else /* integer input */
+    {
+      vector<int> isamples (count * m_n_channels);
+
+      sf_count_t r_count = sf_readf_int (m_sndfile, &isamples[0], count);
+
+      if (sf_error (m_sndfile))
+        return Error (sf_strerror (m_sndfile));
+
+      /* reading a wav file and saving it again with the libsndfile float API will
+       * change some values due to normalization issues:
+       *   http://www.mega-nerd.com/libsndfile/FAQ.html#Q010
+       *
+       * to avoid the problem, we use the int API and do the conversion beween int
+       * and float manually - the important part is that the normalization factors
+       * used during read and write are identical
+       */
+      samples.resize (r_count * m_n_channels);
+      const double norm = 1.0 / 0x80000000LL;
+      for (size_t i = 0; i < samples.size(); i++)
+        samples[i] = isamples[i] * norm;
+    }
 
   return Error::Code::NONE;
 }
@@ -135,3 +165,109 @@ SFInputStream::close()
       m_state = State::CLOSED;
     }
 }
+
+static sf_count_t
+virtual_get_len (void *data)
+{
+  SFVirtualData *vdata = static_cast<SFVirtualData *> (data);
+
+  return vdata->mem->size();
+}
+
+static sf_count_t
+virtual_seek (sf_count_t offset, int whence, void *data)
+{
+  SFVirtualData *vdata = static_cast<SFVirtualData *> (data);
+
+  if (whence == SEEK_CUR)
+    {
+      vdata->offset = vdata->offset + offset;
+    }
+  else if (whence == SEEK_SET)
+    {
+      vdata->offset = offset;
+    }
+  else if (whence == SEEK_END)
+    {
+      vdata->offset = vdata->mem->size() + offset;
+    }
+
+  /* can't seek beyond eof */
+  vdata->offset = bound<sf_count_t> (0, vdata->offset, vdata->mem->size());
+  return vdata->offset;
+}
+
+static sf_count_t
+virtual_read (void *ptr, sf_count_t count, void *data)
+{
+  SFVirtualData *vdata = static_cast<SFVirtualData *> (data);
+
+  int rcount = 0;
+  if (size_t (vdata->offset + count) <= vdata->mem->size())
+    {
+      /* fast case: read can be fully satisfied with the data we have */
+      memcpy (ptr, &(*vdata->mem)[vdata->offset], count);
+      rcount = count;
+    }
+  else
+    {
+      unsigned char *uptr = static_cast<unsigned char *> (ptr);
+      for (sf_count_t i = 0; i < count; i++)
+        {
+          size_t rpos = i + vdata->offset;
+          if (rpos < vdata->mem->size())
+            {
+              uptr[i] = (*vdata->mem)[rpos];
+              rcount++;
+            }
+        }
+    }
+  vdata->offset += rcount;
+  return rcount;
+}
+
+static sf_count_t
+virtual_write (const void *ptr, sf_count_t count, void *data)
+{
+  SFVirtualData *vdata = static_cast<SFVirtualData *> (data);
+
+  const unsigned char *uptr = static_cast<const unsigned char *> (ptr);
+  for (sf_count_t i = 0; i < count; i++)
+    {
+      unsigned char ch = uptr[i];
+
+      size_t wpos = i + vdata->offset;
+      if (wpos >= vdata->mem->size())
+        vdata->mem->resize (wpos + 1);
+      (*vdata->mem)[wpos] = ch;
+    }
+  vdata->offset += count;
+  return count;
+}
+
+static sf_count_t
+virtual_tell (void *data)
+{
+  SFVirtualData *vdata = static_cast<SFVirtualData *> (data);
+  return vdata->offset;
+}
+
+SFVirtualData::SFVirtualData() :
+  io {
+    virtual_get_len,
+    virtual_seek,
+    virtual_read,
+    virtual_write,
+    virtual_tell
+  }
+{
+}
+
+Error
+SFInputStream::open (const vector<unsigned char> *data)
+{
+  m_virtual_data.mem = const_cast<vector<unsigned char> *> (data);
+  return open ([&] (SF_INFO *sfinfo) {
+    return sf_open_virtual (&m_virtual_data.io, SFM_READ, sfinfo, &m_virtual_data);
+  });
+}

src/sfinputstream.hh

@@ -19,18 +19,33 @@
 #define AUDIOWMARK_SF_INPUT_STREAM_HH
 
 #include <string>
+#include <functional>
 
 #include <sndfile.h>
 
 #include "audiostream.hh"
 
+/* to support virtual io read/write from/to memory */
+struct SFVirtualData
+{
+  SFVirtualData();
+
+  std::vector<unsigned char> *mem    = nullptr;
+  sf_count_t                  offset = 0;
+  SF_VIRTUAL_IO               io;
+};
+
 class SFInputStream : public AudioInputStream
 {
+private:
+  SFVirtualData m_virtual_data;
+
   SNDFILE    *m_sndfile = nullptr;
   int         m_n_channels = 0;
   int         m_n_values = 0;
   int         m_bit_depth = 0;
   int         m_sample_rate = 0;
+  bool        m_read_float_data = false;
 
   enum class State {
     NEW,
@@ -39,10 +54,12 @@ class SFInputStream : public AudioInputStream
   };
   State       m_state = State::NEW;
 
+  Error open (std::function<SNDFILE* (SF_INFO *)> open_func);
 public:
   ~SFInputStream();
 
   Error               open (const std::string& filename);
+  Error               open (const std::vector<unsigned char> *data);
   Error               read_frames (std::vector<float>& samples, size_t count) override;
   void                close();
 

src/sfoutputstream.cc

@@ -30,7 +30,15 @@ SFOutputStream::~SFOutputStream()
 }
 
 Error
-SFOutputStream::open (const string& filename, int n_channels, int sample_rate, int bit_depth, size_t n_frames)
+SFOutputStream::open (const string& filename, int n_channels, int sample_rate, int bit_depth, OutFormat out_format)
+{
+  return open ([&] (SF_INFO *sfinfo) {
+    return sf_open (filename.c_str(), SFM_WRITE, sfinfo);
+  }, n_channels, sample_rate, bit_depth, out_format);
+}
+
+Error
+SFOutputStream::open (std::function<SNDFILE* (SF_INFO *)> open_func, int n_channels, int sample_rate, int bit_depth, OutFormat out_format)
 {
   assert (m_state == State::NEW);
 
@@ -41,18 +49,26 @@ SFOutputStream::open (const string& filename, int n_channels, int sample_rate, i
   sfinfo.samplerate = sample_rate;
   sfinfo.channels   = n_channels;
 
+   switch (out_format)
+     {
+       case OutFormat::WAV:  sfinfo.format = SF_FORMAT_WAV;
+                             break;
+       case OutFormat::FLAC: sfinfo.format = SF_FORMAT_FLAC;
+                             break;
+       default:              assert (false);
+     }
   if (bit_depth > 16)
     {
-      sfinfo.format = SF_FORMAT_WAV | SF_FORMAT_PCM_24;
+      sfinfo.format |= SF_FORMAT_PCM_24;
       m_bit_depth   = 24;
     }
   else
     {
-      sfinfo.format = SF_FORMAT_WAV | SF_FORMAT_PCM_16;
+      sfinfo.format |= SF_FORMAT_PCM_16;
       m_bit_depth   = 16;
     }
 
-  m_sndfile = sf_open (filename.c_str(), SFM_WRITE, &sfinfo);
+  m_sndfile = open_func (&sfinfo);
   int error = sf_error (m_sndfile);
   if (error)
     {
@@ -122,3 +138,13 @@ SFOutputStream::n_channels() const
 {
   return m_n_channels;
 }
+
+Error
+SFOutputStream::open (vector<unsigned char> *data, int n_channels, int sample_rate, int bit_depth, OutFormat out_format)
+{
+  m_virtual_data.mem = data;
+
+  return open ([&] (SF_INFO *sfinfo) {
+    return sf_open_virtual (&m_virtual_data.io, SFM_WRITE, sfinfo, &m_virtual_data);
+  }, n_channels, sample_rate, bit_depth, out_format);
+}

src/sfoutputstream.hh

@@ -23,9 +23,16 @@
 #include <sndfile.h>
 
 #include "audiostream.hh"
+#include "sfinputstream.hh"
 
 class SFOutputStream : public AudioOutputStream
 {
+public:
+  enum class OutFormat { WAV, FLAC };
+
+private:
+  SFVirtualData m_virtual_data;
+
   SNDFILE    *m_sndfile = nullptr;
   int         m_bit_depth = 0;
   int         m_sample_rate = 0;
@@ -38,10 +45,12 @@ class SFOutputStream : public AudioOutputStream
   };
   State       m_state = State::NEW;
 
+  Error open (std::function<SNDFILE* (SF_INFO *)> open_func, int n_channels, int sample_rate, int bit_depth, OutFormat out_format);
 public:
   ~SFOutputStream();
 
-  Error  open (const std::string& filename, int n_channels, int sample_rate, int bit_depth, size_t n_frames);
+  Error  open (const std::string& filename, int n_channels, int sample_rate, int bit_depth, OutFormat out_format = OutFormat::WAV);
+  Error  open (std::vector<unsigned char> *data, int n_channels, int sample_rate, int bit_depth, OutFormat out_format = OutFormat::WAV);
   Error  write_frames (const std::vector<float>& frames) override;
   Error  close() override;
   int    bit_depth() const override;

src/testconvcode.cc

@@ -21,20 +21,10 @@
 #include <random>
 
 #include <assert.h>
-#include <sys/time.h>
 
 using std::vector;
 using std::string;
 
-static double
-gettime()
-{
-  timeval tv;
-  gettimeofday (&tv, 0);
-
-  return tv.tv_sec + tv.tv_usec / 1000000.0;
-}
-
 vector<int>
 generate_error_vector (size_t n, int errors)
 {
@@ -210,14 +200,14 @@ main (int argc, char **argv)
       while (in_bits.size() != 128)
         in_bits.push_back (rand() & 1);
 
-      const double start_t = gettime();
+      const double start_t = get_time();
       const size_t runs = 20;
       for (size_t i = 0; i < runs; i++)
         {
           vector<int> out_bits = conv_decode_hard (block_type, conv_encode (block_type, in_bits));
           assert (out_bits == in_bits);
         }
-      printf ("%.1f ms/block\n", (gettime() - start_t) / runs * 1000.0);
+      printf ("%.1f ms/block\n", (get_time() - start_t) / runs * 1000.0);
     }
   if (argc == 3 && string (argv[2]) == "table")
     conv_print_table (block_type);

src/testhls.cc

+/*
+ * Copyright (C) 2018-2020 Stefan Westerfeld
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <string.h>
+#include <stdio.h>
+
+#include <regex>
+
+#include "utils.hh"
+#include "mpegts.hh"
+#include "wavdata.hh"
+#include "wmcommon.hh"
+#include "hls.hh"
+#include "sfinputstream.hh"
+#include "hlsoutputstream.hh"
+
+using std::string;
+using std::regex;
+using std::vector;
+using std::map;
+using std::min;
+
+class WDInputStream : public AudioInputStream
+{
+  WavData *wav_data;
+  size_t   read_pos = 0;
+public:
+  WDInputStream (WavData *wav_data) :
+    wav_data (wav_data)
+  {
+  }
+  int
+  bit_depth() const override
+  {
+    return wav_data->bit_depth();
+  }
+  int
+  sample_rate() const override
+  {
+    return wav_data->sample_rate();
+  }
+  int
+  n_channels() const override
+  {
+    return wav_data->n_channels();
+  }
+  size_t
+  n_frames() const override
+  {
+    return wav_data->n_values() / wav_data->n_channels();
+  }
+  Error
+  read_frames (std::vector<float>& samples, size_t count) override
+  {
+    size_t read_count = min (n_frames() - read_pos, count);
+
+    const auto& wsamples = wav_data->samples();
+    samples.assign (wsamples.begin() + read_pos * n_channels(), wsamples.begin() + (read_pos + read_count) * n_channels());
+
+    read_pos += read_count;
+
+    return Error::Code::NONE;
+  }
+};
+
+class WDOutputStream : public AudioOutputStream
+{
+  WavData *wav_data;
+  vector<float> samples;
+public:
+  WDOutputStream (WavData *wav_data) :
+    wav_data (wav_data)
+  {
+  }
+  int
+  bit_depth() const override
+  {
+    return wav_data->bit_depth();
+  }
+  int
+  sample_rate() const override
+  {
+    return wav_data->sample_rate();
+  }
+  int
+  n_channels() const override
+  {
+    return wav_data->n_channels();
+  }
+  Error
+  write_frames (const std::vector<float>& frames) override
+  {
+    samples.insert (samples.end(), frames.begin(), frames.end());
+    return Error::Code::NONE;
+  }
+  Error
+  close() override
+  {
+    wav_data->set_samples (samples); // only do this once at end for performance reasons
+    return Error::Code::NONE;
+  }
+};
+
+int
+mark_zexpand (WavData& wav_data, size_t zero_frames, const string& bits)
+{
+  WDInputStream in_stream (&wav_data);
+
+  WavData wav_data_out ({ /* no samples */ }, wav_data.n_channels(), wav_data.sample_rate(), wav_data.bit_depth());
+  WDOutputStream out_stream (&wav_data_out);
+
+  int rc = add_stream_watermark (&in_stream, &out_stream, bits, zero_frames);
+  if (rc != 0)
+    return rc;
+
+  wav_data.set_samples (wav_data_out.samples());
+
+  return 0;
+}
+
+int
+test_seek (const string& in, const string& out, int pos, const string& bits)
+{
+  vector<float> samples;
+  WavData wav_data;
+  Error err = wav_data.load (in);
+  if (err)
+    {
+      error ("load error: %s\n", err.message());
+      return 1;
+    }
+
+  samples = wav_data.samples();
+  samples.erase (samples.begin(), samples.begin() + pos * wav_data.n_channels());
+  wav_data.set_samples (samples);
+
+  int rc = mark_zexpand (wav_data, pos, bits);
+  if (rc != 0)
+    {
+      return rc;
+    }
+
+  samples = wav_data.samples();
+  samples.insert (samples.begin(), pos * wav_data.n_channels(), 0);
+  wav_data.set_samples (samples);
+
+  err = wav_data.save (out);
+  if (err)
+    {
+      error ("save error: %s\n", err.message());
+      return 1;
+    }
+  return 0;
+}
+
+int
+seek_perf (int sample_rate, double seconds)
+{
+  vector<float> samples (100);
+  WavData wav_data (samples, 2, sample_rate, 16);
+
+  double start_time = get_time();
+
+  int rc = mark_zexpand (wav_data, seconds * sample_rate, "0c");
+  if (rc != 0)
+    return rc;
+
+  double end_time = get_time();
+
+  info ("\n\n");
+  info ("total time %7.3f sec\n", end_time - start_time);
+  info ("per second %7.3f ms\n", (end_time - start_time) / seconds * 1000);
+
+  return 0;
+}
+
+int
+main (int argc, char **argv)
+{
+  if (argc == 6 && strcmp (argv[1], "test-seek") == 0)
+    {
+      return test_seek (argv[2], argv[3], atoi (argv[4]), argv[5]);
+    }
+  else if (argc == 4 && strcmp (argv[1], "seek-perf") == 0)
+    {
+      return seek_perf (atoi (argv[2]), atof (argv[3]));
+    }
+  else if (argc == 4 && strcmp (argv[1], "ff-decode") == 0)
+    {
+      WavData wd;
+      Error err = ff_decode (argv[2], wd);
+      if (err)
+        {
+          error ("audiowmark: hls: ff_decode failed: %s\n", err.message());
+          return 1;
+        }
+      err = wd.save (argv[3]);
+      if (err)
+        {
+          error ("audiowmark: hls: save failed: %s\n", err.message());
+          return 1;
+        }
+      return 0;
+    }
+  else
+    {
+      error ("testhls: error parsing command line arguments\n");
+      return 1;
+    }
+}
+

src/testlimiter.cc

@@ -21,7 +21,6 @@
 #include <assert.h>
 #include <math.h>
 #include <string.h>
-#include <sys/time.h>
 
 #include "sfinputstream.hh"
 #include "sfoutputstream.hh"
@@ -33,15 +32,6 @@ using std::vector;
 using std::max;
 using std::min;
 
-static double
-gettime()
-{
-  timeval tv;
-  gettimeofday (&tv, 0);
-
-  return tv.tv_sec + tv.tv_usec / 1000000.0;
-}
-
 int
 perf()
 {
@@ -52,13 +42,13 @@ perf()
   vector<float> samples (2 * 1024);
 
   int n_frames = 0;
-  double start = gettime();
+  double start = get_time();
   for (int i = 0; i < 100000; i++)
     {
       n_frames += samples.size() / 2;
       vector<float> out_samples = limiter.process (samples);
     }
-  double end = gettime();
+  double end = get_time();
   printf ("%f ns/frame\n", (end - start) * 1000 * 1000 * 1000 / n_frames);
   return 0;
 }
@@ -114,7 +104,7 @@ main (int argc, char **argv)
       fprintf (stderr, "testlimiter: open input failed: %s\n", err.message());
       return 1;
     }
-  err = out.open (argv[2], in.n_channels(), in.sample_rate(), 16, in.n_frames());
+  err = out.open (argv[2], in.n_channels(), in.sample_rate(), 16);
   if (err)
     {
       fprintf (stderr, "testlimiter: open output failed: %s\n", err.message());

src/testmpegts.cc

+/*
+ * Copyright (C) 2018-2020 Stefan Westerfeld
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <string.h>
+#include <stdio.h>
+
+#include <array>
+#include <regex>
+
+#include "utils.hh"
+#include "mpegts.hh"
+
+using std::string;
+using std::vector;
+using std::map;
+using std::regex;
+
+int
+main (int argc, char **argv)
+{
+  if (argc == 5 && strcmp (argv[1], "append") == 0)
+    {
+      printf ("append: in=%s out=%s fn=%s\n", argv[2], argv[3], argv[4]);
+      TSWriter writer;
+
+      writer.append_file (argv[4], argv[4]);
+      Error err = writer.process (argv[2], argv[3]);
+      if (err)
+        {
+          error ("ts_append: %s\n", err.message());
+          return 1;
+        }
+    }
+  else if (argc == 3 && strcmp (argv[1], "list") == 0)
+    {
+      TSReader reader;
+
+      Error err = reader.load (argv[2]);
+      for (auto entry : reader.entries())
+        printf ("%s %zd\n", entry.filename.c_str(), entry.data.size());
+    }
+  else if (argc == 4 && strcmp (argv[1], "get") == 0)
+    {
+      TSReader reader;
+
+      Error err = reader.load (argv[2]);
+      for (auto entry : reader.entries())
+        if (entry.filename == argv[3])
+          fwrite (&entry.data[0], 1, entry.data.size(), stdout);
+    }
+  else if (argc == 3 && strcmp (argv[1], "vars") == 0)
+    {
+      TSReader reader;
+
+      Error err = reader.load (argv[2]);
+      map<string, string> vars = reader.parse_vars ("vars");
+      for (auto v : vars)
+        printf ("%s=%s\n", v.first.c_str(), v.second.c_str());
+    }
+  else if (argc == 3 && strcmp (argv[1], "perf") == 0)
+    {
+      for (int i = 0; i < 1000; i++)
+        {
+          TSReader reader;
+
+          Error err = reader.load (argv[2]);
+          if (i == 42)
+            for (auto entry : reader.entries())
+              printf ("%s %zd\n", entry.filename.c_str(), entry.data.size());
+        }
+    }
+  else
+    {
+      error ("testmpegts: error parsing command line arguments\n");
+    }
+}

src/testrandom.cc

@@ -18,20 +18,9 @@
 #include "utils.hh"
 #include "random.hh"
 
-#include <sys/time.h>
-
 using std::vector;
 using std::string;
 
-static double
-gettime()
-{
-  timeval tv;
-  gettimeofday (&tv, 0);
-
-  return tv.tv_sec + tv.tv_usec / 1000000.0;
-}
-
 int
 main (int argc, char **argv)
 {
@@ -43,13 +32,13 @@ main (int argc, char **argv)
     }
 
   uint64_t s = 0;
-  double t_start = gettime();
+  double t_start = get_time();
   size_t runs = 25000000;
   for (size_t i = 0; i < runs; i++)
     {
       s += rng();
     }
-  double t_end = gettime();
+  double t_end = get_time();
   printf ("s=%016lx\n\n", s);
 
   printf ("%f Mvalues/sec\n", runs / (t_end - t_start) / 1000000);

src/utils.cc

@@ -18,9 +18,21 @@
 #include "utils.hh"
 #include "stdarg.h"
 
+#include <sys/time.h>
+
 using std::vector;
 using std::string;
 
+double
+get_time()
+{
+  /* return timestamp in seconds as double */
+  timeval tv;
+  gettimeofday (&tv, 0);
+
+  return tv.tv_sec + tv.tv_usec / 1000000.0;
+}
+
 static unsigned char
 from_hex_nibble (char c)
 {
@@ -108,6 +120,35 @@ vec_to_hex_str (const vector<unsigned char>& vec)
   return s;
 }
 
+static string
+string_vprintf (const char *format, va_list vargs)
+{
+  string s;
+
+  char *str = NULL;
+  if (vasprintf (&str, format, vargs) >= 0 && str)
+    {
+      s = str;
+      free (str);
+    }
+  else
+    s = format;
+
+  return s;
+}
+
+string
+string_printf (const char *format, ...)
+{
+  va_list ap;
+
+  va_start (ap, format);
+  string s = string_vprintf (format, ap);
+  va_end (ap);
+
+  return s;
+}
+
 static Log log_level = Log::INFO;
 
 void
@@ -121,12 +162,10 @@ logv (Log log, const char *format, va_list vargs)
 {
   if (log >= log_level)
     {
-      char buffer[1024];
-
-      vsnprintf (buffer, sizeof (buffer), format, vargs);
+      string s = string_vprintf (format, vargs);
 
       /* could support custom log function here */
-      fprintf (stderr, "%s", buffer);
+      fprintf (stderr, "%s", s.c_str());
       fflush (stderr);
     }
 }

src/utils.hh

@@ -27,6 +27,8 @@ std::string      bit_vec_to_str (const std::vector<int>& bit_vec);
 std::vector<unsigned char> hex_str_to_vec (const std::string& str);
 std::string                vec_to_hex_str (const std::vector<unsigned char>& vec);
 
+double get_time();
+
 template<typename T>
 inline const T&
 bound (const T& min_value, const T& value, const T& max_value)
@@ -58,6 +60,8 @@ enum class Log { ERROR = 3, WARNING = 2, INFO = 1, DEBUG = 0 };
 
 void set_log_level (Log level);
 
+std::string string_printf (const char *fmt, ...) AUDIOWMARK_PRINTF (1, 2);
+
 class Error
 {
 public:
@@ -103,4 +107,19 @@ private:
   std::string m_message;
 };
 
+class ScopedFile
+{
+  FILE *m_file;
+public:
+  ScopedFile (FILE *f) :
+    m_file (f)
+  {
+  }
+  ~ScopedFile()
+  {
+    if (m_file)
+      fclose (m_file);
+  }
+};
+
 #endif /* AUDIOWMARK_UTILS_HH */

src/wavdata.cc

@@ -55,6 +55,8 @@ WavData::load (const string& filename)
 Error
 WavData::load (AudioInputStream *in_stream)
 {
+  m_samples.clear(); // get rid of old contents
+
   vector<float> m_buffer;
   while (true)
     {
@@ -77,7 +79,7 @@ WavData::load (AudioInputStream *in_stream)
 }
 
 Error
-WavData::save (const string& filename)
+WavData::save (const string& filename) const
 {
   std::unique_ptr<AudioOutputStream> out_stream;
   Error err;

src/wavdata.hh

@@ -37,7 +37,7 @@ public:
 
   Error load (AudioInputStream *in_stream);
   Error load (const std::string& filename);
-  Error save (const std::string& filename);
+  Error save (const std::string& filename) const;
 
   int                         sample_rate() const;
   int                         bit_depth() const;
@@ -52,6 +52,11 @@ public:
   {
     return m_samples.size();
   }
+  size_t
+  n_frames() const
+  {
+    return m_samples.size() / m_n_channels;
+  }
   const std::vector<float>&
   samples() const
   {

src/wmcommon.cc

@@ -39,6 +39,8 @@ Format Params::output_format    = Format::AUTO;
 RawFormat Params::raw_input_format;
 RawFormat Params::raw_output_format;
 
+int    Params::hls_bit_rate = 0;
+
 std::string Params::input_label;
 std::string Params::output_label;
 

src/wmcommon.hh

@@ -69,6 +69,8 @@ public:
   static           RawFormat raw_input_format;
   static           RawFormat raw_output_format;
 
+  static           int hls_bit_rate;
+
   // input/output labels can be set for pretty output for videowmark add
   static           std::string input_label;
   static           std::string output_label;
@@ -161,6 +163,7 @@ randomize_bit_order (const std::vector<T>& bit_vec, bool encode)
   return out_bits;
 }
 
+int add_stream_watermark (AudioInputStream *in_stream, AudioOutputStream *out_stream, const std::string& bits, size_t zero_frames);
 int add_watermark (const std::string& infile, const std::string& outfile, const std::string& bits);
 int get_watermark (const std::string& infile, const std::string& orig_pattern);