Merge branch 'speed-detection-v2'

* speed-detection-v2:
  Produce better error messages for some commands.
  Use similar wording for --help options summary.
  Add verbose reporting to gen-speed-mk.sh.
  Fix speed detection for short payloads.
  TODO~
  Fix test script (keep watermarked files for which cmp failed).
  Use sync quality threshold for speed detection to minimize false positives.
  Fix comment.
  Add test script option to copy files if watermark detection fails.
  Merge analysis window generation into one function.
  Use n-best search for patient speed detection as well.
  Normalize speed detection quality scores.
  Export sync finder quality normalization function.
  Fix sync threshold comparision for equal sync quality scores.
  Merge sync finder / speed detection raw bit quality function.
  Fix bug in n-best selection (if three scores have the same value).
  Make test script exit with non-zero exit code if something goes wrong.
  TESTS: add test for audiowmark --key and --test-key
  Support audiowmark cmp --expect-matches 0 (true if no matches found).
  TESTS: use audiowmark cmp --expect-matches option
  Implement audiowmark cmp --expect-matches option for tests.
  Use smooth score function for speed detection to improve accuracy.
  TESTS: use audiowmark test function to improve debugging output
  TESTS: support make V=1 for make check
  TESTS: run sample-rate-test.sh during make check
  Add test for non-standard sample rates.
  Add audiowmark test-resample helper for sample rate testing.
  TESTS: add synchronization tests
  TESTS: add test for short payload
  BUILD: update EXTRA_DIST to make make distcheck work
  Fix option parser: single dash (-) is a valid filename (stdin / stdout).
  TESTS: add test for watermark add / cmp from pipe
  Improve error messages for command line argument parser.
  BUILD: add EXTRA_DIST to make make distcheck work
  Add scripts for make check.
  Add audiowmark helper commands for make check.
  Fix speed Makefile generator bugs.
  Fix overflow for time computation.
  SFInputStream: fixes related to number of frames.
	 - use size_t for n_frames -> support larger files
	 - map SF_COUNT_MAX to unknown length
  Support --try-speed in gen-speed-mk.sh.
  Support --try-speed in ber-test.sh.
  Document --detect-speed-patient.
  Update help to include --detect-speed-patient.
  TODO--
  Avoid message expansion in if --strict is used.
  TODO--
  Refactor payload parsing (avoid crashes for cmp with invalid bits).
  Introduce rounding for speed detection again, avoid lrint() though.
	While for long clips rounding doesn't help (much), it does for short
	clips.
  TODO~
  Test patient speed detection in speed test Makefile generator.
  Add patient speed detection to test script (AWM_SPEED_PATIENT=1).
  Add speed difference debugging output.
  Add speed range debugging function.
  Only accept --json option for cmp/get.
  Avoid lrint() in speed detection (rounding doesn't improve results).
  Reuse last value for speed compare bits start search (performance).
  Refactor speed detection compare loop into optimized extra function.
  Split speed detection compare loop into two loops (performance).
  Process sync bits in sorted order during speed detection (performance).
  Print total speed detection time for performance debugging.
  Update TODO.
  Cleanup window related FIXME in speed detection.
	Tests show: the hamming window is a good choice here.
  Refactor speed detection search.
  Only output speed detection timing information during cmp.
  Introduce SpeedSearch class.
  Throw away speed interpolation code.
  Get rid of extra speed_scan function.
  Cleanup SpeedSync API.
  Fix patient speed scan clip location size bug.
  Minor SpeedSync API cleanup.
  Refactor speed detection run_search() function.
  Refactor closest speed sync search.
  Fix problems / corner cases in speed detection n-best selection function.
  Skip 2nd search step for patient speed detection.
  Refactor speed_scan function.
  Initial implementation for --detect-speed-patient.
  Add extra function for speed score average computation.
  Reuse already prepared SpeedSync for 3rd speed detection step.
  Reduce search range for 3rd speed detection step.
  Implement n-best search for speed detection.
  Merge both speed scan passes into one function.
Signed-off-by: Stefan Westerfeld <stefan@space.twc.de>

Makefile.am

-SUBDIRS = src
+SUBDIRS = src tests
 ACLOCAL_AMFLAGS = -I m4
 
 EXTRA_DIST = README.adoc Dockerfile

README.adoc

@@ -137,6 +137,7 @@ Use watermarking key from file <filename> (see <<key>>).
 Set the watermarking strength (see <<strength>>).
 
 --detect-speed::
+--detect-speed-patient::
 Detect and correct replay speed difference (see <<speed>>).
 
 --json <file>::
@@ -302,6 +303,10 @@ the search is automatically run in parallel using many threads on systems with
 many cpu cores. So on good hardware it makes sense to always enable this option
 to be robust to replay speed attacks.
 
+There are two versions of the speed detection algorithm, `--detect-speed` and
+`--detect-speed-patient`. The difference is that the patient version takes
+more cpu time to detect the speed, but produces more accurate results.
+
 == Short Payload (experimental)
 
 By default, the watermark will store a 128-bit message. In this mode, we

configure.ac

@@ -102,7 +102,7 @@ AC_LANG_POP([C++])
 # Less cluttered build output
 m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])
 
-AC_CONFIG_FILES([Makefile src/Makefile])
+AC_CONFIG_FILES([Makefile src/Makefile tests/Makefile tests/test-common.sh])
 AC_OUTPUT
 
 # Output summary message

src/TODO

 speed detection possible improvements:
-- refine with 128 sync step instead of 256?
-- use bigger initial region (50s) --detect-speed-harder
-- use window hann instead of hamming for sync bits (or all bits)?
-- remove lrint and QInterpolator?
 - detect silence instead of total volume?
 - split big region into two smaller ones?
 - connected search on even smaller regions?
-- reorder memory access in compare to be more cache efficient
-- n-best search?
-- soft sync?
 
 possible improvements:
 - dynamic bit strength
+- merge infinite wav header generator
 
 videowmark:
 opus length ffprobe -show_format phil.mkv

src/ber-test.sh

 #!/bin/bash
 
+# set -Eeuo pipefail # -x
+set -Eeo pipefail
+
 TRANSFORM=$1
 if [ "x$AWM_TRUNCATE" != "x" ]; then
   AWM_REPORT=truncv
@@ -23,6 +26,17 @@ if [ "x$AWM_PATTERN_BITS" == "x" ]; then
   AWM_PATTERN_BITS=128
 fi
 
+audiowmark_cmp()
+{
+  audiowmark cmp "$@" || {
+    if [ "x$AWM_FAIL_DIR" != "x" ]; then
+      mkdir -p $AWM_FAIL_DIR
+      SUM=$(sha1sum $1 | awk '{print $1;}')
+      cp -av $1 $AWM_FAIL_DIR/${AWM_FILE}.${SUM}.wav
+    fi
+  }
+}
+
 {
   if [ "x$AWM_SET" == "xsmall" ]; then
     ls test/T*
@@ -80,13 +94,19 @@ do
 
       [ -z $SPEED_SEED ] && SPEED_SEED=0
       SPEED=$(audiowmark test-speed $SPEED_SEED --test-key $SEED)
-      ((SPEED_SEED++))
+      SPEED_SEED=$((SPEED_SEED + 1))
       echo in_speed $SPEED
 
       sox -D -V1 ${AWM_FILE}.wav ${AWM_FILE}.speed.wav speed $SPEED
       mv ${AWM_FILE}.speed.wav ${AWM_FILE}.wav
 
+      if [ "x$AWM_SPEED_PATIENT" != x ]; then
+        TEST_SPEED_ARGS="--detect-speed-patient --test-speed $SPEED"
+      elif [ "x$AWM_TRY_SPEED" != x ]; then
+        TEST_SPEED_ARGS="--try-speed $SPEED"
+      else
         TEST_SPEED_ARGS="--detect-speed --test-speed $SPEED"
+      fi
     else
       TEST_SPEED_ARGS=""
     fi
@@ -128,29 +148,29 @@ do
       for CLIP in $(seq $AWM_MULTI_CLIP)
       do
         audiowmark test-clip $OUT_FILE ${OUT_FILE}.clip.wav $((CLIP_SEED++)) $AWM_CLIP --test-key $SEED
-        audiowmark cmp ${OUT_FILE}.clip.wav $PATTERN $AWM_PARAMS --test-key $SEED $TEST_CUT_ARGS $TEST_SPEED_ARGS
+        audiowmark_cmp ${OUT_FILE}.clip.wav $PATTERN $AWM_PARAMS --test-key $SEED $TEST_CUT_ARGS $TEST_SPEED_ARGS
         rm ${OUT_FILE}.clip.wav
         echo
       done
     elif [ "x$AWM_REPORT" == "xtruncv" ]; then
       for TRUNC in $AWM_TRUNCATE
       do
-        audiowmark cmp $OUT_FILE $PATTERN $AWM_PARAMS --test-key $SEED $TEST_CUT_ARGS $TEST_SPEED_ARGS --test-truncate $TRUNC | sed "s/^/$TRUNC /g"
+        audiowmark_cmp $OUT_FILE $PATTERN $AWM_PARAMS --test-key $SEED $TEST_CUT_ARGS $TEST_SPEED_ARGS --test-truncate $TRUNC | sed "s/^/$TRUNC /g"
         echo
       done
     else
-      audiowmark cmp $OUT_FILE $PATTERN $AWM_PARAMS --test-key $SEED $TEST_CUT_ARGS $TEST_SPEED_ARGS
+      audiowmark_cmp $OUT_FILE $PATTERN $AWM_PARAMS --test-key $SEED $TEST_CUT_ARGS $TEST_SPEED_ARGS
       echo
     fi
     rm -f ${AWM_FILE}.wav $OUT_FILE # cleanup temp files
   done
 done | {
   if [ "x$AWM_REPORT" == "xfer" ]; then
-    awk 'BEGIN { bad = n = 0 } $1 == "match_count" { if ($2 == 0) bad++; n++; } END { print bad, n, bad * 100.0 / n; }'
+    awk 'BEGIN { bad = n = 0 } $1 == "match_count" { if ($2 == 0) bad++; n++; } END { print bad, n, bad * 100.0 / (n > 0 ? n : 1); }'
   elif [ "x$AWM_REPORT" == "xferv" ]; then
-    awk 'BEGIN { bad = n = 0 } { print "###", $0; } $1 == "match_count" { if ($2 == 0) bad++; n++; } END { print bad, n, bad * 100.0 / n; }'
+    awk 'BEGIN { bad = n = 0 } { print "###", $0; } $1 == "match_count" { if ($2 == 0) bad++; n++; } END { print bad, n, bad * 100.0 / (n > 0 ? n : 1); }'
   elif [ "x$AWM_REPORT" == "xsync" ]; then
-    awk 'BEGIN { bad = n = 0 } $1 == "sync_match" { bad += (3 - $2) / 3.0; n++; } END { print bad, n, bad * 100.0 / n; }'
+    awk 'BEGIN { bad = n = 0 } $1 == "sync_match" { bad += (3 - $2) / 3.0; n++; } END { print bad, n, bad * 100.0 / (n > 0 ? n : 1); }'
   elif [ "x$AWM_REPORT" == "xsyncv" ]; then
     awk '{ print "###", $0; } $1 == "sync_match" { correct += $2; missing += 3 - $2; incorrect += $3-$2; print "correct:", correct, "missing:", missing, "incorrect:", incorrect; }'
   elif [ "x$AWM_REPORT" == "xtruncv" ]; then

src/gen-speed-mk.sh

@@ -3,20 +3,24 @@
 SEEDS=$(seq 10)
 STRENGTHS="10 15"
 CLIPS="15 30"
+MODES="0 1 2"
 
 echo -n "all:"
 
 for SEED in $SEEDS
 do
+  for MODE in $MODES
+  do
     for STRENGTH in $STRENGTHS
     do
       # clips
       for CLIP in $CLIPS
       do
-      echo -n " speed-$CLIP-$STRENGTH-$SEED"
+        echo -n " speed-$CLIP-$STRENGTH-$MODE-$SEED"
       done
       # full file
-    echo -n " speed-full-$STRENGTH-$SEED"
+      echo -n " speed-full-$STRENGTH-$MODE-$SEED"
+    done
   done
 done
 
@@ -25,22 +29,32 @@ echo
 
 for SEED in $SEEDS
 do
+  for MODE in $MODES
+  do
+    if [ "x$MODE" == "x0" ]; then
+      MODE_ARGS=""
+    elif [ "x$MODE" == "x1" ]; then
+      MODE_ARGS="AWM_SPEED_PATIENT=1"
+    elif [ "x$MODE" == "x2" ]; then
+      MODE_ARGS="AWM_TRY_SPEED=1"
+    fi
     for STRENGTH in $STRENGTHS
     do
       # clips
       for CLIP in $CLIPS
       do
-      FILE="speed-$CLIP-$STRENGTH-$SEED"
+        FILE="speed-$CLIP-$STRENGTH-$MODE-$SEED"
         echo "$FILE:"
-      echo -e "\t( cd ..; AWM_RAND_PATTERN=1 AWM_SET=huge2 AWM_PARAMS='--strength $STRENGTH' AWM_SPEED=1 AWM_SPEED_PRE_MP3=128 AWM_CLIP='$CLIP' AWM_SEEDS=$SEED AWM_FILE='t-$FILE' ber-test.sh mp3 128 ) >x$FILE"
+        echo -e "\t( cd ..; AWM_REPORT=ferv AWM_RAND_PATTERN=1 AWM_SET=huge2 AWM_PARAMS_ADD='--strength $STRENGTH' AWM_SPEED=1 $MODE_ARGS AWM_SPEED_PRE_MP3=128 AWM_CLIP='$CLIP' AWM_SEEDS=$SEED AWM_FILE='t-$FILE' ber-test.sh mp3 128 ) >x$FILE"
         echo -e "\tmv x$FILE $FILE"
         echo
       done
       # full file
-    FILE="speed-full-$STRENGTH-$SEED"
+      FILE="speed-full-$STRENGTH-$MODE-$SEED"
       echo "$FILE:"
-    echo -e "\t( cd ..; AWM_RAND_PATTERN=1 AWM_SET=huge2 AWM_PARAMS='--strength $STRENGTH' AWM_SPEED=1 AWM_SPEED_PRE_MP3=128 AWM_SEEDS=$SEED AWM_FILE='t-$FILE' ber-test.sh mp3 128 ) >x$FILE"
+      echo -e "\t( cd ..; AWM_REPORT=ferv AWM_RAND_PATTERN=1 AWM_SET=huge2 AWM_PARAMS_ADD='--strength $STRENGTH' AWM_SPEED=1 $MODE_ARGS AWM_SPEED_PRE_MP3=128 AWM_SEEDS=$SEED AWM_FILE='t-$FILE' ber-test.sh mp3 128 ) >x$FILE"
       echo -e "\tmv x$FILE $FILE"
       echo
     done
+  done
 done

src/sfinputstream.cc

@@ -68,7 +68,7 @@ SFInputStream::open (std::function<SNDFILE* (SF_INFO *)> open_func)
     }
 
   m_n_channels  = sfinfo.channels;
-  m_n_values    = sfinfo.frames * sfinfo.channels;
+  m_n_frames    = (sfinfo.frames == SF_COUNT_MAX) ? N_FRAMES_UNKNOWN : sfinfo.frames;
   m_sample_rate = sfinfo.samplerate;
 
   switch (sfinfo.format & SF_FORMAT_SUBMASK)

src/sfinputstream.hh

@@ -42,7 +42,7 @@ private:
 
   SNDFILE    *m_sndfile = nullptr;
   int         m_n_channels = 0;
-  int         m_n_values = 0;
+  size_t      m_n_frames = 0;
   int         m_bit_depth = 0;
   int         m_sample_rate = 0;
   bool        m_read_float_data = false;
@@ -74,7 +74,7 @@ public:
   size_t
   n_frames() const override
   {
-    return m_n_values / m_n_channels;
+    return m_n_frames;
   }
 };
 

src/syncfinder.cc

@@ -76,6 +76,7 @@ SyncFinder::init_up_down (const WavData& wav_data, Mode mode)
     }
 }
 
+/* safe to call from any thread */
 double
 SyncFinder::normalize_sync_quality (double raw_quality)
 {
@@ -89,6 +90,29 @@ SyncFinder::normalize_sync_quality (double raw_quality)
   return raw_quality / min (Params::water_delta, 0.080) / 2.9;
 }
 
+/* safe to call from any thread */
+double
+SyncFinder::bit_quality (float umag, float dmag, int bit)
+{
+  const int expect_data_bit = bit & 1; /* expect 010101 */
+
+  /* convert avoiding bias, raw_bit < 0 => 0 bit received; raw_bit > 0 => 1 bit received */
+  double raw_bit;
+  if (umag == 0 || dmag == 0)
+    {
+      raw_bit = 0;
+    }
+  else if (umag < dmag)
+    {
+      raw_bit = 1 - umag / dmag;
+    }
+  else
+    {
+      raw_bit = dmag / umag - 1;
+    }
+  return expect_data_bit ? raw_bit : -raw_bit;
+}
+
 double
 SyncFinder::sync_decode (const WavData& wav_data, const size_t start_frame,
                          const vector<float>& fft_out_db,
@@ -118,24 +142,7 @@ SyncFinder::sync_decode (const WavData& wav_data, const size_t start_frame,
               frame_bit_count++;
             }
         }
-      /* convert avoiding bias, raw_bit < 0 => 0 bit received; raw_bit > 0 => 1 bit received */
-      double raw_bit;
-      if (umag == 0 || dmag == 0)
-        {
-          raw_bit = 0;
-        }
-      else if (umag < dmag)
-        {
-          raw_bit = 1 - umag / dmag;
-        }
-      else
-        {
-          raw_bit = dmag / umag - 1;
-        }
-
-      const int expect_data_bit = bit & 1; /* expect 010101 */
-      const double q = expect_data_bit ? raw_bit : -raw_bit;
-      sync_quality += q * frame_bit_count;
+      sync_quality += bit_quality (umag, dmag, bit) * frame_bit_count;
       bit_count += frame_bit_count;
     }
   if (bit_count)
@@ -227,9 +234,10 @@ SyncFinder::sync_select_by_threshold (vector<Score>& sync_scores)
           if (i + 1 < sync_scores.size())
             q_next = sync_scores[i + 1].quality;
 
-          if (sync_scores[i].quality > q_last && sync_scores[i].quality > q_next)
+          if (sync_scores[i].quality >= q_last && sync_scores[i].quality >= q_next)
             {
               selected_scores.emplace_back (sync_scores[i]);
+              i++; // score with quality q_next cannot be a local maximum
             }
         }
     }

src/syncfinder.hh

@@ -81,7 +81,6 @@ private:
   std::vector<std::vector<FrameBit>> sync_bits;
 
   void    init_up_down (const WavData& wav_data, Mode mode);
-  double  normalize_sync_quality (double raw_quality);
   double  sync_decode (const WavData& wav_data, const size_t start_frame,
                        const std::vector<float>& fft_out_db,
                        const std::vector<char>&  have_frames,
@@ -99,6 +98,9 @@ private:
 public:
   std::vector<Score> search (const WavData& wav_data, Mode mode);
   std::vector<std::vector<FrameBit>> get_sync_bits (const WavData& wav_data, Mode mode);
+
+  static double bit_quality (float umag, float dmag, int bit);
+  static double normalize_sync_quality (double raw_quality);
 private:
   void sync_fft (const WavData& wav_data,
                  size_t index,

src/wmadd.cc

@@ -591,30 +591,9 @@ info_format (const string& label, const RawFormat& format)
 int
 add_stream_watermark (AudioInputStream *in_stream, AudioOutputStream *out_stream, const string& bits, size_t zero_frames)
 {
-  auto bitvec = bit_str_to_vec (bits);
+  auto bitvec = parse_payload (bits);
   if (bitvec.empty())
-    {
-      error ("audiowmark: cannot parse bits %s\n", bits.c_str());
-      return 1;
-    }
-  if (Params::payload_short && bitvec.size() != Params::payload_size)
-    {
-      error ("audiowmark: number of message bits must match payload size (%zd bits)\n", Params::payload_size);
-      return 1;
-    }
-  if (bitvec.size() > Params::payload_size)
-    {
-      error ("audiowmark: number of bits in message '%s' larger than payload size\n", bits.c_str());
     return 1;
-    }
-  if (bitvec.size() < Params::payload_size)
-    {
-      /* expand message automatically; good for testing, maybe not so good for the final product */
-      vector<int> expanded_bitvec;
-      for (size_t i = 0; i < Params::payload_size; i++)
-        expanded_bitvec.push_back (bitvec[i % bitvec.size()]);
-      bitvec = expanded_bitvec;
-    }
 
   /* sanity checks */
   if (in_stream->sample_rate() != out_stream->sample_rate())
@@ -638,8 +617,8 @@ add_stream_watermark (AudioInputStream *in_stream, AudioOutputStream *out_stream
     }
   else
     {
-      int orig_seconds = in_stream->n_frames() / in_stream->sample_rate();
-      info ("Time:         %d:%02d\n", orig_seconds / 60, orig_seconds % 60);
+      size_t orig_seconds = in_stream->n_frames() / in_stream->sample_rate();
+      info ("Time:         %zd:%02zd\n", orig_seconds / 60, orig_seconds % 60);
     }
   info ("Sample Rate:  %d\n", in_stream->sample_rate());
   info ("Channels:     %d\n", in_stream->n_channels());

src/wmcommon.cc

@@ -20,6 +20,10 @@
 #include "convcode.hh"
 #include "shortcode.hh"
 
+using std::string;
+using std::vector;
+using std::complex;
+
 int    Params::frames_per_bit  = 2;
 double Params::water_delta     = 0.01;
 bool   Params::mix             = true;
@@ -27,6 +31,7 @@ bool   Params::hard            = false; // hard decode bits? (soft decoding is b
 bool   Params::snr             = false; // compute/show snr while adding watermark
 bool   Params::strict          = false;
 bool   Params::detect_speed    = false;
+bool   Params::detect_speed_patient = false;
 double Params::try_speed       = -1;
 double Params::test_speed      = -1;
 int    Params::have_key        = 0;
@@ -36,6 +41,7 @@ int    Params::test_cut        = 0; // for sync test
 bool   Params::test_no_sync    = false; // disable sync
 bool   Params::test_no_limiter = false; // disable limiter
 int    Params::test_truncate   = 0;
+int    Params::expect_matches  = -1;
 
 Format Params::input_format     = Format::AUTO;
 Format Params::output_format    = Format::AUTO;
@@ -45,37 +51,41 @@ RawFormat Params::raw_output_format;
 
 int    Params::hls_bit_rate = 0;
 
-std::string Params::json_output;
-std::string Params::input_label;
-std::string Params::output_label;
-
-using std::vector;
-using std::complex;
+string Params::json_output;
+string Params::input_label;
+string Params::output_label;
 
 FFTAnalyzer::FFTAnalyzer (int n_channels) :
   m_n_channels (n_channels),
   m_fft_processor (Params::frame_size)
 {
-  /* generate analysis window */
-  m_window.resize (Params::frame_size);
+  m_window = gen_normalized_window (Params::frame_size);
+}
+
+/* safe to call from any thread */
+vector<float>
+FFTAnalyzer::gen_normalized_window (size_t n_values)
+{
+  vector<float> window (n_values);
 
+  /* generate analysis window */
   double window_weight = 0;
-  for (size_t i = 0; i < Params::frame_size; i++)
+  for (size_t i = 0; i < n_values; i++)
     {
-      const double fsize_2 = Params::frame_size / 2.0;
-      // const double win =  window_cos ((i - fsize_2) / fsize_2);
-      const double win = window_hamming ((i - fsize_2) / fsize_2);
+      const double n_values_2 = n_values / 2.0;
+      // const double win =  window_cos ((i - n_values_2) / n_values_2);
+      const double win = window_hamming ((i - n_values_2) / n_values_2);
       //const double win = 1;
-      m_window[i] = win;
+      window[i] = win;
       window_weight += win;
     }
 
   /* normalize window using window weight */
-  for (size_t i = 0; i < Params::frame_size; i++)
+  for (size_t i = 0; i < n_values; i++)
     {
-      m_window[i] *= 2.0 / window_weight;
+      window[i] *= 2.0 / window_weight;
     }
-
+  return window;
 }
 
 vector<vector<complex<float>>>
@@ -211,3 +221,33 @@ frame_count (const WavData& wav_data)
 {
   return wav_data.n_values() / wav_data.n_channels() / Params::frame_size;
 }
+
+vector<int>
+parse_payload (const string& bits)
+{
+  auto bitvec = bit_str_to_vec (bits);
+  if (bitvec.empty())
+    {
+      error ("audiowmark: cannot parse bits '%s'\n", bits.c_str());
+      return {};
+    }
+  if ((Params::payload_short || Params::strict) && bitvec.size() != Params::payload_size)
+    {
+      error ("audiowmark: number of message bits must match payload size (%zd bits)\n", Params::payload_size);
+      return {};
+    }
+  if (bitvec.size() > Params::payload_size)
+    {
+      error ("audiowmark: number of bits in message '%s' larger than payload size\n", bits.c_str());
+      return {};
+    }
+  if (bitvec.size() < Params::payload_size)
+    {
+      /* expand message automatically; good for testing, not so good in production (disabled by --strict) */
+      vector<int> expanded_bitvec;
+      for (size_t i = 0; i < Params::payload_size; i++)
+        expanded_bitvec.push_back (bitvec[i % bitvec.size()]);
+      bitvec = expanded_bitvec;
+    }
+  return bitvec;
+}

src/wmcommon.hh

@@ -48,6 +48,7 @@ public:
   static           int  have_key;
 
   static           bool detect_speed;
+  static           bool detect_speed_patient;
   static           double try_speed;               // manual speed correction
   static           double test_speed;              // for debugging --detect-speed
 
@@ -70,6 +71,7 @@ public:
   static           bool test_no_sync;
   static           bool test_no_limiter;
   static           int test_truncate;
+  static           int expect_matches;
 
   static           Format input_format;
   static           Format output_format;
@@ -128,6 +130,8 @@ public:
 
   std::vector<std::vector<std::complex<float>>> run_fft (const std::vector<float>& samples, size_t start_index);
   std::vector<std::vector<std::complex<float>>> fft_range (const std::vector<float>& samples, size_t start_index, size_t frame_count);
+
+  static std::vector<float> gen_normalized_window (size_t n_values);
 };
 
 struct MixEntry
@@ -147,6 +151,8 @@ int frame_count (const WavData& wav_data);
 int sync_frame_pos (int f);
 int data_frame_pos (int f);
 
+std::vector<int> parse_payload (const std::string& str);
+
 template<class T> std::vector<T>
 randomize_bit_order (const std::vector<T>& bit_vec, bool encode)
 {

src/wmget.cc

@@ -264,19 +264,13 @@ public:
       }
   }
   int
-  print_match_count (const string& orig_pattern)
+  print_match_count (const vector<int>& orig_bits)
   {
     int match_count = 0;
 
-    vector<int> orig_vec = bit_str_to_vec (orig_pattern);
     for (auto p : patterns)
       {
-        bool        match = true;
-
-        for (size_t i = 0; i < p.bit_vec.size(); i++)
-          match = match && (p.bit_vec[i] == orig_vec[i % orig_vec.size()]);
-
-        if (match)
+        if (p.bit_vec == orig_bits)
           match_count++;
       }
     printf ("match_count %d %zd\n", match_count, patterns.size());
@@ -587,7 +581,7 @@ public:
 };
 
 static int
-decode_and_report (const WavData& wav_data, const string& orig_pattern)
+decode_and_report (const WavData& wav_data, const vector<int>& orig_bits)
 {
   ResultSet result_set;
   double speed = 1.0;
@@ -601,10 +595,10 @@ decode_and_report (const WavData& wav_data, const string& orig_pattern)
    * The reason to do it this way is that the detected speed may be wrong (on short clips)
    * and we don't want to loose a successful clip decoder match in this case.
    */
-  if (Params::detect_speed || Params::try_speed > 0)
+  if (Params::detect_speed || Params::detect_speed_patient || Params::try_speed > 0)
     {
-      if (Params::detect_speed)
-        speed = detect_speed (wav_data, !orig_pattern.empty());
+      if (Params::detect_speed || Params::detect_speed_patient)
+        speed = detect_speed (wav_data, !orig_bits.empty());
       else
         speed = Params::try_speed;
 
@@ -639,21 +633,38 @@ decode_and_report (const WavData& wav_data, const string& orig_pattern)
   if (Params::json_output != "-")
     result_set.print();
 
-  if (!orig_pattern.empty())
+  if (!orig_bits.empty())
     {
-      int match_count = result_set.print_match_count (orig_pattern);
+      int match_count = result_set.print_match_count (orig_bits);
 
       block_decoder.print_debug_sync();
 
+      if (Params::expect_matches >= 0)
+        {
+          printf ("expect_matches %d\n", Params::expect_matches);
+          if (match_count != Params::expect_matches)
+            return 1;
+        }
+      else
+        {
           if (!match_count)
             return 1;
         }
+    }
   return 0;
 }
 
 int
 get_watermark (const string& infile, const string& orig_pattern)
 {
+  vector<int> orig_bitvec;
+  if (!orig_pattern.empty())
+    {
+      orig_bitvec = parse_payload (orig_pattern);
+      if (orig_bitvec.empty())
+        return 1;
+    }
+
   WavData wav_data;
   Error err = wav_data.load (infile);
   if (err)
@@ -675,10 +686,10 @@ get_watermark (const string& infile, const string& orig_pattern)
     }
   if (wav_data.sample_rate() == Params::mark_sample_rate)
     {
-      return decode_and_report (wav_data, orig_pattern);
+      return decode_and_report (wav_data, orig_bitvec);
     }
   else
     {
-      return decode_and_report (resample (wav_data, Params::mark_sample_rate), orig_pattern);
+      return decode_and_report (resample (wav_data, Params::mark_sample_rate), orig_bitvec);
     }
 }

tests/.gitignore

+test-common.sh

tests/Makefile.am

+check: detect-speed-test block-decoder-test clip-decoder-test \
+       pipe-test short-payload-test sync-test sample-rate-test \
+       key-test
+
+EXTRA_DIST = detect-speed-test.sh block-decoder-test.sh clip-decoder-test.sh \
+       pipe-test.sh short-payload-test.sh sync-test.sh sample-rate-test.sh \
+       key-test.sh
+
+detect-speed-test:
+	Q=1 $(top_srcdir)/tests/detect-speed-test.sh
+
+block-decoder-test:
+	Q=1 $(top_srcdir)/tests/block-decoder-test.sh
+
+clip-decoder-test:
+	Q=1 $(top_srcdir)/tests/clip-decoder-test.sh
+
+pipe-test:
+	Q=1 $(top_srcdir)/tests/pipe-test.sh
+
+short-payload-test:
+	Q=1 $(top_srcdir)/tests/short-payload-test.sh
+
+sync-test:
+	Q=1 $(top_srcdir)/tests/sync-test.sh
+
+sample-rate-test:
+	Q=1 $(top_srcdir)/tests/sample-rate-test.sh
+
+key-test:
+	Q=1 $(top_srcdir)/tests/key-test.sh

tests/block-decoder-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=block-decoder-test.wav
+OUT_WAV=block-decoder-test-out.wav
+
+audiowmark test-gen-noise $IN_WAV 200 44100
+audiowmark_add $IN_WAV $OUT_WAV $TEST_MSG
+audiowmark_cmp --expect-matches 5 $OUT_WAV $TEST_MSG
+
+rm $IN_WAV $OUT_WAV
+exit 0

tests/clip-decoder-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=clip-decoder-test.wav
+OUT_WAV=clip-decoder-test-out.wav
+CUT_WAV=clip-decoder-test-out-cut.wav
+
+audiowmark test-gen-noise $IN_WAV 30 44100
+audiowmark_add $IN_WAV $OUT_WAV $TEST_MSG
+audiowmark_cmp --expect-matches 1 $OUT_WAV $TEST_MSG
+# cut 1 second 300 samples
+audiowmark cut-start $OUT_WAV $CUT_WAV 44300
+audiowmark_cmp --expect-matches 1 $CUT_WAV $TEST_MSG
+
+rm $IN_WAV $OUT_WAV $CUT_WAV
+exit 0

tests/detect-speed-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=detect-speed-test.wav
+OUT_WAV=detect-speed-test-out.wav
+OUTS_WAV=detect-speed-test-out-spd.wav
+
+audiowmark test-gen-noise detect-speed-test.wav 30 44100
+for SPEED in 0.9764 1.0 1.01
+do
+  audiowmark_add $IN_WAV $OUT_WAV $TEST_MSG
+  audiowmark test-change-speed $OUT_WAV $OUTS_WAV $SPEED
+  audiowmark_cmp $OUTS_WAV $TEST_MSG --detect-speed --test-speed $SPEED
+  audiowmark_cmp $OUTS_WAV $TEST_MSG --detect-speed-patient --test-speed $SPEED
+done
+
+rm $IN_WAV $OUT_WAV $OUTS_WAV
+exit 0

tests/key-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=key-test.wav
+KEY1=key-test-1.key
+KEY2=key-test-2.key
+OUT1_WAV=key-test-out1.wav
+OUT2_WAV=key-test-out2.wav
+
+TEST_MSG2=0123456789abcdef0123456789abcdef
+
+audiowmark test-gen-noise $IN_WAV 30 44100
+audiowmark gen-key $KEY1
+audiowmark gen-key $KEY2
+
+audiowmark_add --key $KEY1 $IN_WAV $OUT1_WAV $TEST_MSG
+audiowmark_add --key $KEY2 $IN_WAV $OUT2_WAV $TEST_MSG2
+
+# shouldn't be able to detect watermark without correct key
+
+audiowmark_cmp --key $KEY1 --expect-matches 1 $OUT1_WAV $TEST_MSG
+audiowmark_cmp --key $KEY2 --expect-matches 0 $OUT1_WAV $TEST_MSG
+audiowmark_cmp --expect-matches 0 $OUT1_WAV $TEST_MSG
+
+audiowmark_cmp --key $KEY2 --expect-matches 1 $OUT2_WAV $TEST_MSG2
+audiowmark_cmp --key $KEY1 --expect-matches 0 $OUT2_WAV $TEST_MSG2
+audiowmark_cmp --expect-matches 0 $OUT2_WAV $TEST_MSG2
+
+rm $OUT1_WAV $OUT2_WAV
+
+# double watermark with two different keys
+audiowmark_add $IN_WAV $OUT1_WAV $TEST_MSG
+audiowmark_add --test-key 42 $OUT1_WAV $OUT2_WAV $TEST_MSG2
+
+audiowmark_cmp --expect-matches 1 $OUT2_WAV $TEST_MSG
+audiowmark_cmp --test-key 42 --expect-matches 1 $OUT2_WAV $TEST_MSG2
+
+rm $IN_WAV $KEY1 $KEY2 $OUT1_WAV $OUT2_WAV
+exit 0

tests/pipe-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=pipe-test.wav
+OUT_WAV=pipe-test-out.wav
+
+audiowmark test-gen-noise $IN_WAV 200 44100
+cat $IN_WAV | audiowmark_add - - $TEST_MSG > $OUT_WAV || die "watermark from pipe failed"
+audiowmark_cmp --expect-matches 5 $OUT_WAV $TEST_MSG
+cat $OUT_WAV | audiowmark_cmp --expect-matches 5 - $TEST_MSG || die "watermark detection from pipe failed"
+
+rm $IN_WAV $OUT_WAV
+exit 0

tests/sample-rate-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=sample-rate-test.wav
+OUT_WAV=sample-rate-test-out.wav
+OUT_48000_WAV=sample-rate-test-out-48000.wav
+
+audiowmark test-gen-noise $IN_WAV 200 32000
+audiowmark_add $IN_WAV $OUT_WAV $TEST_MSG
+audiowmark_cmp --expect-matches 5 $OUT_WAV $TEST_MSG
+audiowmark test-resample $OUT_WAV $OUT_48000_WAV 48000
+audiowmark_cmp --expect-matches 5 $OUT_48000_WAV $TEST_MSG
+
+rm $IN_WAV $OUT_WAV $OUT_48000_WAV
+exit 0

tests/short-payload-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=short-playload-test.wav
+OUT_WAV=short-playload-test-out.wav
+
+audiowmark test-gen-noise $IN_WAV 200 44100
+audiowmark_add --short 12 $IN_WAV $OUT_WAV abc
+audiowmark_cmp --short 12 $OUT_WAV abc
+audiowmark_add --short 16 $IN_WAV $OUT_WAV abcd
+audiowmark_cmp --short 16 $OUT_WAV abcd
+audiowmark_add --short 20 $IN_WAV $OUT_WAV abcde
+audiowmark_cmp --short 20 $OUT_WAV abcde
+
+rm $IN_WAV $OUT_WAV
+exit 0

tests/sync-test.sh

+#!/bin/bash
+
+source test-common.sh
+
+IN_WAV=sync-test.wav
+OUT_WAV=sync-test-out.wav
+CUT_WAV=sync-test-cut.wav
+
+audiowmark test-gen-noise $IN_WAV 200 44100
+audiowmark_add $IN_WAV $OUT_WAV $TEST_MSG
+# cut 20 seconds and 300 samples
+audiowmark cut-start $OUT_WAV $CUT_WAV 882300
+audiowmark_cmp --expect-matches 3 $CUT_WAV $TEST_MSG
+
+rm $IN_WAV $OUT_WAV $CUT_WAV
+exit 0

tests/test-common.sh.in

+# program locations
+
+AUDIOWMARK=@top_builddir@/src/audiowmark
+TEST_MSG=f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0
+
+# common shell functions
+
+die()
+{
+  echo >&2 "$0: $@"
+  exit 1
+}
+
+audiowmark()
+{
+  if [ "x$Q" == "x1" ] && [ -z "$V" ]; then
+    AUDIOWMARK_Q="-q"
+  else
+    echo >&2 ==== audiowmark "$@" ====
+  fi
+  $AUDIOWMARK --strict "$@" || die "failed to run audiowmark $@"
+}
+
+audiowmark_add()
+{
+  if [ "x$Q" == "x1" ] && [ -z "$V" ]; then
+    AUDIOWMARK_Q="-q"
+  else
+    echo >&2 ==== audiowmark add "$@" ====
+  fi
+  $AUDIOWMARK $AUDIOWMARK_Q --strict add "$@" || die "failed to watermark $@"
+}
+
+audiowmark_cmp()
+{
+  if [ "x$Q" == "x1" ] && [ -z "$V" ]; then
+    AUDIOWMARK_OUT="/dev/null"
+  else
+    AUDIOWMARK_OUT="/dev/stdout"
+    echo >&2 ==== audiowmark cmp "$@" ====
+  fi
+  $AUDIOWMARK --strict cmp "$@" > $AUDIOWMARK_OUT || die "failed to detect watermark $@"
+}