Article - Your benchmarks suck!

  • Metadata:
    • author: Steffen Mueller
    • title: Your benchmarks suck! Steffen Mueller [blogs.perl.org]
    • url: Weblink
  • A big problem with Benchmarking Software are ==outliers==.
  • Due to the mostly unpredictable nature of CPU caches and virtual memory, no two runs are the same. Usually, this gives normally distributed time measurements.
  • Sometimes however, big jumps in the execution time can be witnessed, but there are no such outliers in the other direction. That means that our distribution is skewed towards higher runtimes.
  • How to fix these outliers:
    • calculate the median and the median-absolute-deviation
    • remove every measurement that deviates more than x times the mad from the median
    • recalculate the median and the mad based on the truncated dataset
  • this is better than removing the n largest and smallest results, as that could not deal with outliers that are biased in one side