These are chat archives for PerfDotNet/BenchmarkDotNet
I see BDN uses
Benchmark attribute on methods to benchmark, and that is all! Doesn't that mean some reflection is going behind the scenes? I would assume you're using expression trees behind the scenes to compile the
Benchmark methods during runtime to get acceptable numbers. Fair enough, but compiled expressions are always wee bit slower than directly running methods (I would assume compiling expression trees at runtime is not as optimized as the IL generated by compile time compiler in release mode).
Whatever it is (refelcttion/expression), my question is, if your benchmarking approach is going to run my test methods wee bit slower than it can actually run, doesn't that mean I dont get absolute/raw value (by value I mean time taken to complete the run)? Or is BDN always meant to give relative values (given the said performance bottleneck is true for all the
Benchmark attributed methods)?
How do you measure elapsed time? Using
Stopwatch class one can get time elapsed, but there is also
Process.GetCurrentProcess().TotalProcessorTime which gives the CPU time. Which method does BDN prefer, or we can choose?
I have often found that when running a series of benchmark code I see different results based on the order of execution even when I run warmups (I always ran only one or two warmups instead of a million times, so that could be the reason). That is if I run Test1 and Test2 in order, often Test2 is faster, and if I reverse the order, Test1 is faster. What does BDN do to minimize this effect? Run hundreds of thousands of warmups? Or you randomize the testing order?
@AndreyAkinshin thanks for response, good to know.
Can BDN produce charts? Excel or not. Or any kind of visualization aid?
RPlotExporter.
Regarding point 1, I am just wondering what are the implications of BDN's methodology running at reduced speed (that is if I assume compiled function at runtime is never as performant as compile time compiled function, which is almost always the case). Wouldn't it be better to expose a method that accepts an
Action as the test method? Something like:
public double Benchmark(Action action) { for............. action(); } Benchmark(() => myTestMethod());
This way I assume, the an additional layer of delegate redirection has lesser effect on the quality of ouput than BDN's current approach?
And looks like it supports .NET Core as well, I am sold! Thanks guys!
Yep. Only RC1/RC2 for now, but we are planning to include RTM support in 0.9.8.
For plots, just got to install R, that's it?
And use a config which include RPlot export like the following:
public class Config : ManualConfig { public Config() { Add(CsvMeasurementsExporter.Default); Add(RPlotExporter.Default); } }
RPlotExporterwill generate a R script and automatically run it. But you can also add some additional logic and rerun script manually.