My most recent post has a few comments, but because I’ve been on holiday I didn’t reply to them in a timely manner. Sorry about that folks, I did read all of them.
And time has moved on so if I reply to those comments in the comments section, I’m guessing the original commenters won’t read the response. Having said that, they might not read this post either, but I think that is less likely.
Therefore this is a post to respond to the commenters, and to keep up with the IronMan schedule even though I’m not sure I’m being measured.
First of all, Roman, thanks for the comment. The figures you provide are interesting and yours is a second complaint about opaque error messages. That would bother me as well – I get enough of them from emacs macros and I have at least got tools (well, a tool – macroexpand) to debug those.
@Max – Hi. I’m not sure how you got the impression that I was interested in object creation time. We have primarily been talking about Moose start-up overhead here. I think I’m measuring the right thing.
@Anonymous – when you say "Run each benchmark item for 5 seconds or more." I think you missed this bit of the post:
Benchmark: timing 200 iterations of 1just-moose, 2all-includes, 3nocreate, 4create...
1just-moose: 100.196 wallclock secs ( 0.05 usr + 0.25 sys = 0.30 CPU) @ 673.40/s (n=200)
2all-includes: 279.252 wallclock secs ( 0.08 usr + 0.34 sys = 0.42 CPU) @ 475.06/s (n=200)
3nocreate: 318.256 wallclock secs ( 0.06 usr + 0.28 sys = 0.34 CPU) @ 581.40/s (n=200)
4create: 320.979 wallclock secs ( 0.06 usr + 0.27 sys = 0.33 CPU) @ 611.62/s (n=200)
Not one of my benchmarks runs for less than a minute and a half. Most of them are around the five minute mark. And I have deadlines!
So where is this message coming from? I had a look in Benchmark.pm.
print " (warning: too few iterations for a reliable count)\n"
if $n < $Min_Count # 4
|| ($t->real < 1 && $n < 1000)
|| $t->cpu_a < $Min_CPU; # 0.4
My iterations is 200 so $Min_Count is not the problem. $t->real is at least 100 so that leaves $Min_CPU. Hmmm… yeah, I would have had to double the number of iterations to be sure. Look at 4create – 321 wallclock seconds of which 0.3 is CPU. No worries, I’m good – the warnings were spurious as I originally thought.
With Benchmark.pm, instead of using a run count, why not use time limit? Pass a negative number and it will run for at least abs($number) seconds.
No more warnings, ever. Not even if your next computer is a super multi Ghz beast.
Hi Pedro,
I did that initially, but in this case I would have had to wait maybe 30 – 35 minutes or more to get a result and 200 iterations provided enough accuracy.
That’s a good suggestion for something with a better CPU to IO ratio though, thanks.
Hi Jared,
I think measuring Moose prior using it is pretty reasonable approach. One note though, Moose is doing quite a lot of heavy lifting that I would otherwise need to implement, so I like to think that startup time is paid off by that.
Per error messages, they seem to be improving over time, although sometimes you still get a message from somewhere inside Moose.
Hi Roman,
Yes, measuring is a start, but like with all these things you won’t know whether it was a good time investment or not until after you have invested a significant amount of time (kind of like learning a new language).
And as you say, if Moose implements a lot of stuff you would otherwise have to implement yourself, why not use it.
[...] Spurious Benchmark.pm Warnings [...]