Comparing Frameworks with Inheritance Benchmarking
Over on Ajaxian yesterday was a post about benchmarking inheritance methods that was pretty interesting (here’s the post that Ajaxian was covering). In it, various methods for extending objects in JavaScript are compared:
- Ad hoc inheritance – This is a common(?) homebrew technique for allowing prototypes to leverage the code in objects further up the prototype-food chain. Methods are overridden by keeping a reference to the parent method in a separate property , which can then be invoked as needed. It’s fast but not very pretty, and it’s arguable whether or not this qualifies as real “OO” inheritance.
- Prototype-style inheritance – Prototype uses a strategy inspired by Alex Arenell’s Inheritance library. Subclass methods declare a “$super” argument that is set up by Prototype to reference the superclass’ method.
- Base2-style inheritance – Dean Edwards’ library. Subclass methods invoke “this.base()” to call their superclass’ implementation.
- John Resig inheritance – JR, of jquery fame, experimented with a Base2 variant which he published on his blog. It’s a bit simpler than Base2, but seemed worth testing.
The results here are a little questionable. As one person points out in the comments of the post:
There is something wrong with JSLitmus and last executed test which for some reason is always slower than others (I simply moved a couple of tests from top to end and vice-versa, results are systematically slower if placed at the bottom).
I think to get completely valid results we’ll need a Slickspeed-ish implementation but still, these benchmarks are a start. As I pointed out in the original post (to which Ajaxian linked), it bothers me that performance is being benchmarked against code Resig played with but isn’t actually in use in production. If jQuery were to use Resig’s content and encounter use cases that dictated that they add functionality they would very likely have different results than this theoretical example. When you run benchmarks against the other methodologies listed here you are testing more than just inheritance (assuming that most of the initialization mythologies apply any additional logic beyond what’s necessary for inheritance to support additional extensibility – for instance, MooTools’ mutator logic).
If you really wanted to test only inheritance then each framework would need to be cleansed of any code that facilitates additional code that does anything else. Assuming that no one is going to do that, I’m inclined to take Resig’s code example as an academic pursuit, and not something that’s designed for production and tested against the use cases that such could would endure.
Someone did take the time to put MooTools into the test suite (it was excluded because Prototype was included and the two don’t play well – another example of where we need a Slickspeed-ish solution). Given that MooTools is so similar to Dean Edwards’ Base2 (and is greatly inspired by it), it’s no surprise that we perform basically as well.
As you can see, Moo performs quite well.
These results should be taken with a grain of salt of course. They are interesting but they don’t have a lot of meaning to me (perhaps because we perform well). It’s also worth nothing that unless you’re creating a few thousand instances of classes you probably won’t ever notice the actual performance differences in these methodologies.
Update
Per the comment below, here’s a link to a terrific followup that digs deeper into this topic. Great work.
Follow @clientcide on twitter to get notified of new posts.
To follow me personally on twitter, follow @anutron.
February 19th, 2009 at 3:47 pm
I think you should take a look at the work of Andrea Giammarchi about this same topic. I think it goes deeper than the one you have linked.
February 19th, 2009 at 4:38 pm
@Al
The link you sent is awsome, people should have a look.
February 19th, 2009 at 4:44 pm
That is a great link @AI. I’ve added it to the body of my post.
February 19th, 2009 at 5:48 pm
Aaron, I’m glad you found the inheritance article interesting. I’m pleasantly surprised at the amount of feedback it’s generated.
I’m a little concerned by the tone of your post however. I fail to see how the results of this exercise are questionable. Yes, Resig’s code is best treated as experimental, but I make no bones about that. This was never intended, nor presented, as a head-to-head comparison of libraries. It was simply an examination of performance for a few, notable inheritance implementations. Resig provided a useful and informative example of what a streamlined implementation by someone well-versed in writing nuanced code might do. Had I drawn conclusions on how awesome (or shitty) MooTools or other libraries were based on that then that would be questionable. But the raw numbers I preset, and the one conclusion I come to (“Prototype is slow”) seem pretty irrefutable.
What it is more concerning to me is the assertion that JSLitmus is unreliable. As the author of JSLitmus I like to think I’ve created a tool that people will find useful and valuable. I am “appropriately” concerned by the report from WebReflection that he is having issues with it, but I have so far been unable to reproduce the problem he describes. Until I have more information from him, or corroboration from at least one other person experiencing the same issue, I’d prefer not to sound the “JSLitmus sucks!” alarm. It would be really helpful to me if you could at least give the tool a try and share your first-hand experiences with your readers rather than fanning the flames. I’m hopeful that rather than implying it is unreliable, you might be a little impressed at the value it provides in it’s fairly small form-factor.
Finally, I agree that a SlickSpeed-like test for JavaScript performance would be useful. It’s why I created JSLitmus. I felt it was important for people to be able to create interesting tests, and even more important for them to be able to quickly and easily share those tests so they could, oh I dunno, verify the reports and conclusions came to. I’m pretty confident that JSLitmus is about as close to “reliable” as you can get with a pure javascript implementation for a testing tool, so it pains me to see people implying it can’t be trusted.
February 19th, 2009 at 6:11 pm
Hi Robert,
I don’t mean to dismiss your contribution to our community. Benchmarking suites are very useful in driving innovation and I’m glad to see you taking it on. That said, I didn’t spend a ton of time with your test environment and all I meant to highlight was that I, personally, don’t know how much to trust the results as I haven’t investigated the methodologies closely. For instance, the fact that you can’t run MooTools and Prototype together tells me that you aren’t putting these things in Iframes (as Slickspeed does) which means that they might in some ways poison each others’ results. That’s very concerning to me.
My issue with including Resig’s code in the example is that 1) if I remember correctly it’s also very similar to Dean’s work (which is not a bad thing) but that 2) Resig never really put it to use. It’s an unfair comparison to say that the inheritance system is faster or slower than any other when you are testing more than that. MooTools does other things when you instantiate a class including providing hooks for mutations and customizations. So you aren’t really testing inheritance methods here so much as implementations that enable inheritance as well as other things. JavaScript has more than one way to facilitate inheritance – as your tests illustrate – but you aren’t testing that. You’re testing the programming patterns in various frameworks that happen to make use of inheritance as part of that pattern. But not Resig’s code, which was mostly an academic pursuit.
Still, I find this topic immensely interesting and informative. No doubt the Prototype guys and others will learn from these findings and improve their speed which is great for their communities. I don’t mean to cast dispersions on your work or your findings, but in communicating my own thoughts, I stand my statement that they are questionable – as in, I have questions. Unfortunately, it’s not very fair that I don’t have time to find answers myself. My apologies if you found this unfounded or unfair. I hope you keep this kind of work up as it helps everyone in the long run, questions or not.