Sizzle is 2x Faster than MooTools Selector Engine? Say Wha?
David Walsh posted yesterday a bit of code that will let you use Sizzle in MooTools if you want to. This brought a comment from John Resig (author of jQuery and Sizzle):
John Resig:
Umm… your performance numbers are horribly out of date (over 2 months, at this point). In fact, Sizzle is now over two times faster than MooTools (as of jQuery 1.3). In fact, it looks like Sizzle is going to become another 15-25% faster in the release corresponding with jQuery 1.3.2.
There’s a full thread discussing this over on Hacker News, where I address this point and more (concerning the integration).
David Walsh:
Thank you for bringing the speed improvements to my attention. As I’m sure you noted, I’m simply linking to the MooTools blog’s Slickspeed. Do you have an alternate Slickspeed link available?
I mean no offense with this post — in fact, I showed users how to make the choice themselves.
John Resig:
@david: No harm felt! Just wanted to clear up any misconceptions. A full breakdown of the methodology, the tests used, and the resulting data can be found in the jQuery 1.3 Performance notes.
Now, anyone who read Valerio’s in-depth and thoughtful post on why MooTools doesn’t use Sizzle will recall this bit:
Third, as some of you might already know, MooTools post-processes every node resulting from any query. This tends to make things slower. Sizzle however is a pure engine, therefore makes no post-processing at all. This results in a very unfair comparison.
In an effort to test the true efficiency of our CSS selectors engines, I have made a modification to SlickSpeed, so that it runs every test only once, and a couple of modifications to the MooTools code. This special testing version of MooTools no longer “extends” the resulting nodes (I did that by simply adding a method that passes an optional parameter to the default function getElements, as I cannot pass parameters using SlickSpeed), and it uses querySelectorAll where available, just like Sizzle does. This way we can have a true comparison between engines, instead of frameworks and engines.
So John says David’s performance numbers are “horribly out of date” – but are they? Well, see for yourself:
This is the same test Valerio put together 2 months ago except with the latest version of Sizzle (last modified 3 days ago). As you can see, it’s not faster than MooTools by any significant margin. Some selectors are faster in Sizzle than in MooTools and vice versa. When I run the tests, they repeatedly come out very close to each other. 2x faster? Indeed.
Update
I’ve posted an update: Sizzle and MooTools by the Numbers.
Follow @clientcide on twitter to get notified of new posts.
To follow me personally on twitter, follow @anutron.
February 12th, 2009 at 2:03 am
Thanks for shedding some light on the discussion & compiling an updated SlickSpeed.
Torrough as always!
February 12th, 2009 at 5:29 am
You may want to read up on the methodology used to arrive at those results.
1) It used a representatively complex web page (in this case, Yahoo.com). The current test page is not a good sample page (it’s just a long HTML 1-style page, H1s, p, and some divs).
2) It tests selectors that people actually use. Right now MooTools optimizes for the wrong selectors – it focuses on making things like :nth-child faster but without improving the performance of basic selectors. Sizzle is optimized to work best on the selectors that developers actually use – and the test represents that (see the methodology link for more information on those selectors).
3) It runs the tests a statistically significant amount of times. The test that Valerio constructed runs the test only once (!). Especially in browsers like Internet Explorer you need to run tests many times in order to get an accurate picture.
4) “When I run the tests, they repeatedly come out very close to each other. ” When testing the performance of selector engines you can’t just look at a single browser – you have to take a general look (with a heavy focus on Internet Explorer 6 and 7). Selector engines should be optimized for the browsers that people use.
None of these points are represented in the results or test that you show.
February 12th, 2009 at 5:59 am
@ejohn:
The current test page is a very big page with many nested elements. Different types of tags are not required. The (very) high number of nodes, and the (very) nested nature of those nodes are what makes Slickspeed’s testpage a very good test page.
We do not “optimize” our selector engine based on which selector you are currently passing. Every selector gets treated the same way by MooTools, to make the code simple, and to make it (much) less error prone. MooTools makes no difference between “.class” and “whatever.long.selector:you(might[use])”. This means we dont use getElementsByClassName. It is a design decision, and based on the tests I conducted, it works (really) well, for both simple and complex selectors.
This is because Slickspeed’s goal is to test the selector engine performance, not how well the selector’s engine caching system works. If your tests include any kind of caching, you’re not testing the engine.
I couldnt agree more. For instance, run this test page in Firefox 2. You might want to revisit some selectors, as they’re taking close to 3 seconds (!) to run. Something you obviously cannot see with a stupid test page with no significant node nesting like Yahoo’s.
February 12th, 2009 at 6:23 am
So I see the results for engine vs. engine and read the results on jQuery’s website for framework vs. framework. But is there a “current” (without the caching of Sizzle) slickspeed test for jQuery 1.3 vs Mootools 1.2.1? It seems that test is what we, as endusers, would be more concerned with since we are implementing the full framework core in our projects.
Not that the results would really sway me away from mootools for CSS selectors alone. Just curious what the results from actual implementation. if Mootools as a framework post-processes every node which makes the framework slower, why are we doing that? (total lack of understanding)
February 12th, 2009 at 6:31 am
@Valerio: You outline why you feel that the W3C selectors page is a good test page – but not why you think it’s representative of the page’s that developers create. I think it’s very important to develop tests that are comparable to the pages that developers are actually creating. Wrong test data gives you the wrong results – which leads you to create code that doesn’t optimize for the correct cases.
“We do not “optimize” our selector engine based on which selector you are currently passing.”
That seems like a huge mistake to me. Making no attempts to actually optimize the selectors that are being passed in will lead to slower selectors. That’s kind of a strange justification for having slow selectors.
“This is because Slickspeed’s goal is to test the selector engine performance, not how well the selector’s engine caching system works. If your tests include any kind of caching, you’re not testing the engine.”
As I’ve outlined before – Sizzle does not have any query caching in it. That was removed almost 2 months ago. Running a test once and hoping for some result that is meaningful is futile. Running a test once is not statistically significant in any way – it’s a broken test suite.
“For instance, run this test page in Firefox 2. You might want to revisit some selectors, as they’re taking close to 3 seconds (!) to run.”
Uhhh – no? I ran the test suite linked to in this post and got 552ms for Sizzle, 468ms for MooTools (in Firefox 2). I ran the test suite that I created and I’m seeing jQuery 1.3 coming in at 232ms, MooTools in at 305ms (in Firefox 2). There are absolutely no “3 second” results – you may want to check your computer.
February 12th, 2009 at 6:32 am
@Ryan: “without the caching of Sizzle” Sizzle does not cache queries – that was removed 2 months ago and was not integrated into jQuery.
February 12th, 2009 at 7:30 am
@john I thought that was the way it was but with “caching” being thrown around I felt it needed to be cleared up.
I just put together a slickspeed page comparing the frameworks for Jquery 1.3.1 to Mootools 1.2.1 which, to me, shows what we should be testing the speed on since this is how I implement them when I have a project. (correct me if I’m wrong)
In FF 3.0.6 the two seem fairly equal.
IE7 shows faster times for jQuery
Safari 3.2.1 also shows faster times for jQuery
none of the results I would deem slow.
http://sandbox.intereactive.org/slickspeed
February 12th, 2009 at 7:44 am
@Ryan: Thanks for pulling that together, it’s good to have for comparison. Just to clarify for those who are following along, there are 5 test suites being discussed:
1) There is the original MooTools Slickspeed test suite (hosted on their web site) which is outdated at this point (old versions of MooTools and jQuery are used).
2) There is the test suite that Valerio posted back in December. This was a modified version of #1 that only ran tests once AND stripped out the extra work that MooTools had to do to extend elements. The end result was the core Sizzle engine (which had caching, at the time) vs. the core MooTools engine. This introduced the problem of running the tests a statistically insignificant number of times.
3) The test suite posted in this blog post. This is an updated version of #2 that uses a more-current version of Sizzle (with no caching).
4) The test suite posted by Ryan (above) which is an updated version of #1 that is comparing jQuery and MooTools.
All the suites in 1-4 use the same selectors and the same test page.
5) The test suite posted and used in the jQuery 1.3 release notes. This builds on #1 but uses a different test page (Yahoo.com – which we feel is more representative of the pages that developers build) and different selectors (we collected a number of selectors that developers actually use and tested against that). Please see the above notes on our methodology for more information.
People seem to be getting caught up in the 2x figure. Obvious Sizzle/jQuery isn’t 2x faster in a single engine (in fast browsers like Safari or Firefox there is a tiny difference – at best) – but where the difference becomes prominent is in engines like Internet Explorer. As we outlined in our release notes the cumulative results from all of the major browsers (on the test suite that we constructed – which is a representative test page running representative selectors) ends up being – in total – over 2x faster than the results collected from MooTools.
February 12th, 2009 at 7:50 am
@Ryan: Thanks for posting that. Here are the results I’m seeing using your version of SlickSpeed on the browsers I have installed:
FF 3.0.0.6 fairly equal as you mentioned jQuery 328 MooTools: 335
FF 3.1b2 jQuery: 96 MooTools: 243
FF 3.1b3pre jQuery: 86 MooTools: 217
IE 7: jQuery 222 MooTools: 487
Safari 3.2.1: jQuery: 133 MooTools: 203
Again, TO CLARIFY: The above results were derived using Ryan’s version of SlickSpeed:
http://sandbox.intereactive.org/slickspeed
February 12th, 2009 at 8:16 am
I feel like you are trying to win a war with a difference of 50 microseconds
February 12th, 2009 at 8:19 am
@Chris: We’re not trying to win anything. We’re clarifying some statements about selector performance. That’s it.
I didn’t know we were at war in the first place.
February 12th, 2009 at 8:24 am
It’s a lot of talk over single-digit milliseconds. Trivial, really.
February 12th, 2009 at 8:33 am
@David: I guess it wasn’t so trivial since it prompted this blog post.
February 12th, 2009 at 8:37 am
faster is faster and when we are talking about a web and web applications that will continue to rely heavily on JS for UI animations, we have to be concerned about speed.
Just running a selector by itself… sure milliseconds have little effect on that. But when we string them together in animations and series on animations the result can be jerky and poor quality UI.
the difference in short and long duration tweens (in Mootools) is in milliseconds and can really make a lot of difference in the UI. So yes… I think ms are important in regards to frameworks and selectors.
February 12th, 2009 at 8:51 am
Hi all, just woke up to the comment storm I guess I expected.
First, I want to point out that we are not at war as both John and others acknowledge.
But I posted this quote because I think it’s unfair for John to show up on David’s blog where he’s showing people how to use Sizzle and say that MooTools’ selector engine is 2x slower than Sizzle, which is, to use his words, “horribly inaccurate.” MooTools is slower at some selectors, yes, but so is Sizzle. In my own post about sizzle, I make one point (of several) that I think that selector engines have gotten fast enough. When I profile my own pages it’s not the selectors that are slowing it down.
jQuery has a design pattern that uses a lot of css selectors – that’s where the “Query” comes from if I’m not mistaken. This is a fine design pattern! But MooTools doesn’t use this pattern and our own design patterns cache selector results into classes which maintain state. I don’t find myself ever running the same selector twice. Further, I personally write my html at the same time that I’m developing the JavaScript in any UI, which means that I can tailor the html to make for easy selection. I never use nth child type selectors.
So again, my basic opinion is that a faster selector engine at this point is not the most important thing to me.
Valerio’s post was only saying that there was no real value in using Sizzle in MooTools (from his perspective). Not that jQuery was better or worse or faster or slower than MooTools. jQuery is faster at selectors because MooTools has a design pattern that extends Elements. In Rey’s comment above he compares MooTools to jQuery and jQuery is faster than MooTools, yes. But Sizzle isn’t faster than MooTools’ selector engine, which is our only point.
As far as our test being inaccurate because it only runs once, well, it was altered to only run once because Sizzle was caching when we made it. Now that it isn’t, we can use the regular slickspeed again. But there’s nothing stopping you from running the test I posted above more than once, so John’s comment that it runs a statistically insignificant number of times is only true if you decide to only run it once.
All of this is, to me, just not important. As a user of MooTools I feel like it’s selector engine is very fast. Further iteration at this point might shave a few milliseconds off my app’s load time, but not a noticeable amount. For John to show up on a blog post showing people how to use his product and make what I feel is an inaccurate statement (that is comparing MooTools to jQuery but not Sizzle to MooTools’ selector engine) is inappropriate. My point in posting this here was just to refute that claim.
For what it’s worth, I don’t think there’s any hard feelings here between anyone. I am not angry at anybody and I hope that’s true of everyone here. This kind of discussion is what ultimately compels us to reevaluate both what we do and what we say. It’s made me wonder if MooTools needs to add a case where it uses getElementsByClassName as that’s a common pattern and our results could be a little bit faster.
But ultimately I feel like the speed of the various js frameworks now are governed by DOM performance and JS engines more than anything. CSS selectors are, in my opinion, fast enough for production use. Making them faster is at this point a decreasing return on investment.
February 12th, 2009 at 9:03 am
@aaron
“Making them faster is at this point a decreasing return on investment.”
And you believe this holds true even on IE7 (and IE8) and it’s seemingly poorer performance based on Slickspeed? Seems like lots of room for improvement on a browser that still holds majority of the share. Even on a base query of the div element in IE7 I get a stand 16ms slower. Again.. not a big deal as it stands on its own… but to say that it doesn’t need improvement against ~0ms seems strange, especially on something common like that.
I’m by no means arguing against what you are saying. Yes.. it is def fast as it sits. But to say it doesn’t need improvement is some areas of selectors seems a bit off.
February 12th, 2009 at 9:22 am
I’m looking at the performance historically, I suppose. Selectors used to be terrible. Valerio authored Slickspeed expressly to see which frameworks were performing better than our work so that we could learn from them. Go run Slickspeed against early versions of all the frameworks – sloooowwww. When Slickspeed came out you saw this huge, almost exponential speed up in all the selector engines. Our frameworks actually don’t have any other benchmarking method that I know of – selectors are the only thing. There’s no benchmark for CPU usage during animation or whatever.
But now the improvements are small. The initial speed-ups were orders of magnitude. We’re now squabbling over milliseconds. In a real world app, we’re talking about something like, maybe 50ms or so difference in performance for all the selectors running on a page. That’s what I mean by it not being a priority. Yes, maybe we could make MooTools perform 50ms faster, or a 100ms, but at this point, it’s not where we feel the biggest return on our investment lies.
February 12th, 2009 at 9:22 am
The problem with the Slickspeed all use for their benchmarking is that it only run the query once, which on a fast computer almost always gives a 0 ms result! Thats because timing in JavaScript is limited to milliseconds. For reliable results each test should run multiple times that takes many milliseconds, then divide by the iteration count.
When testing our selector we benchmark the correct way. The difference in what many think is a fast selector becomes very obvious.
February 12th, 2009 at 10:55 am
So I’m just curious, what are the use cases for super-lightening-fast selectors in Javascript? It seems like people are really excited about them, but I use non-ID selectors maybe once or twice every thousand lines of JS (and I work to get rid of them).
If I need to change some style to lots of elements, then I set a class on the parent and use CSS to change the elements. If I need to add an event to a bunch of elements, I add it to their parent and use delegation.
Am I missing something? Maybe I’ve been Doing It Wrong™ this whole time.
February 12th, 2009 at 10:57 am
I find it really boring you are still there at speed.
Reread the W3C specs and please tell me which framework give “document ordered”, result set and I will immediately change to that framework. I don’t understand why you always skip this argument.
Does skipping this make you faster ? NO ! Accurate/Consistent ? NO !
Speed in my selector engine is boosted by Mutation Events and it’s several times faster where available. Even with one iteration NWMatcher is faster, knowing that it always return “document ordered” result sets and odes many more checks.
I would very like see comparisons on compliance, features and browser compatibility and possibly a way of comparing my “match()” method too…
@Ryan, I agree with you…the milliseconds spared here will be extremely important when people will start using proper event delegation..where queries will be executed continuously on particular events like “mouseover” and “mouseout”. But for that they will also need a “match()” method, “select()” or “$$()” are a lot less useful then.
After events, selectors are the next important tools we have to script the Web, and yes there is still space for improvements both in speed and the way the traversal is done.
Diego
February 12th, 2009 at 1:34 pm
@Diego, base2 sorts nodes into document order. The downside is that comma separated selectors are slower for some platforms (MSIE). But who does that anyway?
I think that most selector engines are now fast enough. The reason we are so hung up on selector speed is because we execute lots of queries during one event (DOMContentLoaded). For most libraries all this initialisation occurs within one execution context. Complex initialisation can lead to the screen freezing for a fraction of a second. The next trick is to rely less on the DOMContentLoaded event and offer live event bindings.
February 12th, 2009 at 2:43 pm
Dean,
clever Dean, I know you couldn’t miss such a feature. It is nice to know I am not alone thinking the “document order” is important. For me it has shown to be necessary in extracting text from different elements in a document, others have found problems in serializing form values or building pages TOC.
I am already playing with Base2 (for events) but I was not up to date on that bit of information about your fast selector engine (which is part of my benchmarks now)…
Yes I was talking about live event binding (delegation) but a “match()” method is more suitable for that and is way faster than collecting all elements matching and then loop through them searching the one element. Further, to do a selection of element require all elements already being loaded, since it is impossible to do a $$() before then.
The point here is understanding the road is still long and open to new things…
Diego
February 12th, 2009 at 3:56 pm
It would be awesome if you guys spent all the time you’re wasting arguing with each other doing something else like say, developing your libraries. Then we, the end users, could choose the one we wanted to use on a given project and use it. And you know, we might even use different libraries depending on the application we’re developing. Yep, crazy I know but sometimes we just want different things. So thanks to you all for building these libraries that save us so much time and help us to create ace new apps, they really do provide such a huge benefit to us all, but seriously, nobody is the best. So shut up.
February 12th, 2009 at 4:17 pm
@Brian, your post got a chuckle from me. Though this post is all about Slickspeed, jQuery, and MooTools, I agree, this is a non-issue now. And, for what it’s worth, we – the authors of jQuery, MooTools (and YUI, Prototype, Dojo and others) are busy at work making our frameworks better.
This post (and my followup) are as much about the fact that a) selectors are not really where we need to focus and b) why Sizzle isn’t right for MooTools.
February 12th, 2009 at 9:59 pm
k’mon fellas. Can we please just hold hands, sing some strawberry alarm clock and focus on what we all know is actually important? You know, super cool slickified animations and effects and stuff!!!!!
February 13th, 2009 at 4:06 pm
@brian
These discussions produce more from each different framework then all just glad-handing other users of the same frameworks at meetups.
the day the dev teams of the languages/frameworks I code with quit defending their product and/or quit caring enough to enter into debate with people of other thoughts.. is the day I quit using it. I care for the code I write and so should they.
your comment is rude and moreover is laced with stupidity and lack of understanding.
February 13th, 2009 at 4:33 pm
@brian, @ryan, let’s settle down here. No one needs to “shut up” and let’s lay off the “stupidity” remarks.
February 15th, 2009 at 11:21 pm
No problem Aaron, my comments certainly weren’t meant to hold any venom. As I said, I totally appreciate all the work done on the respective libraries and clearly, without vigorous and passionate discussion on topics related to creating such libraries, we’d all be the worse off for it. Ryan is clearly one of those passionate developers, as are others who’ve commented in this thread.
My issues was more about the merits of attempting to declare who has the better library, methodology, selector engine, hair cut, etc. These are the ways we all get sidetracked and lost in defense of our own work which is of course a natural reaction, none of us want to think that we’re creating anything but the best we can. As we’ve mentioned though, there are courses and there are horses and each should ride their own.
Of course it doesn’t help when lead developers of some projects going repeatedly blowing their own trumpets all over town…
February 16th, 2009 at 8:32 am
As I’ve said repeatedly, my opinion of jQuery, Prototype, Dojo, YUI and other frameworks is very high. I don’t see them as being better or worse than MooTools or each other. These things are all tools. Some tools are right for certain jobs, some tools are right for certain people. I would argue that none of these frameworks are better or worse than the others on their own merits, just different. The fact that they are different is the only reason that they are as good as they are. We all learn from each other’s progress. Selector engines is just one example where this has happened, which is one of the reasons MooTools doesn’t use Sizzle. That’s not saying anything bad about Sizzle, I think.