There’s been a lot of churn on selector engines in the last six months or so. The jQuery guys released a stand alone library (Sizzle) which was adopted by a lot of other frameworks (it’s now a part of the Dojo Foundation) including Prototype, Dojo, jQuery (obviously) and others.

There was a lot of hullabaloo when MooTools decided not to use it. One of the arguments that Valerio made in his post about that was this:

Let’s face it: every selector engine, every part of our libraries has benefited from the others. Where they diverge is not an indicator of which framework is superior to another. Rather, they are differences in philosophy. If everyone were to use the same, shared codebase, these awesome open source contributions and general advancements will stop, and users wouldn’t be able to choose the approach which works best for them. I don’t want to see that happening.

That notion – that if all the frameworks adopted the same selector engine that both innovation will suffer – was refuted generally by the advocates of everyone using Sizzle. John Resig left a long and thoughtful comment on my post about Sizzle that included this statements:

With so many minds at the table (all the major libraries) it’ll mean that this library will be in front of the majority of JavaScript users, in one way or another. It will be compelled to become faster.

…

Frankly, not using Sizzle will mean that MooTools will always be playing a game of catch-up. … Right now jQuery, Prototype, Dojo, and YUI are all looking at using the library – that only leaves one odd library out. I’m not attempting to put undue pressure on your team – it’s absolutely your decision – but you’ll definitely be in a position, if all the libraries use Sizzle, of constantly trying to catch-up to what is implemented in the de facto implementation.

I refuted this statement and so did members of the MooTools communities (and others). John’s argument that because MooTools is the odd one out that we’ll always be playing catch up seemed especially off the mark, because it implies that Sizzle will, by virtue of having so many people using it and contributing to it, always step ahead of MooTools, and that all MooTools would hope to do is catch up – that is, implement changes to match Sizzle’s progress rather than out pace it. In reality, that’s not how technology moves forward; competing parties take turns passing each other and drive the baseline forward.

Well, today it looks like it’s Sizzle’s turn to play catch up.

Introducing Sly

MooTools contributor Harald Kirschner today released his own selector engine, Sly. This is a stand alone library (much like Sizzle) that is only 3K gzipped (8K minified without gzip). This is on par with Sizzle (which claims that gzipped is 4K). It offers the same sort of flexibility and support for custom selectors plus all the CSS3 selectors. Form the announcement article on Harald’s site.

If you’re a code geek like me, you might want skip the details and go directly to the source, the speed tests or the specs.

Sly awesomeness:

• Pure and powerful JavaScript matching algorithm for fast and accurate queries
• Extra optimizations for frequently used selectors and latest browser features
• Works uniformly in DOM documents, fragments or XML documents
• Utility methods for matching and filtering of elements
• Standalone selector parser to produce JavaScript Object representations
• Customizable pseudo-classes, attribute operators and combinators
• Just 3 kB! (minified and gzipped, 8 kB without gzip)
• No dependencies on third-party JS libraries, but developers can override internal methods (like getAttribute) for seamless integration.
• Code follows the MooTools philosophy, respecting strict standards, throwing no warnings and using meaningful variable names

But what you probably care about most is the performance:

But enough babbling: I know you just want speed and validity results. The following results were measured using a list of frequently used selectors, searching on a copy of yahoo.com, running in slickspeed.

(See also the DOM fragment speed graph) If you are interested, run the various speed tests on your own system and post your results.

The adapted version of slickspeed also supports other test cases, like querying on a DOM fragment or an XML document. They make sure that Sly and other engines work the same in all environments.

Wait, But This Isn’t MooTools?

That’s right. This is a project Harald was working on and finished and released. This isn’t the official MooTools selector engine and won’t be. MooTools has it’s own new selector engine that is in development. I won’t go into why MooTools isn’t working on a different one (I’ll leave that to its authors to discuss when they are ready to), but the basic point is valid here: having more than one of these things is good competition and it moves the base line forward. Harald’s work has fed directly into the MooTools engine and, perhaps, will also influence Sizzle, who knows.

Why It Kinda Doesn’t Matter

I’ve been saying for a while now that the speed of selector engines is just not an issue any more. The speed of the current selector engines of jQuery, MooTools, and Sly look pretty different on a graph next to each other, but they are all blazing fast. Compared to what they were like a year ago, all of them are miles ahead now and at this point we’re quibbling over milliseconds. There are other things we need to optimize and selector engines are, at this point, plenty fast for nearly all tasks. That’s not to say that there isn’t always room for improvement, but at this point it’s more about features and flexibility than it is speed, and to a great extent all these engines are extensible, flexible, and powerful.

I’m happy to see people out there still cracking this problem open to look at it from fresh perspectives and I’m sure it’s gratifying to write something that performs better than the competition. Congrats to Harald for a fine job well done.

Over on Ajaxian yesterday was a post about benchmarking inheritance methods that was pretty interesting (here’s the post that Ajaxian was covering). In it, various methods for extending objects in JavaScript are compared:

• Ad hoc inheritance – This is a common(?) homebrew technique for allowing prototypes to leverage the code in objects further up the prototype-food chain. Methods are overridden by keeping a reference to the parent method in a separate property , which can then be invoked as needed. It’s fast but not very pretty, and it’s arguable whether or not this qualifies as real “OO” inheritance.
• Prototype-style inheritance – Prototype uses a strategy inspired by Alex Arenell’s Inheritance library. Subclass methods declare a “$super” argument that is set up by Prototype to reference the superclass’ method.
• Base2-style inheritance – Dean Edwards’ library. Subclass methods invoke “this.base()” to call their superclass’ implementation.
• John Resig inheritance – JR, of jquery fame, experimented with a Base2 variant which he published on his blog. It’s a bit simpler than Base2, but seemed worth testing.

Read the rest of this entry »

Dylan Schiemann sent me a nice email response to this post that clears up some details:

Hey Aaron,

We’re actually voting on something different, which is whether or not
the Dojo Foundation will accept Sizzle as a project (Dojo Toolkit,
cometD, DWR, Persevere, Lucid Desktop, and OpenRecord are the other Dojo
Foundation projects).

So why does this matter? Well, it means that Sizzle will have all of
its code contributed under CLA, so it can actually be used in Dojo. It
also means the process for becoming a contributor, and making it into
what you need it to become is much easier than if it was just Sizzle
Copyright John Resig.

We literally went from discussion to vote in a total of about 3 hours
yesterday, so this all happened rather quickly.

On the toolkit level, we will start with a branch and evaluate the
results. Assuming things work great, we’ll switch over asap. If not,
then we wouldn’t make an immediate switch, but we’d work with the people
contributing to Sizzle to get us to the point of sharing code on this.
Certainly getting more of the Ajax community working together on things
we’re all duplicating across the board anyways isn’t a bad thing (it’s
really why Dojo was started).

If you have any questions or comments, don’t hesitate to ask…

Regards,
- -Dylan

This is what Dojo is doing, but I’ll also note that John (Resig), who has some comments below which I’ve responded to, sent us an email before I made my post asking if we, the MooTools team, would be interested in adopting Sizzle. So while the Dojo news on Ajaxian is about them adopting the project the question still remains regarding MooTools potential use of Sizzle.

To be clear, in my earlier post I am not advocating one way or the other. I think there are benefits and drawbacks and I’m curious as to what others think about it. Garrick Cheung weighed in with his own blog post, saying:

Aaron brings up very good pros and cons. I think we should not follow the trend and switch to Sizzle but instead take in what we can and be inspired to improve upon it. We should already be doing that. This would require our awesome MooTools dev team to be up-to-date, but aren’t they continuously learning?

I should also note that both John in his comments below and Tommocchino have pointed out that Sizzle is very extensible. The mechanism – the interface – for extending Sizzle with custom selectors is different from the current MooTools methods, but the capability is the same. Here’s what Tom wrote in the MooTools user’s group, which has had a lively discussion:

…you should note that it is extremely extensible, and things like custom pseudo
selectors are no problem at all.

Resig Responds

Over on my previous post, John Resig wrote a lengthy response to my post. I’m going to quote it and respond to it here so I can insert my comments after his various points.

Just to reply to – and clarify – you ‘Con’ points:

CSS selectors are, by and large, all about as fast as the next now. There’s not much benefit to be had as far as speed is concerned.

While I may agree that selectors aren’t going to get much faster – the thing that is changing is the standardization of those selectors. This comes in two forms:

1. From standards bodies, like the W3C, with new methods (for example, querySelectorAll). If the libraries collaborate on a single engine it becomes fundamentally much easier for these new methods to come into being and propagate. It’s taking far too long for querySelectorAll to reach all of the current libraries – it should be happening much faster.
2. In custom selectors. There is no standardization on custom CSS selectors that libraries have. For example – there is the common custom selector [attr!=value]. What does this mean? Is it equivalent to :not([attr=value])? If that’s the case then it’ll match elements that don’t even have the attribute to begin with. Maybe it should be equivalent to [attr]:not([attr=value]). But, again, that’s not standardized anywhere. Having a unified engine allows us to standardize this in one place with a de facto implementation.

I agree with John on these points, I think. Consolidating our behavior would send a nice message to the browser vendors and the standards groups. As for some of the more complex selectors, on a personal level I don’t find myself using them that often. I think this is a product of some of the differences in MooTools and jQuery or maybe it’s my development style. I tend to prepare my DOM for easy selection without a lot of attribute selecotrs and then manage state in my classes. I’m not saying this is better than using them – just that I don’t tend to use them often. Consequently, on a personal level, my interested is just less engaged on the topics of exotic selectors and so I’m content to let others weigh in on their merits and how they should be structured. I guess what I’m saying here is that I agree with John’s first point and don’t have an opinion on the second.

Frankly, not using Sizzle will mean that MooTools will always be playing a game of catch-up. Right now jQuery, Prototype, Dojo, and YUI are all looking at using the library – that only leaves one odd library out. I’m not attempting to put undue pressure on your team – it’s absolutely your decision – but you’ll definitely be in a position, if all the libraries use Sizzle, of constantly trying to catch-up to what is implemented in the de facto implementation.

Now here I disagree. As I’ve stated previously, I think the speed that most selection engines now have are only marginally different. Sizzle is faster (esp when it caches), but I’m quite happy with the selection engine that’s present in MooTools now and don’t feel that we need to do any catching up. Yes, we could add more selector options, but if users need that they can extend our engine if they like. And yes, it would be nice to add a caching option, but I think most users wouldn’t notice it in most cases. Besides, if you’re writing your code in a MooTools style, you shouldn’t really be selecting the same thing twice unless you’ve loaded new content into the DOM, in which case the caching wouldn’t help you.

So I don’t feel like there’s a huge difference between the options that Sizzle has and the query engine in MooTools has. I also think that, by and large, these things work well enough and until browsers address these needs it’s unlikely that we’ll see any big leaps in performance, as I’ve stated elsewhere.

Sizzle caches selector results, which we could add to MooTools if we liked and we might, but perhaps this isn’t something everyone wants.

That’s something that we’ve been discussing amongst the different implementers. I see no problem with this piece of code being made optional.

Cool. I don’t think the caching thing in Sizzle is a bad idea. As I state above, I don’t ever run the same selector twice in my code unless the DOM has changed. Having this as an option would basically accommodate code styles that encourage re-selecting on the DOM. That’s cool.

The MooTools coding philosophy is definitely set on having extensible code. Anyone can author custom selectors for the MooTools selector engine and the code in the engine is very clean and understandable to any MooTools user. We’d loose some of this if we moved to Sizzle.

This is the point that I’m most confused about. I will put money down, right now, that Sizzle is more extensible than any other CSS selector engine in any of the major libraries – MooTools included. jQuery selectors have been, traditionally, very extensible but I went far beyond that with Sizzle’s implementation and you can extend any selector in virtually any direction. If there are any specific extension concerns I’d love to address them but as far as I know there really aren’t any.

I stand corrected. I’ve since spent a little more time on the topic and John is totally correct that Sizzle is very extensible. My bad there.

I don’t know about “more extensible than any other CSS selector engine” – seems to me that either the engine is extensible or it isn’t. I haven’t seen any complaints about MooTools extensibility in this regard, so we haven’t encountered a use case that our engine doesn’t handle. But regardless, I was incorrect in that statement in my previous post. My bad.

There’s something to be said about autonomy. By having its own selector engine, MooTools can add functionality and manage its own needs.

You can absolutely add your own functionality to Sizzle – in fact a number of libraries are already doing it. For example jQuery has :hidden and :visible – but keeps them within its engine – they are not a part of Sizzle. Both Prototype and MochiKit are looking to do this, as well. That’s the beauty of the extensibility here. Such an extensible architecture means that you’ll be able to add your snap-in your own functionality.

There’s a difference between being able to extend a technology and writing it yourself. There are reasons that there are so many JavaScript frameworks. There are a lot of ways to do this stuff and none of them are wrong or right. It’s a matter of personal taste. My point above is just to say that autonomy has some benefits. Again, I want to reiterate that I’m not advocating publicly whether I think MooTools should or should not adopt Sizzle. Just trying to kick off the discussion in our community.

While adding Sizzle might be a nice gesture to the community, illustrating that we can all get along together, it may not offer any other big benefits (if you don’t consider any of the Pros above big benefits).

I’m not sure what is being referred to here. It’s important to note that it’s not just illustrating that we can get along together – it’s actually getting along together. A lot can be said for standardization. If there wasn’t some form of standardization between browser vendords, for example, we’d be in a much rougher place right now.

My point here is that coming together has some nice benefits, but from a use case perspective, it’s not that big of a deal. If $$(‘div.foo’) works, our developers aren’t going to really care one way or the other. Personally, I see the benefits of using Sizzle as the selector engine for MooTools as being a net zero – it’s got benefits and negatives. In my mind, they kind of cancel each other out. That’s why I’m so keen to see what our users think of it. I could go either way.

One could argue that healthy competition is the reason we have fast CSS engines like Sizzle in the first place. If we all consolidate then maybe that innovation slacks off. One could counter that by saying that selector engines are so fast now that it’s really quite negligible and that any likely speed increase won’t come through further iteration on the JavaScript.

Two points:

1. You’re assuming that the only driving force of performance improvements is a specific outside force (such as another implementation). But you’re forgetting the biggest outside force of all: The users.

I’m not assuming anything nor am I forgetting anyone, thank you. But I think that competition is healthy and all of the frameworks have benefited from seeing how we all solve the same problems. MooTools has certainly benefited from Dean Edward’s work, Prototype’s work, and jQuery’s work. As Stravinsky said, ‘good composers borrow; great composers steal.’

Continuing with John’s point:

In the jQuery project we’ve been making constant performance improvements to a variety of methods and components – because it’ll make the code faster for the users – NOT because there’s some 3rd party test suite that demands we have faster numbers. With so many minds at the table (all the major libraries) it’ll mean that this library will be in front of the majority of JavaScript users, in one way or another. It will be compelled to become faster.

Again, speed, at this point, I think is moot. At least for MooTools developers. jQuery has a pattern that uses selectors a lot, which is totally cool and accessible to a lot of people writing and learning JavaScript. MooTools encourages developers to select once and manage state. I’m not arguing that one way is better or worse, but as far as I’m concerned, I’m quite satisfied with the efficiency in the MooTools selector engine.

We released SlickSpeed to be able to learn from other frameworks. If we could see that someone else was able to perform a selector much faster than we could, then we knew we had work to do. At this point the numbers are so low for most selectors and real world use cases that speed isn’t the concern anymore. Any gains we hope to make at this point are going to come from the browsers.

While the competition has been good, it also has its ugly side. For example, not a single major JavaScript library implements document-order multiple selectors (e.g. returning “h1, h2, h3? in the order in which they’re in the document – not all h1s followed by all h2s, etc.). No library does this because they would get instantly decimated on SlickSpeed. It’s going to be really important, going forward, that we start to match the published specifications (Selectors API) more closely, otherwise we are going to confuse users and break sites. Competition certainly hasn’t helped libraries to progress in this nature.

I’ll still take that healthy competition any day, and I don’t know that there has really been an ugly side. Returning results in the manner that John describes is, to me, a need the browsers need to support as doing it in JavaScript just isn’t practical. Further, I’ve written a LOT of JavaScript (though I’m not claiming that it’s more than others for sure) and only rarely – if ever – do I find myself wishing I had this kind of selector capability.

My final thoughts are that MooTools could benefit from Sizzle and being a consumer of and contributor to this common layer for all the frameworks. But there aren’t that many compelling reasons to do this right now. If it were my choice (and it most certainly isn’t) I’d be inclined to take a wait-and-see approach. If Dojo adopts the project (did that vote already pass? I’m assuming it will) then I’d want to see what Sizzle looks like on the next release. I know John will continue to be a big contributor but by handing it over to Dojo – an awesome move I think – he’s basically saying that everyone should own it. I want to see what it looks like when everyone does. Maybe then the benefits will be much more clear to me.

Jan Kassens, a MooTools contributor, has posted a nifty little trick to help you automatically bind methods to a class. Usually when we reference a method of a class when adding an event, we bind “this” to it to keep our reference to the class (this.addEvent(‘onComplete’, this.complete.bind(this))). This is great until we want to remove that event with removeEvent, which requires we reference the function again. the .bind method creates a copy of the function, so we have to store a reference to it. It’s all very messy. Introducing the Binds mutator. Look for it in the next MooTools patch release, but you can start using it now if you just include the 10 lines Jan has posted.

I wasn’t quite happy with this solution, because it’s too verbose: why always use this.bound.myFn when I always want the bound one? Secondly, I don’t want to do the binding of all these functions by hand. After some discussions on different solutions this is what I come up with as the optimum between speed (don’t worry its faster than the solution above) and usability.

A new so called “class mutator” named Binds. Most of you probably didn’t hear of class mutators before, but you sure have used a class mutator before when you’ve written one or another class. Built in mutators are Implements and Extends...

Precaching isn’t anything new; we were cramming 1×1 pixel versions of our graphics into the footer of our home page back in the day so that after the user clicked through from our splash screen the home page would load quickly. Ahhh, the good ole’ days.

Anyway, Ajaxian has a post up today on a new twist: anticipating what your user is about to do based on what they are doing on the current page. In this case, it’s Yahoo anticipating that if, hey, you’re typing something my search boxen, then you’re probably gonna end up on my search results page. So why not go ahead and load some of those dependencies now?

Yahoo! Search does an interesting bit of caching. To see it in action, go to the main search page with Firebug enabled and ready (or any tool that lets you see network traffic). Then type any character into the search input box and you will see some traffic kick off to download items such as:

http://us.js2.yimg.com/us.js.yimg.com/lib/s3/ysch_srp_clean_200711061918.css

http://us.js2.yimg.com/us.yimg.com/i/us/sch/el/att_hdspr_1.6t.png

http://us.js2.yimg.com/us.js.yimg.com/lib/s3/ysch_srp_clean_200711051714.js

http://us.js2.yimg.com/us.yimg.com/i/us/sch/gr2/sprt_srp_core_6.gif

http://us.js2.yimg.com/us.yimg.com/i/us/sch/el/ng_bg.png

What are these files? They are artifacts for the results page. So, Yahoo! groks that you obviously are going to do a search once you start to type something in, so why not go ahead and preload the files that are needed as part of the results page? Nicely done.