Refocusing WSCCI

As Dries has already reported, we held a summit meeting at the Acquia offices in Boston last week. It was a good sprint for a couple of reasons. For one, a large number of leading core developers got more clearly on the same page about the direction of Drupal core. For another, we were able to break the "too big to swallow" logjam that has been plaguing the Web Services and Context Core Initiative (WSCCI).

Let's quickly look back at WSCCI's original charter. There were four parts to the original plan: 1) Centralize our request/context information; 2) Introduce a universal plugin system; 3) Expose real, modern, solid web services from Drupal; 4) Use those tools to build a "blocks everywhere" multi-layout rendering system.

Yeah, that's big. :-)

Conclusions

The first important thing we decided in Boston was that in order to achieve those goals, we had to split the initiative in half. Moving forward, WSCCI will be focusing on parts 1 and 3: That is, "make Drupal a first-class REST server." Parts 2 and 4 are being split off to a separate effort. (Stay tuned.)

The second key point is that the WSCCI rethink using Symfony components got a green light, and will be moving forward.

Why Symfony2? Well, quite simply, its low-level components are already solid REST-based systems, exactly what we've been trying to build. We should not under-estimate the work that goes into getting that "right"; HTTP is not a trivial spec. Open Source is all about sharing, collaborating, and not reinventing the wheel. With Symfony2 Components, we get access to a really solid set of code as well as a really solid community behind it.

The latter point should not be under-stated, either. For instance, at the sprint we ran across a bug in Symfony's Controller Resolver system that Fabien fixed, on the spot, in 20 minutes flat. Several Symfony developers have reached out to us for feedback on revising their session handling system to make it more useful for Drupal, especially their "flash message" system (a junior version of drupal_set_message(), basically), which is being updated as we speak. They've even asked for use cases from Drupal to help justify a particular change so that we can leverage that directly. That's the sort of relationship that can only be beneficial to both projects. (Read: Go help make Symfony 2.1 better! That can only help Drupal.)

Next steps

The new WSCCI roadmap, then, is roughly as follows:

1. Get the basic request object in place. This begins to allow us to untangling our (ab)use of globals and super globals, and lays groundwork for a full request/response workflow. That is only a stop-gap, however, until...
2. Wrap our current menu/routing logic with Symfony2's HttpKernelInterface and related components. This is mostly to get things broken up and lay the groundwork for limited non-GET-html handling, but leads directly to:
3. Replace our routing resolution system (currently over-coupled with the menu system) with Symfony's HttpKernel implementation, including the EventDispatcher component, and potentially the Symfony2 routing library. (EventDispatcher is the Symfony equivalent of hooks, essentially, and let us leverage not just Symfony's general workflow but it's complete, mature request handling layer.) This is what gets us the real power, and ability to respond to all kinds of HTTP requests with rich HTTP responses. We may retain some of the underlying elements, like the menu_router table. We'll work that out as we go and see what works best. This part may get merged with the previous step, depending on how difficult it is to separate.
4. With that new flexibility, begin cleaning up our existing non-GET-html logic (form submission, Ajax commands, etc.) and adding new functionality (standard libraries for exposing entities as REST, etc.). Details of this part will be worked out later when we get there.

Some of those steps may get compressed, depending on what short-cuts we're able to take in implementation that make it easier to combine steps than not.

There are lots of related topics we should be exploring that we would be able to leverage with this new under-the-hood power. Most notable are Web Sockets and Server Sent Events and Standard Export Formats, both of which should be made possible by these efforts. If you want to help, those are good places to start.

For module developers, we'll need to stop thinking in terms of pages and start thinking in terms of requests and responses. What we now call page callbacks will mutate into "body-like blocks", which will be on the same level as response controllers for JSON, SVG, SOAP, or whatever else we feel like sending. HTML is a just special case of REST.

That in turn means routing will need to become more robust, which likely means heavy changes to hook_menu(). It may even cease to be a hook and turn into configuration objects from the CMI initiative. We'll work that out as we go.

Some low-level operations, such as HTTP error handling and fancier menu tricks, will also shift to a more Symfony-based approach. That should only affect a small number of module developers, however.

Global state variables, such as $user and $language, will be eliminated in favor of the Symfony2 request object. The super-globals will also all be deprecated in favor of the request object, as that is exactly what it is designed to do. That in turn allows us to make portions of code that can run anywhere, even in fake internal HTTP requests, Edge Side Include-generated requests, or pure unit tests (which are several orders of magnitude faster than integration test).

Things like hooks, Fields, Entities, and so forth should be largely or completely unaffected, save perhaps having a request object or request-derived information passed to them rather than using a global.

Other beverages

Something else that became apparent at the sprint is that some people were seeing "WSCCI" as even bigger than it already was. There are a lot of related "framework-ification" tasks going around, many of which have the same people involved in them as WSCCI, but are not directly related to WSCCI. That doesn't make them unimportant; in fact, many of them are, I believe, critical to Drupal's success, which is why many of the people involved in WSCCI have been involved in them as well.

Among those are the discussions around class loaders, namespaces, converting core code in /includes to PSR-0-compatible classes, dependency injection, and so forth. Those are important, and time permitting I and others will continue to work on those, but they are not part of WSCCI proper. Only issues tagged with "WSCCI" or in the WSCCI sandbox are "part of" WSCCI.

There are also ongoing discussions of other parts of Symfony2 that we could leverage in core. Most of that is not directly related to WSCCI either, but this is a great opportunity to reduce the amount of code we have to maintain. If you're interested in reducing Drupal's Not Invented Here syndrome, that's a great place to jump in.

We have only 9 months left to get most of this work done. I think we can do it, if we commit ourselves to it. Now is the time to do. Let's do.