Drupal in the post-page era

Submitted by Larry on 1 November 2011 - 2:31am

(At BADCamp, several people were asking me to explain what the heck the Web Services core initiative was trying to do, so I got to practice my elevator pitch. This is essentially that pitch in written form.)

Drupal today is very page-oriented. Every request that comes in is responded to with a full HTML page. It is possible to return something else, and finally in Drupal 7 there is, sort of, native support to do so with delivery callbacks, but by and large any non-page response is an after thought at best and a hack at worst. The entire system is built around the assumption that we're returning an HTML page. Why else would we still load the theme system and form system for an auto-complete callback?

In the past, that hasn't been a major issue. The web was a series of pages, in practice, and Drupal is one of if not the most flexible page-generating machine on the web today. Drupal 7 is, arguably, the pinnacle of this page-oriented world.

Just in time for that world to be fading fast.

Tomorrow's web

In case you've been living under a rock, the web is in the midst of a massive upheval. HTML and HTTP, tools built for linking static pages together, are transmogrifying (somewhat painfully and with lots of torn shirts along the way) into a layer of the Internet atop the Internet. Everything is not a page, destined for a known browser. Everything is a response to a request. That is all that can be assumed.

Responsive design and mobile-friendly sites may get all the press, but that's the least of our worries. That's "just" theming (he says as the themers get pitchforks). The web is changing in even more fundamental ways.

Consider a site using Varnish. That can do more than just serve a full page from a RAM cache. Edge-Side Includes (ESI) let it cache most of the page but call on the server to render just certain portions of the page (say, those that are customized to a given user) quickly and stitch them together on the fly. That's next year's high-volume web sites.

Consider FaceBook. They switched to parallel rendering of the page through sub-requests over a year ago. They call it Big Pipe. The general idea, though, is breaking the page up to generate and deliver in segments, in parallel. This is last year's high-volume web site.

Consider My.FCC.Gov. Watch the video. That is a Drupal site. (Can you believe it?) The page is actually being built in the browser, by the client, in Javascript. The individual chunks of the page are being retrieved by multiple parallel Ajax requests. That required a fair bit of custom code and Services work, which means lots of full bootstraps.

Consider Backbone.js, which moves nearly all logic into the browser and treats the server as simply a dumb data store. Panels in the client, anyone?

Consider Web Sockets. Create a persistent bidirectional connection between the client and server and push raw data back out to the browser, not on request but immediately as it becomes available, to be presented to the user in whatever way Javascript dictates.

Now, try to combine those. No pages, just HTML fragments, JSON strings, raw persistent data sockets, etc. In practice, the server may almost never generate an entire page from html tag to html tag.

That's the future. That's also not something Drupal today can handle.

Show some initiative

Supporting that sort of next-generation workflow requires changing the way we approach page handling. It requires treating an incoming request as just that, an abstract request, which may be returning a page, or a fragment, or JSON, or establishing a persistent connection, or any number of things we can't even imagine yet. It requires leveraging all of the HTTP protocol, not just the tiny subset that browser HTML lets us access.

Fortunately, work is already under way. We also can get outside help. One of the reasons I am so excited that we've begun adopting Symfony2 components in core is that the Symfony2 framework is built on exactly those principles. By leveraging the work of others, in good open source fashion, we can save time both in coding and in architecting basic components. I expect we will pull in more Symfony2 components as time goes on, as reinventing the wheel in these critical areas would accomplish nothing but ensure that we don't get to the real work.

Today, Drupal loads nearly all of its code base for every request. We need to allow most of the system to lazy-load on demand instead. Fortunately, we can now do that by converting code to PSR-0 compatible classes in core.

Today, Drupal figures out what function will handle a page based purely on the path alone. We need to enable it to look at the full HTTP request and then some to do the complex routing we need. Or perhaps this is another area we could punt to Symfony2 components to save time and debugging.

Today, Drupal figures out what sort of response it is going to return after it's already done the work. We need to make determining what the response will be before we've build a single render array element. That depends on not just the request itself, but on the Drupal-derived information from it. That is, the Context of the request.

Today, we lay out a page as a blob of content with smaller blobs of content arrayed around it. We need to move to a model where one block on a page is no more special than any other, and we approach the page outside in, not inside out. Fortunately we have the model of Panels to follow.

Today, when laying out a page in Panels we have to deal with the Panels UI, which is widely recognized to be not up to the task. Fortunately some smart people are already trying to think through how that could be re-imagined to make it more approachable for site builders and content editors alike. It will take a lot more directed thought and planning,though, to re-invision the UI.

Web sockets may not be straightforward to implement in PHP, but we're already trying to work out how we could enable them from Drupal. (If you've done any development with Web Sockets, please share your experience!)

See something you could help with? We'll see you in the issue queues.

Web Sockets
Web services
Symfony2
Drupal
Architecture
WSCCI

SS (not verified)

1 November 2011 - 6:18am

Permalink

It is high time

John_B (not verified)

1 November 2011 - 8:29am

Permalink

Interesting post.

But taking a step back, it seems to me Drupal is already suffering from trying to be almost all things to almost all people, with a creep in the direction of being adapted to mid-sized government and corporate sites (and increasingly out of reach for the blogger who wants a bit more than WP, who is also still a target market). So deciding what Drupal is for might be a good idea. Otherwise, every additional bit of sophistication could make it ever more diffuse and difficult to understand.

In State of Drupal, Dries spoke about the challenge of combining ease of use with complexity. Ease of use is already lagging. Once you have complex routing, where you say to users 'switch this bit on if you are storing some of your tables in MongoDB' and 'switch this bit on of you are routing abc requests to Varnish cache depending on conditions xyz', what happens to entry-level users? Look at the d.o. support forum at the guys like the one using Drupal to develop sites on free(!) shared hosting where you cannot even upgrade PHP version, or the guy who posted a critical bug to Varnish module, because no one had explained to him that he needed Varnish on his server: then I wonder, is Drupal is trying to do too many different things? I know this point has been made before, but it just gets more urgent with all these exciting developments. Maybe usability, and definition of what Drupal is for, should be an even higher priority than integrating cutting-edge technology?

Great article.
Drupal is a modern CMS and I had a lot of "there is a module for it" experiences. I dream for a long time of a "web" that is just a large "computer" spreaded out over many real computers or whatever may come.
I'm curious about what drupel will become or which way it'll go to meet the needs of the "no page"-world.
The future maybe not as bright for others as it is for drupal : http://buytaert.net/sitecore-fud ;-)

I watched the video at My.FCC.Gov and was amazed. Nice usebility, a modern theme, a flexible and javascripty ui.

Can't wait to see more what's coming in the drupal world.

Thanks for this article.

Thanks for the informative post!
The things you're mentioning, is what I saw when someone demoed Day / Adobe CQ to me. You can output content as HTML, XML, JSON or whatever filter you write for it.
If this can be achieved in drupal 8, it will be a great leap forward.

And it gives me the willies to be honest.

I wrote a KoolAid approved app last year during a vacation. HTML5 boilerplate + Node + dnode (with web sockets), mongodb, I even played around with backbone a bit.

It was fun, and a bit challenging lots{ of{ freaking { brackets {... While I'm not sure node.js is the answer to a lot of the problems people are using it for, it was obvious to me that this model is the future, and PHP is the past.

So where does that put me? I'm a Drupal expert, I've been in the game for awhile. I'm invested. Sure, I can learn other platforms and languages, I'm actually quite good at that, but that sucks! Even with all its libraries, building most sites in node, rails, django or grails is not going to be cheaper or faster than Drupal.

And 90% of websites are better off with a page model IMO and it is just fine. It's cheaper to produce, fast enough and you can make it run anywhere.

But I don't like to work on those 90%... Can we really make Drupal work for the 10%? Should we even try? Is PHP going to let us go there with node.js, closure, twisted, etc waiting in the wings?

We won't know until we try. :-)

To be clear, not all URLs on the web are going to cease to be pages. Even for tricked out sites using layered varnish, most of them will still be pages to the user, just not build page-at-a-time.

However, that's a leading edge case. If we can build something like the above, then it means we can implement ESI or similar right within Drupal itself. That's a performance win right there, which can benefit every sight, right down to this blog.

Don't buy into too much hype, though. node.js is great for certain tasks, but it's not the answer to all problems either. It's best for high-volume, low-complexity tasks. Rendering a modern web page is not a low-complexity task, but there are low-complexity parts of it that might be able to be pushed off to it. Ruby Rails was supposed to kill off PHP in 2004. I'm still waiting. :-)

Drupal already plays decently well with MongoDB. Witness Examiner.com.

PHP is not going anywhere any time soon. Neither is Drupal. But for Drupal to remain tomorrow's dominant platform, we need to be able to adapt to tomorrow's needs, both high end and low end.

Great post, Larry. You already know I broadly agree with you.

I see some concern among above posters that a move to this will either increase complexity (presumably "too much"), or that backbone-style client-side logic is too big a shift for Drupalists who want to use all the greatness of today's Drupal to build sites quickly, and well, and cost-effectively for site owners.

These may all be risks. But I feel confident that the nature of the Drupal community-driven development model will find the right middle-ground. There's a similarity between this process & the process of listening to pundits about ___ (finance | politics | Drupal): There are - and probably need to be - extremists on either end of a spectrum that highlight the best, and worst case scenario of a situation. The function of these is to both reveal possibility, and risk, so that the entirety can find the working middle where the real solution lives.

So instead of a full client-driven model (where no content is sent from Drupal, only JS widgets that come back and get ALL content) which can't be crawled by Google to index a page, Drupal should probably send the node body, and maybe some simple, predictable things in blocks, but have other blocks pulled by the client based on client-side context. And Drupal will use server-side context to decide which of these things to send.

And Drupal will find a way for existing modules start to take advantage of new capabilities in core as they emerge, as well as a faster, better way for new modules to do the same, or new things.

Your concepts also form the foundation of what I see needing to happen to develop a first-rate Web Experience Management Drupal distribution.

Instead of worrying about whether these are the right things to do, my bigger concern is that the whole of the vision you present here is both an important thing to do, but simultaneously a big chunk of work that has tentacles that reach many places in Drupal - not just, e.g. in WSCCI - and that this work may not get done as fast as the web is moving. And that possible module improvements that take advantage of this are gated by getting this initial work done. I want Drupal to keep moving fast enough so that new developers looking at these new, fun approaches of web programming are attracted to Drupal as the best platform for doing so - lest we lose our momentum in achieving world wide web domination.

So I hope you can marshall a significant bunch of core devs to help you, and not be forced to carry the ball yourself. I encourage you to continue to evangelize this future vision broadly. It is important, it needs lots of hands to help, and those hands won't appear until they're excited & passionate about the possibilities. You're right the my.fcc.gov video is inspiring, and having talked to the developer about it, I'm inspired too. The help you need that will come when more people can see how this kind of stuff is the leading-edge work that developers like to play with.

I hope we can make this contagious.

my.fcc.gov is seriously bad. It loads 12 (!!) separate JS files. Concurrent or not, that's awful, awful, awful. The whole thing of making the client side more difficult is horrible, especially in "Tomorrow's web" where everything is loaded on mobile phones with a quarter of the processing power and a tenth of the network speed. We should be making the web SIMPLER, not insanely more difficult.

You should try to make Drupal easier to understand for everybody, not harder.