The "smallcore" debate has heated up again of late, with all its various tendrils. However, I still see a lot of the same misunderstands that I've been seeing for weeks. That can only mean one thing. Time for me to write something. :-)
I do not consider myself a "smallcore" advocate, for one very simple reason: It's horribly misleading. Allow me to repeat what I already said there:
The goals of the "smallcore" movement can be obtained without removing one single module from Drupal core.
Instead, I look at the question of Drupal's future direction from a simple architectural perspective: Is Drupal going to be an additive system, or a subtractive system?
Additive systems
In an additive system, you start from as minimal a set of assumptions as possible. You have a lot of generic tools at your disposal, designed to work together, and you add pieces, features, and functionality until you arrive at what you want to build. In an additive system, the emphasis is on the pieces, the components, and how they fit together. You don't know when you start building pieces how they're going to be put together, but you build them deliberately in such a way that they can be put together in more ways than you can think of at the time. If you can think of all of the potential uses for a given component in advance, then you haven't built it properly. :-)
Think of this as the "bucket of Legos" approach.
Subtractive systems
In a subtractive system, even if it supports pluggable extensions (such as Drupal modules), it is fundamentally one type of system. You can add to it in some cases, but to do anything different than it was built for you must take stuff away from it first. That stuff you're taking away is frequently not easy to remove, either.
Think of this approach as the late-90s Lego kits that came with big, use-specific pieces. Sure, you could build a ship or a castle with it, but you could not build a ship out of the pieces that came with the castle and the pieces that came with the ship were essentially useless if you weren't making a ship.
Just as with Legos, here's the key point: You can build a subtractive system out of an additive system. You cannot build an additive system out of a subtractive system.
Comparative systems: Firefox
I've see comparisons made to Firefox as an example of what Drupal should evolve into, as a user-focused system. However, such comparisons I think miss a key fact of Firefox's existence: Firefox is really nothing special, interesting, or useful. Gecko is.
Firefox is a complete system. It is a web browser, and only a web browser. You can write plugins for it, sure. Those plugins can do an awful lot. But at the end of the day, no matter how many plugins you add, it's still a web browser. For a plugin to remove web browser behavior is quite difficult. You can write all sorts of extensions for it, even games or IRC clients, but you really cannot get away from the fact that it is a web browser.
It's not Firefox that makes Firefox powerful, though. It's the engine underneath it, Gecko. Gecko is an engine that supports rendering all kinds of common Internet standards, like HTML, CSS, SVG, RDF, MathML, XSLT, etc. It has pluggable support for more markup systems, such as XForms. It supports Javascript natively for manipulating documents in all of those languages.
And that's it. Gecko provides no UI. It provides no UX. It provides no reason for an end user to give a damn that it exists. What it does provide is XUL: The XML User interface Language. It is an XML-based markup language for defining the structure of an arbitrary user interface which is then manipulated using Javascript. It is a framework for building stuff that leverages Internet technologies: (X)HTML, CSS, SVG, XSLT, Javascript, etc.
Gecko began life as the engine of the big and slow and annoying Netscape 6. Eventually, some developers said "screw this", took the engine, and build a new, leaner, fewer-featured-but-extensible web browser out of it. They called it Phoenix, because it was a browser rising from the ashes, but due to various trademark disputes it was renamed several times until it eventually settled on Firefox. But the real power of the system, and the reason it is so extensible, is Gecko and XUL.
Those technologies, however, are not bound to just web pages. Because Gecko is a generic framework, it can be used for a wide variety of applications: Firefox the web browser, Thunderbird the mail client, Seamonkey the internet suite, Songbird the music player, Sunbird the calendaring program, Komodo the multi-language IDE, DevHelp the documentation browser, the Celtx Media pre-production suite, all are based on that same core framework of Gecko. And only one of them looks anything like a web browser, but all of them are extensible. And most of them, I wager, the original Gecko developers back in 1998 never even dreamed of enabling.
That's the main point of the "pro-framework" type arguments of late. Drupal lacks a Gecko. Right now, it's a CMS with lots of extensibility. But you cannot make it not be a CMS. With a lot of work you can sort of make it look like something else, like a project management tool, but it's really not hard to see the CMS underneath and the contortions that had to be gone through to get that far.
With a proper framework, with components underneath that can be arbitrarily rearranged, with generic Lego blocks, we could build a real project tracking tool. Or a real BBS. Or a real inventory tracking intranet system. Or a real e-commerce site. Or, dare I say it, a real CMS rather than a Framework/Application that tries to be a framework and an application in the same UI and fails. It just becomes a Framlication, which is as silly as the word sounds.
Yes, Drupal-as-framework is the clearest path to Drupal-the-click-and-drool CMS becoming a reality.
D7UX
A lot of people have been very frustrated by the D7UX project, including one of its leaders. As Leisa points out, part of the problem was that D7UX targeted the wrong users. I would identify an even larger disconnect:
The D7UX project was focused on a particular class of CMS, and most of the code that came out of it was in the form of large single-purpose Lego blocks.
The dashboard module, toolbar module, shortcut module, etc. are all big chunks of ship hull. If you're building a ship of a certain size with your Legos, sure, they're great. But what if you want to build a different type of ship? Or what if you're building a castle? Then they're really not useful at all. And if you can't get rid of them, such as the new contextual edit-in-place links that we're trying to make disable-able, then you end up with the bow of a ship sticking out of the side of your Lego castle.
Bojhan, one of our two new UX leads (woohoo!), commented:
Yet in most discussions, developers assume that design proposals are ill-informed – even if the design proposals truly are informed (by user observations and knowledge about Drupal inner workings)
To which Jeff Eaton replied:
And designers, in most discussions, assume that developers are visually illiterate and uneducated about UX matters. It’s a double-edged sword.
I'm going to give both camps the benefit of the doubt and observe that the User eXperience you want for the class of CMS that D7UX was targeted at is fundamentally different than the UX you want for a project management tool, or a community BBS, or an inventory tracking intranet. While there are common elements, those all require fundamentally different approaches to UX. If you build a great UX for one of those, you harm another. For so many types of application, all of the work that went into D7UX is useless... because those applications are a different use case than the one D7UX tried to address, and the work that came out of it is not framework-y and additive enough to be reusable for the others.
D7UX focused on the web browser, at the expense (in time and resources and optimization) of the music player and IDE. That naturally upset people who try to use Drupal to build music players and IDEs. We just had no idea how to communicate that problem adequately.
Where do we go from here?
Where we as Drupal need to put our efforts for Drupal 8 is making that underlying engine, Drupal's Gecko, really shine. That means having flexibility in core that we do not use in core. That means architecting systems so that they can be reused in ways we can't think of, but hey, someone might be able to reuse this so let's build it to that possibility just in case. That means removing assumptions, stripping away any hard-coded logic that assumes a Drupal-unsavvy-admin-centric CMS. That means forcing ourselves to think outside the box, collaborate, and find ways to enable others to do things with Drupal that have nothing to do with a CMS.
And then rebuild the perfect CMS on top of that, using those tools in ways that can be easily removed, using all of the knowledge gained from D7UX for how that CMS should work. Let that CMS use the workflow Mark and Leisa developed, but let that work flow be reconfigurable or override-able for when you're not building a CMS. Build up a dashboard for admins, but do so using lots of reusable rectangular blocks rather than giant chunks of castle wall.
And. Keep. It. All. In. Core.
That can be done without removing poll module, or blog module, or Open ID, or any of the other modules we love to dump on at times. Keep the default download CMS-centric, but make it easy to rip apart and reconfigure that system without hacking code. Removing modules or functionality is not the point. Opening up a vast array of possibilities with Drupal by removing not code or modules but assumptions is the end goal. Allowing Drupal to become what the Data API Design Sprint envisioned Drupal becoming: a generic web services server with infinite extensibility, out of which can be built a user friendly "CMS for everyone", a personal blog, a web front-end to a legacy system, an enterprise-grade ticketing system, a corporate intranet, or anything else you can imagine... All with the same code base, all without writing massive amounts of code, all without hacking core.
Or we could just settle for being a pretty cool CMS application with some add-on modules.
I'm not one for settling.
Simply spot on
I think this is the kind of ideas that kept, keeps and will keep Drupal on the right track. Thanks for writing this!
When D6 was released the big problem was that top contrib modules (ie. CCK, Views, etc) didn't get a release until later on and that hurt a lot. For D7 we did our homework.
D7 is great, but it could have been a lot better if what you wrote here had been known 1 year ago.
Nothing really new
The whole smallcore discussion sounds very old to me. I'm really surprised that people are getting so worked up about one of the oldest discussion around drupal : "how do I change this?"
I try to help as much as I can in the forums since years, and I found that most of the questions are about how to undo stuff. I'm more focused on the design and theming aspect of things, but I'm sure it's the same discussions developpers have since years : Drupal comes with some items that are (for now) difficult to undo.
A simple example : the search box in D6.
For whatever reason, the search box in D6 suddenly came out with a big "search this site" label, and you could see, even until today people wondering how to undo stuff
Before that, it was the book pagers
Before that it was the login form
Before that...etc... This discussion, on How to change Drupal is being going on for years. The difference is that Drupal is becoming more mature with D7, but suddenly ships with tons of new stuff, that more and more people are wondering how to undo already. So we now dream about an "easy to undo" CMS, and would like to make Drupal this dream CMS.
This can be done, but I wouldn't put a fancy name like "SmallCore" on it. I think it only comes from the fact that we need to stop assuming. Whatever we add, let's make it easy to remove. If we add edit-links, let's make a checkbox to deactivate them.
Our biggest enemies are people that say "Why would you want to do that?"
The reason we are pushing for packaging changes.
It allows us to more cleanly draw the line with what the framework needs and what the application needs, and enforce that split more easily.
It also frees the application from the release cycle of the framework, so it can develop and improve faster.
Firefox is a great example, and you can see what I mean here : http://www.mozilla.org/projects/
The primary download is still firefox, or any of the other applications we will make possible, but tucked away somewhere is the rendering engine and XULrunner that you can use to build other applications.
In my mind, not only will you have the drupal core framework download as a separate download, but you will also have access to the individual modules that are used for the CMS to build your own related building blocks from.
The name #smallcore is a bad one, but it developed organically without anyone giving it any thought. A more correct name would be 'concise core' , in that we want to give core one and only one purpose.
An good description of #smallcore
You're an advocate of smallcore the architecture, but not of the name it seems :)
It's clear that a couple of issues tagged smallcore about moving modules on Drupal.org do seem to have coloured the debate and confused some people.
But we've yet to come up with a better name. At the end of the day this change in architecture does leave us with a small core – an "API Library" plus a few other essentials perhaps – and Drupal-CMS on top of that. So smallcore does describe the concept well.
I think all we can do is keep repeating that this is NOT about dumping drupal 'core' modules into contrib, it's about architectural changes that will help us build a better Drupal – and much more besides.
Sooner or later that message will sink in and we can move on to the specifics.
It is indeed confusing that 2
It is indeed confusing that 2 out of the 7 smallcore issues on d.o are about moving modules from core to contrib. Since that is not what smallcore is about, it would be best to remove those tags, or to remove those posts from http://smallcore.org.
However it be, I think we're all in agreement that this is the future of Drupal.
Obvious things out first.
Rome was not built in a day. Path to smallcore is less painful if it is iterative. Removing clearly extra modules from core and moving them to contrib is a very good place to start, since it's an easy decision that everybody can agree on.
Not so easy
If it were easy decision that everyone agreed on, those modules would already be in contrib. :-)
We may find that while refactoring code to make it more flexible and framework-y, certain modules become entirely redundant. That's fine, and it's happened before; "page" and "story" used to be their own slightly different modules until node module got the ability to define and manage arbitrary admin-defined node types, and then those went away. I suspect further abstraction and framework-ification, as we build Drupal's Gecko, will produce similar redundancy that we can then eliminate. That's a good thing.
But let's not put the cart before the horse.
Lego bricks and UI modules
It's interesting to see you bring up the Lego analogy because I've long thought it a very fitting one for where Drupal needed to head. Lego sets always came with a set of instructions for building the primary design (fire station, post office, race track, moon base), but they also came with suggested alternate designs; encouraging us to experiment. Mostly I would end up following the main instructions because designing something from scratch is difficult. But my Lego towns also had a reasonable amount of custom design in them too. For example, my moon base got turned into a construction site.
For me, those instructions are analogous to the install profiles. I expect most people to go with the default instructions, but the freedom is there to take the bricks and change as little or as much as we would like.
On the subject of UI separation, my favourite example is the Views UI module as something distinct from the core Views module. Views is kind of an obvious case for this separation because it is useful even without the admin UI. However, I also believe that this kind of separation is useful far more widely. Drupal modules are frequently full of great business logic that I could use, but they are far too tightly integrated in with the UI which doesn't work for my project. This doesn't just mean turning off the UI is difficult, but it also makes building on top of it impossible. A prime example is when a database query is made in a submit function instead of calling the module's API. I think it would pay to formalise the separation of API and UI so that we can use more modules like library code. In other words, encourage the #smallcore concept downstream into contrib.
Yes!
It's fine if what you get out of the box is even a pre-built lego kit, ready to play with, as long as it's built using smaller rectangular blocks rather than large chunks of hull. That makes it easy to rip the pieces apart and put back together as a castle instead of having to make so many new pieces and trying to hide the hull beneath it.
I highly recommend Jeff Eaton's "good APIs" session from DrupalCon DC. Views vs. Views UI is only part of the question. But yes, any module that has real logic in its form submit handler is by definition buggy. :-)
The space in between core and contrib
Thank you for this comprehensive and concise post, Larry. I applaud your assessment and agree largely with your conclusion.
With an important exception: I don't think we can "keep it all in core". I argue that we need to open a space between core and contrib.
1. We need different speeds in core
Seconding Adrian on this thread and chx on his recent blog post, I would add that we need to allow for different speeds in core.
Aggregator, Blog, Book, Help, Forum, Poll, Garland etc. are suffocating alongside Drupal the Framework that requires a slower release cycle.
One of the major reasons why Aron Novak and I retreated on core aggregator work was because I saw that not only we had to do refactoring on core piece-meal, but once all these changes where done, we would wait a year until we could use our work in everyday projects and then necessary adjustments would be submitted to yet another 1 year cycle (!)
On top of that Aron and I, who work for the same company, were essentially the only guys dedicating substantial time to it. And while there were reviewers (I nod to chx, Dries, catch, mustafau and others), I was hoping for a ferocious and productive fight with fellow aggregation nerds that would deliver an awesome result. We were left with the cost but without the benefits. I learned the hard way that *for some projects*, contrib is the better place.
And mind: I am not arguing to move modules like blog simply to contrib. I am arguing for a different release cycle for the CMS portion of Drupal. However, this will require separate project nodes and branches for every one of the CMS modules and themes.
Recent progress with packaging on Drupal.org would allow us to continue to offer *at the top of the line* a community built Drupal CMS just like today. Having better Legos, this CMS will be even better than what we have today.
2. Clear distinction between Framework and CMS portion
Further, I do see a tremendous value in a very clear distinction on what's the framework and what's the CMS portion of core. And what could be a clearer distinction than having these portions established as their own projects and release cycles? Heck, we do have includes/ and modules/ today, but that's a fuzzy line in the sand (hint: system module). Having pieces in different projects will establish a clear line and won't allow for convenient decisions breaking the model.
3. Silos
Finally, I know that there are serious concerns around creating more silos in Drupal. In my mind, the silos currently are core and contrib. Opening the space in between: the CMS portion of Drupal, would rather create more traffic between these silos than less.
As someone who is in the UX
As someone who is in the UX issue queue, I dare to say that #smallcore has no actual UX strategy behind it. It is and merely will be a development strategy, saying that it allows you to be more open in your choices for the application your building does not mean that you or anyone else will devote time to making the core CMS product it's UX better.
I am somewhat struck surprised by the examples proposed, which to me outline the fundamental misunderstanding of the scale of UX problems that we face. Which is not that, D7UX was targeted at a particular user - that your applications don'tt have (they likely do) - it's that the D7UX concepts where not adaptable enough to different context. Every #smallcore post seems to point out that the build interactions where not agile enough to also fit your purpose, sorry - but you had all the possibilities to make them so (including contextual links) - the lack of your involvement made them non-reusable. Because we never wan't them to be non-reusable, its in our best interest to have them so.
D7UX is to you, what Fields UI is to the UX-team - a mess that didn't got enough collaboration from both developers as designers, to be as valuable as it should be. To me #smallcore is only to enhance those practices, please if you can convince me otherwise - I am all for #smallcore.
Drupal should indeed not have so many assumptions in its core, but no one is stopping you from taking those out? All the leaders in #smallcore have incredible abilities to already make it happen - I do not see them in my issue queue, and that is very troublesome if you want to make reusable admin interface elements. Catch lays it out very well in - http://developmentseed.org/blog/2009/oct/28/smallcore-manifesto-help-us… . That most of this arguing is #bettercore and none of it is truly related to UX.
Yes and no
Bojhan, I want to reiterate as I said the other night that I am not blaming you or anyone else involved in D7UX. It is a mindset problem that created a process that resulted in systems that only benefit one market segment at the expense of others. As chx and webchick have pointed out, we as always have a major labor shortage which doesn't help matters. I can't be in every issue queue. I am pointing out the problem with the result, not laying blame for the cause.
As for "but no one is stopping you from taking those out?", that's actually not entirely true. I have another blog post coming that touches on that, but the problem right now is that Drupal collectively thinks too application-centric. The perspective from which patches are reviewed is usually application centric, which does stop many framework-centric patches from getting in for fear that they'd make one particular use case rougher rather than seeing it as an opportunity to improve our framework capabilities at large.
But really, think of how much better the Drupal CMS UX could be if the underlying tools let you experiment and try out different things and reconfigure the system at will without writing code. That would have allowed much more rapid iteration, much more feedback, much more agile refining of the overall UX of the CMS application... And, with it, an easier time ripping those Legos apart to build a different CMS application or different type of application all together without writing code, too.
Imagine being able to do for overall UX what Views and Panels let you do for a single page, or CCK lets you do for a data model. No, that's not easy, but that's where we need to go.
Right, but I fail to see how
Right, but I fail to see how the problem was not solvable by a change in mindset? On the application-centric point, I'm inclined to agree since I am not in those issues.
Obviously Drupal CMS UX would be better with all those mentioned, however realistically I dont see that happening anytime soon? Since UX can't wait, neither can you - we have to find a way to do this constructively parallel.
almost agreed
Good summary, almost agreed on all points. I'd call you out for these two paragraphs though:
I'm going to give both camps the benefit of the doubt and observe that the User eXperience you want for the class of CMS that D7UX was targeted at is fundamentally different than the UX you want for a project management tool, or a community BBS, or an inventory tracking intranet. While there are common elements, those all require fundamentally different approaches to UX. If you build a great UX for one of those, you harm another. For so many types of application, all of the work that went into D7UX is useless... because those applications are a different use case than the one D7UX tried to address, and the work that came out of it is not framework-y and additive enough to be reusable for the others.
D7UX focused on the web browser, at the expense (in time and resources and optimization) of the music player and IDE. That naturally upset people who try to use Drupal to build music players and IDEs. We just had no idea how to communicate that problem adequately.
While I totally agree that many of the new D7UX components might not be useful for different types of distros, some will. The contrib admin.module already born out of a slight diversion of the d7ux admin header and is shipped and enabled with Open Atrium. While you were not there (unfortunately), some other fine core developers helped out in how we can generalize these features better, so they integrate with admin_menu and other tools for example. I think many of the new D7UX tools could be useful for other types of sites, so we should somehow enable other distros to pull in these lego bricks selectively. This should be an interesting question for how a core-built-in distro is organized when separated from the framework.
Also, I completely agree that the Drupal CMS should better be separated from the Drupal Framework (whether in-core or out is the next question) and I do think that one of the positives of that separation is that we can do more of the D7UX type targeted work on that CMS distro, not less. If you think this is done on the expense of the music player and IDE, then maybe you do not agree with keeping it in core.
Which I could sympathize with, since I'm sharing affection for the different release process concept with Alex. Locking down the CMS distro to core's currently 2 year release cycle proved to just hold it down. Firefox is again a good example, Gecko has its own release cycles, and you'll not face the Gecko download when going to download, but instead the products built on top of it. Drupal.org would be better aware of these products instead of just its Gecko + Firefox combo.
Yay, Ubercore!
Saw your link... thanks. ; ) Disclaimer: d7uc thinking is still evolving.
But to the point... part of our discussion from the beginning is that the core module needs to have freedom to grow and change unhindered by essential modules that provide features we consider indispensable but aren't themselves providing core systems. I'm playing loose w/ the term "irreducible complexity" to say that the Ubercore module will be the minimum systems and entities without which we cannot have a coherent whole (i.e. an e-commerce system, however limited it might be) and which we need to build all our other features (i.e. recurring payments is not part of a core system but builds on orders, users, and payments to operate).
We're going to target things as "Essential Non-core", meaning they'll be actively maintained either by the Ubercore maintainers or devoted sub-maintainers, and when it comes time to package up a point release of "Ubercart", they'll be bundled in with the "Ubercore". We're still shooting for a rocking e-commerce system that's feature rich, and we'll still package it up together, but we're decoupling the development of the Ubercore from the other modules. This will speed up development of both, and we'll be able to do things like put out a minor release of "Ubercart" that adds a feature to an essential non-core module without waiting on the core code to get through a major release lifecycle.
This worked out perfectly when we ejected UC Recurring from Ubercart 2.0 and got an awesome maintainer for the project (with real use cases for the feature). UC Recurring has flourished and Ubercart 2.0 came much quicker. The decoupling has helped both advance faster.
All that said, we'll still have to make major version updates to Ubercore, and we'll have to wait for the essential non-core modules to catch up before we package up a new major version of Ubercart. It remains to be seen how this will work out. : )
Dual Release Cycle
Has there been any discussion about doing a dual release cycle to give attention to both framework and CMS? Obviously we want to get the framework in the hand of the developers as soon as possible so they can port modules and/or create new ones based on new APIs. Then 4 - 6 months (whenever it's ready) after that release the CMS core of that version.
The idea behind this is that it's usually the hardcore coders that jump on the bandwagon early to test out new releases. These people usually overwrite much of the default CMS and build their own UI to fit client needs. The work that these adventurous developers do at this stage can accelerate and bring new ideas to the upcoming CMS release. This will also help stagger issue queues after a major release and allow developers who like to work on both the framework and CMS. Probably free up some time to assist themers/designers work out technical issues making the new Drupal attractive and user friendly.
forking core modules to contrib
There's been a fair bit of discussion about forking core modules into contrib - and having 'feature branches' during each core release cycle. For example chx and boombatower just this week announced that http://drupal.org/project/simpletest with have Drupal 7 contrib module. The same has more or less happened for http://drupal.org/project/tracker2. I think this has a lot of possibilities, but also needs to be handled carefully - i.e. merging back when the next core release cycle opens, leaving data migration paths open between the versions (or ideally no migration for important data). This isn't a dual release cycle as such, but potentially allows us to have things both ways at least for some core modules.
to late and time wasted
The leader who finally 'realized' the target user was wrong is like the emperor figuring out he wasn't wearing clothes months after he executed his advisors who mentioned it early on.
When several people tried to point out this initial narrowness of use case, we were dismissed out of hand, insulted and pretty much dismissed out of hand. As a result the d7ux suffered a major lack of experienced contributors who couldn't be bothered to waste their time in such a hostile environment.
The result? The wrong target user and wasted time, money, trust and oportunity.
Keep work on
And then rebuild the perfect CMS on top of that, using those tools in ways that can be easily removed, using all of the knowledge gained from D7UX for how that CMS should work. Let that CMS use the workflow Mark and Leisa developed, but let that work flow be reconfigurable or override-able for when you're not building a CMS.
Build up a dashboard for admins, but do so using lots of reusable rectangular blocks rather than giant chunks of castle wall.