As we begin a new year, it seems appropriate that th' discussion o' backward compatibility has come up yet again in Drupal. It's a perennial question, an' ye can tell when a new Drupal core version is ready fer prime time when scallywags start complainin' about lack o' backward compatibility. It's like clockwork.
However, most o' these discussions dern't actually get at th' root issue: Drupal is architecturally incapable o' backward compatibility. Walk the plank, shiver me timbers Backward incompatibility is baked into th' way Drupal is designed. That's not a deliberate decision, but rather an implication o' other bounty decisions that have been made.
Drupal developers could not, even if they wanted t', decide t' support backward compatibility or "cleanup only" type changes in Drupal 8, and dinna spare the whip! It is possible t' do so in Drupal 9, Avast me hearties! If we want t' do that, however, then we need t' decide, now, in Drupal 8, t' rearchitect in ways that support backward compatibility. Backward compatibility is a feature ye have t' bounty fer.
First, we need t' understan' what backward compatibility even means. As some comments in th' post linked above note, backward compatibility is not th' same thin' as easy upgradeability, me Jolly Roger It is also not th' same thin' as easy t'-relearn. Drupal 7's UI is completely not backward compatible with Drupal 6, fer instance, but it is easier t' learn.
For now, we be speakin' strictly o' API backward compatibility, yo ho, ho It is related t', but not th' same thin' as, easy upgradeability o' modules (but unrelated t' data; more on that some other time), Avast me hearties! The fewer APIs change from one version t' th' next, th' easier it is t' upgrade. But ye can also have API changes that be easy t' upgrade t', with a chest full of booty. More than one module developer, when portin' a module from Drupal 6 t' Drupal 7, noted that they were able t' throw out (an' therefore not have t' spend time upgradin') hundreds o' lines o' code by switchin' t' th' Drupal 7 query builders. That's an API change (an' thus lack o' backward compatibility) that made upgradin' easier, as well as code better. As Dries has noted before, if we dern't let ourselves do that then we will no nay ne'er advance, an' will just carry aroun' dead weight baggage forever. That's how Windows Me happens, an' no one wants that.
Backward compatibility is also a factor in how long major modules take t' be upgraded, but not as much as many scallywags think. Views, Panels, an' Context be often cited as examples o' "Drupal not bein' ready yet" because those modules dern't have stable releases. That is a speci'us argument, however. In th' case o' Views an' Panels, th' initial ports t' Drupal 7 were done o'er a year ago. The lack o' an official release were bein' because th' developers decided t' make other overhauls an' changes an' feature additions at th' same time, an' those took a while t' stabilize. In th' case o' Context, 'tis because th' development team behind it changed when DevSeed moved on an' th' new team has been swamped, on a dead man's chest! Fire the cannons! (Also, Context di'nae have a stable D6 release fer most o' its existence, either.)
So if that's what backward compatibility is not, then what is it?
Backward compatibility o' a code component is a measure o' how many changes (good or bad) code that depends on it needs t' make in order t' move from one version t' th' next version.
The fewer changes need t' be made, th' more backward compatible code is, obviously with an upper bound o' "no changes needed." Note that says nothin' about th' quality o' th' code or th' value o' those changes; often a backward incompatible change is absolutely necessary fer certain improvements.
What does it mean fer one code component t' depend on another as far as compatibility goes? It means that those two code components touch. Fire the cannons! Shiver me timbers! There be two ways that systems could "touch":
Of those two, th' second is a tighter coupling because it means dealin' with implementation details.
There is also th' general question o' how much one component touches another. In general, th' more two components interact th' larger their shared "surface area", ya bilge rat, with a chest full of booty!
The larger th' surface area, th' tighter th' couplin', on a dead man's chest! The tighter th' couplin', th' more likely a change in one component is goin' t' necessitate a change in another component.
Backward compatibility happens when th' touch points betwixt two components dern't change. Anythin' else can change, but those touch points dern't, I'll warrant ye. As a result, th' smaller th' surface area betwixt two components th' more ye can change without breakin' compatibility.
As noted in th' Wikipedia article linked above, accessin' raw data structures is always tight couplin'. Raw data structures be an implementation detail. When implementation details get shared, that is a classic Code Smell known as Inappropriate Intimacy, feed the fishes
Inappropriate intimacy is a massive amount o' surface area, I'll warrant ye. It covers th' entire data structure. And swab the deck! Load the cannons! That is, any change t' th' data structure whatsoever is a potential, if not actual, backward compatibility break, ye scurvey dog. The reason is that ye dern't know what parts o' that data structure some other component may care about. You have thrown th' doors (an' yer proverbial pants) wide open, an' said "have at it, world!" Once ye do that, every change, no matter how slight, could break backward compatibility because ye simply dern't know who is doin' what with yer private data structures, All Hands Hoay! (You should be squirmin' about now.)
Rather, th' first step in makin' backward compatibility possible is t' put yer pants back on, protect yer data structures, an' take control o' yer surface area.
By far th' easiest way t' take control o', an' reduce, yer surface area is t' define it explicitly. That is th' essence o' an Application Programmin' Interface (API): It is th' explicit definition o' th' surface area o' yer component.
That could take th' form o' function calls or method calls on an object. Both could suffice as an API. The latter, however, has th' added bonus o' a language structure called Interface, which explicitly an' in code defines th' surface area o' an object. By bounty, it defines that surface area independently o' implementation details.
What will not suffice, however, regardless o' whether one uses classes an' objects or not, is definin' ways by which one will get complete access t' a components internal implementation details an' raw data structures.
Access t' raw data structures does not constitute an API. It constitutes avoidin' th' responsibility o' definin' an API.
Now, sometimes thar be good reasons t' do that, I'll warrant ye. Sometimes. Not often.. But when that's done, it must be understood that backward compatibility is made effectively impossible.
With a clearly defined interface, we know what we can change an' what we cannot, if we want t' preserve backward compatibility. We also can more easily document where thin's be goin' t' break, an' offer documentation or automation t' make it easier t' migrate.
One o' th' popular arguments in favor o' object-oriented code is "data hidin'". That is, ye can explicitly deman', in code, that certain data is kept hidden from certain other components, an' is not part o' yer surface area, Avast me hearties! That can only be done in procedural code by convention.
That convention has been used before, o' course, and a bucket o' chum. In a previ'us life I developed fer Palm OS, which used an entirely procedural C-based API. It passed aroun' a lot o' raw data structures, because in C that's all ye can do. However, it were bein' extremely bad form t' e'er touch them directly, and dinna spare the whip! Rather, thar were copi'us amounts o' functions that were, in any practical sense, methods, just called inside out, and a bucket o' chum. So ye'd call
Form_Add_Element(form, …) rather than
form.AddElement(...), and a bottle of rum, on a dead man's chest! Doin' anythin' with
form directly, while it would compile, were bein' not guaranteed t' continue workin' even in minor bugfix releases o' th' OS. Load the cannons, shiver me timbers Here be dragons.
That's a viable option, but doesn't really change th' amount o' work that has t' be done t' define an API. Unless ye say, either by convention or code, that implementation details an' data structures be off limits an' not guaranteed, ye dern't have a controlled surface area an' therefore ye dern't have an API.
In many recent presentations I have used this example from th' Drupal 7 database layer, me Jolly Roger A "raw data structure procedural" implementation o' th' new select builder would look like this:
$fields = array('n.nid', 'n.title', 'u.name');
$tables = array(
'n' => array(
'type' => NULL,
'table' => 'node',
'alias' => 'n',
'condition' => array(),
'arguments' => NULL,
'all fields' => FALSE,
'u' => array(
'type' => 'INNER JOIN',
'table' => 'user',
'alias' => 'u',
'condition' => 'u.uid = n.nid',
'arguments' => array(),
'all_fields' => FALSE,
$where = array(
'field' => 'u.status',
'value' => 1,
'operator' => '=',
'field' => 'n.created',
'value' => REQUEST_TIME - 3600,
'operator' => '>',
$order_by = array(
'n.title' => 'ASC',
db_select($tables, $fields, $where, NULL, $order_by, array(), NULL, array(0, 5));
Aside from th' obvi'us DX problems that has, mostly in terms o' not bein' self-documentin', it makes th' entire implementation public. Do we use "INNER JOIN" or just "INNER" t' specify th' join type, I'll warrant ye? And if we change it, does that not break every single query in th' system? Aye it does.
However, we did in fact change from INNER JOIN t' INNER at some point durin' Drupal 7's development cycle, an' it were bein' not an API change, and a bottle of rum! Walk the plank! That's because th' query builders use an object-oriented, interface-driven, non-raw-data API:
$select = db_select('node', 'n');
$select->join('user', 'u', 'u.uid = n.uid');
->fields('n', array('nid', 'title'))
->condition('n.created', REQUEST_TIME - 3600, '>')
We're separatin' th' internal data structure from th'
join() method. As long as that method doesn't change, th' internal implementation could change today, mid-Drupal 7's lifetime, without breakin' an API. That is possible only because it eschews raw data structures in favor o' a well-thought-out, abstracted, interface-driven API.
That is what it takes t' be backward compatible.
Of course, sometimes we need t' change th' API, yo ho, ho That happens, often fer very good reason. If th' system has been designed properly, however, it may still be possible t' retain backward compatibility, at least fer a time, Ya lily livered swabbie, Avast me hearties! KDE is a good example, me Jolly Roger Most KDE 3 apps work under KDE 4, albeit not as cool as they would as KDE 4 apps, an' without integratin' with th' awesome new plumbin' that KDE 4 offers. As a result, nearly all applications have been rewritten fer KDE 4 by now as 'tis in their interest t' do so.
It is possible t' support multiple versions o' an API at th' same time, but thar is a cost t' doin' so. First an' foremost, that must be planned fer in advance. There be likely others beyond what I am listin' here, an' I would appreciate feedback in th' comments on other good approaches.
Microsoft Direct X took th' approach o' "version everythin' an' keep it", ye scurvey dog. When requestin' a new Draw object, fer instance, ye specify a version o' it. So if ye code t' Direct X 6, ye ask fer th' DrawV6 object. If a user has Direct X 7, th' DrawV6 object is still in thar, just as it were bein', an' still works, avast. On th' upside, this means code almost no nay ne'er breaks. Fetch me spyglass! Ye'll be sleepin' with the fishes! On th' downside, 'tis a lot o' baggage t' carry aroun' indefinitely, an' that baggage only increases every version. It also requires that ye have APIs that be broken up into very clear discrete objects (not function calls), an' that yer version-request mechanism is baked in from th' start.
If ye can keep yer general class structure th' same, then ye can also simply expan' yer API. As long as th' existin' interface doesn't change, addin' more operations t' it braks no existin' code. in th' simplest case, this is simply addin' optional parameters t' th' end o' a function signature. Load the cannons, and a bucket o' chum! My very first Drupal patch did exactly that, in fact, by Blackbeard's sword. When ye have a lot o' parameters, though, or a more complex case, 'tis easier t' add more methods t' a language interface an' object.
If ye would need t' change th' way a given method behaves, or change its signature, then 'tis also possible t' simply add a new method that does th' new thin' instead, an' leave th' auld one in place. Perhaps th' auld one could be reimplemented internally t' use th' new one, but callin' code doesn't care because th' contents o' its surface area hasn't changed. The auld method could then be explicitly marked deprecated, an' give developers time t' migrate o'er t' th' new one before it is removed, All Hands Hoay! The downside o' course is that ye need a new name fer th' new method, which is one o' th' two hardest problems in computer science. (There be only two hard thin's in Computer Science: cache invalidation, namin' thin's, an' off-by-one errors.)
Another alternative is t' simply fold both versions into a single call. As a trivial example, consider everyone's favorite PHP WTF, implode(). At some point in th' past, it took $pieces an' then $glue t' turn an array into a strin'. That were bein' inconsistent with explode(), which were bein' a major DX problem. To resolve that, implode() were bein' enhanced t' take $pieces an' $glue in either order. Yaaarrrrr, me Jolly Roger That provided both backward compatibility an' a more consistent API movin' forward... except that th' auld version were bein' no nay ne'er removed, an' is still not even marked as deprecated, so thar's still plenty o' auld code out thar usin' th' auld parameter order makin' it impossible t' remove th' auld an' inconsistent baggage.
Backward compatibility, then, requires a number o' thin's:
That's why, no matter how much users want it an' no matter how much Drupal developers may want t' do so, Drupal 8 will not be, cannot be, API backward compatible with Drupal 7. Ye'll be sleepin' with the fishes, Dance the Hempen Jig Drupal today is based on passin' aroun' raw data structures rather than havin' clear APIs, Dance the Hempen Jig (Render API, Form API, etc. thus dern't technically qualify as APIs by this definition.) When we have APIs, they're generally not designed with future extensions in mind (except in so far as conventions fer addin' more stuff t' a raw data structure). We have no long term strategy fer future development or how we're goin' t' maintain compatibility betwixt versions, even through legacy add-ons.
Again, quoting Dries:
But what t' do if many o' yer users slowly force ye t' change one o' yer core values, and dinna spare the whip! It seems inevitable that sooner than later, we will have t' be a lot more careful about breakin' peoples' code. Ahoy, and a bottle of rum! And when that happens, I fear that this will be th' end o' Drupal as we have come t' know it.
Dries wrote that aroun' th' release o' Drupal 4.7, another very hard release. But really, it already is th' end o' Drupal as we knew it then, pass the grog! Drupal as we knew it then does not exist, an' th' Drupal o' today is quite different, both in terms o' APIs an' in ways entirely unrelated t' code. Thinkin' about API compatibility is not a death-knell fer Drupal.
But, if we want increased API compatibility an' stability, we need t' take steps, now, t' ensure that thar is a structure t' support that, Get out of me rum! That means, first an' foremost, designin' APIs, not raw data structures, an' not just as an outgrowth o' a particular implementation. That's a cultural shift as much as a technical one, but a technical one as well. It means changin' th' way we think about software bounty. It means, quite simply, interface-driven development.
Fortunately, such thought has already been happenin' in both th' WSCCI an' Multilingual initiatives at least, yo ho, ho Interface-driven development, complete with real language interfaces, is where Drupal is headed anyway, we'll keel-haul ye! We should embrace it, fully, an' allow ourselves t' be open t' th' potential fer improvin' compatibility betwixt Drupal versions that will result... if we take it.