Drupal 7 gets introspective code registry

At DrupalCon Sunnyvale 2007, Rasmus Lerdorf chided Drupal on spending over half of its request time on just the bootstrap process. As a GHOP Task , Cornil did a performance analysis of Drupal and found its two largest performance drains were the bootstrap process and the theming layer. Quite simply, Drupal spends too much time including code.

Drupal 6 has the beginnings of a solution. Page handlers, the most unused code in Drupal, can now be split out into conditional include files and the menu system is able to conditionally load just the file it needs for a given page request. Based on earlier benchmarks, just that code shuffling netted Drupal 6 a 20% performance boost. The downside, however, is that it does require the module author to explicitly specify file to be included, and the syntax for it is just a little bit annoying what with the file name and file path being separate keys on the menu handler.

Fortunately, Drupal 7's self-learning code registry system has just landed, which should obliterate most of the wasted bootstrap cost.

I started toying with the registry concept last fall, but it didn't really get useful until February when chx visited Chicago for two weeks as part of the Data API Design Sprint. It's gone through several iterations in the past three months, but the final version is really slick. All a module author need do is specify in the .info file what loadable code files the module provides, including the .module file itself. The rest is automatic.

The heart of the registry is a token_get_all() call. That's a PHP function that parses a file of PHP code into a huge array. From that, it's trivial to build an index of all functions and classes in the system and save that to the database, along with information about, for instance, what function maps to what hook, and therefore what hook implementations exist. That then eliminates the need to load code and use function_exists(), since we can now dip into the database to determine if a function or class exists and if so where. Add some caching to taste. Now no code needs to be loaded until it's actually needed. Nifty.

So what do you need to do in order to make your module Drupal 7-ready? First, list all of your loadable code files in your .info file, in the files[] array. After that, it's just a matter of cuting and pasting code for optimal loading and refreshing the cache on the admin/settings/performance page.

Any class or any function that is called indirectly (via module_invoke(), module_invoke_all(), drupal_get_form(), theme(), etc.) can live anywhere; functions (but not classes) that are called directly need to either live in the .module file or else be lazy-loaded first, like so:

If a hook supports optional dynamic forms, such as hook_form_alter(), make use of the dynamic version (e.g., hook_form_$form_id_alter()). That way, you can load just the alters you need. If you make use of classes, those will lazy-load from anywhere thanks to PHP 5's autoload capabilties.

So what makes optimal organization? Well, that will require some experimentation. Registry-style hooks like hook_menu() and hook_theme() are loaded only once in a blue moon, so those can easily be moved to an example.registry.inc file. Form alters for rarely-used forms, such as admin forms, can also be moved out to their own files, or more likely the example.pages.inc and example.admin.inc files that most modules already have as of Drupal 6. Node pseudo-hooks can easily be placed in a file per node type. (Yes, that means .node files are now a reality! Or can be if you want.) Any form that appears in the body of the page can be moved out as well.

Beyond that, we will need to do some benchmarking and profiling. While the intent is to reduce the amount of code that needs to be parsed on each page load, we need to avoid ballooning the number of files that are loaded, too. File IO is expensive. If no one else looks into it I will try to do some investigation into some good guidelines, although it will vary a bit by module. (Feel free to post your own investigation results, though!) chx and Morbus have said they will be working on leveraging the registry to clean up the bootstrap process and possibly even include hook weights. (Oooo...)

Huge thanks to Justin Randell, Peter Wolanin, Moshe, and everyone else who joined the issue queue and helped make the registry really rock.