Front Seed Labs

I have created a preferred configuration for hosting Drupal sites on the excellent nginx web server. I tend to prefer nginx over Lighttpd and/or Apache for at least over a year now and I'm extremely happy with its reliability and performance.

Apart from configuring a little more and getting used to the fact that no run-time configurations are available through .htaccess files, nginx with php-fpm is really a viable option for hosting Drupal sites.

'Nuff said, you can head over to GitHub to check out my configuration. The files have many smart ideas taken from yhager's nginx config, however I added more features and tailored the configuration for my needs.

It has the following features:

• Multisite Drupal installations
• Boost module support
• Advanced CSS/JS Aggregation support
• Supports fake CNAME-based cdns with imagecache and advagg (see previous post about null-session support on fake CDNs)
• Proper MIME types for @font-face fonts
• An overall config file structure to minimize per-vhost file volume

Anyway feel free to add patches or comments: Drupal hosting on nginx

So you want your Drupal site perform as quick as possible and you have set up CDN and patched imagecache and everything is working fine. Then you realize that any asynchronously generated static resource (e.g. image, stylesheet, etc.) will naturally send a session cookie to your visitors.

The very promising Advanced CSS/JS Aggregation and ImageCache both handle requests in a unique way: if the requested resource is not generated yet, it will run through Drupal, create the resource and place it in the path exactly where the request points to. This way, subsequent requests does not need to go through Drupal and PHP again, it can be served as a static resource.

However since the first request runs through Drupal, it will start a session (Drupal 6 by default starts sessions for anonymous users) This is okay if you use a single domain to serve files from. Things get annoying when you start using alternate host names to speed up page loading times, and you realize that these precious alternate host names start sending out session cookies to your visitors, and you are basically doubling (or with 3 cname domains, that is quadrupling) your session rows if a visitor happens to hit a static resource for the first time.

CDNs - even if they are just fake CNAME records pointing to the same webroot - should operate cookieless to keep the http request headers as short as possible for subsequent requests.

One could (eliminate anonymous sessions)[http://2bits.com/articles/reducing-server-load-eliminating-anonymous-sessions-drupal-6x.html] easily from the entire site, but that might not be your cup of cake because anonymous sessions are needed so much say on an Ubercart site to track cart contents.

My solution is to use an alternate session handler which in fact does not handle sessions at all. Configure this session handler to be fired only at your cdn domains and you're set to go.

Using Lighttpd as a proxy for Apache is a popular solution to achieve smaller web server memory footprint and to speed up websites. However things can get messy when it comes to Drupal and the brilliant Imagecache module. Imagecache transforms images using presets and then caches them.

An imagecache url is something like: http://domain.tld/sites/default/files/imagecache/presetname/photo.jpg This will take the the image placed at sites/default/files/photo.jpg and transforms it according to presetname. After succesful operation the image will be available as a static file at sites/default/files/imagecache/presetname/photo.jpg

No need to say, this functionality breaks when the frontend web server only forwards requests to php files so imagecache images won't show up and will not be generated.

One solution is: set up a separate subdomain to serve these images and redirect all imagecache images to this subdomain. This needs some simple LUA scripting, Lighttpd will forward an imagecache url if it hasn't created yet, and will serve it directly if it's already there.

Four steps to get this working:

Configure Lighttpd

Create this little LUA script:

attr = lighty.stat(lighty.env['physical.path'])
if (not attr) then
   cuthere = string.find(lighty.env['uri.authority'], '.', 1, true) + 1
   redirhost = string.sub(lighty.env['uri.authority'], cuthere)
   lighty.header["Location"] = "http://" .. redirhost  .. 
lighty.env["request.orig-uri"]
   return 302
end

This script will stat for the file requested and if not found, redirect the request to the main domain. Link the lua script to the vhost configuration (and enable mod_magnet of course if you haven't done already):

magnet.attract-physical-path-to = ( "/etc/lighttpd/filecheckredirect.lua" )

Set up the subdomain

Configure your subdomain or multiple subdomains for imagecache hosted images and set this in the Drupal settings.php:

/**
 * Alternate hosts for imagecache created images.
 * A string can be set for single hosts,
 * An array of hosts can be set for multiple hosts.
 */
$conf['static_file_hosts'] = array(
  'ic1.example.com',
  'ic2.example.com',
);
//$conf['static_file_hosts'] = 'ic.example.com';

Override imagecache theme function

/**
 * Replace the domain part of imagecache urls with 'static_file_hosts' if
 * available.
 *
 * 'static_file_hosts' can be set from settings.php.
 */
function YOURTHEME_imagecache($presetname, $path, $alt = '', $title = '', $attributes = NULL, $getsize = TRUE) {
  // Check is_null() so people can intentionally pass an empty array of
  // to override the defaults completely.
  if (is_null($attributes)) {
    $attributes = array('class' => 'imagecache imagecache-'. $presetname);
  }
  if ($getsize && ($image = image_get_info(imagecache_create_path($presetname, $path)))) {
    $attributes['width'] = $image['width'];
    $attributes['height'] = $image['height'];
  }
  $attributes = drupal_attributes($attributes);

  $domain = variable_get('static_file_hosts', $GLOBALS['base_url']);
  if (is_array($domain) && !empty($domain)) {
    if (count($domain) > 1) {
      $domain = $domain[rand(0, count($domain) - 1)];
    }
    else {
      $domain = $domain[0];
    }
  }
  if ($domain !== $GLOBALS['base_url']) {
    $base_url_parsed = parse_url($GLOBALS['base_url']);
    $imagecache_url = str_replace($GLOBALS['base_url'], $base_url_parsed['scheme'] . '://' . $domain, imagecache_create_url($presetname, $path));
  }
  else {
    $imagecache_url = imagecache_create_url($presetname, $path);
  }

  return '<img src="http://frontseed.com/%27.%20%24imagecache_url%20.%27" alt="'. check_plain($alt) .'" title="'. check_plain($title) .'" '. $attributes .' />';
}

download this snippet.

Patch imagecache:

The patch below is for imagecache version 6.x-2.0-beta10:

--- imagecache.module   19 Aug 2009 20:59:07 -0000  1.112.2.5
+++ imagecache.module 
@@ -318,7 +318,15 @@
   $args = array('absolute' => TRUE, 'query' => empty($bypass_browser_cache) ? NULL : time());
   switch (variable_get('file_downloads', FILE_DOWNLOADS_PUBLIC)) {
     case FILE_DOWNLOADS_PUBLIC:
-      return url($GLOBALS['base_url'] . '/' . file_directory_path() .'/imagecache/'. $presetname .'/'. $path, $args);
+      $domain = variable_get('static_file_hosts', $GLOBALS['base_url']);
+      if ($domain !== $GLOBALS['base_url']) {
+        if (is_array($domain) && !empty($domain)) {
+          $domain = count($domain) > 1 ? $domain[rand(0, count($domain) - 1)] : $domain[0];
+          $base_url_parsed = parse_url($GLOBALS['base_url']);
+          $domain = $base_url_parsed['scheme'] . '://' . $domain;
+        }
+      }
+      return url($domain . '/' . file_directory_path() .'/imagecache/'. $presetname .'/'. $path, $args);
     case FILE_DOWNLOADS_PRIVATE:
       return url('system/files/imagecache/'. $presetname .'/'. $path, $args);
   }

Final steps

Empty the theme registry and watch your imagecache files served from ic.example.com using Apache the first time and using Lighttpd after image generation.

Update

The solution above is only suitable in scenarios where a reverse proxy is used for static file handling. I believe that planting a Lighttpd in front of Apache is not the best way to serve a Drupal page. According to my experience it is better to run either nginx as a reverse proxy or just have nginx handle all your pages via php-fpm. Or you can strip down Apache and use mpm_worker with php in fastcgi mode, however that is not my cup of tea either.

I usually develop and test my webpages on various VirtualBox virtual machines and since I am doing a lot of console work I use a framebuffer which enables using larger resolutions than the default console. Until now I was using Ubuntu 8.04 LTS virtual machines, but recently I upgraded to 9.10. Ubuntu 9.10 (Karmic Koala) is using grub2 so the simple adding of vga=791 or similar value to boot options won't work. Here's how to enable the framebuffer on virtual consoles:

1. Enable kernel modules

$ sudo nano /etc/initramfs-tools/modules

Add two lines at the end of the file enabling two kernel modules:

vesafb
fbcon

Prevent blacklisting the vesafb module:

$ sudo nano /etc/modprobe.d/blacklist-framebuffer.conf

Look for the line blacklist vesafb and comment it out:

#blacklist vesafb

2. Modify grub configuration

$ sudo nano /etc/default/grub

Remove the comment from the GRUB_GFXMODE line and set the desired resolution:

GRUB_GFXMODE=1024x768

There's no need to set bit depth, just the screen resolution. Then open the very first config file in /etc/grub.d

$ sudo nano /etc/grub.d/00_header

Look for the code set gfxmode=${GRUB_GFXMODE} and insert one line after it so the code looks like this:

if loadfont `make_system_path_relative_to_its_root $(){GRUB_FONT_PATH}` ; then
  set gfxmode=${GRUB_GFXMODE}
  set gfxpayload=keep
  insmod gfxterm
  insmod ${GRUB_VIDEO_BACKEND}

3. Update your boot configuration

$ sudo update-grub2

$ sudo update-initramfs -u

That's it! Now reboot and watch your enhanced virtual console with the framebuffer.

Meet Lighttpd: it is a fast, small footprint and higly configurable webserver with advanced features. It's often a common alternative to Apache. At the time of writing the latest versions from the 1.4.x branch is considered stable while Lighttpd 1.5.0 is an ongoing work with new features.

One new feature is the mod_deflate server module which is a modified version of mod_compress shipped with 1.4.x versions. mod_deflate can compress any output from lighttpd static or dynamic - that is, lighttpd can compress the output of dynamically generated pages with this.

Websites with lots of dynamically generated content receive significant bandwidth saving with dynamic output compression, and visitors of those websites benefit faster page loading times.

There are patches available which makes building a version of Lighttpd 1.4.x from source which includes mod_deflate.

1. Building and packaging Lighttpd on Ubuntu

Before building on an Ubuntu hardy system, some prerequisites need to be installed. After getting lighttpd source, it needs to be patched to support mod_deflate. Standard debian packages are to be built which can be easily installed on any target system (assuming the same OS architecture)

Install tools for building and get the Lighttpd source from the package repository:

$ sudo apt-get install build-essential fakeroot
$ sudo apt-get install debhelper cdbs libssl-dev libbz2-dev libattr1-dev libpcre3-dev libmysqlclient15-dev libgamin-dev libldap2-dev libfcgi-dev libgdbm-dev libmemcache-dev liblua5.1-0-dev dpatch patchutils pkg-config uuid-dev libsqlite3-dev libxml2-dev libkrb5-dev
$ mkdir ~/lighttpd
$ cd ~/lighttpd
$ apt-get source lighttpd

Download the patch and apply it:

$ wget http://redmine.lighttpd.net/attachments/download/632/lighttpd-1.4.19.mod_deflate.patch
$ cd lighttpd-1.4.19
$ patch -p1 < ../lighttpd-1.4.19.mod_deflate.patch

Build and install the package:

$ fakeroot dpkg-buildpackage

After building completes you will see the freshly built deb packages in the parent directory:

$ cd ..
$ ls -lgG
total 1336
drwxr-xr-x 8   4096 Aug 16 08:05 lighttpd-1.4.19
-rw-r--r-- 1  66387 Oct  8  2008 lighttpd-1.4.19.mod_deflate.patch
-rw-r--r-- 1  70422 Aug 16 08:07 lighttpd-doc_1.4.19-0ubuntu3_all.deb
-rw-r--r-- 1  11536 Aug 16 08:08 lighttpd-mod-cml_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1  10958 Aug 16 08:08 lighttpd-mod-magnet_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1   6918 Aug 16 08:08 lighttpd-mod-mysql-vhost_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1   8492 Aug 16 08:08 lighttpd-mod-trigger-b4-dl_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1  19002 Aug 16 08:08 lighttpd-mod-webdav_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1  38501 Aug 16 08:03 lighttpd_1.4.19-0ubuntu3.diff.gz
-rw-r--r-- 1   1106 Aug 16 08:03 lighttpd_1.4.19-0ubuntu3.dsc
-rw-r--r-- 1   2433 Aug 16 08:08 lighttpd_1.4.19-0ubuntu3_i386.changes
-rw-r--r-- 1 267770 Aug 16 08:08 lighttpd_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1 815568 Mar 12  2008 lighttpd_1.4.19.orig.tar.gz

Lighttpd depends on libterm-readline-perl-perl so install this as well:

$ sudo apt-get install libterm-readline-perl-perl

If apt-get is whining about libterm-readkey-perl as a dependency try:

$ sudo apt-get -f install libterm-readkey-perl libterm-readline-perl-perl

Then you can install lighttpd from the freshly built deb package:

$ sudo dpkg -i lighttpd_1.4.19-0ubuntu3_i386.deb

If you need additional packages, install them as well:

$ sudo dpkg -i lighttpd-mod-magnet_1.4.19-0ubuntu3_i386.deb
$ sudo dpkg -i lighttpd-mod-webdav_1.4.19-0ubuntu3_i386.deb

Copy mod_deflate module to its place:

$ sudo cp -d ~/lighttpd/lighttpd-1.4.19/debian/tmp/usr/lib/lighttpd/mod_deflate.so* /usr/lib/lighttpd/

2. Configuring mod_deflate

Load the module in /etc/lighttpd/lighttpd.confand comment out mod_compress:

    server.modules = (
                "mod_access",
                "mod_alias",
                "mod_accesslog",
    #           "mod_compress",
                "mod_deflate",
    #           "mod_rewrite",
    #           "mod_redirect",
    

Also comment out mod_compress configuration directives:

    #### compress module
    #compress.cache-dir          = "/var/cache/lighttpd/compress/"
    #compress.filetype           = ("text/plain", "text/html", "application/x-javascript", "text/css")

Configure the mod_deflate module:

    # mod_deflate settings
    deflate.enabled = "enable"
    deflate.compression-level = 9
    deflate.mem-level = 9
    deflate.window-size = 15
    deflate.bzip2 = "enable"
    deflate.min-compress-size = 200
    #deflate.sync-flush = "enable"
    deflate.output-buffer-size = 4096
    deflate.work-block-size = 512
    deflate.mimetypes = ("text/html", "text/plain", "text/css", "text/javascript", "text/xml")
    deflate.debug = "enable"

Detailed information about configuration options is available on the Lighttpd mod_deflate documentation page. The last line is only to check if mod_deflate is working.

Fire up Lighttpd:

$ sudo /etc/init.d/lighttpd start

Visit a static page: http://your.server/index.lighttpd.html and check the lighttpd error log at /var/log/lighttpd/error.log:

2009-08-16 09:27:22: (mod_deflate.c.1232) Content-Type: text/html
2009-08-16 09:27:22: (mod_deflate.c.1239) mime-type: text/html
2009-08-16 09:27:22: (mod_deflate.c.1267) enable compression for  /index.lighttpd.html
2009-08-16 09:27:22: (mod_deflate.c.1283) add Vary: Accept-Encoding for  /index.lighttpd.html
2009-08-16 09:27:22: (mod_deflate.c.305) output-buffer-size: 4096
2009-08-16 09:27:22: (mod_deflate.c.307) compression-level: 9
2009-08-16 09:27:22: (mod_deflate.c.309) mem-level: 9
2009-08-16 09:27:22: (mod_deflate.c.311) window-size: -15
2009-08-16 09:27:22: (mod_deflate.c.313) min-compress-size: 200
2009-08-16 09:27:22: (mod_deflate.c.315) work-block-size: 512
2009-08-16 09:27:22: (mod_deflate.c.1350) Compress all content and use Content-Length header: uncompress len= 3574
2009-08-16 09:27:22: (mod_deflate.c.948) compress: in_queue len= 3574
2009-08-16 09:27:22: (mod_deflate.c.881) compress file chunk: offset= 0 , toSend= 3574
2009-08-16 09:27:22: (mod_deflate.c.351) gzip_header len= 10
2009-08-16 09:27:22: (mod_deflate.c.386) compress: in= 3574 , out= 0
2009-08-16 09:27:22: (mod_deflate.c.1003) compressed bytes: 3574
2009-08-16 09:27:22: (mod_deflate.c.451) flush: in= 0 , out= 1555
2009-08-16 09:27:22: (mod_deflate.c.489) gzip_footer len= 8
2009-08-16 09:27:22: (mod_deflate.c.919)  in: 3574  out: 1563
2009-08-16 09:27:22: (mod_deflate.c.1020) finished uri: /index.lighttpd.html , query:

3. Install and configure php for Lighttpd

Install the CGI version of PHP

$ sudo apt-get install php5-cgi
$ sudo lighty-enable-mod fastcgi

Place a php file named info.php file in your server root directory:

<?php phpinfo(); ?>

Now visit http://your.server/info.php and check the log:

2009-08-16 09:34:56: (mod_deflate.c.1232) Content-Type: text/html
2009-08-16 09:34:56: (mod_deflate.c.1239) mime-type: text/html
2009-08-16 09:34:56: (mod_deflate.c.1267) enable compression for  /info.php
... snip ...
2009-08-16 09:34:56: (mod_deflate.c.919)  in: 42054  out: 7057
2009-08-16 09:34:56: (mod_deflate.c.1020) finished uri: /info.php , query:

That's it. If it works, be sure to comment out deflate.debug in lighttpd.conf as it's generating a fairly large amount of entries in your server log.

Caveats

mod_deflate doesn't support caching compressed files in a directory like mod_compress does. This means that every http request matching a configured mime-type will trigger mod_deflate compression. This usually produces higher cpu utilization in favor of lower bandwidth usage. If you experience high cpu load, try lowering the deflate.compression-level.