Lighttpd

Using Drupal Imagecache with Lighttpd proxied Apache

Using Lighttpd as a proxy for Apache is a popular solution to achieve smaller web server memory footprint and to speed up websites. However things can get messy when it comes to Drupal and the brilliant Imagecache module. Imagecache transforms images using presets and then caches them.

An imagecache url is something like: http://domain.tld/sites/default/files/imagecache/presetname/photo.jpg This will take the the image placed at sites/default/files/photo.jpg and transforms it according to presetname. After succesful operation the image will be available as a static file at sites/default/files/imagecache/presetname/photo.jpg

No need to say, this functionality breaks when the frontend web server only forwards requests to php files so imagecache images won't show up and will not be generated.

One solution is: set up a separate subdomain to serve these images and redirect all imagecache images to this subdomain. This needs some simple LUA scripting, Lighttpd will forward an imagecache url if it hasn't created yet, and will serve it directly if it's already there.

Four steps to get this working:

Configure Lighttpd

Create this little LUA script:

attr = lighty.stat(lighty.env['physical.path'])
if (not attr) then
   cuthere = string.find(lighty.env['uri.authority'], '.', 1, true) + 1
   redirhost = string.sub(lighty.env['uri.authority'], cuthere)
   lighty.header["Location"] = "http://" .. redirhost  .. 
lighty.env["request.orig-uri"]
   return 302
end

This script will stat for the file requested and if not found, redirect the request to the main domain. Link the lua script to the vhost configuration (and enable mod_magnet of course if you haven't done already):

magnet.attract-physical-path-to = ( "/etc/lighttpd/filecheckredirect.lua" )

Set up the subdomain

Configure your subdomain or multiple subdomains for imagecache hosted images and set this in the Drupal settings.php:

/**
 * Alternate hosts for imagecache created images.
 * A string can be set for single hosts,
 * An array of hosts can be set for multiple hosts.
 */
$conf['static_file_hosts'] = array(
  'ic1.example.com',
  'ic2.example.com',
);
//$conf['static_file_hosts'] = 'ic.example.com';

Override imagecache theme function

/**
 * Replace the domain part of imagecache urls with 'static_file_hosts' if
 * available.
 *
 * 'static_file_hosts' can be set from settings.php.
 */
function YOURTHEME_imagecache($presetname, $path, $alt = '', $title = '', $attributes = NULL, $getsize = TRUE) {
  // Check is_null() so people can intentionally pass an empty array of
  // to override the defaults completely.
  if (is_null($attributes)) {
    $attributes = array('class' => 'imagecache imagecache-'. $presetname);
  }
  if ($getsize && ($image = image_get_info(imagecache_create_path($presetname, $path)))) {
    $attributes['width'] = $image['width'];
    $attributes['height'] = $image['height'];
  }
  $attributes = drupal_attributes($attributes);

  $domain = variable_get('static_file_hosts', $GLOBALS['base_url']);
  if (is_array($domain) && !empty($domain)) {
    if (count($domain) > 1) {
      $domain = $domain[rand(0, count($domain) - 1)];
    }
    else {
      $domain = $domain[0];
    }
  }
  if ($domain !== $GLOBALS['base_url']) {
    $base_url_parsed = parse_url($GLOBALS['base_url']);
    $imagecache_url = str_replace($GLOBALS['base_url'], $base_url_parsed['scheme'] . '://' . $domain, imagecache_create_url($presetname, $path));
  }
  else {
    $imagecache_url = imagecache_create_url($presetname, $path);
  }

  return '<img src="http://frontseed.com/%27.%20%24imagecache_url%20.%27" alt="'. check_plain($alt) .'" title="'. check_plain($title) .'" '. $attributes .' />';
}

download this snippet.

Patch imagecache:

The patch below is for imagecache version 6.x-2.0-beta10:

--- imagecache.module   19 Aug 2009 20:59:07 -0000  1.112.2.5
+++ imagecache.module 
@@ -318,7 +318,15 @@
   $args = array('absolute' => TRUE, 'query' => empty($bypass_browser_cache) ? NULL : time());
   switch (variable_get('file_downloads', FILE_DOWNLOADS_PUBLIC)) {
     case FILE_DOWNLOADS_PUBLIC:
-      return url($GLOBALS['base_url'] . '/' . file_directory_path() .'/imagecache/'. $presetname .'/'. $path, $args);
+      $domain = variable_get('static_file_hosts', $GLOBALS['base_url']);
+      if ($domain !== $GLOBALS['base_url']) {
+        if (is_array($domain) && !empty($domain)) {
+          $domain = count($domain) > 1 ? $domain[rand(0, count($domain) - 1)] : $domain[0];
+          $base_url_parsed = parse_url($GLOBALS['base_url']);
+          $domain = $base_url_parsed['scheme'] . '://' . $domain;
+        }
+      }
+      return url($domain . '/' . file_directory_path() .'/imagecache/'. $presetname .'/'. $path, $args);
     case FILE_DOWNLOADS_PRIVATE:
       return url('system/files/imagecache/'. $presetname .'/'. $path, $args);
   }

Final steps

Empty the theme registry and watch your imagecache files served from ic.example.com using Apache the first time and using Lighttpd after image generation.

Update

The solution above is only suitable in scenarios where a reverse proxy is used for static file handling. I believe that planting a Lighttpd in front of Apache is not the best way to serve a Drupal page. According to my experience it is better to run either nginx as a reverse proxy or just have nginx handle all your pages via php-fpm. Or you can strip down Apache and use mpm_worker with php in fastcgi mode, however that is not my cup of tea either.

Installing Lighttpd with mod_deflate on Ubuntu Hardy

Meet Lighttpd: it is a fast, small footprint and higly configurable webserver with advanced features. It's often a common alternative to Apache. At the time of writing the latest versions from the 1.4.x branch is considered stable while Lighttpd 1.5.0 is an ongoing work with new features.

One new feature is the mod_deflate server module which is a modified version of mod_compress shipped with 1.4.x versions. mod_deflate can compress any output from lighttpd static or dynamic - that is, lighttpd can compress the output of dynamically generated pages with this.

Websites with lots of dynamically generated content receive significant bandwidth saving with dynamic output compression, and visitors of those websites benefit faster page loading times.

There are patches available which makes building a version of Lighttpd 1.4.x from source which includes mod_deflate.

1. Building and packaging Lighttpd on Ubuntu

Before building on an Ubuntu hardy system, some prerequisites need to be installed. After getting lighttpd source, it needs to be patched to support mod_deflate. Standard debian packages are to be built which can be easily installed on any target system (assuming the same OS architecture)

Install tools for building and get the Lighttpd source from the package repository:

$ sudo apt-get install build-essential fakeroot
$ sudo apt-get install debhelper cdbs libssl-dev libbz2-dev libattr1-dev libpcre3-dev libmysqlclient15-dev libgamin-dev libldap2-dev libfcgi-dev libgdbm-dev libmemcache-dev liblua5.1-0-dev dpatch patchutils pkg-config uuid-dev libsqlite3-dev libxml2-dev libkrb5-dev
$ mkdir ~/lighttpd
$ cd ~/lighttpd
$ apt-get source lighttpd

Download the patch and apply it:

$ wget http://redmine.lighttpd.net/attachments/download/632/lighttpd-1.4.19.mod_deflate.patch
$ cd lighttpd-1.4.19
$ patch -p1 < ../lighttpd-1.4.19.mod_deflate.patch

Build and install the package:

$ fakeroot dpkg-buildpackage

After building completes you will see the freshly built deb packages in the parent directory:

$ cd ..
$ ls -lgG
total 1336
drwxr-xr-x 8   4096 Aug 16 08:05 lighttpd-1.4.19
-rw-r--r-- 1  66387 Oct  8  2008 lighttpd-1.4.19.mod_deflate.patch
-rw-r--r-- 1  70422 Aug 16 08:07 lighttpd-doc_1.4.19-0ubuntu3_all.deb
-rw-r--r-- 1  11536 Aug 16 08:08 lighttpd-mod-cml_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1  10958 Aug 16 08:08 lighttpd-mod-magnet_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1   6918 Aug 16 08:08 lighttpd-mod-mysql-vhost_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1   8492 Aug 16 08:08 lighttpd-mod-trigger-b4-dl_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1  19002 Aug 16 08:08 lighttpd-mod-webdav_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1  38501 Aug 16 08:03 lighttpd_1.4.19-0ubuntu3.diff.gz
-rw-r--r-- 1   1106 Aug 16 08:03 lighttpd_1.4.19-0ubuntu3.dsc
-rw-r--r-- 1   2433 Aug 16 08:08 lighttpd_1.4.19-0ubuntu3_i386.changes
-rw-r--r-- 1 267770 Aug 16 08:08 lighttpd_1.4.19-0ubuntu3_i386.deb
-rw-r--r-- 1 815568 Mar 12  2008 lighttpd_1.4.19.orig.tar.gz

Lighttpd depends on libterm-readline-perl-perl so install this as well:

$ sudo apt-get install libterm-readline-perl-perl

If apt-get is whining about libterm-readkey-perl as a dependency try:

$ sudo apt-get -f install libterm-readkey-perl libterm-readline-perl-perl

Then you can install lighttpd from the freshly built deb package:

$ sudo dpkg -i lighttpd_1.4.19-0ubuntu3_i386.deb

If you need additional packages, install them as well:

$ sudo dpkg -i lighttpd-mod-magnet_1.4.19-0ubuntu3_i386.deb
$ sudo dpkg -i lighttpd-mod-webdav_1.4.19-0ubuntu3_i386.deb

Copy mod_deflate module to its place:

$ sudo cp -d ~/lighttpd/lighttpd-1.4.19/debian/tmp/usr/lib/lighttpd/mod_deflate.so* /usr/lib/lighttpd/

2. Configuring mod_deflate

Load the module in /etc/lighttpd/lighttpd.confand comment out mod_compress:

    server.modules = (
                "mod_access",
                "mod_alias",
                "mod_accesslog",
    #           "mod_compress",
                "mod_deflate",
    #           "mod_rewrite",
    #           "mod_redirect",
    

Also comment out mod_compress configuration directives:

    #### compress module
    #compress.cache-dir          = "/var/cache/lighttpd/compress/"
    #compress.filetype           = ("text/plain", "text/html", "application/x-javascript", "text/css")

Configure the mod_deflate module:

    # mod_deflate settings
    deflate.enabled = "enable"
    deflate.compression-level = 9
    deflate.mem-level = 9
    deflate.window-size = 15
    deflate.bzip2 = "enable"
    deflate.min-compress-size = 200
    #deflate.sync-flush = "enable"
    deflate.output-buffer-size = 4096
    deflate.work-block-size = 512
    deflate.mimetypes = ("text/html", "text/plain", "text/css", "text/javascript", "text/xml")
    deflate.debug = "enable"

Detailed information about configuration options is available on the Lighttpd mod_deflate documentation page. The last line is only to check if mod_deflate is working.

Fire up Lighttpd:

$ sudo /etc/init.d/lighttpd start

Visit a static page: http://your.server/index.lighttpd.html and check the lighttpd error log at /var/log/lighttpd/error.log:

2009-08-16 09:27:22: (mod_deflate.c.1232) Content-Type: text/html
2009-08-16 09:27:22: (mod_deflate.c.1239) mime-type: text/html
2009-08-16 09:27:22: (mod_deflate.c.1267) enable compression for  /index.lighttpd.html
2009-08-16 09:27:22: (mod_deflate.c.1283) add Vary: Accept-Encoding for  /index.lighttpd.html
2009-08-16 09:27:22: (mod_deflate.c.305) output-buffer-size: 4096
2009-08-16 09:27:22: (mod_deflate.c.307) compression-level: 9
2009-08-16 09:27:22: (mod_deflate.c.309) mem-level: 9
2009-08-16 09:27:22: (mod_deflate.c.311) window-size: -15
2009-08-16 09:27:22: (mod_deflate.c.313) min-compress-size: 200
2009-08-16 09:27:22: (mod_deflate.c.315) work-block-size: 512
2009-08-16 09:27:22: (mod_deflate.c.1350) Compress all content and use Content-Length header: uncompress len= 3574
2009-08-16 09:27:22: (mod_deflate.c.948) compress: in_queue len= 3574
2009-08-16 09:27:22: (mod_deflate.c.881) compress file chunk: offset= 0 , toSend= 3574
2009-08-16 09:27:22: (mod_deflate.c.351) gzip_header len= 10
2009-08-16 09:27:22: (mod_deflate.c.386) compress: in= 3574 , out= 0
2009-08-16 09:27:22: (mod_deflate.c.1003) compressed bytes: 3574
2009-08-16 09:27:22: (mod_deflate.c.451) flush: in= 0 , out= 1555
2009-08-16 09:27:22: (mod_deflate.c.489) gzip_footer len= 8
2009-08-16 09:27:22: (mod_deflate.c.919)  in: 3574  out: 1563
2009-08-16 09:27:22: (mod_deflate.c.1020) finished uri: /index.lighttpd.html , query:

3. Install and configure php for Lighttpd

Install the CGI version of PHP

$ sudo apt-get install php5-cgi
$ sudo lighty-enable-mod fastcgi

Place a php file named info.php file in your server root directory:

<?php phpinfo(); ?>

Now visit http://your.server/info.php and check the log:

2009-08-16 09:34:56: (mod_deflate.c.1232) Content-Type: text/html
2009-08-16 09:34:56: (mod_deflate.c.1239) mime-type: text/html
2009-08-16 09:34:56: (mod_deflate.c.1267) enable compression for  /info.php
... snip ...
2009-08-16 09:34:56: (mod_deflate.c.919)  in: 42054  out: 7057
2009-08-16 09:34:56: (mod_deflate.c.1020) finished uri: /info.php , query:

That's it. If it works, be sure to comment out deflate.debug in lighttpd.conf as it's generating a fairly large amount of entries in your server log.

Caveats

mod_deflate doesn't support caching compressed files in a directory like mod_compress does. This means that every http request matching a configured mime-type will trigger mod_deflate compression. This usually produces higher cpu utilization in favor of lower bandwidth usage. If you experience high cpu load, try lowering the deflate.compression-level.