• Qmail IPv6 support
  • tcpserver and rblsmtpd
  • sslserver
  • jgreylist
  • jms1 combined patch

ucspi-ssl

The ucspi-ssl package provides the sslserver program. sslserver accepts incoming SSL connections and passes them to another program such as qmail-smtpd. sslserver is almost identical to tcpserver except that it deals with encrypted SSL traffic rather than clear text.

I could not find an IPv6 patch for sslserver, however I was able to port the tcpserver patch to sslserver. You can easily apply my ucspi-ssl-0.70-ipv6.patch:

sslserver

To understand what the sslserver portion of the IPv6 patch does, you should be familiar with the sslserver man page and read about how tcpserver handles IPv6 on Fefe’s ucspi-tcp page. Essentially, if a client connects via IPv4, sslserver exhibits it’s normal behavior. If a client connects with IPv6, the PROTO environment variable will be set to “SSL6” instead of “SSL“.

The patch also supports using IPv6 addresses in your tcprules files. A new rule may look like this:

After applying this patch, you may notice that your logs are filled with addresses similar to: ::ffff:192.168.1.1. This is because internally sslserver treats every IP as an IPv6 address. IPv4 addresses are represented using their IPv4 mapped address.

Summary

If you have any comments, corrections, or questions, please feel free to leave a comment below. Remember Gentoo users can apply all of my qmail patches automatically by using the ebuilds in my gentoo-overlay.

Once you’ve patched sslserver, you should move on to patching jgreylist for IPv6.

• Qmail IPv6 support
  • tcpserver and rblsmtpd
  • sslserver
  • jgreylist
  • jms1 combined patch

ucspi-tcp

The ucspi-tcp package provides the tcpserver and rblsmtpd programs. tcpserver accepts incoming TCP connections and passes them to another program such as qmail-smtpd. rblsmtpd blocks connections from RBL listed IPs.

Thanks to Fefe, a patch has been around for a while that adds IPv6 support to tcpserver. Fefe’s patch does not touch rblsmtpd, however.

I’ve modified Fefe’s patch to patch rblsmtpd as well. You can easily apply my ucspi-tcp-0.88-ipv6.patch:

tcpserver

To understand what the tcpserver portion of the IPv6 patch does, you should read Fefe’s ucspi-tcp page. Essentially, if a client connects via IPv4, tcpserver exhibits it’s normal behavior. If a client connects with IPv6, the PROTO environment variable will be set to “TCP6“.

The patch also supports using IPv6 addresses in your tcprules files. A new rule may look like this:

After applying this patch, you may notice that your logs are filled with addresses similar to: ::ffff:192.168.1.1. This is because internally tcpserver treats every IP as an IPv6 address. IPv4 addresses are represented using their IPv4 mapped address.

rblsmtpd

When patching rblsmtpd for IPv6 support, I had to decide how to lookup IPv6 addresses. As far as I know, there aren’t any IPv6 blacklists yet. There isn’t a spec on how these addresses should be queried. My patch will use a new namespace, ipv6, when querying RBLs as described here. This means if you connect via 2001:470:1f0f:350::1, a TXT DNS lookup will be made to:

If anyone knows of a working RBL that differs, please let me know.

Summary

If you have any comments, corrections, or questions, please feel free to leave a comment below. Remember Gentoo users can apply all of my qmail patches automatically by using the ebuilds in my gentoo-overlay.

Once you’ve patched tcpserver, you should move on to patching ucspi-ssl (sslserver) for IPv6.

I’m not trying to defend IPv6. I realize there are many people with strong feelings towards the subject, including qmail’s author. Switching to IPv6 is a monumental task. It may never happen, but something needs to – we can’t keep NATing forever.

Many software projects have already added support for IPv6. My Gentoo box has been on an IPv6 network, via Hurricane Electric’s free tunnel broker service for a while now. Mac OS X has support for IPv6, as do the latest versions of Windows. Even Windows XP can support IPv6 if enabled. Postfix, Exim, and Sendmail all support IPv6.

In this series of posts, I will outline the steps I took to add IPv6 support to qmail. I use John Simpson’s combined patch for qmail as well as many other tools and methodologies described on his site, however many of the patches and instructions in these posts will work for other versions of qmail as well.

Components

John Simpson has an excellent illustration of a typical qmail system, Anatomy of a typical qmail system [PDF], on his website. The following articles describe the steps I took to enable IPv6 for each of the necessary components:

• tcpserver and rblsmtpd
• sslserver
• jgreylist
• jms1 combined patch

For Gentoo users, the patches described in each of the above posts can be installed automatically using the ebuilds in my gentoo-overlay. For others, I’ve listed all the IPv6 patches on my qmail patches page.

Testing

Testing your IPv6 enabled qmail setup can be a little confusing. There aren’t that many IPv6 enabled mail servers out there. Even worse, most people don’t have IPv6 connections.

Hurricane Electric provides a free IPv6 tunnel broker service that will allocate a /64 block of addresses that you can use. I host my personal mail server on a Linode which, despite being an excellent VPS, doesn’t have native IPv6. To get around this I set up a tunnel broker and enabled AAAA entries in DNS.

To test my setup, I had to install two separate qmail installs on different servers. Email addresses on my bltweb.net domain are now IPv6 enabled. If you’d like to use them to test, feel free to shoot me an email. Perhaps one day I’ll set up some type of reflector to automatically test.

IPv6 email experience

I’ve been running IPv6 mail servers at home and work for a few months now. I haven’t been keeping detailed statistics, but for the most part the only connections I’ve seen over IPv6 thus far have been spam

Still, enabling IPv6 in qmail wasn’t as hard as I thought it was going to be, thanks to the pre-existing patches on the internet. Hopefully more and more companies will start to enable IPv6 on their networks, such as Netflix. While email may still be even further out it never hurts to be ready.

Hopefully these posts have helped you add IPv6 support to your qmail install. Feel free to leave comments or questions below and I’ll do my best to address them.

Benchmark Setup

To compare mod_php to FastCGI with a PHP opcode cache, I used the ab tool on my desktop running PHP 5.3.0, mod_fastcgi 2.4.6 and Apache 2.2.11 on Gentoo linux. Apache had a limited number of modules enabled (actions alias authz_host dav deflate dir expires filter headers log_config mime rewrite setenvif status vhost_alias). The hardware consisted of a Intel Core2 Quad Q6600 (2 x 4MB L2 cache) with 2GB of RAM.

I chose to benchmark against the main page of a stock WordPress 2.8.2 install with no plugins enabled. I chose WordPress because it’s extremely easy to set up and many people are familiar with it. I’ve read that WordPress is very liberal with queries to the database (especially when plugins are involved). Without database queries, I would expect to see a bigger difference between results with an opcode cache and results without one. That being said, most PHP applications today make database queries as part of a typical request and thus I think these results are realistic of what you might experience on your server.

I executed the following command:

in four different environments:

• mod_php with a 30MB APC opcode cache (Apache MPM prefork)
• mod_php without an opcode cache (Apache MPM prefork)
• mod_fastcgi, suEXEC with a 30MB APC opcode cache (Apache MPM worker)
• mod_fastcgi, suEXEC without an opcode cache (Apache MPM worker)

Between each benchmark, I reconfigured and restarted the server. I then loaded the main page in a web browser to prime the APC cache.

Benchmark Results

Requests

Requests per second

The graph above shows that mod_php and FastCGI behaved very similar in terms of requests per second. Using an opcode cache increased the request rate by almost 3 times in both scenarios. As I mentioned earlier, I’d expect to see an even bigger increase with fewer database queries.

I saw similar results when increasing the number of requests to 10,000, or changing the concurrency level. I never saw any failed requests. I was happy to see there wasn’t a significant performance difference moving from mod_php to FastCGI.

Memory Usage

Memory usage

The graph above shows the approximate memory usage for each of the 4 benchmarking scenarios. This data was gathered by running the ps_mem.py script immediately after the ab tool completed. This script outputs the private and shared memory usage for each program running on the system. Because I ran my benchmarks in isolation, with nothing else accessing the host, these results should be pretty accurate.

The results surprised me. From the graph you can see that disabling the opcode cache resulted in more memory usage. I think this is misleading. Typically, Linux keeps memory resident until it is needed by another process. I believe the increased memory usage for the non-opcode cache scenarios is actually discarded memory that the Linux kernel hasn’t reclaimed. Since each process is having to recompile the PHP script on each request, you see a lot more built up memory reported as private but most likely free to be reused. I suspect running this on a low memory server would yield slightly different results.

You can clearly see the difference in memory usage for the Apache processes (green in the graph) between FastCGI and mod_php. The FastCGI Apache processes are much leaner. Since Apache doesn’t need the PHP bloat, it can serve static content much more effectively and quickly.

The scenarios with an opcode cache each used a similar amount of RAM. There is a catch to these results however: I only benchmarked a single user. In a shared environment, each user gets their own PHP instance and opcode cache. With mod_php, all users share the same. Thus I would expect every user to roughly double the RAM used in a FastCGI scenario, while mod_php memory usage would stay relatively constant. This increased memory usage is the main trade-off when going with FastCGI.

Raw Results

For those who want even more detail, below are the raw results from my benchmark tests:

mod_php without an opcode cache:

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)

Server Software:        Apache
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        5474 bytes

Concurrency Level:      30
Time taken for tests:   28.507 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      5717000 bytes
HTML transferred:       5474000 bytes
Requests per second:    35.08 [#/sec] (mean)
Time per request:       855.200 [ms] (mean)
Time per request:       28.507 [ms] (mean, across all concurrent requests)
Transfer rate:          195.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:   138  849 214.5    842    1608
Waiting:      138  848 214.3    842    1601
Total:        139  849 214.4    843    1608

Percentage of the requests served within a certain time (ms)
  50%    843
  66%    911
  75%    975
  80%   1004
  90%   1112
  95%   1230
  98%   1355
  99%   1426
 100%   1608 (longest request)

mod_php with opcode cache:

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)

Server Software:        Apache
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        5474 bytes

Concurrency Level:      30
Time taken for tests:   10.784 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      5717000 bytes
HTML transferred:       5474000 bytes
Requests per second:    92.73 [#/sec] (mean)
Time per request:       323.530 [ms] (mean)
Time per request:       10.784 [ms] (mean, across all concurrent requests)
Transfer rate:          517.70 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:    66  319  62.9    316     580
Waiting:       66  318  62.9    316     580
Total:         67  319  62.8    316     580

Percentage of the requests served within a certain time (ms)
  50%    316
  66%    339
  75%    354
  80%    364
  90%    395
  95%    424
  98%    464
  99%    482
 100%    580 (longest request)

FastCGI without an opcode cache:

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)

Server Software:        Apache
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        5474 bytes

Concurrency Level:      30
Time taken for tests:   28.170 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      5695000 bytes
HTML transferred:       5474000 bytes
Requests per second:    35.50 [#/sec] (mean)
Time per request:       845.104 [ms] (mean)
Time per request:       28.170 [ms] (mean, across all concurrent requests)
Transfer rate:          197.43 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:   105  832  82.9    832    1004
Waiting:      105  832  82.9    832    1004
Total:        107  832  82.8    832    1004

Percentage of the requests served within a certain time (ms)
  50%    832
  66%    856
  75%    873
  80%    882
  90%    908
  95%    927
  98%    954
  99%    971
 100%   1004 (longest request)

FastCGI with opcode cache:

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)

Server Software:        Apache
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        5474 bytes

Concurrency Level:      30
Time taken for tests:   10.917 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      5695000 bytes
HTML transferred:       5474000 bytes
Requests per second:    91.60 [#/sec] (mean)
Time per request:       327.509 [ms] (mean)
Time per request:       10.917 [ms] (mean, across all concurrent requests)
Transfer rate:          509.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:    46  323  36.6    325     426
Waiting:       46  323  36.6    325     426
Total:         47  323  36.5    325     426

Percentage of the requests served within a certain time (ms)
  50%    325
  66%    335
  75%    342
  80%    346
  90%    359
  95%    368
  98%    383
  99%    389
 100%    426 (longest request)

Memory Usage (in MB):

Conclusion

I run my blog on a Linode VPS with a very limited amount of RAM. I also host other applications and a few friends’ sites on this same server. Thus, I am very concerned about memory usage. Still, the security advantages provided by FastCGI far outweigh the increased RAM usage.

I hope you have seen that the performance of FastCGI with an opcode cache is just as good as with mod_php. It’s also evident how important an opcode cache is to running PHP scripts.

If you have any thoughts about these benchmarks, or the performance of FastCGI vs mod_php, feel free to leave a comment or question below!

This is a long post, ready to jump right in? Skip the history!

The evolution of mod_php to FastCGI

In the early days of all-you-can eat shared hosting, administrators served PHP via mod_php. mod_php loads the PHP interpreter into every web server process during server startup, thus alleviating the expense of starting an interpreter each time a script executes. This allowed executing PHP scripts relatively fast.

mod_php came with a few drawbacks:

• Every server process, even those serving static files such as images and CSS scripts, contained the PHP interpreter. This caused a lot of bloat in the web server’s memory footprint. It also eliminated the ability to use mutil-threaded web servers as many PHP extensions are not thread safe.
• Every PHP script ran as the same user as the web server. While web servers typically run as a non-privileged user such as nobody, multiple mutually untrusting shared accounts could easily access, disrupt or destroy each other by executing a PHP script.

FastCGI loads the PHP interpreter into a separate process. This process is still persistent across connections, but, using a mechanism such as suEXEC, can run as a different user. Static files can be served by a lightweight multi-threaded web server process while PHP scripts are served by a single-threaded FastCGI process. What’s more, if PHP crashes, it doesn’t bring down the entire web server.

In the shared hosting context, each user’s PHP scripts are executed with the user’s credentials. This leads to a more secure environment for both the host and the shared user.

The opcode cache

One of the easiest and most effective things you can do to speed up your PHP scripts is to enable an opcode cache such as APC, XCache or eAccelerator. An opcode cache caches the compiled state of PHP scripts in shared memory. Thus each time a PHP script is run, the server doesn’t have to waste time compiling the source code. Opcode caches can speed up execution of scripts by up to 5 times and decrease server load.

In my opinion running PHP on a webserver without an opcode cache is like restarting your car’s engine at every stop sign. You can still get where you’re going but it’s going to take longer and put a lot more wear and tear on your engine. An opcode cache is so important that APC is going to be included in the core of PHP 6.

An opcode cache requires that the PHP interpreter process persist between connections. Both mod_php and FastCGI satisfy this requirement. An opcode cache requires RAM, a precious commodity on a shared hosting server. By default, each cache allocates 30MB of shared memory. This can be easily configured up or down depending on the scripts you are running.

Combining FastCGI with an opcode cache

So if we agree that FastCGI and opcode caches are good (a must IMHO), why do most shared hosting providers only enable one? The answer is two-fold:

1. RAM. Each opcode cache is typically 30MB. Each PHP process gets its own opcode cache. Each user must run its own PHP process for security. Thus each user requires at least 30MB of RAM on top of the RAM required for the PHP interpreter (a lot). All you can eat shared hosting companies typically oversell their servers. Overselling usually works when it comes to bandwidth, I/O and CPU time, however overselling RAM is harder. Remember the PHP processes stay in memory between connections. So a small site only getting a 100 hits a day still hogs the same amount of RAM as a busy site. This breaks the overselling model.
2. FastCGI. In a typical configuration, FastCGI spawns many separate PHP processes per user. Each PHP process needs its own opcode cache. Instead of maintaining one opcode cache (per user), the server maintains multiple caches. This reduces the effectiveness of the cache and increases the strain on server resources.

Solving problem #1 is hard. Some have suggested a single cache that can be shared across multiple processes and users and still provide assurance that different users cannot mess with each other. This blog post is not about #1. There are many reasons to use unlimited shared hosting providers. Opcode caches are not one.

This blog post is about how to solve problem #2. The goal is to have a reasonable system that utilizes suEXEC, FastCGI and the APC opcode cache. Each user should have one and only one opcode cache. The administrator should be able to adjust the size of the cache for each individual user based on their needs (and monthly fee). Finally, the solution should decrease script load time and increase server performance while maintaining security and privacy between accounts.

mod_fastcgi vs mod_fcgid

I run Apache on my server. Many people suggest running a more lightweight server such as lighttpd. One day I may switch, but for now I’ve tuned my Apache server to be as fast as I need.

There are two modules to implement FastCGI on Apache – mod_fastcgi and the newer mod_fcgid. Both are binary compatible with each other and do basically the same thing. mod_fcgid sports better control over spawning processes and error detection. mod_fastcgi has been around longer. Both support suEXEC, and both separate PHP from Apache, thus allowing Apache to run threaded workers if desired.

As I mentioned in the combining FastCGI with an opcode cache section, the typical behavior of FastCGI is to spawn multiple PHP interpreters. The FastCGI process monitors each child process, kicking out processes with errors, restarting failed processes and sending incoming requests to the least busy child. This is usually the preferred behavior, and mod_fcgid implements it particularly well.

Opcode caches throw a wrench in this however, because of their inability to share the cache across FastCGI processes. Hopefully one day this will be remedied. Luckily, in the meantime, PHP is capable of playing “process manager” and a single PHP process can spawn several children to handle requests. This way the parent PHP process can instantiate the opcode cache and its children can share it. You’ll see this later when we set the PHP_FCGI_CHILDREN environment variable.

Both mod_fcgid and mod_fastcgi can be told to limit the number of PHP processes to 1 per user. The PHP process can then be told how many children to spawn. Unfortunately mod_fcgid will only send one request per child process. The fact that PHP spawns its own children is ignored by mod_fcgid. If we use mod_fcgid with our setup, we can only handle one concurrent PHP request. This is not good. A long running request could easily block multiple smaller requests.

mod_fastcgi will send multiple simultaneous requests to a single PHP process if the PHP process has children that can handle it. This is the reason we must use mod_fastcgi to achieve our goal of one cache per user.

Implementation

This section describes the steps I took to enable suEXEC FastCGI with a single APC opcode cache per user on Apache 2.2. These instructions may vary by Linux distribution and are not intended to be a cut-and-paste howto. I use Gentoo, so most steps will be geared towards a Gentoo install but the general idea should work on any distribution.

1. Install php-cgi and disable mod_php

The PHP interpreter can run in three different modes: as an Apache module, as a CGI binary or as a command line command. Typically, separate binaries are built for the CGI and CLI modes, php-cgi and php respectively. On Gentoo, each mode is associated with a USE flag: apache2 for mod_php, cgi for a CGI binary, and cli for command-line PHP. The cgi USE flag must be enabled. If it isn’t, add it to /etc/make.conf or /etc/portage/package.use and recompile PHP. On other distributions, search for a php-cgi binary.

You will want to disable mod_php (if it was enabled) before implementing FastCGI. This can be done by commenting out the appropriate LoadModule line in your Apache configuration file:

On Gentoo, this can be easily done by removing PHP5 from the APACHE2_OPTS variable in /etc/conf.d/apache2.

2. Install and enable mod_fastcgi Apache module

We already discussed why we must use mod_fastcgi instead of mod_fcgid. On Gentoo, installing mod_fastcgi can easily be done by running:

For other distributions, try installing a mod_fastcgi package or see the FastCGI Installation Notes.

Make sure your Apache conf file contains the line:

On Gentoo, this line is found in /etc/apache/modules.d/20_mod_fastcgi.conf. mod_fastcgi is enabled by adding FASTCGI to the APACHE2_OPTS variable in /etc/conf.d/apache2.

3. Install and configure the APC Opcode Cache

To install APC on Gentoo, simply run:

For other distributions, see the Alternative PHP Cache installation instructions.

Once installed, look for the apc.ini file in your php extension configuration directory (e.g. /etc/php/cgi-php5/ext-active). The default apc.ini works with one exception. You need to comment out apc.shm_size="30" (line 5 below). Commenting this line will enable us to set it per user later.

My apc.ini file looks like:

extension=apc.so
apc.enabled="1"
apc.shm_segments="1"
;commenting this out allows you to set it in each fastcgi process
;apc.shm_size="30"
apc.num_files_hint="1024"
apc.ttl="7200"
apc.user_ttl="7200"
apc.gc_ttl="3600"
apc.cache_by_default="1"
;apc.filters=""
apc.mmap_file_mask="/tmp/apcphp5.XXXXXX"
apc.slam_defense="0"
apc.file_update_protection="2"
apc.enable_cli="0"
apc.max_file_size="1M"
apc.stat="1"
apc.write_lock="1"
apc.report_autofilter="0"
apc.include_once_override="0"
apc.rfc1867="0"
apc.rfc1867_prefix="upload_"
apc.rfc1867_name="APC_UPLOAD_PROGRESS"
apc.rfc1867_freq="0"
apc.localcache="0"
apc.localcache.size="512"
apc.coredump_unmap="0"

4. Install/enable Apache suEXEC

Apache 2.2 contains built-in support for executing CGI programs as a different user id and group id than the webserver. This support must be compiled into Apache. On Gentoo, use the suexec USE flag and recompile apache. On other distributions, see Configuring & Installing suEXEC.

5. Create wrapper scripts

The Apache suEXEC security model requires that the CGI binary meet some pretty stringent requirements concerning file ownership and permissions. Rather than copying the php-cgi binary for each user, we create multiple wrapper scripts around the php-cgi binary. These wrapper scripts allow us to set options on a per-user basis.

I keep my wrapper scripts in /var/www/bin, though you may keep yours wherever you want. Each user has a directory in /var/www/bin, for example:

Inside each user’s bin directory is a single wrapper script, php-fastcgi:

I’ve shown the ls -l output to show the file and directory ownership and permissions. These are important, and Apache suEXEC will not work correctly if the owner and permissions are not correct.

The contents of the php-fastcgi file in each user’s bin directory (see below for an explanation):

#!/bin/sh

PHP_FCGI_CHILDREN=5
export PHP_FCGI_CHILDREN
PHP_FCGI_MAX_REQUESTS=500
export PHP_FCGI_MAX_REQUESTS

umask 0022
exec /usr/bin/php-cgi -d apc.shm_size=25

PHP_FCGI_CHILDREN
This variable tells PHP how many child processes it should spawn. As we discussed earlier, our PHP process will act as “process manager” and pass incoming requests to its children. The parent will maintain a single opcode cache which each child will share. The PHP_FCGI_CHILDREN variable tells PHP how many children to spawn. Another way to think of this is the number of concurrent PHP requests that can be handled per user.

PHP_FCGI_MAX_REQUESTS
PHP is known for memory leaks in long running processes. This variable causes each child process to be restarted once it has served a given number of requests (e.g. 500). Only the child process is restarted, the parent process remains. Since the parent process maintains the opcode cache, the opcode cache persists.

umask 0022
This sets the umask the PHP binary will run under. Some people may prefer a stricter umask such as 0077, however I’ve found 0022 works best as it allows the Apache server running as nobody to read static files written earlier by a suEXEC’d PHP process. Some PHP applications (WordPress plugins) do not do a good job with permissions, and a strict umask can cause applications to fail.

exec /usr/bin/php-cgi -d apc.shm_size=25
This line calls the php-cgi binary and modifies the APC cache size. It is possible to configure this to use a separate php.ini file instead of setting configuration parameters on the command line, however I like the ability to share a single php.ini file.

6. Edit global Apache settings

There are two sets of settings you must configure in Apache: those that affect all users and those that affect a specific user. This section describes global settings that affect all users.

I like to keep my global settings in my /etc/apache/modules.d/20_mod_fastcgi.conf file, but these can go in any part of your http.conf file. Most of the time you do not want this in a VirtualHost section. My global mod_fastcgi settings look like this (see below for an explanation):

<IfDefine FASTCGI>
LoadModule fastcgi_module modules/mod_fastcgi.so

FastCgiConfig -idle-timeout 20 -maxClassProcesses 1
FastCgiWrapper On

AddHandler php5-fcgi .php
Action php5-fcgi /cgi-bin/php-fastcgi

<Location "/cgi-bin/php-fastcgi">
   Order Deny,Allow
   Deny from All
   Allow from env=REDIRECT_STATUS
   Options ExecCGI
   SetHandler fastcgi-script
</Location>

</IfDefine>

FastCgiConfig
The FastCgiConfig configuration directive sets parameters for all dynamic FastCGI processes. The idle-timeout causes FastCGI to abort a request if there is no activity for more than 20 seconds. The maxClassProcesses option is very important: it tells FastCGI to only spawn one php-cgi process regardless of how many requests are pending. Remember that our PHP process will spawn its own children, so FastCGI only needs to spawn one. Until this APC bug is fixed, this is necessary to allow sharing the APC cache among children.

FastCgiWrapper
The FastCgiWrapper configuration directive is needed to allow suEXEC to work.

AddHandler / Action
The AddHandler and Action configuration directives tell Apache to handle all files ending in .php with the php-fastcgi script in cgi-bin. In the next step, you’ll see how we alias this cgi-bin directory for each individual user.

Location
The Location directive tells Apache how to handle requests to /cgi-bin/php-fastcgi. The Allow from env=REDIRECT_STATUS on line 13 prevents users from executing this script directly. With this line, the only way to execute php-fastcgi is by requesting a file ending in .php.

7. Edit per-user Apache settings

On my host, every virtual host is associated with one user. And every user has exactly one opcode cache. A single user can have multiple virtual hosts, but these virtual hosts share the same opcode cache.

For each virtual host, I add the following lines, customized for the user associated with that virtual host:

<VirtualHost *:80>
ServerName www.sue.bltweb.net
...
<IfModule mod_fastcgi.c>
   SuexecUserGroup sue sue
   Alias /cgi-bin/ /var/www/bin/sue/
</IfModule>
...
</VirtualHost>

When combined with the global apache settings and the wrapper scripts, this will launch the php-cgi binary using suEXEC to execute as the appropriate user and group whenever a .php file is requested.

There are several different ways to call the FastCGI binary. On my hosts, users don’t have access to their cgi-bin directory. The /var/www/bin directory is not accessible by ordinary users. This doesn’t have to be the case, the cgi-bin directory could be stored in the user’s directory. It is important to note that allowing the user to modify php.ini values allows them to modify their opcode cache size, which could have severe repercussions on RAM usage.

Pros and Cons

The implementation described above is only one of many ways to implement FastCGI and APC. In my opinion, it is the best way to meet my goals, but in this section I’ll try to outline some of the advantages and disadvantages of my setup.

Advantages

• Different users can have different APC cache sizes
• Multiple concurrent PHP requests can be handled simultaneously
• RAM usage is predictable as a product of the number of users on the host
• Server is better secured against attacks from the inside since PHP processes run as the user who owns the script
• Resource usage can be monitored since each user has a separate PHP process
• A PHP crash doesn’t mean an Apache crash. If a PHP process crashes it is restarted automatically.

Disadvantages

• The process manager built into mod_fastcgi isn’t used. One of the motivations behind mod_fcgid was to improve upon the process manager in mod_fastcgi.
• The newer mod_fcgid cannot be used as it will only send one request at a time to the PHP process, thus multiple requests can’t be handled simultaneously
• Maintaining separate opcode caches per user uses a considerably larger amount of RAM than a single opcode cache used with mod_php
• Users cannot alter php.ini files
• If a PHP script crashes it has potential to take down all of the PHP requests currently being processed for that user

Performance

In my next post I’ll try to cover RAM usage, performance, benchmarks, compatibility and best practices. This post is already way too long; I’m surprised you are even still reading it!

Stay tuned for more information on using FastCGI with a PHP APC opcode cache. In the meantime let me know what you think of this approach. Have you tried it? Know of a better way? Found any bugs or problems? Leave a comment below!

Support for DomainKeys in qmail has existed for a while thanks to a patch by Russel Nelson. Kyle Wheeler created a set of wrapper scripts that can be used to provide support for DKIM and DomainKeys. Mihai Secasiu has some wrapper scripts similar to Kyle’s that provide support for DKIM via the libdkim library instead of Perl’s Mail::DKIM module.

The current methods take different approaches to implement DKIM and DomainKeys. The DomainKeys patch creates a single program, qmail-dk that is called before qmail-queue. This program signs or verifies all incoming messages (that may later become outbound) based on the existence of the DKSIGN and DKVERIFY variables. The DKIM wrapper scripts wrap qmail-remote to sign messages and wrap qmail-queue (or qmail-dk) to verify incoming messages. This can be easier understood by looking at the qmail big picture.

I tend to agree with separate programs for signing outbound messages and verifying inbound messages as this allows signing all outbound messages, even those (such as NDRs) that never pass through qmail-queue. I also prefer patching qmail as it tends to be a little easier and requires less configuration after qmail is installed.

In this post I will show you how to patch qmail to support DKIM as well as DomainKeys. My qmail DKIM/DomainKeys patch uses neither Russel Nelson’s DomainKeys patch nor Kyle Wheeler’s DKIM/DomainKey wrappers, but borrows ideas from both. My patch uses the libdomainkeys and libdkim libraries to do the actual signing and verifying. Rather than creating two new programs, I patch qmail-smtpd (for verifying) and qmail-remote (for signing) directly.

I’ll do my best to provide step by step instructions for patching and installing for you non-Gentoo users, but in my next post I’ll share my ebuild which does it all for you.

1. Install libdomainkeys

The libdomainkeys library is used to sign and verify DomainKeys signatures.

2. Install libdkim

The libdkim library is used to sign and verify DKIM signatures. You’ll need g++ to compile this on your system. The library claims to be portable, but I needed to patch it to get it to compile on my Gentoo box. I’ve also included a (slightly modified) patch from Mihai Secasiu that makes working with libdkimtest much easier.

3. Patch and install qmail

I’m currently using John Simpson’s qmail Combined Patch Set for my qmail installation. The instructions below highlight how to apply my DKIM/DomainKeys patch on top of John’s combined patch. I’d highly recommend checking out John’s combined patch as it is about as close as you can get to an actively maintained qmail.

I’m not attempting to describe or document John’s patch in anyway in this post, as John runs an excellent site about qmail (qmail.jms1.net) that contains far more information than is contained here. Do not attempt to proceed without reading through John’s documentation as well as the rest of this post.

4. Configure DKIM/DomainKeys signing

Signing is done by qmail-remote and is controlled by the dksign control file. Signatures are created using a private key on your system, and verified by a public key stored in the DNS for the email domain.

Generate keys

Before you can sign an email, you must create at least one public/private key pair. You should create key pairs for every domain you wish to sign. To create keys for example.com:

It is very important that the default file be readable only by root and the group which qmailr (the qmail-remote user) belongs to. This is the private key used for signing messages and, if compromised, would allow others to sign messages as your domain.

Now add a TXT entry to the DNS for default._domainkey.example.com containing the quoted part in the /etc/domainkeys/example.com/default.pub. NOTE: You normally want to include the quotes!

Configure control files

Create a file /var/qmail/control/dksign containing one line:

The % will be replaced with the domain name in the From: header (or the Sender: header if it exists). If no file exists for the given domain, parent domains will be tried. For example if the message is from foo@bar.example.com, /etc/domainkeys/bar.example.com/default will be tested first. If the file does not exist, /etc/domainkeys/example.com/default will be tested. If no key can be found, the message will not be signed. If a key exists, but cannot be read or contains invalid data, the message will not be sent and will remain in the queue until the problem is fixed.

If you do not create the /var/qmail/control/dksign file, no messages will be signed.

Test outbound signing

Now that DKIM/DomainKeys signing is configured, you can test it by sending an email to sa-test (at) sendmail dot net. This reflector will reply (within seconds) to the envelope sender with a status of the DomainKeys and DKIM signatures.

If you experience problems, consult the qmail-remote man page or post a comment below and I’ll try to help.

5. Configure DKIM/DomainKeys verification

Verification is performed by qmail-smtpd and is controlled by the DKVERIFY environment variable. Messages are only verified if DKVERIFY is set and RELAYCLIENT is not set. You may control which IP addresses are verified using the tcpserver access file (sometimes stored in /etc/tcprules.d/tcp.qmail-smtp).

When verifying a message, the contents of DKVERIFY are checked against the status of the DomainKeys and DKIM results. Each test result is represented by a letter. DKVERIFY should contain a series of letters for DomainKeys results, a comma, and then a series of letters for the DKIM results. If the letter is uppercase, the message will be rejected (hard error). If the letter is lowercase, the message will be deferred (soft error). The DKVERIFY variable can be set but empty, in which case messages will be verified and an Authentication-Results: header will be added but all messages will be accepted regardless of status.

The letters for DomainKeys results are:

The letters for the DKIM results are:

I recommend a DKVERIFY value of DEGIJKfh,CGHIJMQRkl. This will only reject improperly formatted messages. Messages that don’t verify will still be allowed. I would advise against rejecting messages that don’t verify as there are still some problems with DomainKeys and DKIM (such as mailing lists). Rather than rejecting bad signatures, incorporate the Authentication-Results header into your broader spam prevention strategy.

The Authentication-Results header

All messages received by qmail-smtpd when DKVERIFY is set will add an Authentication-Results header to the incoming message. This header conforms to the IETF internet draft. Here’s an example from one of my emails:

Authentication-Results: bltweb.net; domainkeys=pass (ok); dkim=pass (ok)

6. Examples

Here are some examples to help you configure your box. Anything that normally should be private is made up.

Keys

For my bltweb.net domain name, here’s what my keys look like (these are not the actual keys installed on my system, those are private):

Signing Configuration

To sign emails for all domains for which I have a key in /etc/domainkeys, I set the control/dksign configuration file:

Verify Configuration

Here’s an example of my /etc/tcprules.d/tcp.qmail-smtp file. Make sure you regenerate the cdb file after editing your tcp.qmail-smtp file!

7. Finished

That’s it. You should now have a qmail installation capable of signing and verifying messages. More information is contained in the qmail-smtpd and qmail-remote man pages.

If you have any comments or find any bugs, please feel free to post a comment below.