<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" ""> <html> <head> <title>Postfix Stress-Dependent Configuration</title> <meta http- </head> <body> <h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix Stress-Dependent Configuration</h1> <hr> <h2>Overview </h2> <p> This document describes the symptoms of Postfix SMTP server overload. It presents permanent <a href="postconf.5.html">main.cf</a> changes to avoid overload during normal operation, and temporary <a href="postconf.5.html">main.cf</a> changes to cope with an unexpected burst of mail. This document makes specific suggestions for Postfix 2.5 and later which support stress-adaptive behavior, and for earlier Postfix versions that don't. </p> <p> Topics covered in this document: </p> <ul> <li><a href="#overload"> Symptoms of Postfix SMTP server overload </a> <li><a href="#concurrency"> Service more SMTP clients at the same time </a> <li><a href="#time"> Spend less time per SMTP client </a> <li><a href="#hangup"> Disconnect suspicious SMTP clients </a> <li><a href="#legacy"> Temporary measures for older Postfix releases </a> <li><a href="#adapt"> Automatic stress-adaptive behavior </a> <li><a href="#feature"> Detecting support for stress-adaptive behavior </a> <li><a href="#forcing"> Forcing stress-adaptive behavior on or off </a> <li><a href="#other"> Other measures to off-load zombies </a> <li><a href="#credits"> Credits </a> </ul> <h2><a name="overload"> Symptoms of Postfix SMTP server overload </a></h2> <p>. </p> <p> SMTP server overload may be caused by a surge of legitimate mail (example: a DNS registrar opens a new zone for registrations), by mistake (mail explosion caused by a forwarding loop) or by malice (worm outbreak, botnet, or other illegitimate activity). </p> <p> Symptoms of Postfix SMTP server overload are: </p> <ul> <li> <p> Remote SMTP clients experience a long delay before Postfix sends the "220 hostname.example.com ESMTP Postfix" greeting. </p> <ul> <li> <p> NOTE: Broken DNS configurations can also cause lengthy delays before Postfix sends "220 hostname.example.com ...". These delays also exist when Postfix is NOT overloaded. </p> <li> <p> NOTE: To avoid "overload" delays for end-user mail clients, enable the "submission" service entry in <a href="master.5.html">master.cf</a> (present since Postfix 2.1), and tell users to connect to this instead of the public SMTP service. </p> </ul> <li> <p> The Postfix SMTP server logs an increased number of "lost connection after CONNECT" events. This happens because remote SMTP clients disconnect before Postfix answers the connection. </p> <ul> <li> <p> NOTE: A portscan for open SMTP ports can also result in "lost connection ..." logfile messages. </p> </ul> <li> <p> Postfix 2.3 and later logs a warning that all server ports are busy: </p> <pre> Oct 3 20:39:27 spike postfix/master[28905]: warning: service "smtp" (25) has reached its process limit "30": new clients may experience noticeable delays Oct 3 20:39:27 spike postfix/master[28905]: warning: to avoid this condition, increase the process count in <a href="master.5.html">master.cf</a> or reduce the service time per client </pre> </ul> <p> Legitimate mail that doesn't get through during an episode of Postfix SMTP server overload is not necessarily lost. It should still arrive once the situation returns to normal, as long as the overload condition is temporary. </p> <h2><a name="concurrency"> Service more SMTP clients at the same time </a> </h2> <p>. </p> <ul> <li> <p> You increase the number of SMTP server processes either by increasing the <a href="postconf.5.html#default_process_limit">default_process_limit</a> in <a href="postconf.5.html">main.cf</a> (line 3 below), or by increasing the SMTP server's "maxproc" field in <a href="master.5.html">master.cf</a> (line 10 below). Either way, you need to issue a "postfix reload" command to make the change effective. </p> <li> <p> Process limits above 1000 require Postfix version 2.4 or later, and an operating system that supports kernel-based event filters (BSD kqueue(2), Linux epoll(4), or Solaris /dev/poll). </p> <li> <p> More processes use more memory. You can reduce the Postfix memory footprint by using <a href="CDB_README.html">cdb</a>: lookup tables instead of Berkeley DB's hash: or btree: tables. </p> <pre> 1 /etc/postfix/<a href="postconf.5.html">main.cf</a>: 2 # Raise the global process limit, 100 since Postfix 2.0. 3 <a href="postconf.5.html#default_process_limit">default_process_limit</a> = 200 4 5 /etc/postfix/<a href="master.5.html">master.cf</a>: 6 # ============================================================= 7 # service type private unpriv chroot wakeup maxproc command 8 # ============================================================= 9 # Raise the SMTP service process limit only. 10 smtp inet n - n - 200 smtpd </pre> <li> <p> NOTE: older versions of the <a href="SMTPD_POLICY_README.html">SMTPD_POLICY_README</a> document contain a mistake: they configure a fixed number of policy daemon processes. When you raise the SMTP server's "maxproc" field in <a href="master.5.html">master.cf</a>, SMTP server processes will report problems when connecting to policy server processes, because there aren't enough of them. Examples of errors are "connection refused" or "operation timed out". </p> <p> To fix, edit <a href="master.5.html">master.cf</a> and specify a zero "maxproc" field in all policy server entries; see line 6 in the example below. Issue a "postfix reload" command to make the change effective. </p> <pre> 1 /etc/postfix/<a href="master.5.html">master.cf</a>: 2 # ============================================================= 3 # service type private unpriv chroot wakeup maxproc command 4 # ============================================================= 5 # Disable the policy service process limit. 6 policy unix - n n - 0 spawn 7 user=nobody argv=/some/where/policy-server </pre> </ul> <h2><a name="time"> Spend less time per SMTP client </a></h2> <p>. </p> <ul> <li> <p> Eliminate non-functional RBL lookups (blocklists that are no longer in operation). These lookups can degrade performance. Postfix logs a warning when an RBL server does not respond. </p> <li> <p> Eliminate redundant RBL lookups (people often use multiple Spamhaus RBLs that include each other). To find out whether RBLs include other RBLs, look up the websites that document the RBL's policies. </p> <li> <p> Eliminate <a href="postconf.5.html#header_checks">header_checks</a> and <a href="postconf.5.html#body_checks">body_checks</a>, and keep just a few emergency patterns to block the latest worm explosion or backscatter mail. See <a href="BACKSCATTER_README.html">BACKSCATTER_README</a> for examples of the latter. <li> <p> Group your <a href="postconf.5.html#header_checks">header_checks</a> and <a href="postconf.5.html#body_checks">body_checks</a> patterns to avoid unnecessary pattern matching operations: <pre> 1 /etc/postfix/header_checks: 2 if /^Subject:/ 3 /^Subject: virus found in mail from you/ reject 4 /^Subject: ..other../ reject 5 endif 6 7 if /^Received:/ 8 /^Received: from (postfix\.org) / reject forged client name in received header: $1 9 /^Received: from ..other../ reject .... 10 endif </pre> </ul> <h2><a name="hangup"> Disconnect suspicious SMTP clients </a></h2> <p> Under conditions of overload you can improve Postfix SMTP server responsiveness by hanging up on suspicious clients, so that other clients get a chance to talk to Postfix. </p> <ul> <li> <p>. </p> <li> <p>: </p> <pre> 1 /etc/postfix/<a href="postconf.5.html">main.cf</a>: 2 <a href="postconf.5.html#smtpd_client_restrictions">smtpd_client_restrictions</a> = 3 <a href="postconf.5.html#permit_mynetworks">permit_mynetworks</a> 4 <a href="postconf.5.html#reject_rbl_client">reject_rbl_client</a> zen.spamhaus.org=127.0.0.10 5 <a href="postconf.5.html#reject_rbl_client">reject_rbl_client</a> zen.spamhaus.org=127.0.0.11 6 <a href="postconf.5.html#reject_rbl_client">reject_rbl_client</a> zen.spamhaus.org 7 8 <a href="postconf.5.html#rbl_reply_maps">rbl_reply_maps</a> =} </pre> <p> Although the above example shows three RBL lookups (lines 4-6), Postfix will only do a single DNS query, so it does not affect the performance. </p> <li> <p> "<a href="postconf.5.html#smtpd_hard_error_limit">smtpd_hard_error_limit</a> = 1" as described below. </p> <li> <p> You can automatically turn on the above overload measure with Postfix 2.5 and later, or with earlier releases that contain the stress-adaptive behavior source code patch from the mirrors listed at <a href=""></a>. Simply replace line above 8 with: </p> <pre> 8 <a href="postconf.5.html#rbl_reply_maps">rbl_reply_maps</a> = ${stress?hash:/etc/postfix/rbl_reply_maps} </pre> </ul> <p> More information about automatic stress-adaptive behavior is in section "<a href="#adapt">Automatic stress-adaptive behavior</a>". </p> <h2><a name="legacy"> Temporary measures for older Postfix releases </a></h2> <p> See the next section, "<a href="#adapt">Automatic stress-adaptive behavior</a>", if you are running Postfix version 2.5 or later, or if you have applied the source code patch for stress-adaptive behavior from the mirrors listed at <a href=""></a>. </p> <p> The following measures can be applied temporarily during overload. They still allow <b>most</b> legitimate clients to connect and send mail, but may affect some legitimate clients. </p> <ul> <li> <p> Reduce <a href="postconf.5.html#smtpd_timeout">smtpd_timeout</a> (default: 300s). <b>most</b> legitimate clients to connect and send mail, but may delay mail from some clients. No mail should be lost, as long as this measure is used only temporarily. </p> <li> <p> Reduce <a href="postconf.5.html#smtpd_hard_error_limit">smtpd_hard_error_limit</a> > Use an <a href="postconf.5.html#smtpd_junk_command_limit">smtpd_junk_command_limit</a> of 1 instead of the default 100. This prevents clients from keeping idle connections open by repeatedly sending NOOP or RSET commands. </p> </ul> <blockquote> <pre> 1 /etc/postfix/<a href="postconf.5.html">main.cf</a>: 2 <a href="postconf.5.html#smtpd_timeout">smtpd_timeout</a> = 10 3 <a href="postconf.5.html#smtpd_hard_error_limit">smtpd_hard_error_limit</a> = 1 4 <a href="postconf.5.html#smtpd_junk_command_limit">smtpd_junk_command_limit</a> = 1 </pre> </blockquote> <p> With these measures, no mail should be lost, as long as these measures are used only temporarily. The next section of this document introduces a way to automate this process. </p> <h2><a name="adapt"> Automatic stress-adaptive behavior </a></h2> <p> Postfix version 2.5 introduces automatic stress-adaptive behavior. This is also available as a source code patch for Postfix versions 2.4 and 2.3 from the mirrors listed at <a href=""></a>. </p> <p> It works as follows. When a "public" network service such as the SMTP server runs into an "all server ports are busy" condition, the Postfix <a href="master.8.html">master(8)</a> daemon logs a warning, restarts the service (without interrupting existing network sessions), and runs the service with "-o stress=yes" on the server process command line: </p> <blockquote> <pre> 80821 ?? S 0:00.24 smtpd -n smtp -t inet -u -c -o stress=yes </pre> </blockquote> <p> Normally, the Postfix <a href="master.8.html">master(8)</a> daemon runs such a service with "-omain.cf</a> parameter settings stress adaptive. The following settings are the default with Postfix 2.6 and later. With earlier Postfix versions that have stress-adaptive support, append the lines below to the <a href="postconf.5.html">main.cf</a> file and issue a "postfix reload" command: </p> <blockquote> <pre> 1 <a href="postconf.5.html#smtpd_timeout">smtpd_timeout</a> = ${stress?10}${stress:300}s 2 <a href="postconf.5.html#smtpd_hard_error_limit">smtpd_hard_error_limit</a> = ${stress?1}${stress:20} 3 <a href="postconf.5.html#smtpd_junk_command_limit">smtpd_junk_command_limit</a> = ${stress?1}${stress:100} </pre> </blockquote> <p> Translation: <p> <ul> <li> <p> Line 1: under conditions of stress, use an <a href="postconf.5.html#smtpd_timeout">smtpd_timeout</a> value of 10 seconds instead of the default 300 seconds. most legitimate clients to connect and send mail, but may delay mail from some clients. No mail should be lost, as long as this measure is used only temporarily. </p> <li> <p> Line 2: under conditions of stress, use an <a href="postconf.5.html#smtpd_hard_error_limit">smtpd_hard_error_limit</a>> Line 3: under conditions of stress, use an <a href="postconf.5.html#smtpd_junk_command_limit">smtpd_junk_command_limit</a> of 1 instead of the default 100. This prevents clients from keeping idle connections open by repeatedly sending NOOP or RSET commands. </p> </ul> <p> The syntax of ${name?value} and ${name:value} is explained at the beginning of the <a href="postconf.5.html">postconf(5)</a> manual page. </p> <p> NOTE: Please keep in mind that the stress-adaptive feature is a fairly desperate measure to keep <b>some</b> legitimate mail flowing under overload conditions. If a site is reaching the SMTP server process limit when there isn't an attack or bot flood occurring, then either the process limit needs to be raised or more hardware needs to be added. </p> <h2><a name="feature"> Detecting support for stress-adaptive behavior </a></h2> <p>. </p> <p> The following example is for FreeBSD or Linux. On Solaris, HP-UX and other System-V flavors, use "ps -ef" instead of "ps ax". </p> <blockquote> <pre> $ ps ax|grep smtpd 83326 ?? S 0:00.28 smtpd -n smtp -t inet -u -c -o stress= 84345 ?? Ss 0:00.11 /usr/bin/perl /usr/libexec/postfix/smtpd-policy.pl </pre> </blockquote> <p> You can't use <a href="postconf.1.html">postconf(1)</a> to detect stress-adaptive support. The <a href="postconf.1.html">postconf(1)</a> command ignores the existence of the stress parameter in <a href="postconf.5.html">main.cf</a>, because the parameter has no effect there. Command-line "-o parameter" settings always take precedence over <a href="postconf.5.html">main.cf</a> parameter settings. <p> <p> If you configure stress-adaptive behavior in <a href="postconf.5.html">main.cf</a> when it isn't supported, nothing bad will happen. The processes will run as if the stress parameter always has an empty value. </p> <h2><a name="forcing"> Forcing stress-adaptive behavior on or off </a></h2> <p> You can manually force stress-adaptive behavior on, by adding a "-o stress=yes" command-line option in <a href="master.5.html">master.cf</a>. This can be useful for testing overrides on the SMTP smtp inet n - n - - smtpd 7 -o stress=yes 8 -o . . . </pre> </blockquote> <p> To permanently force stress-adaptive behavior off with a specific service, specify "-omaster.cf</a> command line. This may be desirable for the "submission" submission inet n - n - - smtpd 7 -o stress= 8 -o . . . </pre> </blockquote> <h2><a name="other"> Other measures to off-load zombies </a> </h2> <p> OpenBSD <a href="">spamd</a> implements a daemon that handles all connections from "new" clients. Only well-behaved mail clients are allowed to talk to the mail server. Other clients are tarpitted, and will never get a chance to affect mail server performance. </p> <p> At some point in the future, Postfix may come with a simple front-end daemon that does basic greylisting and pipelining detection to keep zombies and other ratware away from Postfix itself. This would use the "pass" service type which has been available in stable Postfix releases since Postfix 2.5. </p> <h2><a name="credits"> Credits </a></h2> <ul> <li> Thanks to the postfix-users mailing list members for sharing early experiences with the stress-adaptive feature. <li> The RBL example and several other paragraphs of text were adapted from postfix-users postings by Noel Jones. <li> Wietse implemented stress-adaptive behavior as the smallest possible patch while he should be working on other things. </ul> </body> </html>