Setting up SMTP AUTH with sendmail and Cyrus-SASL


SMTP AUTH is a method for authenticating connections and negotiating security layers for the SMTP protocol. SMTP AUTH is defined in RFC 2554 and is based on SASL, the Simple Authentication and Security Layer. SMTP AUTH is supported in sendmail 8.10 and greater. The SASL layer is provided by the Cyrus-SASL libraries.

Goals and Background

Configuring sendmail to use SMTP AUTH is not a basic task. The goal of this document is to show how to successfully install and configure the necessary components to get SMTP AUTH working in sendmail. What we are trying to achieve by configuring SMTP AUTH is to provide a secure means of email relay.

In the early days of the Internet, all mail servers were open relays--that is, any client could relay mail through any server. This was not a problem as the number of hosts on the Internet were not great and the network was not used for commercial purposes. With the advent of easy Internet access for the general populace a problem arose. Spammers could send their unwanted junk mail through a third-party server, thereby shifting the costs of sending the spam to an innocent third-party.

Open relays had to be closed and a method was needed to allow legitimate users to relay mail through a server. One method is to use IP-based authentication whereby only certain IP addresses are allowed to relay. This works when the entire user base is static. Roaming users presented a problem, as their IP address would change based on where they were. Another method of authenticating roaming users is POP-before-SMTP. POP-before-SMTP requires a user to check their POP3 email before being allowed to relay mail. The server would record the incoming IP address of the POP3 request and then temporarily allow relaying from that IP.

POP-before-SMTP, while functional, has its drawbacks. As it is not an official Internet standard most MTAs (Mail Transfer Agents--the server software) and POP servers do not support POP-before-SMTP out of the box, requiring patches and additional configuration steps to enable. Also, there is the remote possibility that an unwanted party will be able to relay mail if the IP address that is authorized by the POP-before-SMTP system is shared among multiple users and computers.

SMTP AUTH provides a better solution--password protected authorization of the client who wishes to relay mail. Each SMTP session requires authorization and there is no possibility of accidentaly relaying unwanted mail. Unfortunately, not all methods of SMTP AUTH are supported by all MUAs (Mail User Agents--the client software) as will be seen.

Necessary Components

The examples that will be shown in this document were created on a system running Slackware 9.1. Slackware is a generic enough Linux distribution (and UNIX-like system in general) that these instructions should be easily transferrable to other Linux and UNIX flavors. Additional software is required to complete the installation, including sendmail, the GNU dbm (gdbm), and the cyrus-sasl implementation as shown in the table below. The configuration shown here will support the AUTH LOGIN mechanism. LOGIN is not necessarily secure, however, it is one of only two methods supported by Microsoft Outlook Express, by far one of the most popular mail clients in use today (this is merely an observation and should by no means be taken as an endorsement of MS Outlook Express by the author).

The AUTH LOGIN method utilizes BASE64 encoding of usernames and passwords. BASE64 is trivial to decode by anyone who might be sniffing network traffic, thereby revealing usernames and passwords. However, the goal here is to prevent unauthorized relaying through an SMTP server, not to provide high-levels of security to roaming users. Also keep in mind that most remote users are already sending their usernames and passwords in cleartext over the Internet via the POP3 protocol.

We will be configuring sasl to utilize the existing user accounts on the system by reading from the /etc/shadow file. There are other options available, including creating separate user accounts through the /etc/sasldb file.

Required Software

Package Name: URL:

Note that while there is a 2.x version of the cyrus-sasl implementation, it is not yet fully supported in sendmail 8.12.x, so be sure to download the most recent version of the 1.5.x tree.

Normally the Berkeley DB is used with cyrus-sasl, however, I found that while compiling against Berkeley DB 3.3.11, cyrus-sasl would not function properly and result in segmentation faults when running the saslpasswd command. A few vague references on mailing lists and newsgroups pointed towards inconsistencies in the Berkeley DB libraries causing these problems. Slackware 9.1 by default installs Berkeley DB 1.x, 2.x, and 3.x, so this may be part of the problem. In addition, there is a newer Berkeley DB 4.x which is not installed as part of the Slackware install.

Instead of using Berkeley DB, I chose to use the GNU DBM, or GDBM. While not normally as popular as the Berkeley DB, it is also part of the Slackware install and does work for our purposes, especially considering we will not be utilizing the sasldb functionality to create user accounts.

Important Assumptions

In order to ensure that this document is not overly long and tedious, I will be making a few important assumptions about the reader. First, we will assume that the reader has some basic knowledge of how to download, untar, and compile software on a UNIX system. Second, and most important, we will assume that the reader also has some knowledge of building files from the M4 macros in a file. The necessary macros that need to be added for SMTP AUTH to function will be presented, however, the subject of building a basic, functional file is beyond the scope of this document.

It is assumed that the primary audience for these instructions is the administrator who already has a functional sendmail infrastructure and wishes to add SMTP AUTH capabilities to their site. In cases like this, most admins will already have a functional file for their site (and hopefully the source M4 macros used to create it, in the form of a file). Provided this assumption is correct, it is a simple matter to add the relevant M4 macros presented below to the existing M4 source to add SMTP AUTH capabilities.

Building the Cyrus-SASL Libraries

First, begin by making sure that gdbm is installed. If it was not installed as part of the Slackware OS install, either install the Slackware .tgz package from the CD, or compile the source code which can be downloaded from GNU.

The cyrus-sasl implementation must be compiled and installed. Download the tar file to a suitable location for building, such as /usr/src. It is assumed that the reader already knows how to expand gzipped tar archives and these obvious steps will be skipped. Take a moment to read over the INSTALL file, and also execute the configure script with the --help option to familiarize yourself with the options available.

Execute configure with the following options to configure the software in preparation for compiling:

# ./configure  --without-pam --enable-login --with-dblib=gdbm
The --without-pam option disables PAM, Pluggable Authentication Modules, which is not used at all in Slackware. The --enable-login option enables the AUTH LOGIN mechanism, which is critical for this configuration. Finally, the --with-dblib=gdbm option specifies the DB library to use, in this cause GNU dbm. Cyrus SASL will install by default in /usr/local. If this is not satisfactory for your system, change with --prefix=/path/to/install.

Once configure is completed, review the output to make sure everything looks fine. Now, compile the software by simply running:

# make
If everything compiles successfully, Cyrus SASL can be installed by running:
# make install
This will install the libraries in /usr/local/lib/sasl, a few binaries in /usr/local/sbin, and some include files in /usr/local/include.

Important Note: It is important that a symbolic link be created in /usr/lib to point to /usr/local/lib/sasl. The install will not do this for you, so it must be done by hand. It is also a good idea to run ldconfig at this point to update the cache used by the dynamic linker when compiling software.

# ln -s /usr/local/lib/sasl /usr/lib/sasl
# ldconfig

Configuring Cyrus-SASL

Each application that utilizes the SASL libraries must have a configuration file. These files are stored in the SASL library directory, in this example /usr/local/lib/sasl. For sendmail, the file should be called Sendmail.conf (Note the capital S in the name). This file will contain one line:

pwcheck_method: shadow
This specifies the method of looking up passwords--in this case utilizing the user's existing username and password via the /etc/shadow file. The various options available for lookup are:

sasldb The user is looked up in /etc/sasldb
passwd The user is looked up via /etc/passwd (For systems w/o shadow passwords)
shadow The user is looked up via /etc/shadow
PAM The user is looked up via PAM
kerberos_v4 The user is looked up via KERBEROS V4
pwcheck The user is looked up through a seperate daemon

A blank /etc/sasldb file should be created at this time, even though it is not used in this specific configuration. The reason for this is that sendmail may complain on startup if the file does not exist. Make sure the file is only readable and writable by the root user and no group or other permissions are granted (chmod 600 /etc/sasldb).

Building sendmail

Download the sendmail tar file to /usr/src and untar it. In order to get sendmail to build against the SASL libraries, a site.config.m4 file must be created. This file resides in the devtools/Site directory, off of the root of the source directory. In our example, this file would be located at /usr/src/sendmail-8.12.10/devtools/Site/site.config.m4. The contents of this file must include at the least the following lines:

APPENDDEF(`conf_sendmail_ENVDEF', `-DSASL')
APPENDDEF(`conf_sendmail_LIBS', `-lsasl')
APPENDDEF(`confLIBDIRS', `-L/usr/local/lib/sasl')
APPENDDEF(`confINCDIRS', `-I/usr/local/include')
Note the unusual look of the single quotes. It is critical that the proper open (`) and close (') quotes be used here. If the SASL libraries were installed in a location other than /usr/local, modify the above lines to point to the correct path. Note that if the SASL libraries were simply installed in /usr (/usr/lib and /usr/include), the last two lines are not necessary.

Next we will compile the sendmail binary. Sendmail is a bit different from other packages in that it does not utilize the now standard GNU autoconf method of building a Makefile. Change directories to the sendmail directory, off of the root of the source directory (/usr/src/sendmail-8.12.10/sendmail). Now, execute the following command to compile sendmail:

sh Build
Note that if you have previously compiled sendmail from this source, you will need to utilize sh Build -c in order to get your new site.config.m4 file to be utilized. You will know things are working if you notice -DSASL in the output while sendmail is compiling. Provided the compile completed properly, the new sendmail binary can be installed with the following command:
sh Build install

Creating the File

It will be necessary to add a few lines to the file in order to generate a new file to use SMTP AUTH. At the minimum, the following lines will need to be added to the .mc file:

define(`confAUTH_OPTIONS', `A')dnl
The AUTH_MECHANISMS line specifies a list of authentication mechanisms which are offered by the server. If a mechanism is not specified here, it will not be offered as an option. Also note, just because an auth mechanism is listed here does not mean it will necessarily be available to the server. The list will be compared with the list of actual mechanisms which are enabled in the SASL libraries and only that subset will be truly available.

The TRUST_AUTH_MECH line specifies which auth mechanisms are trusted to allow relaying through the server.

Finally, the AUTH_OPTIONS line specifies the only option currently available for authentication, A. According to the README file, when A is set, "then the AUTH= parameter for the MAIL FROM command is only issued when authentication succeeded."

The file can be built by issuing the following command from the source's cf directory (/usr/src/sendmail-8.12.10/cf/cf):
sh Build
The newly created file can then be installed via:
sh Build install-cf


The sendmail binary itself can be tested using the following command:

sendmail -d0.1 -bv root | grep SASL
If the command is successful, something similar to the following will be output:
If the string SASL does not appear in the command's output, then there was likely a problem during the compilation stage of game. The most likely culprit is failing to use the -c option during compilation and having the site.config.m4 ignored.

The next test to complete is actually connecting to the daemon and verifying that your specified auth mechanisms appear correctly. Here is a successful output example with the desired string highlighted in red:

slacksite: {/home/slack/public_html} % telnet localhost 25
Connected to localhost.
Escape character is '^]'.
220 ESMTP Sendmail 8.12.11/8.12.10; Thu, 31 Mar 2005 23:59:14 -0500
ehlo localhost Hello IDENT:1007@localhost [], pleased to meet you
250-SIZE 20485760
250 HELP
221 2.0.0 closing connection
Connection closed by foreign host.

If both of these tests are successful then you should be able to fire up an appropriate mail client such as Microsoft Outlook Express, configure it to use SMTP authentication, and test it against the mail server.

If there is still a problem at this point, check the logs to see if there are any hints as to the problem. Messages for sendmail are usually logged to /var/adm/maillog or /var/log/maillog depending on your UNIX flavor. Common problems at this point would be incorrect file permissions on various SASL libraries or configuration files. The log entries will point out where these errors lie.


If all went well, you now have a fully functional SMTP server which supports the SMTP AUTH mechanisms LOGIN and PLAIN. Please keep in mind that PLAIN transmits in clear text and LOGIN is merely BASE64 encoded. Both of these methods will not increase your level of security other than to allow a server to no longer function as an open relay. Other methods should be considered to increase your level of password security, such as SSL Tunnels or STARTTLS, both of which are beyond the scope of this document.