Email Server Security Best Practices to Look Out For

A significant percentage of online cyberattacks are email related. Enterprises are often required to host their own email servers for the sake of compliance and practicality, instead of using one of many third party email services that are often offered by ISPs. But if an email server that your business operates is subject to attack, not only can your business experience lost productivity, you can also be subject to litigation for leaking sensitive data.

Fortunately, there are some basic best practices for configuring your email server in a more secure way, which should significantly reduce your risk. This isn't a comprehensive guide to secure email configuration, and your business may be subject to compliance standards that aren't covered here, depending on your industry. But this is an excellent starting point. Sometimes, even the most experienced IT professionals forget about the basics.

For the sake of this piece, I use the phrase “email server” to refer to backend programs that are mail transfer/transport agents (SMTP), or mail delivery agents (e.g. POP3, IMAP). There are many popular mail transfer agents, and mail delivery agents that businesses use, such as sendmail, Postfix, Microsoft Exchange Server, Apache James, MDaemon, and Citadel. These best practices can be applied to the configuration of most commonly used email server applications, regardless of your OS platform.

Basic steps for secure mail server configuration

Here are the usual steps for setting up a mail transfer agent:

• Install the application of your choice. Generally, you can only have one application for using SMTP per email server. For instance, if you want to use postfix, you may have to uninstall sendmail packages first. Whether your MTA application is opensource or proprietary, installed via repository or removable media, choose carefully. Consider any requirements that compliance and company policy may have. If you can use an application that has added security features, that's probably a better choice. Also make sure that your application will be well supported in the coming years with frequent patches as necessary.
• Configure carefully. It's always more secure to make sure you're running no default settings. Also make sure that you change administrative usernames, and use only complex passwords. Change all passwords according to a set schedule, once every three months is usually good. Some of the settings specific to SMTP include total number of connections, total number of simultaneous connections, and maximum connection rate. Those will need to be tweaked for a while during operation until you find what works.
• Test, test, test! Have some test emails sent through your SMTP application, and commence full operation only when that's successful. But testing for incidents like DoS attacks will probably require the work of penetration testers. If you can arrange it, having a red team work on your SMTP server before you launch it may be a good idea.
• Review the functionaity and security of your SMTP server on a regular basis, to a set schedule that works for your organization.

And here are the usual steps for setting up a mail delivery agent:

• Install the application of your choice. As with mail transfer, you should only have one application installed as a mail delivery agent per email server. So uninstall any existing packages for mail delivery agents that you don't want to use first. Everything else that applies to choosing a mail transfer agent also applies to installing a mail delivery agent. Except for one thing- there's more than one commonly used TCP/IP protocol for mail delivery. Choosing between POP3 and IMAP4 is crucial, and you'll also have to consider the mail user agents your users have.
• As with mail transfer agents, change all of your default settings for your mail delivery agent as well. I recommend reading the documentation for your mail delivery agent thoroughly first, and you'll also have to consider compliance and company policy. Again, passwords should be changed on a regular basis.
• Test your POP3 or IMAP4 server too! The main concern for SMTP is DoS attacks, but the main concern for mail delivery is the ability to receive malicious emails. There are security features specific to mail delivery such as Spam URI Realtime Block Lists. They should be tested too.
• Just as with mail transfer, your mail delivery agent's functionality and security should be reviewed on a regular basis according to what's workable for your organization.

Choose an email delivery protocol

Many IT professionals prefer IMAP4 to POP3, because IMAP4 is the newer standard and has more features. But sometimes organizations may be better off choosing the older POP3 protocol. I would not recommend choosing POP1 or POP2 under any circumstances because those protocols are seldom well supported by modern email clients and webmail applications. It's important for your end users to run current versions of email clients or webmail applications so they can receive recent security patches. Unpatched email software can have huge security vulnerabilities!

• IMAP4- The main security advantage and disadvantage of IMAP is that a user's emails remain stored on the mail delivery agent server. It can be an advantage because an administrator can keep a permanent record of the emails that a user receives. From a defensive security perspective, that facilitates asset management and security auditing, especially if an organization's users sometimes deal with sensitive or proprietary data. The disadvantage is that if a malicious party penetrates an organization's mail delivery agent server, they'll be able to access the email received by all of its users. Securing your IMAP server is paramount. An organization will definitely have to choose IMAP instead of POP3 if a user needs to have multiple client devices (such as PCs and smartphones) for accessing their email.
• POP3- Email delivered by POP3 is deleted from the mail delivery agent as soon as a user receives it. That's great if your server has limited data storage. Some emails can be very large if they have large file attachments, and megabytes turn into gigabytes and terabytes over the cumulative effect of a few years, especially if you have a lot of users. You'll really need to focus on endpoint security in securing POP3 systems, because delivered email stays on client machines unless a user deletes them. POP3 systems can be harder work for an attacker who wants to obtain the emails all of your users receive, because they'll have to penetrate multiple client machines, instead of having one point of attack as IMAP systems have. But as delivered email doesn't stay on the mail delivery agent machine, security auditing the content of emails is difficult. If your users need to use multiple client devices for the same email account, POP3 is not an option.

General best practices for securing email servers

• There are SSL options for POP, IMAP, and SMTP as standard TCP/IP ports. They should always be chosen instead of plaintext email ports, no matter how unimportant you may think your data is. Nearly all modern email clients and server applications support SSL. You can completely eliminate plaintext email sending and receiving rather easily, so why not? Secure POP3 uses port 995, secure IMAP uses port 993, and secure SMTP uses port 465. Encryption is built-in and easy to use.
• Availability is a component of the Confidentiality Integrity and Availability (CIA) triad that's sometimes forgotten. Often DNS problems impact availability. Make sure that the domain that your corporate email uses has at least two MX records, with secondary and tertiary MX records for necessary redundancy when the primary MX record fails.
• SMTP servers are often subject to denial of service attacks. In your SMTP settings, limit your total number of connections, total number of simultaneous connections, and your maximum connection rate as much as you can while still maintaining the functionality that your organization requires. That may require some fine tweaking over time, a bit of trial and error. Every SMTP server's user needs are different.
• Spam is not only a common annoyance, it can also be a security problem with possible phishing and malware attacks. By maintaining a local IP blacklist on your email server machine, you can help a spam filter to harden your network's security.
• Activating reverse DNS lookup can also security harden your SMTP server from spam and other email attacks. Setting it up is well worth the effort!
• The mail relay for your SMTP server should be limited to only authenticated user email accounts in your system. Your mail transfer agent can be an open relay if it's configured that way. Don't do that.
• You can further mitigate email malware and phishing attacks by configuring Spam URI Realtime Block Lists (SURBL). SURBL can filter out malicious hyperlinks that are often embedded in the body of emails. Not all email server applications support the feature, but if yours does, you really ought to set it up. If you're in the process of choosing which email server software to use, you should probably choose applications that support SURBL.

Keeping these best practices for email servers in mind can greatly prevent cybersecurity catastrophes in the future. A little extra work and caution is always well worth it in the long run, especially considering how vital email is to most organizations.