Configuring the Windows Server 2003-based ISA Server 2000 Firewall as a Filtering SMTP Relay

 

An SMTP relay is a computer that handles SMTP email messages between the source and the destination of the messages. The SMTP relay routes SMTP email messages from one SMTP server to another. In most situations, the user sends an email messages to his local SMTP server and the user’s SMTP server forwards the messages to the destination SMTP server. In this example the user’s SMTP server relays messages to the destination SMTP server. The messages are then be placed in the appropriate user mailbox after its received by the destination SMTP server.

 

The user’s local SMTP server does not need to relay messages directly to the final destination SMTP server that is responsible for a particular email domain. Instead of allowing SMTP servers on the Internet to forward SMTP messages destined for your own domains to directly deliver them to your Exchange Server, you may want to use your own SMTP relay to intercept SMTP email messages before the SMTP email reaches the SMTP service on your Exchange Server.

 

There are many advantages to using an inbound SMTP relay:

 

 

ISA Server firewalls can act as SMTP relays. The ISA Server firewall can accept incoming SMTP messages from Internet-based SMTP servers, evaluate the SMTP messages for validity, and then accept or reject the SMTP messages based on the result of the evaluation. ISA Server firewalls have two features that allow you to control SMTP access:

 

 

The ISA Server firewall’s SMTP filter protects against buffer overflow attacks. A number of standard SMTP service commands are included with the filter. Each command is assigned a predefined command length. Malicious commands exceeding this length are dropped. The SMTP filter examines all SMTP traffic moving through a Server Publishing Rule. Packet filters do not expose the SMTP traffic to the SMTP filter’s buffer overflow protection.

 

 

The SMTP Message Screener is an ISA Server firewall component that plugs into an IIS SMTP service. The Message Screener communicates with the ISA Server firewall’s SMTP application filter and extends the functionality of the SMTP filter. While the SMTP filter is only able to examine the SMTP command set, the Message Screener is able to examine the content of the SMTP data. This allows the SMTP Message Screener to block messages based on the message content, attachment types, and source email addresses or domains.

 

Note:
For more information on how to configure the SMTP filter and Message Screener, please see the ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Installing and Configuring the Windows Server 2003-based ISA Server 2000 SMTP Filter and Message Screener on the ISA Server Firewall and Configuring a Secure Internal SMTP Relay with the SMTP Filter and Message Screener.

 

The following procedures are required to enable the ISA Server firewall as an filtering SMTP relay:

 

·         Install Windows Server 2003 on the firewall computer

·         Install the IIS SMTP service on the Windows Server 2003 firewall computer

·         Decide whether to use packet filters or Server Publishing Rules to make the SMTP relay available to Internet hosts and disable socket pooling

·         Configure the SMTP server to support unauthenticated (and optionally authenticated) relay

·         Install ISA Server 2000 on the firewall computer, together with the SMTP Message Screener

·         Configure packet filters or SMTP server Publishing Rules on the ISA Server machine

 

 

The remainder of this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document discusses in detail how to perform these procedures.

 

Install Windows Server 2003 on the Firewall Computer

 

The computer that will become the ISA Server 2000 firewall/SMTP relay should meet the following minimum requirements:

 

 

The ISA Server firewall and Web caching components work very well on very modest hardware. This is true even when the SMTP filter is enabled and protecting the published co-located SMTP server. However, the SMTP Message Screener can be very processor intensive. This is why I recommend that you use a processor with a minimum of rating of 1.5 MHz.

 

Installing the IIS 6.0 SMTP Services on the Windows Server 2003 Firewall Computer

 

The SMTP Message Screener requires the IIS SMTP service. You will need to install the SMTP service because Windows Server 2003 does not install IIS by default. Perform the following steps to install the IIS 6.0 SMTP service:

 

  1. Click Start, point to Control Panel and click the Add or Remove Programs command (figure 1).

 

Figure 1

 


  1. Click the Add/Remove Windows Components button on the left side of the Add or Remove Programs window (figure 2).

 

Figure 2

 

 


  1. In the Windows Components dialog box, click on the Application Server entry (do not put a checkmark in its checkbox!). Click on the Details button (figure 3).

 

Figure 3

 

 


  1. In the Application Server dialog box, click on the Internet Information Services entry (do not put a checkmark in its checkbox!). Click on the Details button (figure 4).

 

Figure 4

 

 


  1. On the Internet Information Services (IIS) dialog box, put a checkmark in the SMTP Service checkbox. The Internet Information Services Manager checkbox will be automatically selected for you (figure 5). Click OK.

 

Figure 5

 


  1. Click OK in the Application Server dialog box (figure 6).

 

Figure 6

 


  1. Click Next on the Windows Components page (figure 7).

 

Figure 7

 


  1. The Windows Components Wizard installs the IIS SMTP service (figure 8).

 

Figure 8

 

  1. Click Finish when the Wizard completes.

 

 

Decide on whether to use Packet Filters or SMTP Server Publishing Rules and Disabling SMTP Service Socket Pooling

 

There are two methods you can use to allow external hosts to connect to the SMTP relay server running on the ISA Server firewall:

 

 

ISA Server packet filters can be configured to allow incoming SMTP messages to be accepted on the external interface of the firewall. The advantage of using an SMTP packet filter is that you do not need to disable socket pooling and you retain the original source IP address of the host sending the message in the SMTP service log files. The disadvantage of using an SMTP packet filter is that incoming SMTP connections are not exposed to the SMTP Application Filter’s buffer overflow protection

 

 

A Server Publishing Rule can be configured to allow incoming SMTP messages to be forwarded to the internal interface of the ISA Server firewall. You disable socket pooling and configure the IIS SMTP service to listen on an IP address bound to the internal interface. The advantage of this approach is that incoming messages are exposed to the SMTP filter’s buffer overflow protection. The disadvantages include the requirement to disable socket pooling and the inability to determine the source IP address of the incoming SMTP messages by analyzing the SMTP service logs. However, SMTP connection attempts are logged in the firewall service log.

 

Note:
Both options allow the SMTP Message Screener to filter messages based on domain name, user account, attachments and key words

 

The packet filter approach is simpler and retains the integrity of the SMTP service log, but the Server Publishing method provides protection against buffer overflow attacks. We will discuss both methods in this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document.

 

You will need to disable socket pooling if you intend to use the Server Publishing method. Perform the following steps to disable socket pooling for the Windows Server 2003 IIS 6 SMTP service:

 

Note:
Socket pooling allows a service to listen on all IP addresses and all interfaces. This prevents Server Publishing Rules from binding to the socket required listen for incoming SMTP messages. Socket pooling for the SMTP service must be disabled before creating an SMTP Server Publishing Rule.

 

 

1.       Click Start and then click the Command Prompt link. In the Command Prompt window, switch to the Inetpub\AdminScripts folder. Then type in the following command and press ENTER (figure 9):

 

Adsutil.vbs set /smtpsvc/1/DisableSocketPooling 1

 

Figure 9

 

2.       If the SMTP service is installed and you entered the command correctly, you should see what appears in figure 10.

 

Figure 10

 

 

3.       Close the command prompt window.

 

The SMTP service will continue to listen on all IP addresses on all interfaces. You must configure the service to listen on specific IP addresses to limit the server to listening on a subset of addresses.

 

 

Configuring the SMTP Service to Support Unauthenticated (and Optionally Authenticated) SMTP Relay

 

You can now configure the SMTP server to act as an SMTP relay. The following procedures are discussed in this section:

 

 

Configure the Default Virtual SMTP Server Properties

 

The Default Virtual SMTP Server listens for incoming messages to email domains you host. Perform the following steps to configure the Default Virtual SMTP Server:

 

1.       Click Start, point to Administrative Tools and click on Internet Information Services (IIS) Manager. In the Internet Information Services (IIS) Manager window, expand your server name and click on the Default SMTP Virtual Server entry in the left pane. Right click on Default SMTP Virtual Server and click on the Properties command (figure 11).

 

Figure 1 (Fig11)

 

2.       In the Default SMTP Virtual Server Properties dialog box, click on the General tab. Click the down arrow in the IP address drop down list box. Note the list of IP addresses include in the list. You should see entries for you external addresses, internal addresses, an (All Unassigned) (figure 12).

 

If you are using packet filters to allow inbound access to your SMTP relay, then select an external IP address.

 

If you are using Server Publishing Rules to allow inbound access to your SMTP relay, then select an internal IP address.

 

Click Apply after selecting an IP address to bind to the Default SMTP Virtual Server.

 

Figure 12

 

3.       Click on the Access tab. You have a number of options available on this tab. Click on the Relay button that’s located in the Relay Restrictions frame (figure 13).

 

Figure 13

 

 

4.       The default setting in the Relay Restrictions allows no machines to relay through this virtual SMTP relay except for authenticated users (figure 14). This is a global setting for the SMTP service. We will override this relay configuration by configuring a Remote Domain on this SMTP server later.

 

We do not want anyone to have “open relay” access to this virtual machine, regardless of their ability to authenticate. Remove the checkmark from the Allow all computers which successfully authenticate to relay, regardless of the list above. Removing this option prevents this virtual server from being able to relay to any mail domain except for those mail domains you create Remote Domain entries for.

 

Click OK.

 

Figure 14

 

 

5.       Click on the Messages tab. You have the option to limit the size of messages that move through the server, the number of messages per connection, and the number of recipients per message. You can also set a location for the badmail directory, which is the directory messages that aren’t destined for any of your remote domains will be deposit. Make sure that you place this directory on a volume that has a generous amount of free space so that your disk does not fill up in the event of a spam flood.

 

Figure 15

 

 

6.       Click on the Delivery tab (figure 16). On this tab you can configure how long the SMTP relay will wait before retrying to send messages to your Exchange SMTP service. This “queuing” of SMTP messages is helpful when you must take your Exchange Server offline. If the SMTP relay cannot immediately deliver the messages to your SMTP server, it will place them in a queue and attempt to redeliver the messages based on the intervals set on this tab.

 

Note that the SMTP relay will continue to resend the mail indefinitely. After the third retry, subsequent delivery attempts are done at an interval based on the Subsequent retry interval (minutes) entry. Even if your Exchange Server is down for a day or more, the SMTP relay will queue mail for you. Once your Exchange Server becomes again, you can restart the SMTP service on the SMTP relay computer and the mail will be delivered to your Exchange Server’s SMTP service immediately.

 

Figure 16

 

 

7.       Click on the Outbound Security button. In the Outbound Security dialog box (figure 17), you have the option to configure credentials the SMTP relay can use to authenticate with the SMTP service on the Exchange Server.

 

We will use this feature to allow the authenticating SMTP relay to authenticate with the Exchange Server’s SMTP service in the ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring a Windows Server 2003-based ISA Server as a Secure Authenticating SMTP Relay.

 

Click Cancel in the Outbound Security dialog box. We want to allow the unauthenticated SMTP relay to anonymously access the Exchange Server. You do not need to worry about spammers using your Exchange Server’s SMTP service as an “open relay”. The SMTP relay server on the ISA Server firewall will only relay messages that are destined for the domains you host on your Exchange Server.

 

Figure 17

 

 

8.       Click Apply and then click OK in the Default SMTP Virtual Server Properties dialog box.

 

 

 

Create Remote Domains to Support Your Email Domains and Enable Relay for Those Domains

 

The SMTP server is now configured to relay no messages. All incoming messages to the SMTP relay server will be dropped. This isn’t a very functional SMTP relay yet. We fix this by creating one or more Remote Domains on the SMTP relay.

 

A Remote Domain is an email domain hosted on another SMTP server. For example, if you are hosting the email domain internal.net, then you want all email messages destined for your users in the internal.net email domain to be relayed to the Exchange Server’s SMTP service on the internal network.

 

Note that the Internet email domains do not need to be the same as your internal network’s Active Directory domain or domains. The email domains accepted by the Exchange Server’s SMTP service are configured in the Recipient Policy of the Exchange Server. For example, the Exchange Server can be configured to receive email destined for users in the domain.com and domain.net domains, even thought it is a member of the internal.net domain.

 

You need to create a Remote Domain for each email domain you want your Exchange Server to receive email for. In the current example, we want to host mail for a single email domain, internal.net.

 

Perform the following steps to create a Remote Domain for the internal.net domain:

 

1.       Click Start, point to Administrative Tools, and click on Internet Information Services (IIS) Manager. In the Internet Information Services (IIS) Manager console, expand your server name and then expand the Default SMTP Virtual Server node. Click on the Domain node and then right click on it. Point to New and click on Domain (figure 18).

 

Figure 18

 

 

2.       On the Welcome to the New SMTP Domain Wizard page of the New SMTP Domain Wizard, select the Remote option (figure 19). Click Next.

 

Figure 19

 

3.       On the Domain Name page, type the name of your email domain in the Name text box. Click Next (figure 20).

 

Figure 20

 

 

4.       The new Remote Domain appears in the right pane of the console (figure 21). Right click on the Remote Domain and click on the Properties command.

 

Figure 21

 

 

5.       In the Remote Domain’s Properties dialog box, click on the General tab (figure 22). On the General tab, put a checkmark in the Allow incoming mail to be relayed to this domain checkbox. This option allows mail addressed to users in this domain to be relayed to the Exchange Server’s SMTP service.

 

You have two options in the Route domain frame:

 

Use DNS to route to this domain This option allows your DNS infrastructure to route requests to your mail domains based on the MX record entries for these domains. In order for this to work correctly, you must have a split DNS infrastructure so that the ISA firewall machine can resolve the names of your email domains to the internal IP address of the Exchange Server computer. If the ISA Server firewall resolves the email domains to the external address of the ISA Server firewall, then the relay will fail.

 

Note:
Please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring DNS to Support Exchange Server Publishing for more information on how to configure your DNS infrastructure to support SMTP server publishing using ISA Server firewalls

 

Forward all mail to smart host This option allows you to enter the IP address of your Exchange Server and have mail for your domains relayed to this IP address. You must put brackets around the IP address. If you do not put brackets around the IP address, the SMTP relay server will attempt to resolve the IP address to an IP address.

 

The Outbound Security button allows you to configure authentication methods the SMTP relay server can use to authenticate with the SMTP service on the Exchange Server. In this example we will not configure the Remote Domain to authenticate with the Exchange Server because only mail destined for the domains under your administrative control are relayed to the server.

 

Click Apply and then click OK.

 

Figure 22

 

6.       In the Internet Information Services (IIS) Manager, right click on the Default SMTP Virtual Server node and click the Stop command (figure 23).

 

Figure 23

 

7.       In the Internet Information Services (IIS) Manager console, right click on the Default SMTP Virtual Server node and click the Start command (figure 24).

 

Figure 24

 

 

The SMTP relay is now ready to relay mail to your mail domain. If you have multiple email domains, you will need to create a Remote Domain for each of your email domains.

 

 

Configure the SMTP Server to Support Authenticated SMTP Relay for Domain Users

 

At this point the SMTP relay allows inbound SMTP mail to the remote domains you’ve configured on the server. This allows any user or SMTP server on the Internet to send mail to the domains you host on your Exchange Server. What this configuration does not allow is for external users or servers to send mail to all addresses on the Internet.

 

For example, you might wish to provide an SMTP server for your remote users. These users may connect to a hotel or other broadband network that does not require them to log onto an ISP. Since these users do not log on to an ISP, it’s unlikely that the facility will provide them an SMTP server to send outbound SMTP messages. This problem applies to both your POP3 and IMAP4 clients, because IMAP4 downloads messages headers and message bodies, but it does not allow for sending mail. IMAP4 depends on SMTP to send SMTP messages.

 

However, you do not want to allow an “anonymous open relay”. Such an open relay allows anonymous users to connect to the SMTP server and relay to any SMTP server on the Internet. Such “open” SMTP relay computers are fodder for spammers and should be shut down immediately.

 

You can provide an “open relay” for your users by requiring authentication to the SMTP server. However, you must be careful about protecting your users’ credentials when they connect to your authenticating SMTP server. If a spammer is able to retrieve user credentials, the spammer will be able to use these credentials to send out literally millions of spam messages.

 

If you would like to provide an authenticating SMTP server for your external POP3 and IMAP4 mail clients, then please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Configuring a Windows Server 2003-based ISA Server as a Secure Authenticating SMTP Relay.

 

 

 

Installing ISA Server 2000 with the SMTP Filter and Message Screener on the Firewall Computer

 

The next step after installing and configuring the SMTP service on the ISA Server firewall is to install ISA Server 2000 with the SMTP Filter and Message Screener on to the Windows Server 2003 computer.

 

The ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Installing ISA Server 2000 on Windows Server 2003 for instructions on how to install ISA Server 2000 on Windows Server 2003. This document provides instructions on how to install all ISA Server 2000 components onto the Server. If you need to remove components, you can remove them later.

 

The ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Installing and Configuring the ISA Server SMTP Filter and Message Screener provides details on how to enable and configure the SMTP filter and Message Screener.

 

 

 

Configuring Packet Filters or Server Publishing Rules on the ISA Server Firewall

 

You can configure the ISA Server firewall to accept the incoming SMTP connections using either packet filters or Server Publishing Rules. We’ll cover both procedures in this section.

 

Configuring SMTP Packet Filters

 

Static packet filters allow external hosts to send packets to the external IP address on the ISA Server firewall’s TCP port 25 from any source port. This allows both SMTP servers and clients on the Internet to send SMTP messages to the SMTP relay on the ISA Server firewall.

 

Perform the following steps to create the SMTP packet filters:

 

 

1.       Open the ISA Management console. Expand the Servers and Arrays node, then expand your server name. Expand the Access Policy node, click on the IP Packet Filters node and then right click on it. Point to New and click Filter (figure 25).

 

Figure 25

 

2.       Type a name for the packet filter in the IP packet filter name text box on the Welcome to the New IP Packet Filter Wizard page (figure 26). Click Next.

 

Figure 26

 

 

3.       Select the Allow packet transmission option on the Filter Mode page (figure 27). Click Next.

 

Figure 27

 

4.       On the Filter Type page, select the Predefined option. Click the drop down list box and select the SMTP option (figure 28). Click Next.

 

Figure 28

 

 

5.       On the