Configuring a Secure Internal SMTP Relay with the SMTP Filter and Message Screener

 

An SMTP relay is a machine that accepts SMTP messages for mail domains for which is it not authoritative. The most common example of an SMTP relay is an SMTP server run by an ISP that accepts SMTP messages destined to other domains. The user logs onto his ISP and his email client software is configured to use the SMTP server at his ISP. The user sends an SMTP message to a user at domain.com, which is not run by the ISP the user connects to. The SMTP message is sent to the ISP’s SMTP server and the ISP’s SMTP server relays the mail to the appropriate SMTP server for the domain.com domain.

 

You can create an SMTP relay on your internal network that accepts incoming mail to the domains that you are responsible for. This relay can leverage the ISA Server 2000 SMTP Message Screener to block messages based on source address or domain, attachment characteristics or keywords. In addition, you can configure a second, secure authenticating SMTP virtual server on the SMTP relay computer to allow your external users to relay mail to any domain.

 

The primary advantage of using an SMTP relay for incoming connections is that you avoid allowing anonymous external computers from creating new inbound SMTP connections to your Exchange Server. The incoming connections are made to the SMTP relay and the SMTP relay forwards legitimate mail to email domains hosted on the Exchange Server to the Exchange Server machine. In addition, the SMTP relay machine absorbs the processing costs involved with the deep inspection of SMTP messages that the ISA Server 2000 SMTP filter performs.

 

You need to carry out the following procedures to create an anonymous inbound SMTP relay that sends mail to domains under your administrative control and create a secure authenticating SMTP relay your external users can use to relay mail to all mail domains:

 

 

The remainder of the ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document covers each of these procedures in detail.

 

 

 

Installing Windows Server 2003 on the Firewall Computer

 

The computer that becomes the ISA Server 2000 firewall must meet the following minimum requirements:

 

 

The ISA Server firewall and Web caching components work very well on very modest hardware. This is true even when the SMTP filter is enabled and protecting the published co-located SMTP server. However, the SMTP Message Screener can be very processor intensive. This is why I recommend that you use a processor with a minimum of rating of 1.5 MHz. This is especially true if you plan on running an authenticating and non-authenticating SMTP relay on the same computer.

 

 

 

Installing ISA Server 2000 on the Firewall Computer

 

Requirements and procedures for installing ISA Server 2000 on a Windows Server 2003 are slightly different from those for installing ISA Server 2000 on a Windows 2000 computer. Please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Installing ISA Server 2000 on Windows Server 2003 for details on how to install ISA Server 2000 on a Windows Server 2003 computer.

 

 

 

Configure DCOM Permissions on the Firewall Computer

 

The SMTP relay on the internal network using DCOM to communicate with the SMTP filter components on the ISA Sever firewall. DCOM permissions must be configured to allow the SMTP relay on the internal network to communicate with the filter. Perform the following steps to configure the proper DCOM permissions:

 

  1. Click Start and then click the Run command. Type dcomcnfg.exe in the Open text box and click OK (figure 1).

 

Figure 1

 


  1. In the Component Services console (figure 2), expand the Component Services node, and then expand the Computers node. Expand the My Computer node and then click on the DCOM Config node. Right click on the VendorDataClass entry in the right pane of the console and click Properties.

 

Figure 2

 


  1. On the General tab, click the down arrow for the Authentication drop down list box and select the None option (figure 3). Click Apply.

 

Figure 3

 


  1. Click on the Location tab (figure 4). Put a checkmark in the run application on the computer where the data is located checkbox. Put a checkmark in the run application on the following computer checkbox. Then type in the name of the SMTP relay computer on the internal network. Make sure that the ISA Server firewall can resolve this name. You can use either a NetBIOS name or a fully qualified domain name. Use WINS to help resolve NetBIOS names and DNS for resolve FQDNs.

 

Figure 4

 


  1. Click on the Security tab (figure 5). Select in the Customize option in the Launch Permissions, Access Permissions and Configuration Permissions frames. Click the Edit button in the Launch Permissions frame.

 

Figure 5

 


  1. In the Launch Permissions dialog box, click the Add button (figure 6). In the Select Users, Computers, or Groups dialog box, type Everyone in the Enter the object names to select text box. Click the Check Names button to confirm that the system recognizes the entry. Click OK in the Select Users, Computers, or Group dialog box.

 

Figure 6

 


  1. Notice that the Everyone group is automatically granted the Allow permission for Launch Permission (figure 7). Click OK in the Launch Permission dialog box.

 

Figure 7

 


  1. Click on the Edit button in the Access Permissions frame (figure 8).

 

Figure 8

 


  1. Click the Add button in the Access Permissions dialog box (figure 9). In the Select users, Computers, or Groups dialog box, type Everyone in the Enter the object names to select text box. Click the Check Names button to confirm that the system recognizes the entry. Click OK in the Select Users, Computers, or Groups dialog box.

 

Figure 9

 


  1. Notice in the Access Permission dialog box (figure 10) that the Everyone group is automatically assigned the Allow permission to the Access Permission permission. Click OK.

 

Figure 10

 


  1. Click the Edit button in the Configuration Permissions frame (figure 11).

 

Figure 11

 


  1. In the Change Configuration Permission dialog box, click the Add button (figure 12). In the Select Users, Computers, or Groups dialog box, type Everyone in the Enter the object names to select text box. Click the Check Names button to confirm that the system recognizes the entry. Click OK in the Select users, Computers, or Groups dialog box.

 

Figure 12

 


  1. Notice in the Change Configuration Permission dialog box that the Everyone group is assigned the Allow permission to the Full Control and Read permissions (figure 13). Click OK.

 

Figure 13

 


  1. Click on the Identify tab (figure 14). Select the This user option. Type in a user name in the User name text box. This user must be an administrator on the local machine, or a domain administrator. Enter this user’s password and then confirm the password. Click Apply and then click OK.

 

Figure 14

 


  1. Close the Component Services console (figure 15)

 

Figure 15

 

 

Install Windows Server 2003 on the SMTP Relay Computer

 

The SMTP relay computer must meet the basic hardware requirements for installing Windows Server 2003. In addition, consider the following options for the SMTP relay machine:

 

 

In this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit article, we will assume that the machine is not a member of the user domain and that you will assign the users a group account to authenticate with the secure authenticating SMTP server. If the SMTP relay were a member of the Windows domain that contained user accounts, you might consider allowing users to use their own user account to authenticate with the SMTP relay computer.

 

Install the SMTP Service on the SMTP Relay Computer

 

For security reasons, IIS 6.0 is not installed by default on a Windows Server 2003 computer. You must install the IIS 6.0 SMTP service on the Windows Server 2003 SMTP relay.

 

Perform the following steps to install the IIS 6.0 SMTP relay computer:

 

1.       Click Start, point to Control Panel and click on Add or Remove Programs. In the Add or Remove Programs window, click the Add/Remove Windows Components button. You will see a Windows Setup dialog box asking your to please wait (figure 16).

 

Figure 16

 

2.       In the Windows Components dialog box (figure 17), click on the Application Server entry (do not put a checkmark in its checkbox!). Click Details.

 

Figure 17

 

3.       In the Application Server dialog box (figure 18), Click on the Internet Information Services (IIS) entry (do not put a checkmark in its checkbox!). Click Details.

 

Figure 18 (fig18)

 

4.       In the Internet Information Services (IIS) dialog box, put a checkmark in the SMTP Service checkbox (figure 19). Click OK.

 

Figure 19

 

5.       Click OK in the Application Server dialog box (figure 20).

 

Figure 20

 

6.       Click Next on the Windows Components dialog box (figure 21).

 

Figure 21

 

7.       A progress bar appears as the application installs (figure 22).

 

Figure 22

 

8.       Click Finish on the Completing the Windows Components Wizard page (figure 23).

 

Figure 23

 

 

 

 

Configure the First Virtual SMTP Server (Non-Authenticating) for Anonymous Inbound Relay

 

We will run both an anonymous inbound SMTP relay that allows Internet SMTP servers to relay SMTP mail to the domains under your administrative control and a secure authenticating SMTP relay that you external users can use to relay mail to all domains, internal and external. We will configure the Default SMTP Virtual Service to provide the anonymous inbound relay. Later we will install a second virtual SMTP server that you will use as the secure authenticating SMTP relay.

 

1.       Click Start and point to Administrative Tools. Click on the Internet Information Services (IIS) Manager entry (figure 24).

 

Figure 24

 

2.       Right click on the Default SMTP Virtual Server entry in the left pane of the console (figure 25). Click the Properties command.

 

Figure 25

 

3.       In the Default SMTP Virtual Server Properties dialog box (figure 26), click the General tab. Click the down arrow for the IP address drop down list box and select the IP address that you want the anonymous inbound SMTP relay to use. This anonymous relay accepts mail for the mail domains that are under your administrative control and drops all other inbound mail. Internet SMTP servers use this machine to send mail to the Exchange Server. Click Apply after selecting the IP address.

 

Figure 26

 

 

4.       Click on the Messages tab (figure 27). You have the following options:

 

Limit message size to (KB)

This is what Microsoft SMTP Service will advertise, in kilobytes, as the maximum message size this SMTP virtual server will accept. If a mail client sends a message that exceeds the limit, the client will get an error. If a remote server supports EHLO, it will detect the advertised maximum message size value when it connects to the SMTP virtual server and won't even attempt to deliver a message that exceeds the limit. Instead it will simply NDR the sender of the message. A remote server that doesn't support EHLO will try to send a message that exceeds the size limit, but will still end up sending an NDR to the sender when the message doesn't go through. The default is 2048 KB. The minimum value is 1KB. To have no limit, clear the check box.

 

Limit session size to (KB)

This is the maximum amount of data, in kilobytes, accepted during the total connection. It is the sum of all messages sent during the connection (applying to the message body only). Type a value larger than the Limit message size to (KB). This maximum should be set carefully, because the connecting message transfer agent (MTA) is likely to resubmit the message repeatedly. The default size is 10240 KB. This value should be greater than or equal to the value entered for Limit message size to (KB). To have no limit, clear the check box.

 

Limit number of message per connection to

When the check box is selected, this option enables you to limit the number of messages sent in a single connection. The default is 20. This feature also provides a method to increase system performance by using multiple connections to deliver messages to a remote domain. Consequently, once the set limit is reached, a new connection is automatically opened and the transmission continues until all messages are delivered. To disable this feature and have no limit, clear the check box.

 

Limit number of recipients per message to

This setting limits the maximum number of recipients for a single message. The default is 100, which is the minimum required number specified in Request for Comments (RFC) 821. To disable this feature and have no limit, clear the check box. Some clients return messages with a non-delivery report (NDR) once an error message is received indicating that the maximum number of recipients has been exceeded. A server running Microsoft SMTP Service does not return messages with an NDR in this instance. It opens a new connection immediately and processes the remaining recipients. For example, if the recipient limit is set to 100 and a message with 105 recipients is being transmitted, the first 100 are delivered in one connection after receipt of the error message. Then a new connection is opened and the message is processed for the remaining five recipients.

 

Send copy of Non-Delivery Report to

When a message is undeliverable, it is returned to the sender with a non-delivery report (NDR). You can designate that copies of the NDR are sent to a specific SMTP mailbox. Type an e-mail address for the mailbox.

 

Badmail Directory

When a message is undeliverable, it is returned to the sender with a non-delivery report (NDR). You can designate that copies of the NDR are sent to a location of your choice.

All NDRs go through the same delivery process as other messages, including attempts to resend the message. If the NDR has reached the retry limit and cannot be delivered to the sender, a copy of the message is placed in the Badmail directory. Messages placed in the Badmail directory cannot be delivered or returned. Check the directory regularly and reconcile the messages, because a full directory may adversely affect Microsoft SMTP Service performance.

 

If you choose to hold mail evaluated by the SMTP message screener, then you might consider moving the location of the Badmail directory. If you server is subject to a spammer’s spam attack, you want to make sure that the held mail doesn’t fill up the operating system partition.

 

Make your configuration changes and click Apply.

 

 

Figure 27

 

5.       Click on the Delivery tab (figure 28). Notice the default entries for the retry intervals. If the SMTP relay is not able to contact the Exchange Server, it will attempt to redeliver the mail based on these intervals. Not that after the third failed attempt, the SMTP relay will continue to try and deliver the mail every 240 minutes. You might want to reduce this value in the event that you need to periodically take the Exchange Server offline for maintenance.

 

Click Apply and then click OK.

 

Figure 28

 

You may have noticed that we didn’t make any configuration changes to the authentication mechanism or the relay characteristics of this virtual SMTP server. The reason is that the default setting is to not relay mail unless a user authenticates. This prevents spammers from hijacking your anonymous inbound SMTP relay while allow users to relay if they authenticate. You’ll create a dedicated virtual SMTP server that will act as both a secure and authenticating SMTP relay.

 

The next step is to configure Remote Domains. You need to create a remote domain for each domain that you want to accept inbound mail for. For example, if you host the mail domains internal.net and domain.com on your Exchange Server, then you need to create a remote domain for internal.net and another remote domain for domain.com. In the current example we’ll create a single remote domain for internal.net.

 

Perform the following steps to create a Remote Domain that you allow anonymous relay to:

 

1.       Open the Internet Information Services (IIS) Manager console and expand your server name (figure 29). Expand the Default SMTP Virtual Server and right click on the Domains node. Point to the New command and click on Domain.

 

Figure 29

 

2.        On the Welcome to the New SMTP Domain Wizard page (figure 30), select the Remote option and click Next.

 

Figure 30

 

3.       On the Domain Name page, type in the name of your mail domain in the Name text box. Click Finish (figure 31).

 

Figure 31

 

4.       Right click on your remote domain in the right pane of the console and click on the Properties command (figure 32).

 

Figure 32

 

5.       On the remote domain’s Properties dialog box, put a checkmark in the Allow incoming mail to be relayed to this domain checkbox. This allows thevirtual SMTP server to relay mail addressed to this remote domain. Remember, this virtual SMTP server does not relay mail and drops all incoming SMTP messages, with the exception being for users who authenticate and for mail addressed to a domain that you’ve configured a remote mail domain for.

 

Select the Forward all mail to smart host option and type in a FQDN or IP address for the Exchange Server on your internal network. If you use a FQDN, make sure this SMTP relay computer can resolve this name to the IP address of the Exchange Server’s virtual SMTP server. If you use an IP address, make sure you surround the address with straight brackets, as seen in figure 33.

 

Click on the Security button.

 

Figure 33

 

6.       By default, this virtual SMTP server does not send credentials to the Exchange Server when it relays mail, and the Exchange Server’s SMTP service does not require credentials. You do have the option of configuring the Exchange Server to require authentication before it will accept the connection from the SMTP relay computer. If you configure the Exchange Server’s SMTP service to require authentication, then you must include valid credentials here. The account and password you enter in this dialog box must match the account you configure on the Exchange Server.

 

In this example, we will allow anonymous connections to the Exchange Server’s SMTP service. The Anonymous access option is select by default and we will leave it at its default. If you make a change on the Outbound Security dialog box (figure 34), click OK. Otherwise, click Cancel.

 

Figure 34

 

 

7.       Close the Internet Information Services console (figure 35).

 

Figure 35

 

 

Create a Second Virtual SMTP Server for Authenticated Inbound Relay for External Users

 

The second virtual SMTP server on the SMTP relay computer is used to allow your external users to relay to both your internal domains and any other domain on the Internet. You do not want to create an anonymous open relay. An open relay can send mail to any domain on the Internet and if it is anonymous, spammers will find your open relay and send gigabytes of spam through it.

 

You can force users to authenticate before they relay. This prevents spammers from using your SMTP relay to send spam, and allows your external users the ability to relay mail to any domain. In addition, you want to secure the messages moving from your external users and the SMTP relay. Some of the information moving from the external user to the SMTP relay may be destined to your internal Exchange Server. These messages may have proprietary information and you want to protect that information from prying eyes.

 

You can prevent intruders from obtaining information from your external users inbound SMTP messages for forcing SSL/TLS encryption. The authenticating SMTP relay can force the external users to negotiate a TLS connection first and after the TLS connection is established, the machine will accept the user’s credentials and accept the transfer of SMTP messages from the external SMTP client. If the external SMTP client does not successfully negotiate a TLS session, then the SMTP relay will drop the connection.

 

Because we want to have two virtual SMTP servers that have different authentication and security requirements, a second IP address will need to be bound to the network interface card.

 

Perform the following steps to add a second IP address to the SMTP relay’s network interface:

 

1.       Right click on the My Network Places icon on the desktop and click on the Properties command (figure 36).

 

Figure 36

 

2.       In the Network Connections windows (figure 37), right click on your network interface and click the Properties command.

 

Figure 37

 

3.       In the connection’s Properties dialog box (figure 38), click on the Internet Protocol (TCP/IP) entry and then click the Properties button.

 

Figure 38

 

4.       In the Internet Protocol (TCP/IP Properties dialog box, click the Advanced button (figure 39).

 

Figure 39

 

5.       In the Advanced TCP/IP Settings dialog box, click the Add button in the IP addresses frame (figure 40).

 

Figure 40

 

6.       Enter the IP address and Subnet mask in the provided text boxes on the TCP/IP Address dialog box (figure 41).

 

Figure 41

 

7.       The second address appears in the IP addresses frame on the IP Settings tab (figure 42). Click OK on the Advanced TCP/IP Settings dialog box and then click OK on the Internet Protocol (TCP/IP) Properties dialog box. Finally, click Close on the connection’s Properties dialog box.

 

Figure 42

 

8.       Close the Network Connections window (figure 43).

 

Figure 43

 

 

Now you can create the second virtual SMTP server and use the second IP address bound to the network interface. Perform the following steps to create the second virtual SMTP server: Its important that each virtual SMTP server listen on a different IP address because you need to have a specific IP address to forward incoming messages from the ISA Server firewall using Server Publishing.

 

Perform the following steps to create the second virtual SMTP server that will be used as an secure authenticating SMTP relay:

 

1.       Open the Internet Information Services (IIS) Manager and expand your server name (figure 44). Right click on the Default SMTP Virtual Server entry, point to New and click on Virtual Server.

 

Figure 44

 

2.       Type in a friendly name for the new virtual SMTP server in the Name text box (figure 45) on the Welcome to the New SMTP Virtual Server Wizard page. You can use nay name you like. This is the name of the virtual SMTP server that will appear in the right pane of the console. Click Next.

 

Figure 45

 

3.       On the Select IP Address page, click the down arrow for the Select the IP address for this SMTP virtual server drop down list box (figure 46). Make sure the IP address is the not the same one used by the anonymous inbound SMTP relay virtual server. Click Next.

 

Figure 46

 

4.       On the Select Home Directory page, type in a path for the home directory of this virtual SMTP server. The Wizard will create the directory if you have not created it already. Enter the path in the Home directory text box (figure 47).

 

Figure 47

 

5.       On the Default Domain page, type in bogus name for a default domain in the Domain text box. You do not want this virtual SMTP server to be an endpoint for any mail so you enter a bogus name. Click F