Secure Exchange 2003 SMTP/Secure SMTP (SMTPS) Publishing

 

You can make the SMTP service on the Exchange Server available to Internet hosts. The most common reason to allow inbound access to your internal Exchange Server is to allow Internet SMTP servers to send SMTP mail to mail domains under your administrative control. For example, if your organization hosts mail for the internal.net domain, you will want to allow Internet SMTP servers to send mail to your Exchange Server’s SMTP service.

 

Another reason to allow inbound access to your Exchange Server is to provide a secure SMTP relay for your users. For example, many of your off-site users are able to connect to the Internet without first establishing a connection to an ISP that provides them access to an SMTP server. You can provide your users access to your SMTP server and force a secure authenticated SMTP connection. This protects the users credentials and data as it moves over untrusted networks and also protects your server from becoming an open relay.

 

In this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document we will cover the following procedures that are required to publish a secure SMTP server for your remote users and publish an anonymous SMTP server that Internet SMTP servers can use to forward mail to domains under your administrative control.

 

The procedures include:

 

·         Creating a second SMTP virtual server on the Exchange 2003 Server

 

You need two virtual SMTP servers on the Exchange Server. One of the virtual SMTP servers is used for authenticated SMTP connections that are protected by TLS encryption. The other virtual SMTP server allows anonymous inbound connections from Internet SMTP servers that need to forward mail to mail domains under your administrative control.

 

It is possible to allow authenticated and secure connections to the same SMTP virtual server that allows anonymous inbound connections. The drawback of this approach is that you cannot enforce authentication and encryption on the single virtual SMTP server. This is because you must allow anonymous connections from Internet SMTP servers that need to relay mail to mail domains under your administrative control. Because you cannot force authentication and encryption, you must depend on the email clients to configure their client software secure.

 

We consider it poor security practice to allow clients the option to use secure communications when they choose to. It’s important to force the SMTP clients to authenticate and use TLS encryption. Forcing encryption on the second virtual SMTP server guarantees that SMTP clients will not be able to create anonymous connections to the Exchange SMTP service. This is especially important because your external users need to use the second virtual SMTP server to relay mail to domains that are not under your organization’s control.

 

·         Requesting and install Web site certificate for the second SMTP virtual server

 

You must have a certificate bound to the SMTP virtual server before you can force TLS encryption for the connection. You use the IIS 6.0 Certificate Request Wizard to issue the request and install the certificate after it has been issued.

 

Note:
Although the certificate is requested using the Web Site Request Wizard, the certificate can be bound to the SMTP service, even though the SMTP service is not a Web site.

 

·         Configuring the first SMTP virtual server

 

The first virtual SMTP server is used to accept incoming anonymous connections from Internet SMTP servers that need to relay SMTP mail to domains under your organization’s control. These domains are set in the Exchange Server’s recipient policy and you have configured MX records for these domains to point to the IP address on the external interface of the ISA Server firewall that is listening for requests to the first virtual SMTP server.

 

Key characteristics of the first virtual SMTP server include:

 

The first SMTP virtual server allows anonymous connections. You do not force authentication on the first SMTP virtual server because Internet SMTP servers cannot authenticate with the first virtual SMTP server when relaying SMTP mail to your domains.

 

The first SMTP virtual server does not allow relay for anonymous connections. You have the option to allow relay for authenticated connections, or you can use disable relay on the first SMTP virtual server and require that users who need to relay use the second virtual SMTP server.

 

·         Configuring the second SMTP virtual server

 

Your remote users use the second virtual SMTP server for secure SMTP access. These users are forced to authenticate and they are forced to use TLS encryption when connecting to this virtual server. TLS encryption protects the user credentials and data. Remote users use this virtual SMTP server to send mail to domains hosted on the Exchange Server and to relay mail to domains not under your administrative control. While it’s not critical that mail destined for remote domains be encrypted, mail destined for your own domains that are hosted on the Exchange Server should be encrypted because there is a higher likelihood that proprietary information to other users within your organization are send via this channel.

 

The second virtual SMTP server has the following characteristics:

 

The second virtual SMTP server requires authentication. If the user cannot authenticate with the second virtual SMTP server, then the connection attempt is rejected. This prevents spammers from using your Exchange Server as a mail relay.

 

The second virtual SMTP server requires TLS encryption. If the user cannot successfully negotiate TLS encryption, then the connection attempt is rejected. TLS encryption protects the user credentials and data. The mail client must have the Root CA certificate of the CA that issued the Web site certificate to the SMTP server in its Trusted Root Certificate Authorities machine certificate store.

 

The second virtual SMTP server can resolve MX domain names itself, or your can use a smart host.

 

The SMTP filter must be disabled on the ISA Server firewall. The reason is the SMTP filter does not allow TLS encrypted sessions to be created between the SMTP client and the published SMTP server.

 

Note:
One solution to this problem is to configure the ISA Server computer as an SMTP relay. When the SMTP filter is enabled, you can configure the secure authenticating SMTP virtual server to listen on the external interface of the ISA Server firewall and configured packet filters to allow inbound access. Because the SMTP filter does not examine packets moving through a static packet filter, the TLS session can be successfully established. You can then bind the non-authenticating, anonymous SMTP server to the internal interface of the ISA Server firewall and the SMTP filter will protect the anonymous SMTP server from buffer overflow.

 

·         Installing Windows Server 2003 on the firewall computer

 

After you’ve configured the Exchange Server’s SMTP virtual servers, you’re ready to install Windows Server 2003 on the firewall computer. A key requirement in this scenario is that you must bind two IP addresses to the external interface of the ISA Server firewall. The reason is that you need to create two SMTP server publishing rules: one SMTP server publishing rule redirects incoming secure SMTP requests to the second virtual SMTP server, and the second SMTP Server Publishing Rule redirect incoming anonymous SMTP requests to the first virtual SMTP server.

 

·         Installing ISA Server 2000 on the firewall computer

 

Install ISA Server 2000 on the firewall computer after Windows Server 2003 has been installed.

 

·         Configuring the ISA Server firewall to support outbound access for the Exchange 2003 SMTP service

 

The ISA Server firewall must be configured to allow outbound access to SMTP and DNS Query and DNS zone transfer. The virtual SMTP servers need outbound access to SMTP to relay SMTP mail to domains not under your organization’s control. The DNS Query and DNS Zone Transfer Protocol Rules are required to allow either the Exchange Server or a DNS server on your internal network to resolve the MX domain name for outbound SMTP mail.

 

·         Creating the SMTP and Secure SMTP (SMTPS) Server Publishing Rules

 

An SMTP Server Publishing Rule supports the incoming anonymous connections to the first virtual SMTP server. This rule redirects incoming TCP 25 connections to the IP address that the first virtual SMTP server listens on.

 

The second SMTP Server Publishing Rule supports incoming SMTP connections requests that require authentication and TLS encryption. A Secure Exchange Server SMTP Server Publishing Rule accepts incoming connections on TCP port 25. The Exchange Server is able to use its TCP port 25 to listen for connection requests for encrypted and non-encrypted connections. This is the reason why you do not need to use the built in SMTPS Server Publishing Rule to publish the TLS secured SMTP server.

 

·         Configuring  the mail client to support SMTP and SMTPS access

 

The SMTP client will need to authenticate with the second virtual SMTP server. In addition, the SMTP client will need to negotiate a secure TLS protected SMTP sessions with the second virtual SMTP server. The client must trust the Root CA that issued the Web site certificate to the second virtual SMTP server.

 

The remainder of this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document discusses the details of each of these steps.

 

Create a Second SMTP Virtual Server on the Exchange 2003 Server

 

The first step is to create the second virtual SMTP server. This second virtual SMTP server will accept the incoming authenticating SMTP connections that are protected with TLS encryption.

 

Perform the following steps to create the second virtual SMTP server:

 

1. Open the Exchange System Manager and expand your organization name. Expand the Servers node and then expand your server name. Expand the Protocols node and expand the SMTP node. Click on the SMTP node and the right click on an empty area in the right pane of the console. Point to New and click SMTP Virtual Server (figure 1).

 

Figure 1

 

2. On the Welcome to the New SMTP Virtual Server Wizard page of the New SMTP Virtual Server Wizard, type in a name for the new virtual SMTP server in the Name text box (figure 2). Click Next.

 

Figure 2

 

3. Click the down arrow for the Select the IP address for this SMTP virtual server drop down list on the Select IP Address page (figure 3). Select the IP address that you want the second virtual SMTP server to listen on. Do not use the same address that the first virtual SMTP server listens on. By default, the first virtual SMTP server listens on all IP addresses. You will configure the first virtual SMTP server to listen on a specific IP address later. Select the address and click Finish.

 

Figure 3

 

4. The new virtual SMTP server appears in the left pane of the Exchange System Manager console (figure 4).

 

Figure 4

 

 

 

Request and install Web site certificate for the second SMTP virtual server

 

The SMTP service needs to obtain and bind a Web site certificate before it can be configured to require TLS encryption. Please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document How to Obtain a Web Site Certificate for detailed information on how to request and install a Web site certificate on the virtual SMTP server.

 

 

Configure the First SMTP Virtual Server

 

Now we can start configuring the virtual SMTP servers. Perform the following steps to configure the first virtual SMTP server; Internet SMTP servers use this virtual SMTP server to relay mail to domains under your administrative control:

 

1.       Open the Exchange System Manager and expand your organization name. Expand the Servers node and then expand your server name. Expand the Protocols node and expand the SMTP node. Right click on the Default SMTP Virtual Server node and click on the Properties command (figure 5).

 

Figure 5

 

2.       The General tab is the first one you see in the Default SMTP Virtual Server Properties dialog box (figure 6). Click the down arrow for the IP address drop down list box and select an IP address for this SMTP virtual server to use. Do not use the same IP address that you use for the second virtual SMTP server. Click Apply after selecting the IP address from the list.

 

Figure 6

 

3.       Click on the Access tab (figure 7). On the Access tab, click on the Authentication button in the Access control frame.

 

Figure 7

 

4.       In the Authentication dialog box (figure 8), you have the option to allow authenticated and/or anonymous connections.

 

Internet SMTP servers do not authenticate with the first virtual SMTP server. You can allow authenticated connections to the first virtual SMTP server if you want to allow authenticated users to relay through this virtual server. However, we recommend that you allow only anonymous connections through this virtual SMTP virtual server and require that users who wish to relay to mail domains not under your administrative control to use the second virtual SMTP server.

 

Click OK after making your selection.

 

Figure 8

 

5.       Click the Relay button in the Relay restrictions frame (figure 9).

 

Figure 9

 

6.       If you wish to allow authenticated users to relay through this virtual SMTP server, confirm that there is a checkmark in the Allow all computer which successfully authenticate to relay, regardless of the list above checkbox (figure 10). Do not add any IP addresses to the list. Controlling relay using IP addresses is not secure, as its relatively easy to spoof an IP address. Click OK.

 

Figure 10

 

7.       Click on the Delivery tab (figure 11). Click the Advanced button.

 

Figure 11

 

8.       You can configure a smart host on the Advanced Delivery dialog box (figure 12). Type in either an IP address or a FQDN for a smart host in the Smart host text box. A smart host is an SMTP server that accepts all SMTP messages sent by the virtual SMTP server and forwards the messages to the appropriate Internet SMTP server after it resolves the MX domain name to an IP address of an SMTP server responsible for that domain’s SMTP mail.

 

You have the option to use an external DNS server. Normally, the Exchange Server’s SMTP service resolves the MX domain name for the destination email domain by using the DNS server configured in the TCP/IP Properties of its network interface card. However, you may want the Exchange Server computer to use an internal DNS server that does not resolve Internet host names; this would be required if you needed the Exchange Server to be a member of a domain but have no internal DNS servers that can resolve Internet host names. In this case, you configure the SMTP service to use another DNS server that can configure Internet host names while still retaining the DNS server settings on the Exchange Server’s network interface card.

 

Note that if you use a smart host, you do not need to worry about MX domain name resolution, as all mail that needs to be relayed is forwarded to the smart host.

 

Click OK and then click Apply and then OK.

 

Figure 12

 

 

 

Configure the Second SMTP Virtual Server

 

The configuration steps for the second SMTP virtual server are similar to the first. However, there are a few critical differences that will be pointed out when we reach them. Perform the following steps to configure the second virtual SMTP server:

 

1.       Right click on the second virtual server node in the left pane of the console and click the Properties command. On the General tab of the second virtual SMTP server’s Properties dialog box, select an IP address to bind to the second virtual SMTP server. This must not be the same IP address that the first virtual SMTP server listens on (figure 13). Click Apply after selecting an IP address.

 

Figure 13

 

2.       Click on the Access tab. On the Access tab, click on the Authentication button in the Access control frame (figure 14).

 

Figure 14

 

3.       On the Authentication dialog box, remove the checkmark from the Anonymous access checkbox and place a checkmark in the Integrated Windows Authentication checkbox (figure 15). You want to force authentication against the second virtual SMTP server. You do not want to allow anonymous connections to the second virtual SMTP server because this server will allow relay to remote SMTP mail domains. Click OK.

 

Figure 15

 

4.       On the Access tab, click on the Communication button in the Secure communication frame (figure 16).

 

Figure 16

 

5.       On the Security dialog box, put a checkmark in the Require secure channel checkbox. Place a checkmark in the Require 128-bit encryption checkbox. These options force a 128-bit TLS secured connection between the SMTP client and the second SMTP virtual server (figure 17). Click OK.

 

Figure 17

 

6.       On the Access tab, click on the Relay button in the Relay restrictions frame. On the Relay Restrictions dialog box, confirm that there is a checkmark in the Allow all computers which successfully authenticate to relay, regardless of the list above checkbox (figure 18). This option allows users who authenticate with the second virtual SMTP server to relay mail to Internet mail domains that are not under your administrative control. Click OK.

 

Figure 18

 

7.       Click on the Delivery tab (figure 19). Click on the Advanced button.

 

Figure 19

 

8.       You can configure a smart host for the second virtual SMTP server (figure 20). You can also configure an external DNS server if you do not want to use a smart host or resolve the name via an internal DNS server. If you have a DNS sever on the internal network that can resolve Internet DNS host names, and the network interface card on the Exchange Server is configured to use that DNS server, then you do not need to use an external DNS server or smart host.

 

Click OK.

 

Figure 20

 

9.       Click Apply and then click OK on the second virtual SMTP server’s Properties dialog box (figure 21).

 

Figure 21

 

 

 

 

Installing Windows Server 2003 on the Firewall Computer

 

The computer that will become the ISA Server 2000 firewall relay must meet the following minimum requirements:

 

• A personal computer with a 1.5 MHz or higher Intel/AMD-compatible CPU
• For the operating system, Windows 2000 Service Pack 4 or Windows Server 2003
• 256 MB of memory (RAM)
• 20 MB of available hard disk space for program files
• Two network adapters that is compatible with Windows 2000 or Windows Server 2003 , for communication with the internal and external networks
• One local hard disk partition that is formatted with the NTFS file system for log files and Web caching (if you wish to run the ISA Server firewall’s Web caching facilities)

 

The ISA Server firewall and Web caching components work very well on modest hardware. This is true even when the SMTP filter is enabled and protecting the published SMTP servers. However, if you run decide to use the SMTP Message Screener on the firewall, or if you use SSL to protect Web Published Web site, or if you use the ISA Server firewall as a VPN server, you need to increase the minimum requirements to support encryption services.

 

 

Install ISA Server 2000 on the Firewall Computer

 

Install ISA Server 2000 after installing Windows Server 2003 onto the firewall computers. You must go through some specific procedures outside of the standard ISA Server 2000 installation when installing the firewall software onto a Windows Server 2003 computer. Please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document Installing ISA Server 2000 on Windows Server 2003.

 

 

 

Configure the ISA Server firewall to support outbound access for the Exchange 2003 SMTP service

 

The Exchange SMTP services need outbound access to the SMTP protocol if they are configured to relay SMTP messages to mail domains that are not under your administrator control. For example, one of the expressed purposes of your secure authenticating SMTP server is to provide SMTP relay services for your authenticated users. The second virtual SMTP server will need outbound access to the SMTP protocol to relay mail to Internet SMTP servers.

 

Outbound access for DNS is also required. If you have an internal DNS server resolving Internet host names, then make sure the internal DNS server has access to both the DNS Query and DNS Zone Transfer protocols.

 

If the Exchange Server is using an External DNS server, then make sure that the Exchange Server has access to a DNS Query and DNS Zone Transfer Protocol Rule that allows it to access the external DNS server.

 

You do not need to create a DNS Protocol Rule if the Exchange Server uses a smart host to handle MX domain name resolution.

 

Please refer to Configuring Outbound Access for the Exchange 2003 SMTP Service for more information on creating SMTP and DNS Protocol Rules to support the internal SMTP servers.

 

 

 

Create the SMTP and Secure Exchange SMTP Server Publishing Rules

 

You need to create Server Publishing Rules to allow inbound access to the Exchange virtual servers on the internal network. You need to create two Server Publishing Rules:

 

• One SMTP Server Publishing Rule forwards the anonymous inbound SMTP connections from Internet SMTP servers forwarding mail to domains under your administrative control
• The other SMTP Server Publishing Rule forwards the secure authenticated SMTP connections to the second virtual SMTP server that requires both authentication and a secure SSL channel.

 

Note:
You will not be able to connect using TLS encryption when the SMTP Filter is enabled. You must disable the SMTP Filter to support TLS encryption for communications between the SMTP client and SMTP server.

 

Perform the following steps to disable the SMTP filter:

 

 

1.       Open the ISA Management console, expand the Servers and Arrays node and expand your server node. Expand the Extensions node and click on the Application Filters node. Right click on the SMTP Filter node and click the Disable command (figure 22).

 

Figure 22

 

2.       Select the Save the changes and restart the services (s) option in the ISA Server Warning dialog box (figure 23). This will allow the Firewall service on the ISA Server firewall machine to restart automatically.

 

Figure 23

 

3.       The SMTP Filter icon in the right pane of the console should now have a “red” down pointing arrow superimposed on it (figure 24).

 

Figure 24

 

 

You can now create your SMTP Server Publishing Rules. Perform the following steps to create a Server Publishing Rule to publish the secure authenticating SMTP virtual server:

 

1.       Open the ISA Management console, expand the Servers and Arrays node and expand the server name. Expand the Publishing node and click on the Server Publishing Rules node. Right click on the Server Publishing Rules node, point to New and click on Rule (figure 25).

 

Figure 25

 

2.       Type a name for the Sever Publishing Rule in the Server publishing rule name text box on the Welcome to the New Server Publishing Rule Wizard page (figure 26). Click Next.

 

Figure 26

 

3.       Type in the IP address used by the secure authenticating SMTP virtual server in the IP address of internal server text box on the Address Mapping page (figure 27). Click the Browse button under the External IP address on the ISA server and select the IP address you want to use on the external interface of the ISA Server firewall to accept incoming connection requests to the secure authentication SMTP virtual server. Click OK after selecting the address in the New Sever Publishing Rule Wizard dialog box.

 

Figure 27

 

4.       Click Next on the Address Mapping page after the external IP address has been entered (figure 28).

 

Figure 28

 

5.       On the Protocol Settings page (figure 29), click the down arrow from the Apply the rule to this protocol drop down list box and select the SMTP Server Protocol Definition. Click Next.

 

Figure 29

 

6.       On the Client Type page, select the Any request option and click Next (figure 30).

 

Figure 30

 

7.       Review the settings on the Complete the New Server Publishing Rule Wizard page (figure 31), and click Finish.

 

Figure 31

 

 

The next step is to create the Server Publishing Rule for the anonymous SMTP virtual server that accepts connections from Internet SMTP servers:

 

1.       Type a name for the Sever Publishing Rule in the Server publishing rule name text box on the Welcome to the New Server Publishing Rule Wizard page (figure 32). Click Next.

 

Figure 32

 

2.       Type in the IP address used by the secure anonymous SMTP virtual server in the IP address of internal server text box on the Address Mapping page (figure 33). Click the Browse button under the External IP address on the ISA server and select the IP address you want to use on the external interface of the ISA Server firewall to accept incoming connection requests to the secure authentication SMTP virtual server. Click OK after selecting the address in the New Sever Publishing Rule Wizard dialog box.

 

Figure 33

 

3.       Click Next on the Address Mapping page after the external IP address has been entered (figure 34).

 

Figure 34

 

4.       On the Protocol Settings page (figure 35), click the down arrow from the Apply the rule to this protocol drop down list box and select the SMTP Server Protocol Definition. Click Next.

 

Figure 35

 

5.       On the Client Type page, select the Any request option and click Next (figure 36).

 

Figure 36

 

6.       Review the settings on the Complete the New Server Publishing Rule Wizard page (figure 37), and click Finish.

 

Figure 37

 

7.       The anonymous and secure authenticating SMTP Server Publishing Rules appear in the right pane of the console (figure 38).

 

Figure 38

 

 

 

 

Configuring the SMTP Client to use TLS Encryption for SMTP Messages

 

The SMTP client must be configured to negotiate a TLS connection with the authenticating SMTP relay. The method used to configure the client to use secure SMTP connections varies with the client. The following ISA Server 2000 Exchange Server 2000/2003 Deployment Kit documents discuss how to configure some popular SMTP client to the SMTP relay using TLS:

 

• Configuring Outlook Express
• Configuring Outlook 2000
• Configuring Outlook 2002
• Configuring Outlook 2003

 

Regardless of the SMTP email client application, all clients will need a copy of the Root CA certificate of the CA that assigned the authenticating SMTP server its Web site certificate. Please refer to ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document How to Import the Root CA Certificate into Email Client Certificate Stores.

 

 

Summary

 

In this ISA Server 2000 Exchange Server 2000/2003 Deployment Kit document we went over the procedures required to make an anonymous and a secure authenticating SMTP virtual server on the Exchange Server available to Internet SMTP servers relaying mail to Internet domain names under your administrative control and for your remote users who require a secure SMTP server connection to send mail to other users in your organizations and to relay mail to Internet mail domains not under your control.