Microsoft Internet Security and Acceleration Server 2000 in Education Deployment Kit

 

Chapter 4

Stopping Unwanted E-mail and E-mail Attachments from Entering the Institution with ISA Server 2000

 

 

 

 

 

 

 

 

Dr. Thomas W Shinder

Debra L. Shinder

December 2003

 

 

Table of Contents

 

Abstract 3

Unwanted e-mail 4

Viruses and Worms. 6

Buffer Overflow Attacks. 7

The ISA Server 2000 Server SMTP Filter and SMTP Message Screener 9

The SMTP Filter 9

The SMTP Message Screener 11

Placing the ISA Server 2000 SMTP Filtering Firewall on Your Network. 13

ISA Server 2000 Front-end Firewall Topology. 13

ISA Server 2000 Back-end Firewall Topology. 15

ISA Server 2000 Front-end and Back-end Firewalls. 17

ISA Server 2000 Application Layer Filtering Web Proxy in the Perimeter Network. 19

SMTP Filter and Message Screener Scenarios: SMTP Relay and Message Screener on the Campus Network  20

SMTP Filter and Message Screener Scenarios: SMTP Relay and Message Screener on the ISA Server 2000 Firewall 22

Installing and Configuring the SMTP Message Screener and SMTP Filter on the ISA Server 2000 Firewall 23

Install the IIS 6.0 SMTP Service on the Windows Server 2003 ISA Server Firewall Computer 23

Disable SMTP Service Socket Pooling. 32

Configure the IIS 6.0 SMTP Service Relay Properties. 34

Create Remote Domains to Support Your E-mail Domains and Enable Relay for Those Domains. 41

Install ISA Server 2000 onto the Windows Server 2003 Firewall Computer 50

Configuring Server Publishing Rules on the ISA Server Firewall 50

Configure the SMTP Filter and SMTP Message Screener Properties. 58

Summary. 73

 

 

 


Abstract

ISA Server 2000 is an advanced application aware firewall. As a sophisticated application aware firewall, ISA Server 2000 examines application layer content of communications moving through it. This means administrators of educational institution networks can use ISA Server 2000’s advanced application layer filtering to help prevent unwanted e-mail, worms and viruses from endangering the campus network.

 

Educational institutions are more vulnerable than are corporate networks because of the generally open nature of their networks, necessitated by access policies that encourage free flow of information. Yet a worm or virus can crash computers or even shut down the entire network, thus interfering with the access that students, faculty and administrators need to do their work.

 

In this ISA Server 2000 Application in Education Deployment Kit document, you will learn about how unwanted e-mail, viruses and worms enter the campus network and how they can harm computer networks and the institutions running them. You will also learn how ISA Server 2000 application layer aware firewalls can protect your network from these threats.

 

ISA Server 2000 application layer filtering firewalls can be placed virtually anywhere on the campus network.  This allows a high level of flexibility for institutions that already have an existing firewall infrastructure that they do not wish to replace, making it easy to add another layer of protection by incorporating ISA Server into the existing security strategy.

 

We will show you where you can place the ISA Server 2000 SMTP filtering firewall on your network and provide step by step details on how to configure the ISA Server 2000 SMTP filtering firewall as a secure filtering gateway to keep out unwanted e-mail and virus/attachments.

 

 

 


Campus networks are under constant attack from external intruders. With many educational institutions engaged in important research, this poses a threat not only to the users of the network, but in some cases to national security. As early as March 2002, Department of Health and Human Services auditors were concerned about security at several university research facilities, due to the possibility of terrorists getting information about hazardous biological and chemical materials.

 

Lax security on campus networks also results in higher costs (for example, in the form of increased insurance premiums), taking away money that could be spent on academic programs. Productivity is reduced and, in the case of public-funded institutions, taxpayer dollars are wasted.

 

There are almost as many different types of attacks as there are attackers. However, the three most common attacks that are attempted against networks and network services revolve around three main areas:

 

  • Unwanted e-mail
  • Viruses and Worms
  • Buffer Overflows

 

A large proportion of the problems encountered on networks today can be traced to one of these issues. Unwanted e-mail, viruses, worms and buffer overflows clog mail servers, disable network services, destroy data, consume available network bandwidth, and cost educational institutions thousands and potentially millions of dollars each year.

Unwanted e-mail

Unsolicited and unwanted commercial e-mail (commonly known as “spam”) is one of the greatest problems facing campus networks and the Internet today. Unwanted e-mail leads to the following problems:

 

  • Wasted bandwidth on Internet connections
  • Increased Internet bandwidth cost
  • Increased non-productive traffic on the campus network
  • Decreased campus employee productivity due to time spent reading and deleting unwanted e-mail
  • Increased administrative costs as campus network administrators attempt to reduce the negative effects of unwanted e-mail
  • Increased disk usage on mail servers
  • Increased processor and memory utilization on mail servers
  • Increased exposure to legal liability secondary to employees and students who may view offensive unwanted e-mail messages
  • Increased risk of campus servers and desktops being used as unwanted e-mail relay stations

 

It has been estimated that unwanted e-mail  messages consume up to 50% of total bandwidth usage on the Internet today. Recent trends suggest there will be an acceleration in the unwanted e-mail volume curve and an increase in the resources required to review, store, report and delete unwanted e-mail messages from mail servers and user workstations. Many educational institutions have already reached a breaking point regarding unwanted e-mail on their mail servers. Unwanted e-mail control and elimination is no longer an optional network activity; it’s a requirement.

 

The challenge is to determine which e-mail messages constitute unwanted e-mail. You want to block unwanted e-mail, but you also must allow valid e-mail messages into your mail systems. An overly aggressive approach to unwanted e-mail control will result in a high number of “false positives” (legitimate e-mail messages being blocked) and this can have an adverse effect on overall productivity and satisfaction. The only thing worse than having “spam” get through to a user’s mailbox is having an important legitimate message blocked by the e-mail filters so that the user never gets it.

 

Why block unwanted e-mail at the firewall? You may already have an anti-spam program installed on your mail server and/or spam blockers implemented on client machines. Like any security strategy, however, effective spam control is a multi-layered job. Filtering is a processor intensive activity that can easily eat up the resources of a server. By stopping some of the unwanted e-mail before it reaches the server, you spread the processing load and increase the performance of the server. A three-tiered approach (with filtering at the firewall, server and client levels) is the best way to ensure that as little unwanted mail as possible reaches the user’s mailbox.

 

There are a number of different methods that can be used to control unwanted e-mail and each of these methods inspect application layer information. In order to be most effective, devices designed to control the influx of unwanted e-mail must be application layer aware. Application layer aware devices can inspect the SMTP messages transporting the unwanted e-mail and evaluate characteristics of the messages, including the following:

 

  • Source e-mail address
  • Source e-mail domain
  • Keywords in the subject line
  • Keywords in the message body
  • Attachment name
  • Attachment extension
  • Attachment size

 

Unwanted e-mail can also be blocked by restricting access to your mail servers from specific e-mail addresses of known spammers or from domains known for hosting spammers. You may want to restrict a specific e-mail address rather than an entire domain if legitimate mail also originates with that domain. However, blocking unwanted e-mail based on source mail domain or e-mail address is only a first step in controlling unwanted e-mail, because spammers often change addresses and domains frequently to circumvent this type of filtering. Nonetheless, address and domain blocking are useful components of a good multi-faceted approach to spam control.

 

Another powerful method used to block unwanted e-mail is keyword matching. Unwanted e-mail can be blocked based on content in the subject line and message body. Unwanted e-mail messages typically contain words never or seldom used in legitimate e-mail. You can leverage this fact by blocking e-mail messages containing targeted keywords in the subject or message body. ISA Server allows you to define character strings (keywords or phrases) that it will search for within the text of the message. If those character strings are found, the message will be marked as unwanted e-mail and blocked.

 

E-mail attachments can be included with both unwanted e-mail and legitimate e-mail. Attachments can be undesirable for several reasons:

 

·         Downloading large attachments can clog network bandwidth and delay other e-mail from getting through.

·         Pornographic pictures and other offensive content are often sent as attachments.

·         Some attachments consist of executable files that run when the user clicks to open them.

 

Executable attachments (including attached documents that contain macros or other embedded executable programs) present a special problem for campus networks and are discussed in the next section on viruses and worms.

 

You may want to block some attachments and allow others. ISA Server allows you to block attachments based on file size and/or file extension.

Viruses and Worms

Viruses and worms cause a tremendous amount of damage to campus networks today. When the Nimda Internet worm hit college networks a few years ago, some of the smaller institutions (for example, Central Wyoming College) had to completely shut down in order to allow IT personnel to recover from the attack.

 

The prevalence of high speed Internet connections (cable and DSL broadband) has greatly increased the usability of the Internet for many students and instructors, but it has also resulted in faster spread of viruses, worms and other malicious code. The popularity of wireless networking on campus and availability of low-cost portable computers has made it easier than ever for network users to disseminate viruses either deliberately or accidentally.

 

Viruses and worm attacks are responsible for:

 

  • Destruction of data on servers and workstations
  • Denial of service attacks on servers and workstations
  • Lost employee and student productivity because a workstation or network server is unavailable
  • Distribution of sensitive departmental or student data via mass mailing worms
  • Increased administrative costs due to repairing damaged workstations and servers
  • Increased bandwidth use on the corporate network and Internet connection secondary to mass mailing worms and denial of service attacks
  • Destruction of corporate Web sites
  • Lost sales because of service unavailability

 

Colleges and universities are subject to civil litigation and liability if attackers use their computers to launch virus attacks or if confidential information is distributed. If you can’t show that you practice due diligence in your efforts to protect the network, you could find the institution involved in a costly lawsuit. It is not necessary for a plaintiff to prove any intent on the part of the institution; mere negligence can result in an expensive judgment. Educational institutions are also bound by federal and state laws safeguarding the privacy of certain records, and breach of that privacy can result in administrative penalties, fines and even criminal charges.

 

Historically, worms and viruses were introduced into campus networks by students and employees (internal users). Floppy disks and CDs containing infected files were the original avenue by which exploits got into the campus network. Floppy disks and CDs now represent a minor source of infection. Internet downloads using a variety of protocols are now responsible for the vast majority of virus and worm infestations.

 

How does virus and worm control at the firewall fit into the campus security plan? It does not take the place of anti-virus software, but “fills in the gaps” that AV programs installed on servers and client systems may not be able to cover. ISA Server 2000 can serve as a front line of defense against malicious code, supplementing whatever AV software you have in place.

 

Traditionally, campus networks have been less likely than corporate networks to have strong perimeter security. This was due less to ignorance or laziness than to philosophy: collaboration with other outside institutions is vital to research and free access to information is a cornerstone of the educational process. College and university officials today are becoming more aware of the importance of good risk management. However, administrators are often wary about limiting access that students, faculty and staff have come to expect, including remote access.

 

Tight budgets force many educational institutions to operate with minimal IT staffs, and with few or no full time security experts. In addition, many institutions are running older computers with operating systems that are more vulnerable and harder to secure. The internal network is often a hybrid one, with a variety of client, server and mainframe computers interoperating in the campus environment. Some of these machines, especially high-dollar research computers, may be so obsolete that they are no longer supported by their vendors and cannot easily be updated.

 

The most common Internet protocol used to download viruses and worms into the network is SMTP, which is used to send e-mail. Virus writers realize that e-mail is a vital function to all business and educational activities and they take advantage of this fact by crafting viruses and worms that spread via e-mail. Both sophisticated and unsophisticated users open e-mail attachments containing dangerous code. The code is released to the user’s computer and then spreads to the rest of the network. A single infected host can damage virtually every networked device in a short period of time.

An effective way to prevent e-mail borne attacks is to block all attachments at the perimeter. An application aware device can then examine e-mail attachments and perform one of the following actions:

 

  • Scan the attachment for viruses, worms and other dangerous code
  • Hold the attachment for later examination; this is sometimes referred to as “quarantine”
  • Forward the message, along with its attachment, to a security administrator’s e-mail account for inspection and analysis
  • Delete the message immediately
  • Forward the message to the mail server if the application layer filter determines that the attachment does not represent a risk to network security

 

Attachment blocking should be performed in conjunction with blocking of HTML mail for most effectiveness, since HTML messages allow attackers to embed viruses in the body of the message itself without relying on users to open attachments.

 

*       Note:
ISA Server can also use application layer inspection for outbound mail. An organization may wish to block outgoing viruses and worms in an effort to protect other Internet connected networks – and to avoid liability. In addition, outbound mail inspection can prevent users from sending attachment documents and other files that contain proprietary or confidential data that should not be disseminated outside the institution’s network.

 

Buffer Overflow Attacks

Attackers use buffer overflow attacks to disable network servers. Unlike the typical virus or worm attack, a buffer overflow cannot be protected against by blocking attachments. Hackers use buffer overflow exploits to disable specific server services with the intent of creating a denial of service – either by disabling a specific service on the target computer or by taking the entire machine offline. More elaborate buffer overflow exploits can be used to disable key security features and allow the attacker to run commands of his choice on the targeted machine.

 

Traditional firewalls will allow most buffer overflow attacks to go through to the network. Application layer filtering is required to catch most of these attacks. Even then, the application gateway must be properly designed. ISA Server’s intrusion detection system is specifically designed to intercept buffer overflow attacks.

 

SearchSecurity.com defines a buffer overflow in this way:

 

“A buffer overflow occurs when a program or process tries to store more data in a buffer (temporary data storage area) than it was intended to hold. Since buffers are created to contain a finite amount of data, the extra information - which has to go somewhere - can overflow into adjacent buffers, corrupting or overwriting the valid data held in them. Although it may occur accidentally through programming error, buffer overflow is an increasingly common type of security attack on data integrity. In buffer overflow attacks, the extra data may contain codes designed to trigger specific actions, in effect sending new instructions to the attacked computer that could, for example, damage the user's files, change data, or disclose confidential information. Buffer overflow attacks are said to have arisen because the C programming language supplied the framework, and poor programming practices supplied the vulnerability.”

 

A buffer, then, represents a contiguous chunk of memory allocated for some purpose. The C programming language doesn’t contain any automatic checks on the boundaries of buffers. Many of its functions don’t perform such checking. Thus, users with some knowledge of C can write programs that try to write data to memory beyond the memory that’s been allocated for the buffer. For example, if a buffer (location in memory) has 16 bytes allocated and a program command is written to copy 20 bytes to that location, an overflow will occur. This makes it possible for hackers to write code to perform whatever actions they want (for example, seizing administrative permissions) and place that code in the overflow area of the buffer.

 

To an extent, developers can prevent buffer overflows by writing program code in such a way that minimizes the use of C functions (such as strcpy) that don’t check buffer boundaries. Unfortunately, the possibility of buffer overflow can’t be entirely prevented. Buffer overflow attacks can be mounted against many different types of systems using different protocols. A common type of buffer overflow attack targets the educational institution’s Simple Mail Transfer Protocol (SMTP) servers and can stop the inflow of mail. The best way to prevent a buffer overflow attack against the SMTP server is to stop the attacker at the network perimeter, before the exploit ever finds its way into the campus network. An application aware device such as ISA Server 2000 can evaluate the SMTP commands sent through the firewall and stop the attack.


The ISA Server 2000 Server SMTP Filter and SMTP Message Screener

ISA Server 2000 is a sophisticated application layer aware firewall that can help solve the problems of unwanted email and dangerous e-mail attachments. The ISA Server 2000 firewall performs deep inspection of SMTP messages moving through it and blocks dangerous code and unwanted e-mail from entering the campus network.

 

ISA Server 2000 firewalls use two technologies to protect the e-mail servers on your campus network:

 

  • The SMTP filter
  • The SMTP Message Screener

 

When used in combination, the SMTP filter and SMTP Message Screener become powerful allies in the war against SMTP attacks and unwanted e-mail.

The SMTP Filter

The ISA Server 2000 SMTP filter examines SMTP commands sent by Internet SMTP servers and clients. This application layer filter intercepts SMTP commands and checks to see if they are larger than they should be. SMTP commands that are larger than RFC limits are assumed to be attacks against the SMTP server and are stopped at the perimeter by the ISA Server 2000 firewall’s SMTP application layer filter. You can configure the properties of the SMTP filter to specify a maximum command length for incoming SMTP commands, on a per-command basis. If this length is exceeded, the command will not be allowed through. You can also configure ISA Server to send an alert notifying you when an attempted buffer overflow is detected.

 


The figure below shows the flow of information and where the SMTP filter blocks the buffer overflow attack.

 

 


the figure below shows a partial list of the default commands included with the SMTP filer.

 

 

Each SMTP command has a Maximum Length associated with it. This length represents the number of bytes allowed for each command and the default values are based on RFC recommendations for the SMTP protocol. If an attacker sends a command exceeding the number of bytes allowed for the command, then the ISA Server 2000 firewall drops the connection and prevents the attacker from communicating with the corporate mail server. You can use the Edit button to modify the maximum command lengths.

 

The attack is also reported to the ISA Server 2000 Event Log and an e-mail can be sent to  administrators. Many pagers and mobile phones are connected to e-mail systems, so administrators can also be informed via pagers and mobile phone immediately.

 

NOTE: Feature Pack 1 includes enhancements to improve performance and allow you to set up an SMTP relay that can be authenticated to by external users without allowing spammers to misuse the relay.

The SMTP Message Screener

The SMTP Message Screener works together with the ISA Server 2000 SMTP filter. The latter is an application filter used to intercept all SMTP traffic arriving on TCP port 25 of the ISA Server 2000 firewall. The filter accepts the traffic, performs deep application layer inspection, and passes it on to the educational institution’s SMTP servers only if it passes inspection based on rules you configure.

 

The SMTP Message Screener component can filter incoming mail based on the following characteristics:

 

  • Source e-mail address
  • Source e-mail domain
  • Keywords in the e-mail subject line
  • Keywords in the e-mail message body
  • Attachment name
  • Attachment file extension
  • Attachment size

 

You can configure ISA Server to generate an alert if mail is received from specific users. In addition, the SMTP Message Screener can be configured to hold the e-mail for later inspection or forward the message to a security administrator’s e-mail box for further examination and analysis.

 

Figure C shows an example of how the SMTP filter can be configured to block keywords. One of the most common unwanted e-mail messages received on campus networks contains the keyword Viagra. This example shows how to block e-mail containing the word Viagra in the message header (subject line) or body. One of three actions can be taken when the e-mail message matches this rule:

 

  • Delete message
  • Hold message
  • Forward message to <specified e-mail address>

 

Figure C

Placing the ISA Server 2000 SMTP Filtering Firewall on Your Network

The ISA Server 2000 firewall can function as the only firewall on your network, or you can integrate ISA Server 2000’s powerful application layer filtering protection with your existing firewall infrastructure. Some common ISA Server 2000 network topologies are:

 

  • ISA Server 2000 acting as front-end firewall
  • ISA Server 2000 acting as back-end firewall
  • ISA Server 2000 acting as front-end and back-end firewalls
  • ISA Server 2000 acting as application layer filtering gateway in a perimeter network

 

 

ISA Server 2000 Front-end Firewall Topology

Smaller educational institutions that do not already have a large investment in a current firewall infrastructure may prefer to make the ISA Server 2000 firewall a front-end firewall. The front-end firewall has two network interfaces: a network interface on the campus network and a network interface directly connected to the Internet. All communications that come into and out of the campus network must go through the ISA Server and are exposed to ISA Server 2000’s deep application layer inspection.

 

The advantages of this configuration include:

 

  • All communications into and out of the campus network are exposed to firewall policy
  • You only need to learn how to configure the ISA Server 2000 firewall software; this avoids the potential for firewall misconfiguration when multiple vendor firewalls are used
  • All inbound and outbound access can be controlled on a granular user or group basis. Users only access the content and servers you want them to access, based on rules you configure
  • This configuration is easy to set up and maintain

 


The figure below shows the network topology for the ISA Server 2000 front-end firewall placement.

 

 

 

 

 


ISA Server 2000 Back-end Firewall Topology

Educational institutions that already have an existing firewall infrastructure may prefer to leave the current firewalls in place and put the ISA Server 2000 firewall behind the current firewalls. This topology allows third party firewalls to provide high speed packet filtering (stateful filtering) before forwarding the remaining packets to the application aware ISA Server 2000 firewall. The network between the third party front-end firewalls and the ISA Server firewall is a perimeter network where publicly accessible servers and services can be placed.

 

Each third-party packet filtering firewall has two network interfaces: an interface directly connected to the Internet and an interface connected to a perimeter network between the third-party packet filtering firewalls and the ISA Server 2000 application layer aware firewall. The ISA Server 2000 firewall also has two interfaces: an interface on the perimeter network and an interface on the protected internal campus LAN.

 

Advantages of this configuration include:

 

  • Educational institutions do not need to perform a major redesign of their current firewall infrastructures
  • Third party hardware-based firewalls can perform high-speed packet filtering. This offloads the packet filtering overhead from the ISA Server 2000 firewall and increases the resources available on the ISA Server 2000 firewall to perform deep application layer inspection
  • Resources located on the campus network are protected by the ISA Server 2000 firewall’s enhanced application layer inspection mechanisms
  • Granular inbound and outbound access control can be done on a user/group basis

 


The figure below shows the topology of the ISA Server 2000 back-end firewall topology.

 


ISA Server 2000 Front-end and Back-end Firewalls

The ISA Server 2000 front-end and back-end firewall configuration uses two ISA Server 2000 computers, one as the Internet edge firewall and the other as the corporate LAN edge firewall. The front-end ISA Server 2000 firewall has an interface directly connected to the Internet and an interface on the perimeter network between the firewalls. The back-end ISA Server 2000 firewall has an interface on the perimeter network and an interface on the protected campus LAN.

 

The advantages of this configuration include:

 

  • A single firewall system; this reduces the training overhead and the probability of a configuration error
  • Sophisticated application layer filtering protecting hosts on the perimeter network and the internal, core campus network
  • You can leverage Web Proxy chaining and firewall chaining to significantly increase access control from perimeter network servers and users on the internal network. This prevents attackers from using compromised servers on the perimeter network  as a launch point for outbound attacks from the perimeter network
  • Granular outbound user/group based access control for hosts on both the campus network and the perimeter network
  • Excellent support for highly secure VPN passthrough, allowing access to protected resources on the campus network

 

*       Note:

Please see kit doc Protecting Departmental/Student LAN segments with ISA Server 2000 for more information about Web proxy and Firewall chaining.

 

The figure below shows the topology for the ISA Server 2000 front-end back-end firewall configuration.

 

 

 


ISA Server 2000 Application Layer Filtering Web Proxy in the Perimeter Network

Some educational institutions already have an existing firewall infrastructure that includes front-end and back-end firewalls. These campuses have a large investment in their current firewall infrastructure and prefer to leave them intact. You can still leverage ISA Server 2000’s application layer filtering features by making the ISA Server an application layer filtering proxy. This ISA Server 2000 proxy can be placed on the perimeter network between the front-end and back-end third party packet filtering firewalls or you can place the ISA Server 2000 application layer proxy on the internal campus network.

 

Advantages of the application layer filtering proxy configuration include:

 

  • The ability to leave the current firewall infrastructure intact; you can “drop in” the ISA Server 2000 application layer filtering proxy virtually anywhere
  • The third party front-end and back-end packet filtering firewalls can pass packets at high speed while allowing ISA Server 2000 to provide a very high level of security for communications passed through its application layer inspection mechanisms
  • A hardened ISA Server 2000 proxy can be placed on the perimeter network segment to reduce the attack surface
  • In reverse Web Proxy scenarios, the ISA Server 2000 application layer filtering proxy can forward user credentials across the back-end firewall to pre-authenticate remote users

 

The figure below shows the topology of the application layer filtering proxy configuration.

 

 

 

SMTP Filter and Message Screener Scenarios: SMTP Relay and Message Screener on the Campus Network

The SMTP filter always runs on the ISA Server 2000 firewall computer. However, you can place the SMTP Message Screener on another computer located on a protected network behind the ISA Server 2000 firewall. The SMTP Message Screener can be installed in any of the following locations:

 

  • On the ISA Server 2000 firewall itself
  • On an independent IIS SMTP relay located behind the ISA Server 2000 firewall
  • On an Exchange Server

 

Message filtering requires