White Paper
Message Systems Performance
Abstract
"The performance of today's email systems hinders productivity. People simply can't accomplish what they need to due to architectural design flaws."
- Theo Schlossnagle
Chief Technology Officer
Message System
Message Systems is a high performance email application server. The Message Systems server, a next generation Message Transfer Agent (MTA) was designed to alleviate the need of excessive machinery to manage large email infrastructures.
Classic MTA design has proven inadequate for large scale email systems given the proliferation of unsolicited bulk email (spam), viruses, worms and phishing attacks.
This paper focuses on the techniques employed by Message Systems to leverage more performance from existing machinery.
Background
In 1982, the RFC draft of the Standard Mail Transfer Protocol (SMTP) was standardized. Since that date, email has become the dominant communication tool for business and personal use. Every day, trillions of email messages transit the networks that comprise the Internet and the volume is increasing month-by-month.
The situation has been compounded by the proliferation of unsolicited bulk email (spam), viruses, worms and other such annoyances that litter email systems with unwanted and potentially hazardous messages. Email system performance has become a necessary concern for entire organizations.
Concurrency
High volume email systems must be able to complete deliveries with high concurrency in order to perform as required. A typical email transaction from one server to another can take as long as a minute. It is absolutely essential that a solution be capable of handling multiple simultaneous transactions in order to sustain the transactional throughput required of an enterprise mail system.
As numbers scale, the need for concurrency becomes much more apparent. If an email system is required to deliver 100 email messages per second and each email requires an average of 10 seconds of wall-clock time to be delivered, the system must be able to concurrently manage 1000 sessions.
Necessity
Due to the proliferation of unwanted emails, companies are deploying a variety of defenses to protect their resources against 'attacks' from remote sites sending spam and viruses. One of the major methods used is known as tarpitting. Tarpitting is a self-defense technique that artificially prolongs the amount of time required to deliver a single message. The intent is to limit the rate at which a remote site can send email while still requiring that site to dedicate resources to the delivery. Servers which have a limited ability to manage concurrent connections or are unable to isolate and contain the resource utilization of a single destination domain are particularly susceptible to being crippled by even a single domain using tarpitting.
Existing Solutions
Typical email systems that exist on the Internet today use one of two key models.
The first is the multi-process model most prevalent on systems running UNIXTM or Linux. Multi-process systems are the oldest known architectural design still used in MTA's today. As can be assumed from its name, the multi-process model utilizes multiple separate system processes to deliver email concurrently. Multi-process architectures are employed by many of the 'standard' MTA's that ship with operating systems.
The second popular architectural paradigm is called a threaded model. The threaded model, quite similar in concept to the multi-process model, utilizes multiple threads within a single system process to deliver mail concurrently. Historically, multi-threading support in many operating systems was not as stable as multiprocessing. Due to this instability, only recent MTA systems use the threading model.
Both models scale poorly. Both processes and threads require operating system resources such as memory and CPU time to start, schedule, and cleanup. When the number of processes or threads approaches magnitudes such as 1000, the resource requirements become excessive and the resources that were being used to deliver email are being used by the system to manage the "model." Issues such as tarpitting exacerbate the problem and truly expose the flaws in these two models, therefore, causing these systems to become crippled.
Hybrid Architecture
A third popular model used for network daemons is called an event model. An event model uses a single thread in a single system process, but requests that the operating system notify it of events. This allows the process to utilize very few system resources such as stack space and scheduling time by only performing actions that will complete immediately using a technique known as non-blocking I/O. Effectively, this offloads the responsibility of "scheduling" the different sessions from the operating system to the application. The event model has a proven track record for applications that do not have computationally intensive or latent tasks - so called blocking operations - to perform.
Email solutions have both computationally intensive tasks (policy enforcement, spam and virus detection) and latent tasks to perform (writing messages to disk to provide integrity guarantees); however, the incredible performance advantages of the event model should not be lost due to this introduced obstacle.
It is clear that the multi-process model has inherent performance limitations due to the concurrency demands of the MTA combined with the resource demands of the model. The threaded model, however, is both light-weight (when used responsibly) and provides an alternate mechanism for performing blocking operations.
By developing a new model that is a hybrid between the high-performance event model for all network activity and the threaded model for all blocking operations, we created a best-of-breed architecture for delivering email.
Building the Hybrid Model
In order to build the hybrid model, the classic event model needed two specific modifications:
Most event-based solutions base their event system around the poll() system call. poll() is extremely portable, meaning that it exists on most systems, which makes coding applications based around poll() easy to port from one operating system to another. In addition to poll(), most UNIX variants have their own specialized event engine designed for optimal performance. Because a fast event system is critical to Message Systems performance, it employs the most efficient event system available on every supported platform: kqueue on FreeBSD, /dev/poll on Solaris, epoll on Linux, and watch_event or kqueue on Mac OS X.
The system was enhanced to allow delegation of blocking operations and computationally intensive tasks to worker threads that dispatch responses upon the completion of those operations. Running blocking operations in separate parallel threads overcomes all the normal bottlenecks present in an event system.
Message Systems unique scheduling engine incorporates network-based events, asynchronous I/O, timed events, and complex user-defined blocking operations such as message validation and modification.
Results
When put to the test, Message Systems implementation drives home the validity of the hybrid event-thread model. In laboratory tests, message throughput was well over 400 messages per second. Perhaps more impressive are the real-world deployments that sustain over 250 messages per second on commodity hardware.