White Paper

Message Systems Extensibility


Abstract

"When implementing a new email security gateway into your infrastructure, the solution should adapt itself to your environment, not the other way around."

- Theo Schlossnagle
Chief Technology Officer
Message System


Message Systems is a high performance email application server. This next generation Message Transfer Agent (MTA) is designed to provide fast and flexible integration points with third-party message filtering solutions and to alleviate the need of excessive machinery to manage large email infrastructures.

Message Systems provides an extensive set of APIs that enable partners and customers to develop powerful custom solutions quickly and efficiently.

Background

Message Systems allows developers to customize the way messages are handled via an extensive Application Programming Interface (API). Potential customizations include:

  • Writing customized logging modules that record the exact audit data you need in the exact format you need.
  • Performs analysis on message failures to diagnose problem domains in real-time.
  • Virus-checking.
  • Spam Prevention.
  • Content Policy Enforcement.

This white paper is an overview of the extensibility of Message Systems solution. The design goals, implementation, and benefits of the Extension API will be covered.

Design Goals

The Message Systems Extension API is set of (3) interfaces for third-party extensions to validate, modify and log messages that are transported through the system. Extensions register callouts, which are functions that will be called on specific events. These callouts are tightly integrated into the core of the Message Systems solution and provide detailed access to both the entire message and any connection associated with it. Extensions are loaded as dynamic shared objects from the configuration file and are designed to have minimal impact.

Offering a C, C++, Java and a Perl interface to these callouts, Message Systems accommodates both the need for enterprise quality extensions from vendors, as well as simple scripted extensions designed by administrators. The tight integration into Message Systems means that even simple scripted extensions can be both powerful and efficient.

The following design goals were the driving forces in designing the extension interface:

  1. Performance. Extensions have minimal impact on Message Systems performance.
  2. Tight integration. Extensions should not require passing a message off to external processes. Extension code should be loaded directly by Message Systems and called as native code.
  3. Flexibility. Every step of the message delivery process, from initial reception to final delivery, should be modifiable.

Message Validation API

Message validation serves two main purposes. First, in the war on spam and viruses a variety of validation techniques are employed to reduce the influx of unwanted email. Second, many companies have policies that require them to audit all outbound messages to ensure, for example, that all outbound mail is audited for acceptable content. The Message Systems' Message Validation API provides a robust interface with hooks that are called at every stage of the SMTP reception process. At every step, the message and its associated connection can be analyzed and accepted, rejected, or modified.

The message validation API is ideal for scalable integration of third-party virus-checking and policy management software. Because message validation happens at the core of Message Systems, extremely high throughput can be maintained with simple validation modules.

Message Manipulation API

Many email solutions require the ability to modify the contents of messages in transit. Common mail-modification needs include incorporating virus-scanning software into the email server, performing policy management on incoming and outgoing messages, generating dynamic content, or guaranteeing that correct disclaimers are placed in all outgoing mail.

In addition to the content modification opportunities provided by the Message Validation API, the Message Manipulation API provides an additional callout that is executed from a pool of asynchronous threads before the message is written to disk. Off-loading this work into a separate thread allows for relatively complex processing and manipulation with minimal impact to the overall performance of the Message Systems solution.

Logging API

Audit logs are an important part of sending email. Not only does Message Systems log in its own native format, but many companies prefer to have their log files imported into a relational database for data mining. Message Sytems provides a full logging API that allows you to construct log files in with arbitrary format or logged to a database in real time. The logging API allows you to take action on a message at all of the following stages:

  • When a message is first received by Message Systems.
  • When a message is delivered to a remote server.
  • When a message encounters a temporary or permanent failure.

Perl for Power and Performance

Message Systems supports an integrated Perl interpreter that exposes the entire extension API. Using a scripting language for writing server extensions has many advantages over using a traditional compiled C extension:

  • Ideal for text. Perl is an ideal language for parsing and manipulating text. This makes it a perfect choice for embedding into an email solution for message processing and manipulation.
  • Fast Prototyping. Often an extension can be prototyped in hours instead of days or weeks. This makes it easy to evaluate the value of an extension without committing large amounts of time.
  • Safe. Coding in C exposes you to the possibility of segmentation faults and memory leaks. Using a high-level interpreted language allows you to focus on the big picture and let the language itself handle all the memory management.

Message Systems uses Perl's ithreads feature to offer full support in a multi-threaded environment. This allows asynchronous callbacks to use a pool of interpreters for maximum performance. When a Message Systems server is started, the callbacks to be registered are compiled and stored in memory for the life of the process as in Apache's mod_perl. This provides a substantial performance boost and invalidates many of the traditional speed issues involved with using an interpreted language.

Benefits

Re-use Your Existing Software. Many companies have invested substantial amounts of resources in tools to manage and process their existing MTA's audit logs. Customized logging modules allow you to ability to have Message Systems write logs in your format, extending the life of your tools.

Re-use Your Existing Software. Many companies have invested substantial amounts of resources in tools to manage and process their existing MTA's audit logs. Customized logging modules allow you to ability to have Message Systems write logs in your format, extending the life of your tools.

Don't Sacrifice Performance or Scalability. Traditional MTA architectures incur a significant performance hit when incorporating external applications. The Message System design allows extensions to execute with minimal overhead.