OpenSMTPD: We deliver!

Éric Faurot

AsiaBSDCon 2013 - March 17, Tokyo, Japan

= Plan = * SMTP * OpenSMTPD ** Configuration, Queue * Internals ** Design, Workflow, Backends * General remarks == SMTP == = SMTP = * Venerable protocol * Exchange Internet messages ** as defined by RFC5322 (used to be 822) ** between mailboxes * Everybody has heard of it * Not going away anytime soon ** outlived Google Wave ** outlived Google Reader ** probably more to come

SMTP

Next relay is found using:

Specific route
DNS lookup for domain MXs

$ dig -t mx poolp.org
 ... some stuff...
 ;; ANSWER SECTION:
 poolp.org.  3589  IN   MX   200  poolp.no-ip.org.
 poolp.org.  3589  IN   MX     0  mx1.poolp.org.
 poolp.org.  3589  IN   MX    50  mx2.poolp.org.
 poolp.org.  3589  IN   MX   100  mx3.poolp.org.
 ... more stuff...

= SMTP Protocol = * Simple Client/Server protocol ** Line-oriented protocol ** RFCs 5321 + some extensions ** Command / Response * Three phases ** Initial handshake (connection and banner) ** Session parameters negociation (tls, auth) ** Mail transfer: SMTP Transaction = SMTP Protocol = Transaction: foo@bar.com mails gilles@poolp.org *C: MAIL FROM: <foo@bar.com> *S: 250 Ok *C: RCPT TO: <gilles@poolp.org> *S: 250 Ok *C: DATA *S: 354 Go ahead *C: message content *C: . *S: 250 OK = SMTP server = * Role of a SMTP server : route mails ** Receive incoming mail ** Deliver to local mailbox ** Transfer to next relay * Sounds easy * Seen as a difficult thing to setup, why? == OpenSMTPD == = OpenSMTPD = * Part of the OpenBSD project * Started a long time ago by gilles@ * 3 main developers * Very active in the last 12 months * Philosophy: ** Make simple things simple ** Routing logic easy to describe ** Clean flexible design ** Nice license (ISC)

Configuration

Minimalist setup

listen on lo0

table aliases db:/etc/mail/aliases.db

accept for local alias <aliases> deliver to mbox
accept for any relay

Configuration

Primary domain

listen on egress

table aliases db:/etc/mail/aliases.db

accept from any for domain "example.org" \
		 alias <aliases> deliver to mbox
accept for local alias <aliases> deliver to mbox
accept for any relay

= Configuration = * Simple ruleset * Syntax inspired by pf.conf * First match wins * Describe the routing logic ** Listeners ** Routing rules: conditions + action

Configuration

Using a smarthost

listen on lo0

# format:  "label   login:password"
table secrets file:/etc/mail/secrets
table aliases db:/etc/mail/aliases.db

accept for local alias <aliases> deliver to mbox
accept for any relay                    \
    via smtps+auth://label@smtps.my.isp \
    auth <secrets>

Configuration

Backup server

listen on egress

table poolp { poolp.org, opensmtpd.org }

accept for local deliver to mbox

accept from any for domain example.org relay \
    backup mx4.example.org
	
accept from any for domain <poolp> relay \
    backup mx2.poolp.org

Configuration

Signing outgoing mail with DKIM proxy

listen on lo0
listen on lo0 port 10029 tag DKIM

accept for local deliver to mbox
accept tagged DKIM for any relay
accept for any relay via smtp://127.0.0.1:10028

Configuration

Authenticating relay

listen on egress port submission tls \
                 certificate my.cert auth

accept from any for domain "opensmtpd.org" \
                 deliver to maildir

accept for any relay

Configuration

Deliver to virtual users

listen on egress

table usr { "alice" = "100:100:/var/vusers/alice",
            "bob" = "100:100:/var/vusers/bob" }
			   
accept from any for domain "wonderland.org" \
       userbase <usr> deliver to maildir
	   
accept for any relay

= Configuration = * Very easy to read * Can express complex routing strategies efficiently * Beware of the rule order though * Quite a lot of features ** authentication in and out, backup server, virtual domains, virtual users, selectable binding address, ... * Flexible table model = Tables = * Abstractions for list and mappings * Provide different services ** domain names, aliases, userbase, creds... ** depends on the context * Implemented by backends ** static ** file ** db ** sqlite, ldap (experimental)

Queue

Set of mails the server is responsible for
Message
- Unique 32bit id
- Content of a mail
Envelope
- Unique 64bit id
- Destination to route the message
- Types: relay, local delivery, bounce

Queue

# find -f /var/spool/smptd/queue
/var/spool/smtpd/queue/90/90fa32a2/90fa32a23acab696
/var/spool/smtpd/queue/90/90fa32a2/90fa32a273da3247
/var/spool/smtpd/queue/90/90fa32a2/message
/var/spool/smtpd/queue/34/3475fa58/3475fa584b7239a9
/var/spool/smtpd/queue/34/3475fa58/message

Queue

# cat /var/spool/smtpd/queue/34/3475fa58/3475fa584b7239a9
version: 1
helo: shear.ucar.edu
hostname: lists.openbsd.org
sockaddr: 192.43.244.163
sender: owner-misc+M130340=mailing=poolp.org@openbsd.org
rcpt: mailing@poolp.org
dest: mailing@poolp.org
ctime: 1363275217
expire: 345600
type: mda
mda-method: mda
mda-usertable: <getpwnam>
mda-buffer: /home/mailing/archiver.py
mda-user: mailing
last-bounce: 1363289617
retry: 113
errorline: "Exception: 403"

= Administration = * Through /usr/sbin/smtpctl ** Local enqueuer (sendmail) ** Queue inspection ** Mail removal ** Pause/resume ** Retrieve stats ** Monitor server activity ** etc. * Talk to /var/run/smtpd.sock

== Internals == = Design = * OpenBSD daemon style ** Multiple processes ** Privilege separation ** Talk via imsg(3) ** FD passing * Split the general task into small units of work

Process layout

= Process layout =

SMTP

* Server-side SMTP sessions * Supports: ** TLS, smtps ** Auth: PLAIN and LOGIN * Automagically limits incoming sessions * Force disconnect if client hog the session * Runs unprivileged * Chrooted to /var/empty = Process layout =

FILTER

* Perform SMTP filtering * Manage chained processes * Mostly there * Unprivileged * Chrooted to /var/empty = Process layout =

TRANSFER

* Client SMTP sessions for relaying * Unprivileged * Chrooted to /var/empty = Process layout =

DELIVERY

* Handle local deliveries ** mbox ** maildir ** mda ** lmtp (new) * All through parent * Unprivileged * Chrooted to /var/empty = Process layout =

CONTROL

* Handle smtpctl connections * Gather statistics * Unprivileged * Chrooted = Process layout =

LOOKUP

* Performs all lookups: ** DNS (asynchronous) ** user credentials ** certificate verification ** ruleset evaluation ** alias expansion * Unprivileged * Not chrooted (needs /etc/resolv.conf) = Process layout =

PARENT

* Setup everything: read config, fork others, open listeners, enqueue offline mails * Specific privileged tasks ** Fork mda for the delivery agent ** Performs privileged auth checking ** Open .forward files * Runs as superuser * Not chrooted = Process layout =

SCHEDULER

* Knows all exisiting envelopes * No persitent data * Decide when to relay/deliver an envelope * Can be queried by the admin ** Get state ** Force scheduling/removal ** Pause/resume * Unprivileged * Chrooted to /var/empty = Process layout =

QUEUE

* Manage access to persistent storage ** messages ** envelopes * Feed the scheduler on startup * Enqueue bounced messages * Unprivileged * Chrooted to spool directory * Can run as a separate user

Enqueueing

Envelope expansion

Scheduling

On temporary failure:

Quadratic delay: d = \frac{k.n^2}{2}
May bounce a warning

= Relaying = * Handled by TRANSFER * Maintain a list of tasks (transactions) per destination * Establish sessions with relays ** Limit connections to the same host ** Wait a bit between each connection ** Sessions pick job * Loop detection = Delivering = * Handled by DELIVERY * Maintain a per-user list of deliveries * Request a fd to write the content ** PARENT forks a mda * Limit number of running deliveries * Loop detection

= Bouncing = * Notification that something went wrong * Bounces happen: ** On permanent errors when delivering/relaying an envelope ** When an envelope expires ** When an envelope is stuck (delayed) * Two types: ** Failure ** Notice

Bouncing

= Bouncing = * How bounces are handled: ** Specific envelope type for bounce request ** Re-injected through internal SMTP session ** Grouping when possible

Bouncing

= Backends = * Queue and scheduler have backends * Functionality implemented using a simple internal API * Enforce a semantics * Any compliant implemention can be used

Backends: queue API

enum queue_op {
  QOP_CREATE,
  QOP_DELETE,
  QOP_UPDATE,
  QOP_WALK,
  QOP_COMMIT,
  QOP_LOAD,
  QOP_FD_RW,
  QOP_FD_R,
  QOP_CORRUPT,
};

struct queue_backend {
  int (*init)(int);
  int (*message)(enum queue_op, uint32_t *);
  int (*envelope)(enum queue_op, uint64_t *, char *, size_t);
};

Backends: scheduler API

struct scheduler_backend {
  void   (*init)(void);
  void   (*insert)(struct scheduler_info *);
  size_t (*commit)(uint32_t);
  size_t (*rollback)(uint32_t);
  void   (*update)(struct scheduler_info *);
  void   (*delete)(uint64_t);
  void   (*batch)(int, struct scheduler_batch *);

  size_t (*messages)(uint32_t, uint32_t *, size_t);
  size_t (*envelopes)(uint64_t, struct evpstate *, size_t);
  void   (*schedule)(uint64_t);
  void   (*remove)(uint64_t);
};

= Backends = * Scheduler backends ** ramqueue (default) ** null * Queue backends ** fs (default) ** null ** ram ** REST (PoC) * Useful to validate APIs and run tests == General remarks == = Security = * Technical means ** Privilege revocation ** Chroot ** No auth on unsecure connections ** Queue can run as specific user ** Limited superuser delivery options * Design principles ** Simple and compact config ** Spot errors just by reading aloud ** Sane defaults = Reliability = Nobody likes to lose mails

* There might be bugs ** Lots have been fixed already ** Well tested ** Please report weird behavior = Reliability = * Transient errors ** Can happen during lookup, IO, etc... ** Rollback is always possible ** Robust to admin intervention * Unlucky shutdown? What might happen: ** Client could not be notified of accepted mail ** Queue missed a transfer/delivery notification ** At worse, a duplicate = Performances = * Not specifically designed for performance * Stands the comparison with other servers * Benchmarks show that bottleneck is the FS * Real-world deployment: ** Large infrastructure ** Relayed millions of mails ** Very satisfying rate ** Low memory usage * Not really an issue = Good and ba^Wless good = * Advantages ** Simple to configure and administer ** Handle complex routing setups ** Low memory footprint * Limitations ** Fewer features (for now) ** Limited built-in spam filtering ** No real support for masquerading

Well suited for lots of use-cases

= Portability = * Main development on OpenBSD * Portable version (thanks chl@) ** Added features (PAM support) ** Runs on multiple systems *** NetBSD, FreeBSD (in ports) *** Linux (gentoo, archlinux, debian) *** MacOS X *** Nokia N900!

We are looking for packagers!

= Roadmap = * First stable release now * Will ship with OpenBSD 5.3 * Next major release in about six months ** More cleanups ** Filter infrastructure in place ** More table backends ** External queue and scheduler backends ==Thank you!
Questions?==