« Back to Self-Hosting

Running your own mail server with OpenBSD and OpenSMTPD

Introduction

Self-hosting your own services such as a blog, website, or email is a valuable skill. It gives you a better understanding in the technology and usually also fun to do. With privacy under pressure and Big Tech continuing to store our data, this might be a good time to host your own mail server.

Email is a great service to host yourself, as it is based on decentral protocols. Each company, society, or individual can run a system of their own. Or, let someone else do it. If you wan to take back some control about your messaging needs, then this is the article for you.

Hosting your own email, together with your own domain name, gives you back the control over your communications. Make as many email addresses as you want and block them when spam comes in. Unique addresses helps also with finding the source of spam and decouple purchased services.

Before we start

This article is meant for those that really want to set up a secure system that can be fully tailored to their needs. You will learn a lot along the way, from DNS records, using SSL certificates, up to configuring the different software components.

Target audience

  • Technical (usually Linux and/or BSD users)
  • People who like to dig deep and understand the full system
  • Anyone just learning about the possibilities or learn something new about OpenBSD or spam filtering

Provide feedback

We are living in a time that AI is creating content on the web. This article is 100% made by a human being and a way to contribute back to the BSD community. It may contain errors, missing information, or unclear sentences. Do you catch something that is unclear or incorrect? Send me any feedback so this article can be further improved.

Software components

This article will cover the following software components:

  • acme-client
  • Dovecot
  • httpd
  • OpenBSD
  • OpenSMTPD
  • Rspamd
  • unbound
  • unwind

Most of these components will be covered in-depth, including its configuration. Some will be mentioned, but covered in a separate guide.

Required knowledge and parts

Setting up your own mail server server requires some technical knowledge. Some parts are fairly easy to set up, others require some attention to detail. If you just run a service daemon to accept mail, then you might end up being blocked by most of the internet. So setting up things carefully and correctly from day 1, will save you some trouble later.

This article will use OpenBSD, so if you have already experience with it, great. Used Linux before, then you might need to learn a few more things, mainly for system administration. Besides some operating system knowledge, you need to know or learn about DNS, some generic concepts of a mail server and mail boxes. While it all not being rocket science, there are definitely a few tricky things to get right. But when you do with the help of this article, you got a smooth running mail server.

Requirements:

  • Domain name
  • Hosted virtual machine
  • Access to DNS
  • Time

Domain name

If you don’t have domain name yet, you may need to require one. Typically this is done with the help of registrar, often a hosting provider. Don’t know who to use as a registrar? Ask around with friends or on Mastodon.

Hosted virtual machine

While you can host a virtual machine yourself, that is probably not the wisest thing to do. Internet connections of consumers are usually flagged as suspicious. In the past they were a frequent source of spam, so internet providers started to block outgoing mail from their customers. Also the receiving end might block incoming connections from dynamic blocks intended for home internet connectivity.

It’s better to use a dedicated hosting company for your virtual machine. Reliability and trustworthiness are important. IP addresses and network blocks can be compared with good or bad neighborhoods. Are all the IP addresses around you spamming, scanning the web, and post bad stuff? Then this might affect the IP address of your virtual machine as well.

For hosting my mail server, I used OpenBSD AmsterdamExternal link , which provides virtual systems running OpenBSD. Their audience is technical and being all BSD oriented, they are typically good netizen.

Disclaimer: I know the owner of OpenBSD Amsterdam, but he did not know upfront that I would become a customer, nor that I would create this article. While he provided feedback, he has no say in these words.

After filling in the form to order a virtual machine, a welcome email was received with the details. It includes IP addresses, relevant hostnames, and tips. OpenBSD Amsterdam gives you access using SSH (public key authentication with your provided SSH key) and access to the console. This way you still manage the system if you lock yourself out, for example after applying too tight firewall rules.

DNS configuration

An important part of running a mail server is configuring the DNS records correctly. This way other mail servers know how to connect yours and to validate several pieces of a bigger puzzle, such as SPF, DMARC, and more.

There are several types of DNS records. If you are less familiar with them, here is an introduction in several common types.

TypePurpose
Alink an IP address to a hostname
AAAAlink an IPv6 address to a hostname
MXdefine which servers are accepting email for the domain
TXTcontains information with a wide variety of goals, such as domain validation, SPF, DMARC, and others
PTRtranslate IP address to a hostname
CAAdefine entities that are allowed to register a certificate

Usually we use a name (hostname) when configuring DNS records. If there is not a specific name, but you want to refer to the domain itself, then use a at-sign (@) as its name. The record type always has to be defined. Depending on the record type, you then fill in also a value, such as an IP-address, hostname, or text string.

First steps before we begin

Don’t just select a cheap hoster and install a system and hope for the best. As shared earlier, the IP address is an important piece in being able to reliably deliver messages to other. Take some preparation steps to validate that nothing is wrong with the IP addresses assigned to your system.

  • Does the hoster already have a bad reputation for hosting spammers?
  • Is the IP address that will be assigned to your virtual machine dedicated to your system only?
  • Does the IP exist on a blacklist (RBL), e.g. Spamhaus?
  • Can you define the reverse DNS record (PTR)?

The target audience of OpenBSD Amsterdam are technical people using OpenBSD. They benefit, like myself, if we all behave. Most of the BSD people like to be friendly netizen and don’t become a source of trouble, like spam and unauthorized scans. That alone makes me believe that this IP block I’m part will have a better-than-average reputation.

There are several online sources to test your IP address and if it listed on a blacklist or blocklist. For example, MXToolBoxExternal link is well-known for this purpose. Check both the IPv4 and IPV6 address. Another interesting source is the website of Talos IntelligenceExternal link to retrieve information about the IP addresses.

Basic configuration of the system

Adjust OpenSSH

Since we use SSH via public key authentication, let’s disable normal password authentication.

rcctl set sshd flags -o PasswordAuthentication=no

Optionally, you can adjust the SSH port in /etc/ssh/sshd_config. It will not make the system more secure, but may reduce log clutter by a lot and even reduce some unneeded traffic.

It is possible to use a flag for the port number similar to that of PasswordAuthentication. Even if the configuration gets lost, it is easy to restore via console access.

Restart the SSH daemon after the change:

rcctl restart sshd

This might be a good time to check in a new terminal window that the connection on the new port still works.

Want to be sure that authentication is turned off? ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password -p PORT USER@HOSTNAME

Firewall: pf

Especially when the system is not actively in use yet, it is a great time to set up a firewall. This way we can poke some hole in the configuration based on actual need, instead of closing things down later on. Just make sure that SSH access is allowed for now. Then later add those additional services, like SMTP and IMAP.

OpenBSD uses pf, a great filtering solution. With just a few lines you may already have a great functional filter to block out the onslaught of bad bots.

Want to more about pf? There are a lot of good tutorials available. If time allows, another one will be published here later.

Good to know and check:

  • Time synchronization: ntpctl -s all
  • Update packages and updates: syspatch -c
  • Retrieve a TXT record to validate DNS lookups functioning: host -t txt openbsd.org

DNS caching with unbound and/or unwind

OpenBSD provides the option to use the unwind service. This light daemon helps with caching (remember) of all DNS requests. DNS is used to translate between computer hostnames and their IP addresses. As a mail server, and especially spam filtering, may cause a lot of DNS requests, setting up caching is a requirement. Don’t skip it and do it right away.

Open /etc/unwind.conf and define the DNS servers that will do DNS lookups. There are few options, but let’s start with an example of what the configuration may look like:

forwarder { 2620:fe::fe DoT 2620:fe::9 DoT }
preference { DoT }

Confirm that the configuration of unwind is correct with unwind -n before starting it.

rcctl enable unwind

Start unwind to it becomes active:

rcctl start unwind

Confirm that /etc/resolv.conf was updated and that it points to localhost (127.0.0.1) with a reference to unwind. Example output:

nameserver 127.0.0.1 # resolvd: unwind
lookup file bind

If you want to go a step further, using unbound is another great option. Configuring unbound is outside the scope of this article. If you decide to take this route, then adjust your /etc/resolv.conf file. Most likely you might need to disable resolvd to prevent dynamic configuration of the DNS server(s) when DHCP is being used.

Another option is to use both. In that case run Unbound on port 5353 so it does not collide with unwind. By using unwind, you don’t have to fiddle with resolvd and when Unbound goes down, Unwind will temporarily select another DNS server. In /etc/unwind.conf configure that our instance is listening on the localhost (IPv6) on port 5353.

forwarder {
    ::1 port 5353
}

Important note: do not use a generic DNS resolver for your configuration. Your mail server may do a lot of DNS records in a short time, based on the checks that are performed. Common DNS resolvers such as those of Google, Cloudflare, and Quad9 might be rate-limited. This may impact your DNS resolution and proper processing of incoming email.

Email security, privacy, and anti-spam

There are several technologies that boost privacy, confidentiality, or the authenticity of email.

SPF

SPF, short for Sender Policy Framework, tells other servers which servers are allowed to send mail for your domain. It is like an allow list and only trusted systems are included.

DKIM

DKIM (DomainKeys Identified Mail) helps by digitally signing messages. Only signed email with the right key, is considered to be legitimate. Your mail server will need to be configured to use the secret key to sign, and the public key is then used to see if it matches. The DKIM signature will be included in the headers of email messages. The receiving system, can then use DKIM to do the validation. If there is a match, you can be (more) sure that the right entity send out the email. A bit like SPF, but on a different level.

DMARC

Using SPF and DKIM is a good start. The receiving server can do several checks and see if things look good on a first glance. With DMARC on top, we can also instruct the receiving party what to do if something is wrong, like an incorrectly signed message, or an IP address sending our mail that was not on the allow list.

DMARC defines what should happen if something is wrong, like still accepting it or rejecting it altogether.

DMARC also has alignment, which makes things a little bit more complex. Email can have a different email address as the sender and what is presented in the ‘From:’ header of the email. This may happen when forwarding email or solutions like running mailing groups. So if you use this functionality, have a good look at the alignment specific setting of DMARC.

A few tips:

  • Read up on DMARC, it has several options
  • Receive messages about DMARC issues with rua
  • Start with p=none for an existing domain, until you got everything configured correctly, then switch to a more strict configuration. Add this to you To Do list. For a new domain, go for strict from the very first day.

MTA-STS

Besides SPF, DKIM, and DMARC, is there also MTA-STS. It is short for Mail Transfer Agent Strict Transport Security and can be compared with HTST, where it forces HTTPS usage. Now with email that translated to we only want to accept encrypted email.

To leverage MTA-STS, a web server is needed. Fortunately, OpenBSD provides the httpd service for this.

URL: https://mta-sts.example.org/.well-known/mta-sts.txtExternal link (replace example.org)

Create the directory structure with the help of mkdir using ‘-p’ to create the full path.

mkdir -p /var/www/htdocs/mta-sts.example.org/.well-known

Change the ownership, including any subdirectories and files.

chown -R www:www /var/www/htdocs/mta-sts.example.org

Create the file mta-sts.txt:

vi /var/www/htdocs/mta-sts.example.org/.well-known/mta-sts.txt

Add enforcement instructions. Example:

version: STSv1
mode: enforce
mx: mail.example.org
mx: mail2.example.org
max_age: 604800

What does this all mean? Let’s have a look:

  • version: version of STS, which is currently STSv1
  • mode: enforce, only allow encrypted connections
  • mx: for each mail server a single line (this example has two)
  • max_age: how long the content of this file can be cached in seconds (604800 is a week)

Create the file and then later we complete it with a virtual host configuration.

Configure DNS

With all this knowledge about DNS, time to set things up. When the DNS configuration is complete, the last parts for SPF/DKIM/DMARC/MTA-STS can be completed.

A/AAAA records

  • Name: mail
  • Type: A / AAAA
  • Points to: IP address of server
  • Related protocols: HTTP/HTTPS/SMTP

Set up a record for IPv4 (A) and one for IPv6 (AAAA).

  • Name: mta-sts
  • Type: A / AAAA
  • Points to: IP address of server
  • Related protocols: HTTPS

Again one for IPv4 and one for IPV6.

MX records

The MX records define our server(s) that can receive the incoming email. If you are serious about your email, you can create a second server that accepts the email when the primary is down.

  • Name: @
  • Type: MX
  • Value: 10
  • Points to: mail.example.org

If you use a backup mail server, it gets a higher number (e.g. 20). The MX record with the lowest number is considered to be tried first. If you have multiple MX records with the same value, then both can be used. Please note that spammers may try to use the backup server, as they may have less spam training or weaker configurations.

CAA record

With the help of Let’s EncryptExternal link we can create some certificated. To define that only this entity may provide certificates, we can lock this down.

  • Name: @
  • Type: CAA
  • Points to: 0 issue “letsencrypt.org”

This specific value consists of a flag, tag and text value:

  • Flag: 0 means ’non-critical’ and that no special flags needs to be set
  • Tag: issue normal certificate (instead of wildcard)
  • CA: define the entities that may create a certificate for our domain

SPF configuration

SPF defines who can send emails that originate from our domain. You have to decide the best match:

  • “v=spf1 -all”: no system may send email originating from my domain (perfect for domains that have no email)
  • “v=spf1 a:mail.example.org -all”: only ‘mail’ can send
  • “v=spf1 a:mailrelay.example.org mx -all”: the mailrelay host can send, so can the systems linked to our MX records (more overhead, as it requires multiple DNS lookups!)
  • “v=spf1 ip4:46.1.2.4 ip6:2a03:6000:abcd:efab::4 -all”: only the mentioned IPv4 and IPv6 addresses mentioned may send out email

I prefer the last one. Explicit and the least amount of DNS lookups required.

DMARC

DMARC helps to define what should happen when something is wrong, such non-matching SPF or DKIM values.

  • Name: _dmarc
  • Type: TXT
  • Point to: v=DMARC1;p=reject;sp=reject;adkim=s;aspf=s;rua=mailto:dmarc@example.org

This sets everything to reject emails that are not properly signed according to DKIM or when the sender system is not listed in the SPF records. Also the alignment is set to strict.

DMARC can also be defined very loose, which might be a good one for existing domains. Just do some initial monitoring, then set it to a stricter policy.

  • Points to: v=DMARC1;p=none;rua=mailto:dmarc@example.org

DMARC has many specific settings and you are advised to read a bit more about it, to better understand what it does.

Primary domain

When using multiple domains, you can have the other domains use the primary domain in the configuration. For the primary domain itself we do a “normal” configuration.

  • Name: @
  • Type: TXT
  • Points to: v=spf1 ip4:1.2.3.4 ip6:2a03:abc:abc:abc -all

Want other domains to use this primary domain for their configuration? Then define an additional record on the primary domain that can be used as an ‘include’ within the other domains: _spf ➞ v=spf1 ip4:1.2.3.4 ip6:2a03:abc:abc:abc -all

Example when using it within the other domains and pointing to the primary domain: @ ➞ v=spf1 include:_spf.example.org -all

PTR records (reverse DNS)

Another important step is setting up reverse DNS, the so-called PTR records. These are to be configured with your (web) hoster, the one owning the IP block. We link the IP address back to the name, that is why we call this reverse DNS. How to configure this is different for each hoster. So look for ‘PTR record’ in the web interface of your hoster or frequently asked questions.

To check if all is set up correctly:

host -t ptr 1.2.3.4

When checking the output of your IP addresses (check both IPv4 and IPv6!), confirm that it points back to the hostname of your mail server.

Web server configuration

Open /etc/httpd.conf and add the virtual hosts. We can add a generic section that will cover the challenges on port 80 by using a wildcard (*). This particular configuration snippet will help proving Let’s Encrypt that we have ownership over this server. Only then it will issue a certificate.

server "*" {
    listen on * port 80

    # ACME http-01 challenge
    location "/.well-known/acme-challenge/*" {
        # /var/www/acme
        root "/acme"
        request strip 2
        directory no auto index
    }
}

server "example.org" {
    listen on * tls port 443

    tls {
        certificate "/etc/ssl/example.org.fullchain.pem"
        key         "/etc/ssl/private/example.org.key"
    }

    gzip-static
    hsts
    root "/htdocs/example.org"
}

server "mta-sts.example.org" {
    listen on * tls port 443

    tls {
        certificate "/etc/ssl/mta-sts.example.org.fullchain.pem"
        key         "/etc/ssl/private/mta-sts.example.org.key"
    }

    hsts
    root "/htdocs/mta-sts.example.org"
}

Start httpd and make it active for the next reboot of the system.

rcctl start httpd && rcctl enable httpd

Optional: feel free to create a generic page (/var/www/htdocs/index.html) with some text.

This configuration now can be used by acme-client, which will request the certificates at Let’s Encrypt.

Certificates for HTTPS

OpenBSD uses the acme-client to request Let’s Encrypt certificates. This way we can provide an HTTPS connection, which is also required when using MTA-STS.

Open /etc/acme-client.conf and define below the authority section the individual hostnames that require a certificate.

authority letsencrypt {
    api url "https://acme-v02.api.letsencrypt.org/directory"
    account key "/etc/acme/letsencrypt-privkey.pem"
}

domain "example.org" {
    alternative names { "www.example.org" }
    domain key "/etc/ssl/private/example.org.key"
    domain full chain certificate "/etc/ssl/example.org.fullchain.pem"
    sign with letsencrypt
}

domain "mail.example.org" {
    domain key "/etc/ssl/private/mail.example.org.key"
    domain full chain certificate "/etc/ssl/mail.example.org.fullchain.pem"
    sign with letsencrypt
}

domain "mta-sts.example.org" {
    domain key "/etc/ssl/private/mta-sts.example.org.key"
    domain full chain certificate "/etc/ssl/mta-sts.example.org.fullchain.pem"
    sign with letsencrypt
}

If all is set, request a certificate, for example for our ‘mail’ hostname:

acme-client -v mail.example.org

With crontab -e we can edit the scheduled tasks and tell it to run a daily/weekly task for the renewals.

30 	4	*	*	1	/usr/local/scripts/renew-certificates

Example script (/usr/local/scripts/renew-certificates):

#!/bin/sh

set -o nounset

NEEDS_HTTPD_RELOADED=0

for D in $(awk '/^domain/ {print $2}' /etc/acme-client.conf | tr -d '"'); do
    #echo "Domain: ${D}"
    /usr/sbin/acme-client "${D}"
    if [ $? -eq 0 ]; then NEEDS_HTTPD_RELOADED=1; fi 
done

# Only reload http daemon when a certificate was renewed or updated
if [ ${NEEDS_HTTPD_RELOADED} -eq 1 ]; then
    /usr/sbin/rcctl reload httpd
fi

exit 0

# EOF

OpenSMTPD

To use SQLite, install the package:

pkg_add opensmtpd-table-sqlite

Create mailbox location

You can choose where you want to store the mailboxes. I prefer them in the home directory of the vmail user. So it has to be created first.

mkdir -p /home/vmail/mailboxes

Create the account:

useradd -c "Mail for virtual users" -d /home/vmail -s /sbin/nologin -u 2000 -g =uid -L staff vmail

Change ownership of the directory:

chown -R vmail:vmail /home/vmail

Define database structure and content

Open with the help of the sqlite3 the database:

sqlite3 /etc/mail/smtpd.sqlite

Create the database table domains for your domains. The table credentials contains the accounts and virtuals the virtual email addresses and aliases.

CREATE TABLE domains (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    domain VARCHAR(255) NOT NULL
);

CREATE TABLE credentials (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    email VARCHAR(255) NOT NULL,
    password VARCHAR(255) NOT NULL,
    is_active INTEGER DEFAULT 1
);

CREATE TABLE virtuals (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    email VARCHAR(255) NOT NULL,
    destination VARCHAR(255) NOT NULL,
    comment VARCHAR(255),
    hyperlink VARCHAR(255),
    is_active INTEGER DEFAULT 1,
    expires_after DATE DEFAULT '2099-12-31'
);

Add a domain into the related domains table using the INSERT statement:

INSERT INTO domains (domain) VALUES ('example.org');

Got more domains? Then repeat this for every domain.

Create one or more accounts. For this we use the email address (for authentication purposes, similar to username) and create a password. The command smtpctl encrypt will wait for you to insert some text, then return the hashed password string

# smtpctl encrypt 
paste-your-password-here
$2b$09$CVUigOvEqpik3GAOcUaZ4.tyYzU1bvOeIRexAlBZ38QLT12AZoxh6

So we use this last line in our SQL query:

INSERT INTO credentials (email, password, is_active) VALUES ("account123@example.org", "$2b$09$CVUigOvEqpik3GAOcUaZ4.tyYzU1bvOeIRexAlBZ38QLT12AZoxh6:", 1);

Just the account is not enough to receive email. Link the email address and any additional aliases.

INSERT INTO virtuals (email, destination, comment, is_active) VALUES("account123@example.org", "vmail", "Primary email address for Account 123", 1);
INSERT INTO virtuals (email, destination, comment, is_active) VALUES("mailalias1@example.org", "account123@example.org", "First alias for Account 123", 1);
INSERT INTO virtuals (email, destination, comment, is_active) VALUES("mailalias2@example.org", "account123@example.org", "Another alias for Account 123", 1);

The first line links the email address to the ‘vmail’ system account. For every account, create such a related entry. Then define any additional aliases, by linking them to the primary email address of the account.

Open cat /etc/mail/sqlite.conf and define the path of the database, including the related database queries.

# Database
dbpath /etc/mail/smtpd.sqlite

# Queries
query_alias       SELECT destination FROM virtuals WHERE email=? AND is_active=1 AND current_date < expires_after;
query_credentials SELECT email, password FROM credentials WHERE email=? AND is_active=1;
query_domain      SELECT domain FROM domains WHERE domain=?;

OpenSMTPD configuration

Open /etc/mail/smtpd.conf to configure OpenSMTPD. Below is an example configuration. I suggest adding a reference to this website (with version), so that you can refer to it if any changes were made.

# OpenSMTPD configuration v2026-01
# Based on the work of https://bsd-audit.com

### Tables

# Primary tables for mailboxes
table aliases_local file:/etc/mail/aliases
table credentials sqlite:/etc/mail/sqlite.conf
table domains sqlite:/etc/mail/sqlite.conf
table virtuals sqlite:/etc/mail/sqlite.conf

# Other tables
table bad_rdns_regexes file:/etc/mail/block-rdns-regexes.list
table bad_sender_from_regexes file:/etc/mail/block-sender-regexes.list
table bad_sender_domains file:/etc/mail/block-sender-domains.list

### Filters

# Filter to combat spam
filter rspamd      proc-exec "filter-rspamd"
#filter senderscore proc-exec "filter-senderscore -junkBelow 70 -slowFactor 5000 -scoreHeader"

filter check_fcrdns                  phase connect match !fcrdns junk
filter check_rdns                    phase connect match !rdns junk
filter check_rdns_for_dynamic_ranges phase connect match rdns regex <bad_rdns_regexes> junk
filter blocklist_sender_from         phase mail-from match mail-from regex <bad_sender_from_regexes> reject "550 bad sender?"

### Special options

smtp sub-addr-delim '_' # use underscore (_) instead of plus sign (+) as a delimiter, as a + is not always accepted and an underscore is typically not seen as a way to create multiple aliases referring to the same e-mail address 

### Certificates

pki mail.example.org cert "/etc/ssl/mail.example.org.fullchain.pem"
pki mail.example.org key "/etc/ssl/private/mail.example.org.key"

### Mail routing

# Listen on localhost
listen on lo0

# Listen on our external interface to receive mail from other systems
listen on egress port  25 tls         pki mail.example.org protocols "secure" ciphers "secure" filter { blocklist_sender_from, check_fcrdns, check_rdns, check_rdns_for_dynamic_ranges, rspamd, senderscore }

# Listen on our external interface to receive mail from trusted users for relaying purposes
# (auth requires authentication and mask-src removes the originator 'from' field)
listen on egress port 587 tls-require pki mail.example.org protocols "secure" ciphers "secure" auth <credentials> mask-src filter { rspamd }

### Actions

action "local_mail" mbox alias <aliases_local>
action "outbound" relay helo mail.example.org
action "store_mail" maildir "/home/vmail/mailboxes/%{dest.domain}/%{dest.user}" virtual <virtuals>

### Matches

# Store mail for valid user aliases
match from any for domain <domains> action "store_mail"

# Local mail can be delivered to local aliases
match from local for local action "local_mail"

# Allow authenticated users or from the local system itself to send out e-mail
match from any auth for any action "outbound"

# The local system may send mail out if needed
match from local for any action "outbound"

# EOF

Notes:

  • SenderScore is a method to check an incoming IP address on a rated trust score. The service that does this lookup is rate-limited and very easily returns code 127 meaning you hit the limit. As I did not get it properly to work, it has been disabled in this configuration and article.

Spam filtering

Besides the basic filters in OpenSMTPD to block spammers, the more powerful Rspamd software can be added. To make them work together, a filter is required. Also the installation of Rspamd and Redis is needed. Redis is used to store dynamic data, such as learning from discovered spam and valid messages (ham), including statistics.

pkg_add opensmtpd-filter-rspamd rspamd redis

Let’s already enable the services for the next reboot.

rcctl enable rspamd
rcctl enable redis

Rspamd will most likely work without any configuration. Unfortunately, it will not be as efficient. So tuning is definitely needed.

Want to better understand Rspamd? An in-depth article about Rspamd is planned. Stay tuned!

Start the services

Before starting or restart OpenSMTPD, perform a configuration test by using the ‘-n’ option.

# smtpd -n
configuration OK

If all is fine, like above, then restart OpenSMTPD.

rcctl restart smtpd

DKIM

Previously it is was common to create RSA keys with 1024 bits. The recommended size is 2048 now, but not every DNS provider allows the length of this longer key size. You may need to split the key over multiple DNS entries.

A key size of 4096 bits is not recommended, due to compatibility and that there is a more modern replace available that allows stronger keys with a smaller footprint (ed25519)

Another option is ed25519, modern and shorter in size. Not all mail servers may support this yet, but most will support it now or in the upcoming years. This support for older and the modern key type ed25519 provides a challenge: what to use? Use both! That provides compatibility for both old and new systems.

Option 1 - Rspamd

If you want to leverage the spam filtering capabilities of Rspamd, then you can let it also do DKIM signing. Rspamd has the option to define multiple selectors for a given domain.

Directory creation

Create the related directory and adjust ownership and file permissions. This way only the rspamd user (_rspamd) can access the DKIM key.

install -d -o root -g \_rspamd -m 770 /etc/mail/dkim

Key generation

Start by creating a key with type of ed25519. Since we will be using multiple keys, concise naming convention will help and avoid confusion later on. For example ‘dkim-YEAR-XY-TYPE’, where year is the full year, a follow number of two numbers, and finally the type. This makes rotation easier, while still clearly defining what the key.

rspamadm dkim_keygen --domain example.org --selector dkim-2026-01-ed25519 --privkey /etc/mail/dkim/example.org-ed25519.key --type ed25519

The default option for the ‘dkim_keygen’ subcommand is to show a DNS output. This information we will use after we created also the RSA key.

Now the RSA can be created.

rspamadm dkim_keygen --domain example.org --selector dkim-2026-01-rsa2048 --privkey /etc/mail/dkim/example.org-rsa2048.key --type rsa --bits 2048

Change DNS records

After the keys have been created, we need to add them to the DNS configuration. One entry will be created for the ed25519 key and another separate one for RSA.

Tips:

  • The name of the DNS record is listed on the left side of the output of rspamadm dkim_keygen
  • Record type is TXT
  • Value is v=DKIM1;k=ed25519;p=ABC
  • Or for RSA: v=DKIM1;k=rsa;p=XYZ
  • When using the ed25519 key, the text (ABC) will fit directly after ‘p=’ (without the quotes). When using the RSA (2048 bits) key, copy the two text strings as one. That is without quotes and if your provider allows you to insert a long string of text.

Check permissions and DKIM configuration

Ensure that user _rspamd can access the generated keys.

Open /etc/rspamd/local.d/dkim_signing.conf to insert the keys. Here is an example for a domain (example.com) with just one selector and another one with the set-up of two selectors.

domain {
  example.com {
    selector = "dkim-2026-01";
    path = "/etc/mail/dkim/dkim-example.com.key";
  }
  example.org {
    selectors [
      {
        path: "/etc/mail/dkim/example.org-ed25519.key";
        selector: "dkim-2026-01-ed25519";
      },
      {
        path: "/etc/mail/dkim/example.org-rsa2048.key"
        selector: "dkim-2026-01-rsa2048";
      }
    ]
  }
}

Another more extensive article about Rspamd will cover the usage of DKIM keys.

Option 2 - manual

Another option is do everything by hand.

Create a RSA key of 2048 bits with the help of the openssl command.

openssl genrsa -out /etc/mail/dkim/dkim-example.org-rsa2048.key 2048
openssl rsa -in /etc/mail/dkim/dkim-example.org-rsa2048.key -pubout -out /etc/mail/dkim-example.org-rsa2048.pub

Check the permissions so that that Rspamd can access the key. Then we need to publish the public key via DNS. As the public key part consists of multiple lines, it needs to be combined to one string of text. Also the key type and version is needed. Here is a one-liner to do so:

printf "v=DKIM1;k=rsa;p=%s\n" $(grep -v '\-\-' /etc/mail/dkim/dkim-example.org-rsa2048.pub | tr -d '\n')

  • Name: dkim-2026-01-rsa2048._domainkey
  • Type: TXT
  • Value: v=DKIM1;k=rsa;p=random-string

Dovecot

After setting up OpenSMTPD and our DNS, we can receive email. It would be great if we can also see those emails, for example via Thunderbird or another IMAP capable client. To provide this IMAP service, we can use the open source Dovecot.

Install Dovecot:

pkg_add dovecot

To grant users the option to automate things using filters, we need another package named dovecot-pigeonhole. It enables the use of services like Sieve and ManageSieve, for example to place spam automatically in the folder ‘Junk’.

pkg_add dovecot-pigeonhole

Open files

With the usage of the Maildir format of mailboxes, the number of open file may increase quickly. Open /etc/login.conf and allow it more open files.

dovecot:\
	:openfiles-cur=1024:\
	:openfiles-max=2048:\
        :tc=daemon:

Customizing Dovecot

Dovecot has a lot of options, so investing time in reading the documentation might be worth it. Knowing about what options there are will help in better customizing the system to your needs, such as additional authentication methods, Sieve support, and generic support for several protocols.

This article focuses on a few things:

  • Use only IMAP for pulling in the email
  • Usage of PLAIN authentication method, but with encryption
  • Describe our default mailbox structure
  • Use of Dovecot and Sieve scripts to automate spam handling

Open /etc/dovecot/local.conf to customize your installation. Example configuration:

# Authentication type, user, and SSL certificates
auth_mechanisms = plain
first_valid_uid = 2000
first_valid_gid = 2000
ssl_cert = </etc/ssl/mail.example.org.fullchain.pem
ssl_key = </etc/ssl/private/mail.example.org.key

# Location of mail boxes
mail_location = maildir:/home/vmail/mailboxes/%d/%n
# Location of plugins
mail_plugin_dir = /usr/local/lib/dovecot

# Sieve
managesieve_notify_capability = mailto
managesieve_sieve_capability = body copy environment fileinto envelope include mailbox stop variables vnd.dovecot.pipe

# Using NFS? Then disable the usage of mmap(), otherwise skip this option
# mmap_disable = yes

# Define how a mailbox should look like, including some folders
namespace inbox {
  inbox = yes
  location =
  mailbox Archive {
    auto = subscribe
    special_use = \Archive
  }
  mailbox Drafts {
    auto = no
    special_use = \Drafts
  }
  mailbox Junk {
    auto = create # only create, don't subscribe to it
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox Trash {
    auto = no
    special_use = \Trash
  }
  prefix =
}

# Plugins for Sieve
plugin {
  # Message marked as SPAM? Learn as SPAM
  imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/learn-as-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Junk
  # Message marked as SPAM but moved? Learn as HAM
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/learn-as-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
  # Script location
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
  # Activate plugins for Sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  # Extra script to mark SPAM marked email and move it to 'Junk' folder
  sieve_after = file:/usr/local/lib/dovecot/sieve/spam-to-folder.sieve
}

# Used protocols (only IMAP and Sieve)
protocols = imap sieve
protocol imap {
  mail_plugins = $mail_plugins imap_sieve
  mail_max_userip_connections = 10
}
protocol sieve {
  log_path = /var/log/dovecot-sieve-errors.log
  info_log_path = /var/log/dovecot-sieve.log
}
service imap-login {
  # Disable IMAP by setting it to 0, this will disable port 143 but still enable IMAPS (port 993)
  inet_listener imap {
    port = 0
  }
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}

Dovecot and using the SQLite database

Open /etc/dovecot/dovecot-sql.conf.ext to define the usage of SQLite and the related database queries.

# Use SQLite
driver = sqlite

# Path to database
connect = /etc/mail/smtpd.sqlite

# Define how the password is stored
default_pass_scheme = BLF-CRYPT

# Authentication query to validate user accounts
password_query = \
  SELECT email AS user, password \
  FROM credentials WHERE email = '%u' \
  AND password NOT NULL AND is_active = 1 \
  LIMIT 1;

# Define the user and where the mail box can be found
user_query = \
  SELECT '/home/vmail/%d/%n' AS home, 2000 AS uid, 2000 AS gid \
  FROM credentials WHERE email = '%u' \
  AND password NOT NULL AND is_active = 1 \
  LIMIT 1;

Dovecot configuration

Fully configured Dovecot? Great! As Dovecot combines default settings with your customized settings, it is wise to confirm that all is properly set up. This can be done using doveconf | less and look at the full configuration.

Sieve configuration

Create /usr/local/lib/dovecot/learn-as-spam.sieve

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment"];
if environment :is "imap.cause" "COPY" {
  pipe :copy "rspamc" ["learn_spam"];
}

To learn normal mail (HAM) create /usr/local/lib/dovecot/learn-as-ham.sieve. When a message goes to the bin, then we take no action. However if we move it to another folder, then it most likely is good email. Then we learn that message being HAM.

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];
if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}
if string "${mailbox}" ["Trash"] {
  stop;
}
pipe :copy "rspamc" ["learn_ham"];

We can also have the spam mails automatically be moved. A great function to keep the inbox clean, especially when a big spam run is active. Create the file /usr/local/lib/dovecot/move-spam-to-junk-folder.sieve.

require ["fileinto","mailbox"];

if header :contains "X-Spam" "Yes" {
  fileinto :create "Junk";
  stop;
}

Scripts need to be compiled first, also after they are changed.

sievec /usr/local/lib/dovecot/learn-as-spam.sieve
sievec /usr/local/lib/dovecot/learn-as-ham.sieve
sievec /usr/local/lib/dovecot/move-spam-to-junk-folder.sieve

After running these commands, you should have three additional files with the file extension .svbin.

By default our rspamc binary is not available in the path where Sieve configuration is looking. This may prevent the ‘pipe’ action to run it. After looking where rspamc is located (whereis rspamc), we can link it.

ln -s /usr/local/bin/rspamc /usr/local/lib/dovecot/sieve/rspamc

Connection with SQLite database

Open /etc/dovecot/conf.d/auth-sql.conf.ext for the configuration with our SQLite database and how authentication should occur.

passdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}

userdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
  override_fields = uid=vmail gid=vmail
}

Open /etc/dovecot/dovecot-sql.conf.ext for defining the database settings.

driver = sqlite

Point to our database to connect:

connect = /etc/mail/smtpd.sqlite

Define how password are saved:

default_pass_scheme = BLF-CRYPT

SSL configuration

Run the command dovecot-mkcert.sh to create a self-signed certificate. We then can replace this for our own certificate.

Sieve support

Probably /etc/dovecot/conf.d/20-sieve.conf already exists that enables sieve support. The reference to port 2000 (name: managesieve_deprecated) can be removed, as it is an older implementation.

protocols = $protocols sieve

service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}

plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
}

Start Dovecot

If the configuration has been completed, finish it by enabling the service and start it.

rcctl enable dovecot
rcctl start dovecot

Accounts

Password generation can be done using the smtpctl command with the encrypt subcommand.

# smtpctl encrypt 
paste-your-password-here
$2b$09$CVUigOvEqpik3GAOcUaZ4.tyYzU1bvOeIRexMlNZ38QLT12AZoxh6

To test if it works:

# doveadm auth test account@example.org 
Password: 
passdb: account@example.org auth succeeded
extra fields:
  user=account@example.org

Additional tips

Testing mails

Test the configuration by sending an email with the mail command. Include the sender, receiver, and a subject. Copy-paste some text and on a new line add just a dot to close the conversation.

# mail -s test -r sender@example.org receiver@example.org
This is a wonderful message, or so I believe.
.

Note: Yes, the ‘-r’ is confusing, it is the actual sender, not receiver

Useful commands

Dovecot

  • doveadm auth test (test authentication)
  • doveadm kick (disconnect users)
  • doveadm mailbox list (show mailboxes)
  • doveadm log errors (show errors)
  • doveadm reload (reload the configuration)
  • doveadm stats (show statistics)
  • doveadm user (show available users)

OpenSMTPD

  • smtpctl encrypt (generate a password for new accounts)

System management

  • syspatch
  • fw_update
  • pkg_add -u
  • sysupgrade

Useful resources

  • internet.nl (check your mail configuration and DNS configuration externally)

Overview of the DNS records

These are the TXT records:

_dmarc		TXT		1 hour	v=DMARC1;p=reject;sp=reject;adkim=s;aspf=s;rua=mailto:dmarc@example.org
_mta-sts	TXT		1 hour	v=STSv1; id=20260605
_smtp._tls	TXT		1 hour	v=TLSRPTv1; rua=mailto:tlsrpt@example.org;
_spf		TXT		1 hour	v=spf1 ip4:1.2.3.4 ip6:2a03:6000:aaaa:bbbb:::123 -all

Most likely you will have the same ones. Something missing? Let me know.

Errors and troubleshooting tips

Errors happen, so here are a few that came up during the configuration.

lookup: table-proc en lka socket closed (OpenSMTPD)

Jun  1 10:43:35 mail smtpd[60534]: lookup: table-proc: unexpected EOF during handshake
Jun  1 10:43:35 mail smtpd[67030]: smtpd: process lka socket closed

This is caused by an incorrect SQL query. The wrong database table was used. Unfortunately nothing useful shows up in OpenSMTPD itself, so you also have to look in any external components linked to OpenSMTPD, such as Rspamd.

Dovecot

Also in Dovecot it is fairly easy to end up with some error message. Especially during the configuration of Dovecot it is useful to check the log file now and then.

Password query failed: No such table

Jun  1 21:18:28 mail dovecot: auth-worker(5269): Error: conn unix:auth-worker (pid=64967,uid=518): auth-worker<2>: sql(myuser@mydomain.eu): Password query failed: no such table: credentials

The database defined in dovecot-sql.conf.ext was incorrect

Password query failed: no such column

Jun  1 21:21:56 mail dovecot: auth-worker(98895): Error: conn unix:auth-worker (pid=63146,uid=518): auth-worker<1>: sql(auth-michael@example.org): Password query failed: no such column: active

Incorrect database field was used, so SQL query did not work. In dit case a missing field ‘active’ that prevented the SQL query to work correctly.

Failed to pipe message to program rspamc

dovecot: imap(account@example.org)<96356><vIZjqddUouktjpAw>: Error: sieve: failed to pipe message to program `rspamc': refer to server log for more information. [2026-06-22 15:31:43]
dovecot: imap(account@example.org)<96356><vIZjqddUouktjpAw>: Error: sieve: Execution of script /usr/local/lib/dovecot/sieve/learn-as-ham.sieve failed

rspamc was not accessible in the defined path within the Dovecot configuration (/etc/dovecot/local.conf).

sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve

To resolve the issue, create a symlink between rspamc and the defined file path.

ln -s /usr/local/bin/rspamc /usr/local/lib/dovecot/sieve/rspamc

Permission denied of DKIM key

2026-06-27 23:56:06 #85922(normal) <004d42>; task; dkim_module_load_key_format: cannot load dkim key /etc/mail/dkim/private/dkim-example.org.key: cannot stat key file: '/etc/mail/dkim/private/dkim-example.org.key' Permission denied

The file permissions might be too tight.

Sources

Many sources cover OpenBSD, so it would be silly to reinvent the wheel. While I got inspired by some of the configuration snippets, I found none that fully covered this topic in-depth, hence the reason to create this extensive article. Below are some of the sources that I found, that might be worth also reading.

Definitions

DefinitionDescription
DNSContains information related to the domain name with focus on translating between hostnames and IP addresses
Domain nameUnique name on the internet to make services available, such as a website
HostnameThe name of a computer or defined name within DNS entry, such as when creating A records
HTTPSEncrypted communication channel between a web server and the web browser
SQLQuery language to manage a database, request, alter or remove data

Some other definition missing that you had to look up? Feedback on this article is appreciated!

Special thanks

The following persons took the time to carefully read this document and provide feedback

  • Misha Peters
  • Jeroen Janssen

Work in progress

  • Monitoring of domain name and DNS records
  • Check and monitor DMARC/MTA-STS
  • Flush pf tables until pf supports automatic expiry
  • Only allow authenticated users to send with their own ‘From:’
           senders <users> [masquerade]
                 Look up the authenticated user in the users mapping table
                 to find the email addresses that user is allowed to
                 submit mail as.  In addition, if the masquerade option is
                 provided, the From header is rewritten to match the
                 sender provided in the SMTP session.

Related articles

Like to learn more? Here is a list of articles within the same category or having similar tags.