Email 101
Email works like most any standard post service, only electronically. There is typically a:
- Sender
- Sending service
- Receiving service
- Receiver
In a post service, the sender would be you. The sending service would be the post service that picks up your package and is in charge of that package reaching its destination. The receiving service would be the same post service at a different location if within the same country or another post service if international shipping is involved. The receiver would be the party you are sending the package to. This same paradigm extends out to email.
An email can be sent from an email client (like GMail or Outlook). Simple Mail Transport Protocol (SMTP) transports that message to Gmail’s email server - a distribution center. After that, SMTP moves it further to the recipient’s email server, let’s say this is Yahoo. In the end, the message reaches the recipient’s inbox via POP3 or IMAP, the carrier.
SMTP
Simple Mail Transfer Protocol (SMTP) is the standard protocol used for sending emails. Allowing mail clients to send or forward messages to a receiving server. Similarly, a sender can instruct the SMTP server to transmit an email
Detailed SMTP Flow
Mail User Agent (MUA): Where an email begins and ends. The webapp application server in Appian acts as a MUA.
Mail Submission Agent (MSA): Takes input mail from a MUA. For a specific network, the SMTP gateway is the MSA that handles mail submission.
Mail Transfer Agent (MTA): Also known as a relay. Receives mail from an MSA or MTA and passes it to the next hop which may be another MTA or an MDA.
Mail Delivery Agent (MDA): Takes input from an MTA and is responsible for getting the email to the final MUA. Dovecot (or anything that provides IMAP) is an example of an MDA.
Simple Mail Transfer Protocol (SMTP): RFC 5321 Internet standard protocol to send and receive email messages
Securing the SMTP Flow
There are a few standards used to secure email flow from sender to recipient (covered in further detail in the below section on Email Security):
Sender Policy Framework (SPF): RFC 7208 Domains can explicitly authorize the hosts that are allowed to use their domain names, and a receiving host can check such authorization.
DomainKeys Identified Mail (DKIM): RFC 6376 permits a person, role, or organization that owns the signing domain to claim some responsibility for a message by associating the domain with the message
Domain-based Message Authentication, Reporting, and Conformance (DMARC): RFC 7489 Mechanism by which a mail-originating organization can express domain-level policies and preferences for message validation, disposition, and reporting, that a mail-receiving organization can use to improve mail handling.
SMTP Statuses
SMTP Status Codes
When sending a message via SMTP, there are a number of statuses a sender can receive once a message is sent.
A basic status code in an SMTP reply consists of a three digit number (transmitted as three numbers) followed by some text. The number is for use by sender email clients to determine what state to enter next; the text portion is for human use typically.
The first digit denotes whether the response is good, bad, or incomplete.:
- 2yz (Positive Completion Reply): The requested action has been successfully completed.
- 3yz (Positive Intermediate Reply): The command has been accepted, but the requested action is being held in abeyance, pending receipt of further information.
- 4yz (Transient Negative Completion Reply): The command was not accepted, and the requested action did not occur. However, the error condition is temporary, and the action may be requested again.
- 5yz (Permanent Negative Completion Reply): The command was not accepted and the requested action did not occur. The SMTP client SHOULD NOT repeat the exact request (in the same sequence).
The second digit encodes responses in specific categories:
- x0z (Syntax): These replies refer to syntax errors, syntactically correct commands that do not fit any functional category, and unimplemented or superfluous commands.
- x1z (Information): These are replies to requests for information.
- x2z (Connections): These are replies referring to the transmission channel.
- x3z : Unspecified.
- x4z : Unspecified.
- x5z (Mail system): These replies indicate the status of the receiver mail system.
SMTP Status Categories
| Status | Description |
|---|---|
| Delivered | Messages that are sent through and delivered. |
| Quarantined | Messages that are quarantined by one of the filters on the receiving side. |
| Blocked | Messages that are blocked because of a virus or executable file. Messages that are blocked cannot be released. |
| Deferred | Message could not be delivered to the recipient’s server. These errors are temporary and normally result in 400 error codes. |
| Bounced | Messages did not get delivered to the recipient server. Messages get sent a Non-Delivery Report (NDR). These are what we call a “hard bounce” normally 500 error codes. These messages will need to be resent. |
Email Bounces
When you send an email, recipient mail servers can reject your message, preventing a percentage of your prospects from receiving your message. When a server rejects a message, it’s called a bounce. A bounce can be hard or soft.
Soft Bounces
A soft bounce is an email that is returned to the sender because the recipient’s mailbox is full or the mail server is temporarily unavailable. A soft bounce message can be delivered at another time or forwarded by the recipient’s network administrator.
Hard Bounces
A hard bounce is an email that permanently bounced because the address is invalid. A hard bounce usually occurs because the domain name doesn’t exist or the recipient is unknown. Some mail servers hard-bounce an email address when the message is suspected as spam.
Common Bounce Codes
A bounce code is a reply from the recipient’s mail server that describes why an email wasn’t delivered. Mail server administrators can customize the messages that accompany codes, so messages can be different from server to server.
Traditional SMTP bounce codes are three digits. Enhanced SMTP bounce codes are three digits separated by decimal points. Popular services such as Gmail and Outlook.com often have their own bounce responses, so make sure to check each service’s documentation for more.
Traditional Bounce Codes
| Code | Description |
|---|---|
| 421 | Service not available, closing transmission channel. |
| 450 | Requested mail action not taken: mailbox unavailable (for example: mailbox busy). |
| 451 | Requested action aborted: error in processing. |
| 452 | Requested action not taken: insufficient system storage |
| 500 | The server could not recognize the command due to a syntax error. |
| 501 | A syntax error was encountered in command arguments. |
| 502 | This command is not implemented. |
| 503 | The server has encountered a bad sequence of commands. |
| 504 | A command parameter is not implemented. |
| 550 | User’s mailbox was unavailable (such as not found). |
| 551 | The recipient is not local to the server. |
| 552 | The action was aborted due to exceeded storage allocation. |
| 553 | The command was aborted because the mailbox name is invalid. |
| 554 | The transaction failed for some unstated reason. |
Enhanced Bounce Codes
| Code | Description |
|---|---|
| 5.0.0 | Unknown issue |
| 5.1.0 | Other address status |
| 5.1.1 | Bad destination mailbox address |
| 5.1.2 | Bad destination system address |
| 5.1.3 | Bad destination mailbox address syntax |
| 5.1.4 | Destination mailbox address ambiguous |
| 5.1.5 | Destination mailbox address valid |
| 5.1.6 | Mailbox has moved |
| 5.1.7 | Bad sender’s mailbox address syntax |
| 5.1.8 | Bad sender’s system address |
| 5.2.0 | Other or undefined mailbox status |
| 5.2.1 | Mailbox disabled, not accepting messages |
| 5.2.2 | Mailbox full |
| 5.2.3 | Message length exceeds the administrative limit. |
| 5.2.4 | Mailing list expansion problem |
| 5.3.0 | Other or undefined mail system status |
| 5.3.1 | Mail system full |
| 5.3.2 | System not accepting network messages |
| 5.3.3 | System not capable of selected features |
| 5.3.4 | Message too big for system |
| 5.4.0 | Other or undefined network or routing status |
| 5.4.1 | No answer from host |
| 5.4.2 | Bad connection |
| 5.4.3 | Routing server failure |
| 5.4.4 | Unable to route |
| 5.4.5 | Network congestion |
| 5.4.6 | Routing loop detected |
| 5.4.7 | Delivery time expired |
| 5.5.0 | Other or undefined protocol status |
| 5.5.1 | Invalid command |
| 5.5.2 | Syntax error |
| 5.5.3 | Too many recipients |
| 5.5.4 | Invalid command arguments |
| 5.5.5 | Wrong protocol version |
| 5.6.0 | Other or undefined media error |
| 5.6.1 | Media not supported |
| 5.6.2 | Conversion required and prohibited |
| 5.6.3 | Conversion required but not supported |
| 5.6.4 | Conversion with loss performed |
| 5.6.5 | Conversion failed |
| 5.7.0 | Other or undefined security status |
| 5.7.1 | Delivery not authorized, message refused |
| 5.7.2 | Mailing list expansion prohibited |
| 5.7.3 | Security conversion required but not possible |
| 5.7.4 | Security features not supported |
| 5.7.5 | Cryptographic failure |
| 5.7.6 | Cryptographic algorithm not supported |
| 5.7.7 | Message integrity failure |
IMAP
IMAP, or Internet Message Access Protocol, is a newer protocol that was designed for faster and constant Internet connections. Essentially, the email messages live on the server and the user downloads copies to their computer. IMAP is the protocol that is used to view and download messages from the mail server.
POP3
POP, or Post Office Protocol, was first developed in early 1984 and is currently in its third version (POP3). POP works by allowing users to retrieve email and download it onto their computer. Because this protocol was developed before constant Internet connections, it is meant to allow users to interact with their email on their computer and then connect to the server to send it.
MIME
Although not a protocol, there are a series of Multipurpose Internet Mail Extensions (just MIME, never “MIMEs”) for various types of email attachments (that are not just simple, plain text).
MAILDIR
Maildir is a format for storing email messages on a mail server. Each user/email address gets its own mailbox to store messages in. Each mailbox folder is a directory and each message a file. This improves efficiency because individual emails can be modified, deleted, and added without affecting the mailbox or other emails, and makes it safer to use on networked file systems such as NFS. See: https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/
Email Security
Email Headers
Email security is based on values contained within the email message header. These fields are used for a variety of purposes in SPF, DKIM, and DMARC as described below.
A great resource on email header fields: https://datawookie.dev/blog/2021/10/fathoming-email-headers/
SPF
Sender Policy Framework (SPF) is an email-authentication technique which is used to prevent spammers from sending messages on behalf of your domain. With SPF an organization can publish authorized mail servers.
An SPF record is a DNS record that has to be added to the DNS hosted zone of your domain. In an SPF record you can specify which IP addresses and/or hostnames are authorized to send email from this specific domain.
The mail receiver will use the “envelope from” address of the mail (the Return-Path header) to confirm that the sending IP address was allowed to do so. This will happen before receiving the body of the message. When the sending email server isn’t included in the SPF record from a specific domain the email from this server will be marked as suspicious and can be rejected by the email receiver.
DKIM
DKIM stands for DomainKeys Identified Mail and is used for the authentication of an email that’s being sent. Like SPF, DKIM is an open standard for email authentication that is used for DMARC alignment.
When an inbound mail server receives a message, it will detect the DKIM signature and look up the sender’s public DKIM key via DNS. The DKIM selector provided in the DKIM signature and the sender’s domain are used to determine where to look for this key. If the key is found, it can be used to decrypt the DKIM signature. This is then compared to the values retrieved from the received mail as the headers received are also hashed. If the hashes match, the DKIM is valid.
A domain owner adds a DKIM record, which is a modified TXT record, to the DNS records on the sending domain. This TXT record will contain the public key that’s used by receiving mail servers to verify a message’s signature.
DMARC
Domain-based Message Authentication Reporting and Conformance (DMARC) is a free and open technical specification that is used to authenticate an email by aligning SPF and DKIM mechanisms.
DMARC alignment requires that the “from” domain in a received email message to match either of the domains used in DKIM or SPF. Only emails that are aligned via matching can pass DMARC.
With DMARC you can tell the world how to handle the unauthorized use of your email domains (via a failed alignment check) by instituting a policy in your DMARC record.
The three DMARC policies are:
| p=none | The Domain Owner requests no specific action be taken regarding delivery of messages. |
| p=quarantine | The Domain Owner wishes to have email that fails the DMARC mechanism check be treated by Mail Receivers as suspicious. Depending on the capabilities of the Mail Receiver, this can mean “place into spam folder”, “scrutinize with additional intensity”, and/or “flag as suspicious”. |
| p=reject | The Domain Owner wishes for Mail Receivers to reject email that fails the DMARC mechanism check. Rejection SHOULD occur during the SMTP transaction. |
Mail Receivers MAY choose to reject or quarantine email even if email passes the DMARC mechanism check. The DMARC mechanism does not inform Mail Receivers whether an email stream is “good”.
Spam
The simplest definition of spam is “unsolicited bulk email.” Unsolicited means that it’s sent without the recipient’s consent, and bulk means that it’s sent to many people at once.
Spam Filters
Although spam filters can work in different ways, they typically examine sending reputation, email content, and recipient engagement when determining if a message is spam.
To determine if a message is spam, most spam filters check for appropriate email authentication and your sending IP reputation. To avoid having your emails considered spam, use email authentication and protect your sending reputation.
Email Authentication
Email authentication helps ISPs determine whether an email is coming from a legitimate source. Setting up your email authentication is crucial for good deliverability. The two most common authentication standards are SPF and DKIM. Every major ISP and most major spam filter providers check for one or both of these standards when determining what to do with an email.
Sending IP Reputation
Your sending IP reputation is based on many factors, including:
- Bounce rates
- Blocklistings
- Spam complaints
- Lack of sending IP history
If your sending IP reputation gets damaged, it can hurt your deliverability. Lower-volume senders typically use shared sending IPs of infrastructure level providers. If a client is blocklisted for spamming, it can affect deliverability for other clients on the same shared IP.