Kernel and PF disagree on TCP RST handling

[Tor Perkins]

Hello tech@,

Spamd does not always detect when a connection is closed by a
legit (non-spoofed) RST packet (i.e.: read() does not return -1).

PF accepts the RST and clears state, but the kernel drops it and
the error condition of ECONNRESET is not set for the socket.

So... PF and the kernel handle some TCP RST packets differently.

I noticed this when using 'block log all' with PF; I saw lots of
outbound blocked FIN packets from spamd. These occur because PF
no longer has state for the connection, but spamd does not know
the connection is gone.

A packet trace reveals that the client's RST has a SEQ that is one
less than the last ACK we sent.

Curious as to how the stack should handle this RST scenario, I
found this timely post that compares RST handling for many OS's:

https://www.snellman.net/blog/archive/2016-02-01-tcp-rst/

The TL;DR from the above is that the stack should require the
exact SEQ to defend against spoofing attacks, but since TCP stack
implementations vary, a SEQ of +1/-1 should also be processed.

PF accepts it because it matches the criteria seen here:

/usr/src/sys/net/pf.c

((th->th_flags & TH_RST) == 0 || orig_seq == src->seqlo || // exact
(orig_seq == src->seqlo + 1) || (orig_seq + 1 == src->seqlo))) { // +1/-1

The kernel drops it because of the criteria seen here:

/usr/src/sys/netinet/tcp_input.c

if (tiflags & TH_RST) {
if (th->th_seq != tp->last_ack_sent
th->th_seq != tp->rcv_nxt &&
th->th_seq != (tp->rcv_nxt + 1))
goto drop;

When I added some printf() output to the above, my observation for
these connections was that last_ack_sent was the same as rcv_nxt,
and, that neither matched the SEQ on the wire (th_seq, which was
less by one).

Parenthetically, The author of the above article refers to the
OpenBSD kernel's +1 check as "slack in the wrong direction".

I think the +1 check is correct, but we should also check for -1:

th->th_seq != (tp->rcv_nxt - 1) &&

I added this to my kernel and spamd now detects these closures.

The article has a grid showing that, for the -1 case (column 16),
older FreeBSD accepted these as a closure. Current FreeBSD and
Linux ACK these. OS X, Windows Server 2012 and OpenBSD (the
kernel, not PF) drop them.

I can attest that there are "real world" hosts sending -1 RST
packets. Inconsistant handling of these leads to stateless
sockets (at least for spamd).

To be more explicit, here's a distilled representation of what the
connection looked like at the end of the packet trace:

Time: 36s, Port 1234 --> 25, Seq: 2, Ack: 8 Len: 1 Flags: ACK (SMTP: 'E')
Time: 36s, Port 25 --> 1234, Seq: 8, Ack: 3 Len: 0 Flags: ACK
Time: 63s, Port 1234 --> 25, Seq: 2, Ack: 8 Len: 1 Flags: ACK (SMTP: 'E')
Time: 63s, Port 25 --> 1234, Seq: 8, Ack: 3 Len: 0 Flags: ACK
Time: 70s, Port 1234 --> 25, Seq: 2, Ack: 8 Len: 0 Flags: RST|ACK

Before the final RST, there were 7 retransmit pairs (with timer
backoff). These retransmits are what follow spamd's initial SMTP
220 greeting (the famous "stutter").

My suspicion is that the spambot silently abandoned the connection
(no RST or FIN) as soon as it detected spamd and what follows is
the spambot's intermediary firewall cleaning things up.

Thanks for your consideration.

- Tor