Designing Packet Protocols

I wrote the original full dynamic NAT code for linux (as part of netfilter) which gave me fairly strong opinions about how a packet protocol should look. These are the first principles from which I considered the lightning protocol spec.

The length doesn’t include the header. Say you use a 4 byte length, then your packet (the simplest header scheme possible). That length should not include itself, otherwise implementers have to worry about underrun and overrun. Cut their worries in half.

Binary values are little-endian. I was part of the team which implemented little-endian Linux on Power at IBM, so I feel your pain. But much as we may love big endian on the wire, it’s purely legacy. You might not even wrap your decoding in endian-conversion macros, though I can’t bring myself to that (yet).

Plan for expansion. You need to have a way to accept unknown fields from the very first implementation. The classic way is to allow extra data in any packet. Another way is to have two bitsets in the initial handshake: “I support” and “You must support”. You can ignore any “I support” bits you don’t understand, but “You must support” bits cause an abort. You’ll eventually want to deprecate ancient versions, and this is how you’ll do it.

Beware crypto. If you can simply tunnel over ssh, do that. Otherwise, libsodium may help; but you still need someone experienced to audit your protocol. I certainly do!

Now, I used Google protobufs (v2) for my initial lightning prototype. It’s nice for prototyping and saves writing decoding/encoding logic (with the associated buffer overruns). But you still need to handle framing, retransmission and crypto yourself, and annoyingly it doesn’t allow you to send fixed-sized blobs (eg. 33-byte keys, or 64-byte signatures). It does allow addition of fields to the protocol (nicely ignored if unknown, but discoverable), which allows for easy expansion later.