What a binary wire protocol taught me about API design
When every byte is yours to define, you start seeing REST APIs differently.
Framing: the problem HTTP hides from you
TCP gives you a stream of bytes, not messages. The first thing every protocol needs is framing — a way to know where one message ends and the next begins. For Flume I used length-prefixed frames:
+----------+----------+------------------+
| length | type | payload |
| uint32 | uint8 | length - 1 bytes |
+----------+----------+------------------+Four bytes of length, one byte of message type, then the payload. That's it. HTTP does the same job with headers and chunked encoding — you just never have to think about it until you build the layer underneath.
Versioning is day-one work
The moment a client and server can be deployed separately, you have a compatibility problem. Kafka's protocol carries an API version in every request header — I copied that, and it saved me the first time I changed the produce message layout. The REST equivalent — versioned routes, additive-only schema changes — is the same discipline wearing different clothes.
What carries over to everyday APIs
Three habits survived the trip back up the stack: never break the frame (make responses self-describing), version from the first release, and treat every field you add as a promise you'll keep forever. JSON lets you be sloppy about all three — binary protocols don't, and the constraint is a good teacher.