Thursday, 8 October 2009

Networking and interprocess communication

Last year I implemented a unicast subset of XTP as a plausible attempt at a new transport-level protocol for use on Blue Gene. XTP has some of the flavour of the Universal Receiver Protocol: an XTP transmitter can cause the same receiver to execute different protocols (eg, datagram, reliable datagram, transaction, or stream) by sending different sets of orthogonal options. Unfortunately, although XTP is simpler and more flexible than TCP/IP it still seemed over-complicated. In particular, it has elaborate state transitions to end a conversation. There are other ambiguities in the specification. An INRIA report on a formal specification of XTP (mechanically verified) was helpful in resolving various problems I noticed when doing my implementation, or validating solutions that I developed independently.

In November, I read the book Patterns in Network Architecture: A Return to Fundamentals, by John Day (ISBN-13: 978-0-13-225242-3). The book packs a lot of material into its 400 pages, with many interesting observations along the way. Several simple points are relevant here. Day reviews the history of protocol development and analyses the components of various protocols. He observes that there might be only three types of protocols:
  1. relaying protocols, including multiplexing protocols
  2. error- and flow-control protocols ("error control" protocols)
  3. application protocols
In Chapter 6 (Divining Layers) Day considers why and how one might structure network protocols, and in particular, the purpose of layers of protocols. He therefore designs a basic internet protocol in stages. He concludes, as Metcalfe observed in 1972, that "networking is interprocess communication", but refines the observation: "networking is distributed IPC and only IPC" (ie, the resulting protocol suite exists only to permit communication between (or amongst) application processes), and "a layer is a distributed IPC facility". Subsequent chapters provide a detailed and more formal construction of a complete new network architecture with those observations in mind. The architecture uses IPC protocols recursively in a way reminiscent of Cheriton's VMTP.

The relationship between networking and IPC is I suppose fairly obvious, yet easily forgotten. Many protocols quickly form arbitrary stacks, layered elaborately but pointlessly, like pyramids of acrobats. The essential exchange of data between applications is almost an afterthought. By contrast, specialised protocols such as IBM's one-sided communication protocol on Blue Gene can indeed support specific IPC schemes, such as ARMCI. (In earlier posts, I dubbed that protocol "Clint": the protocol with no name.) With care, such a protocol can support several compatible IPC regimes efficiently, as Clint does for both ARMCI and MPI.

XTP was designed to support a bigger range of IPC options than we need for Plan 9 in this environment. We should do something more specific, and simpler. Unfortunately, Clint is not adequate because one fundamental IPC in Plan 9 is the 9P file serving protocol, which is conversation oriented.