| # Protocol Specification | |
| The ttrpc protocol is client/server protocol to support multiple request streams | |
| over a single connection with lightweight framing. The client represents the | |
| process which initiated the underlying connection and the server is the process | |
| which accepted the connection. The protocol is currently defined as | |
| asymmetrical, with clients sending requests and servers sending responses. Both | |
| clients and servers are able to send stream data. The roles are also used in | |
| determining the stream identifiers, with client initiated streams using odd | |
| number identifiers and server initiated using even number. The protocol may be | |
| extended in the future to support server initiated streams, that is not | |
| supported in the latest version. | |
| ## Purpose | |
| The ttrpc protocol is designed to be lightweight and optimized for low latency | |
| and reliable connections between processes on the same host. The protocol does | |
| not include features for handling unreliable connections such as handshakes, | |
| resets, pings, or flow control. The protocol is designed to make low-overhead | |
| implementations as simple as possible. It is not intended as a suitable | |
| replacement for HTTP2/3 over the network. | |
| ## Message Frame | |
| Each Message Frame consists of a 10-byte message header followed | |
| by message data. The data length and stream ID are both big-endian | |
| 4-byte unsigned integers. The message type is an unsigned 1-byte | |
| integer. The flags are also an unsigned 1-byte integer and | |
| use is defined by the message type. | |
| +---------------------------------------------------------------+ | |
| | Data Length (32) | | |
| +---------------------------------------------------------------+ | |
| | Stream ID (32) | | |
| +---------------+-----------------------------------------------+ | |
| | Msg Type (8) | | |
| +---------------+ | |
| | Flags (8) | | |
| +---------------+-----------------------------------------------+ | |
| | Data (*) | | |
| +---------------------------------------------------------------+ | |
| The Data Length field represents the number of bytes in the Data field. The | |
| total frame size will always be Data Length + 10 bytes. The maximum data length | |
| is 4MB and any larger size should be rejected. Due to the maximum data size | |
| being less than 16MB, the first frame byte should always be zero. This first | |
| byte should be considered reserved for future use. | |
| The Stream ID must be odd for client initiated streams and even for server | |
| initiated streams. Server initiated streams are not currently supported. | |
| ## Mesage Types | |
| | Message Type | Name | Description | | |
| |--------------|----------|----------------------------------| | |
| | 0x01 | Request | Initiates stream | | |
| | 0x02 | Response | Final stream data and terminates | | |
| | 0x03 | Data | Stream data | | |
| ### Request | |
| The request message is used to initiate stream and send along request data for | |
| properly routing and handling the stream. The stream may indicate unary without | |
| any inbound or outbound stream data with only a response is expected on the | |
| stream. The request may also indicate the stream is still open for more data and | |
| no response is expected until data is finished. If the remote indicates the | |
| stream is closed, the request may be considered non-unary but without anymore | |
| stream data sent. In the case of `remote closed`, the remote still expects to | |
| receive a response or stream data. For compatibility with non streaming clients, | |
| a request with empty flags indicates a unary request. | |
| #### Request Flags | |
| | Flag | Name | Description | | |
| |------|-----------------|--------------------------------------------------| | |
| | 0x01 | `remote closed` | Non-unary, but no more data expected from remote | | |
| | 0x02 | `remote open` | Non-unary, remote is still sending data | | |
| ### Response | |
| The response message is used to end a stream with data, an empty response, or | |
| an error. A response message is the only expected message after a unary request. | |
| A non-unary request does not require a response message if the server is sending | |
| back stream data. A non-unary stream may return a single response message but no | |
| other stream data may follow. | |
| #### Response Flags | |
| No response flags are defined at this time, flags should be empty. | |
| ### Data | |
| The data message is used to send data on an already initialized stream. Either | |
| client or server may send data. A data message is not allowed on a unary stream. | |
| A data message should not be sent after indicating `remote closed` to the peer. | |
| The last data message on a stream must set the `remote closed` flag. | |
| The `no data` flag is used to indicate that the data message does not include | |
| any data. This is normally used with the `remote closed` flag to indicate the | |
| stream is now closed without transmitting any data. Since ttrpc normally | |
| transmits a single object per message, a zero length data message may be | |
| interpreted as an empty object. For example, transmitting the number zero as a | |
| protobuf message ends up with a data length of zero, but the message is still | |
| considered data and should be processed. | |
| #### Data Flags | |
| | Flag | Name | Description | | |
| |------|-----------------|-----------------------------------| | |
| | 0x01 | `remote closed` | No more data expected from remote | | |
| | 0x04 | `no data` | This message does not have data | | |
| ## Streaming | |
| All ttrpc requests use streams to transfer data. Unary streams will only have | |
| two messages sent per stream, a request from a client and a response from the | |
| server. Non-unary streams, however, may send any numbers of messages from the | |
| client and the server. This makes stream management more complicated than unary | |
| streams since both client and server need to track additional state. To keep | |
| this management as simple as possible, ttrpc minimizes the number of states and | |
| uses two flags instead of control frames. Each stream has two states while a | |
| stream is still alive: `local closed` and `remote closed`. Each peer considers | |
| local and remote from their own perspective and sets flags from the other peer's | |
| perspective. For example, if a client sends a data frame with the | |
| `remote closed` flag, that is indicating that the client is now `local closed` | |
| and the server will be `remote closed`. A unary operation does not need to send | |
| these flags since each received message always indicates `remote closed`. Once a | |
| peer is both `local closed` and `remote closed`, the stream is considered | |
| finished and may be cleaned up. | |
| Due to the asymmetric nature of the current protocol, a client should | |
| always be in the `local closed` state before `remote closed` and a server should | |
| always be in the `remote closed` state before `local closed`. This happens | |
| because the client is always initiating requests and a client always expects a | |
| final response back from a server to indicate the initiated request has been | |
| fulfilled. This may mean server sends a final empty response to finish a stream | |
| even after it has already completed sending data before the client. | |
| ### Unary State Diagram | |
| +--------+ +--------+ | |
| | Client | | Server | | |
| +---+----+ +----+---+ | |
| | +---------+ | | |
| local >---------------+ Request +--------------------> remote | |
| closed | +---------+ | closed | |
| | | | |
| | +----------+ | | |
| finished <--------------+ Response +--------------------< finished | |
| | +----------+ | | |
| | | | |
| ### Non-Unary State Diagrams | |
| RC: `remote closed` flag | |
| RO: `remote open` flag | |
| +--------+ +--------+ | |
| | Client | | Server | | |
| +---+----+ +----+---+ | |
| | +--------------+ | | |
| >-------------+ Request [RO] +-----------------> | |
| | +--------------+ | | |
| | | | |
| | +------+ | | |
| >-----------------+ Data +---------------------> | |
| | +------+ | | |
| | | | |
| | +-----------+ | | |
| local >---------------+ Data [RC] +------------------> remote | |
| closed | +-----------+ | closed | |
| | | | |
| | +----------+ | | |
| finished <--------------+ Response +--------------------< finished | |
| | +----------+ | | |
| | | | |
| +--------+ +--------+ | |
| | Client | | Server | | |
| +---+----+ +----+---+ | |
| | +--------------+ | | |
| local >-------------+ Request [RC] +-----------------> remote | |
| closed | +--------------+ | closed | |
| | | | |
| | +------+ | | |
| <-----------------+ Data +---------------------< | |
| | +------+ | | |
| | | | |
| | +-----------+ | | |
| finished <---------------+ Data [RC] +------------------< finished | |
| | +-----------+ | | |
| | | | |
| +--------+ +--------+ | |
| | Client | | Server | | |
| +---+----+ +----+---+ | |
| | +--------------+ | | |
| >-------------+ Request [RO] +-----------------> | |
| | +--------------+ | | |
| | | | |
| | +------+ | | |
| >-----------------+ Data +---------------------> | |
| | +------+ | | |
| | | | |
| | +------+ | | |
| <-----------------+ Data +---------------------< | |
| | +------+ | | |
| | | | |
| | +------+ | | |
| >-----------------+ Data +---------------------> | |
| | +------+ | | |
| | | | |
| | +-----------+ | | |
| local >---------------+ Data [RC] +------------------> remote | |
| closed | +-----------+ | closed | |
| | | | |
| | +------+ | | |
| <-----------------+ Data +---------------------< | |
| | +------+ | | |
| | | | |
| | +-----------+ | | |
| finished <---------------+ Data [RC] +------------------< finished | |
| | +-----------+ | | |
| | | | |
| ## RPC | |
| While this protocol is defined primarily to support Remote Procedure Calls, the | |
| protocol does not define the request and response types beyond the messages | |
| defined in the protocol. The implementation provides a default protobuf | |
| definition of request and response which may be used for cross language rpc. | |
| All implementations should at least define a request type which support | |
| routing by procedure name and a response type which supports call status. | |
| ## Version History | |
| | Version | Features | | |
| |---------|---------------------| | |
| | 1.0 | Unary requests only | | |
| | 1.2 | Streaming support | |