Event-Driven SSL

[Work in progress. Meant to stimulate discussion about SSL API design. Not yet a practical design.]

OpenSSL currently uses a mix of blocking and non-blocking API's. There is no easy way presently to avoid blocking on slow crypto operations in OpenSSL applications. There *is* a way to avoid blocking on network I/O, but it's a bit cumbersome, and might not lend itself to the completion-notification style of network programming (currently used by high-performance Windows NT network applications, and slowly becoming available in the Unix world as well).

It may be possible to rearrange OpenSSL to give the application maximum control over where and how CPU time is spent by switching to a new event-driven API. For backwards compatibility, a compatibility layer would implement the classic OpenSSL API on top of the new API. Only the most demanding of applications will really need the new API, and programmers would be free to continue using the old API (via the compatibility layer), but some programmers might prefer the new API even for less demanding apps.

The new API would have one other benefit: by cleanly separating cryptographic from I/O operations, it might make the innards of OpenSSL significantly cleaner and easier to understand.

The new API would be event-driven, that is, the application and the API would pass each other events; each event would be either an SSL Record Layer record (aka protocol data unit), a block of cleartext, or a crypto engine processing request. Passing all SSL Record Layer records through the event interface lets the application decide exactly how to do its network I/O, and even lets the application use event-driven I/O rather than just blocking or nonblocking I/O. Passing all crypto engine processing requests through the event interface lets the application decide which thread to execute them in.

To avoid name space clashes, all public symbols defined by the new api will start with OPENSSL_ or openssl_.

Events

The events exchanged between the app and the API can be any one of three basic types. (IN and OUT in the following definitions are from the SSL state machine's point of view; IN means "into the SSL state machine", OUT means "out from the SSL state machine".)

typdef union OPENSSL_event_u {
	// type field; may be one of OPENSSL_{ENGINE_CALL,RECORD,CLEARTEXT}_{IN,OUT}_KIND
	int kind;

	OPENSSL_CLEARTEXT c;
	OPENSSL_RECORD r;
	OPENSSL_ENGINE_CALL e;
} OPENSSL_EVENT;

Methods would be provided for allocating and freeing events. Generally, whichever side originates an event receives the same event later when the operation it implies is complete, and can tell by examining the 'kind' field whether it is a new event or a reply. (If it had a kind of ...IN... originally, the reply's kind will be ...OUT..., and vice versa.) Thus the side that originates an event is responsible for freeing it or recycling it as appropriate; the side that receives an event must not free it, and must eventually dispose of it by returning it to the originator.

Example: SSL_write implemented on top of new API

Before I dive into the details of how the API works, here's a short example showing roughly how the classic OpenSSL API could be implemented on top of the proposed API:

int SSL_write(SSL *ssl,const char *buf,int num)
{
	// Wrap cleartext in object.  This copies the input buffer.
	OPENSSL_EVENT *e = openssl_newCleartextInEvent(buf, num);

	// Send it to the state machine.
	openssl_putEvent(e);

	// If blocking behavior desired, wait until reply comes back.
	if (isBlocking(ssl)) {
		openssl_waitForReply(e);
	}
	// else app responsible for grinding state machine elsewhere as described below
}

Example: grinding the SSL state machine

An application or library using this API would contain somewhere in it an event processing routine responsible for doing network I/O and grinding the SSL state machine. Here's a single-threaded example (very handwavy):

// Handle network I/O for one SSL connection
int handleNetworkIO(int fd, int pollevent, openssl_t *openssl)
{
	OPENSSL_EVENT *e;

	// Read from network, if appropriate
	if (pollevent & POLLIN) {
		nbytes = read(fd, buf, len);
		e = openssl_buildFrame(openssl, buf, len);
		// If we've finished building a frame, pass it to the state machine
		if (e) 
			openssl_putEvent(openssl, e);
	}

	// Grind the SSL state machine
	while ((e = openssl_getEvent(openssl)) != NULL) {
		switch (e->kind) {
		case OPENSSL_ENGINE_CALL_OUT_KIND:
			// Just execute the call in this thread for simplicity
			e->e.engine->call(e);
			// and send result back to SSL state machine
			assert(e->kind == OPENSSL_ENGINE_CALL_IN_KIND);
			openssl_putEvent(openssl, e);
			break;

		case OPENSSL_CLEARTEXT_OUT_KIND:
			// SSL has finished decrypting some text for us; pass it to app
			use_cleartext(e->c.len, e->c.buf);
			// and tell SSL state machine we're done with the buffer
			openssl_putEvent(openssl, e);
			break;

		case OPENSSL_RECORD_OUT_KIND:
			// this simple example can't handle more than one frame at once (FIXME)
			assert(!m_outputframe);
			// SSL has a record to send to the peer; remember it
			m_outputframe = e;
			break;
		
		case OPENSSL_RECORD_IN_KIND:
		case OPENSSL_CLEARTEXT_IN_KIND:
			// SSL has finished using some cleartext or an ssl record we sent it; free the buffer
			openssl_freeEvent(e);
			break;
		}
	}

	// Write to network, if appropriate
	if (m_outputframe && (pollevent & POLLOUT)) {
		write some more bytes from m_outputframe to network;
		if no more bytes in m_outputframe {
			// send output frame back to state machine
			openssl_putEvent(openssl, m_outputframe);
			m_outputFrame = NULL;
		}
	}

	// no error
	return 0;
}

A program that wanted to avoid crypto operations blocking might instead enqueue some or all of the OPENSSL_ENGINE_CALL events to a second thread.

Cleartext I/O events

Cleartext I/O events as seen by the state machine are simply batches of bytes on their way into the API from the app or out of the API to the app, stored in the following structure:

typedef struct OPENSSL_cleartext_st {
	// type field; either OPENSSL_CLEARTEXT_IN_KIND or OPENSSL_CLEARTEXT_OUT_KIND 
	int kind;

	// Length of this record.  Max value is somewhat less than 16384.
	int len;

	// The bytes of this unit of cleartext.
	char *buf;
} OPENSSL_CLEARTEXT;

Network I/O events (SSL Record Layer)

Network I/O events as seen by the state machine will be complete SSL Record Layer messages. The process of parsing bytes from the network into complete SSL Record Layer messages, and the reverse process of chipping off parts of SSL Record Layer messages into whatever number of bytes the network can handle at the moment, is performed by a thin framing layer. The results of the framing layer are stored in the following structure.

typedef struct OPENSSL_record_st {
	// type field; either OPENSSL_RECORD_IN_KIND or OPENSSL_RECORD_OUT_KIND 
	int kind;

	// Length of this record.  Max value is 16384 (set by the SSL standard).
	int len;
	// The bytes of this unit of SSL protocol data.
	// First byte is always 'protocol'; next two bytes are version, etc.
	char *buf;
} OPENSSL_RECORD;

Crypto Engine Processing Events

At least initially, rather than redesign the engine API to add nonblocking behavior, it could be used as-is; applications that need nonblocking crypto behavior can devote a thread to running the engine. To accomodate both multithreaded and singlethreaded applications, the proposed API would give the application direct control over which thread calls the engine API methods.

To make it easy to delegate engine operations to other threads, a new structure would be defined to collect the arguments of an engine method API call:

typedef struct openssl_engine_call_st {
	// type field; always one of OPENSSL_ENGINE_CALL_{IN,OUT}_KIND
	int kind;

	// the engine to carry out the operation
	ENGINE *engine;		

	// Error information (filled in by engine)
	ERR_STATE err;

	// which operation
	int optype;

	// Operands for each kind of operation
	union {
		struct sign {
			...
		} s;
		struct verify {
			...
		} v;
		...
	} u;
} OPENSSL_ENGINE_CALL;

A new entry point would be provided in ENGINE which would take a pointer to an OPENSSL_ENGINE_CALL, and execute the indicated call in the familiar blocking fashion.

Alternatively, the engine API could be modified to support nonblocking methods or to be event-driven (i.e. deliver completion notification events). The latter would be a good match for the event-driven SSL API proposed here.