Concurrency and Network IO

Single Threaded Concurrency

Due to the fact that CPython implementation has an embedded GIL, the mainstream python ecosystem is build on the single threading assumption. All major IO libraries are consequently written in a synchronous manner.

In modern python, concurrency is achieved through the use of coroutine, which is implemented via generators and the yield statement (previously iterators).

A function which implements a yield statement become a coroutine, the context switch between the coroutine and the master routine (the caller) is triggered whenever the generator function is called. The yield statement allows the local variables, states, and execution progress of the coroutine to be retained, which makes single threaded concurrency available.

Single-threaded languages have urgent need for asynchronous IO libraries because in a multi-threaded environment, even when the IO of a specific thread is blocked, the overall program is still responsive. The extra burden of single threaded libraries such as python and js is the requirement of asynchronous implementation of IO library. The lack of IO std library is the reason why JS is chosen by Ryan Dahl as the implementation language of NodeJS, largely to achieve asynchronous IO from the bottom. To be noted, NodeJS is a run-time environment packaged with JavaScritpV8 engine rather than a new language. In NodeJS, all IO is performed in the default event-loop (with no event-loop called to be defined/declared explicitly).

The Limitation of Single-threaded Concurrency

Single-threaded concurrency can only solve one type of Blocking: IO-blocking, this is the emphasis of NodeJS and Python tornado. In an environment where all IO is non-blocking, a computationally intensive operation will block the entire server. For the above reason, a single threaded non-blocking IO server must and can only handle computationally intensive operation through Restful Http/RPC call to other service providers (which should be non-blocking). In this chain of service request / providing, except for the first call from client, all the following services should be non-blocking, otherwise the overall service is blocked.

Single-threaded non-blocking IO concurrency can only save CPU resources for the Http Server (The nodejs/tornado server that only handles http requests) in a manner that one thread v.s. multiple threads (may contain suspended threads). This server can only handle tasks that (besides IO operation) will not be CPU intensive enough to block the whole IO chain (a light task).

Blocking/Non-blocking v.s. Synchronous/Asynchronous

Synchronous/Asynchronous describes the identity of the thread who process the IO. In synchronous mode, the main thread process the IO, so all IO must be processed one after another (refers as "blocking") and the main thread cannot do anything until the IO request returns. In asynchronous mode, the main thread assign the IO request to a child thread (or coroutine), so all IO can be processed simultaneously (non-blocking). Synchronous/Asynchronous is used to describe the mechanism of the request processing logic, this terminology is commonly applied from a service provider's viewpoint.

Blocking/Non-blocking is used to describe how requests are processed, this terminology is commonly applied from a IO request caller's viewpoint. Blocking means the service operates in a blocked manner.

All kinds of Blocking

Blocking for single-threaded language includes all executions (computationally or non-computationally) that would suspend the current (main and only) thread. Even sleep(1) in different coroutines (python) will block each other because there is only one real thread running at a given time. In this sense, single-threaded languages tend to be very sensitive to all "blocking" operations.

Single threaded concurrency can not handle cpu blocking (deployment of dl network, syntactic parsing, etc).
When dealing with blockings from other sources: network-io, disk-io, etc, event-loop is more efficient in terms of cpu resources. Event-loop is most suitable for light-weighted http server.

Solution

  1. Blocking/Non-blocking Http Client -> Async Tornado Http Server (Thrift: AsyncClient) -> (Thrift) TThreadPoolServer
  2. Blocking/Non-blocking Http Client -> Actor-based Concurrency Playframework Http Server (Thrift: SyncClient) -> (Thrift) TThreadPoolServer

Note: service requester can have blocking io, only service providers should be able to handle concurrency request (through multi-threading or event-loop).