Async-LCAM: a lock contention aware messenger for Ceph distributed storage system

Ceph, an object-based distributed storage system, has a communication subsystem called Async messenger. In the Async messenger, a worker thread in the thread pool is assigned to each connection in a round-robin fashion and is allowed to process all the incoming or outgoing messages from the connection. Although this thread per connection strategy is easy to implement, it has an inherent problem such that when a connection is overloaded, it results in load imbalance problem among worker threads. In order to mitigate this problem, multiple worker threads can be assigned to a single connection to handle the traffic from the connection. However, this mapping structure induces another overhead related to lock contention since multiple threads contend to access the shared resources in the connection. In this paper, we propose lock contention aware messenger (Async-LCAM) , a messenger that assigns multiple worker threads per connection and is aware of lock contention generated from the threads. By keeping track of the lock contention of each connection every interval, the Async-LCAM dynamically adds or deletes assigned threads to/from the connection in order to balance the workloads among worker threads. The experimental results show that the Async-LCAM improves the throughput and latency of Ceph storage by up to 184 and 65%, respectively, compared to the original Async messenger.