Dealing with Contention

Contention is one of the most challenging problems in distributed systems and concurrent programming. It occurs when multiple processes or threads compete for the same resource simultaneously, leading to race conditions, inconsistent state, and data corruption if not handled properly. From booking the last concert ticket to processing financial transactions, contention management is critical for building reliable systems that maintain data consistency under high concurrency.

Understanding the Contention Problem: Consider a simple scenario: buying concert tickets online. There’s one seat remaining for a popular concert, and two users—Alice and Bob—both click “Buy Now” at precisely the same moment. Without proper coordination, both requests might read that one seat is available, both proceed to payment, and both receive confirmation for the exact same seat. This race condition occurs because there’s a gap between reading the current state and updating it based on that reading. In that tiny window, which might be mere microseconds in memory or milliseconds over a network, the world can change, leading to inconsistent system state.

The fundamental issue is that read and write operations aren’t inherently atomic. When Alice’s request reads “1 seat available” and Bob’s request also reads “1 seat available” before either has updated the count, both believe they can proceed with the purchase. By the time Bob’s update executes, Alice may have already decremented the count to zero, but Bob’s logic was based on the stale reading of one available seat. The result is that the seat count goes negative, both users are charged, and both receive confirmations for the same physical seat—a scenario that creates angry customers and requires costly remediation.

This problem intensifies dramatically as systems scale. With thousands of concurrent users competing for limited resources, even small race condition windows create massive conflicts. As organizations grow and need to distribute data across multiple databases or geographic regions, coordinating updates across these boundaries adds layers of complexity that make contention management even more challenging.

Single Database Solutions: When all your data resides in a single database, several well-established patterns can effectively manage contention. The key is choosing the right mechanism based on your specific access patterns and consistency requirements.

Atomicity and Transactions: The foundation of contention management is atomicity—ensuring that groups of operations either all succeed or all fail together, with no partial completion. Database transactions provide this guarantee. A transaction groups multiple operations into a single unit, starting with BEGIN TRANSACTION and ending with either COMMIT to save all changes or ROLLBACK to undo everything. For a money transfer between accounts, atomicity ensures that both the debit from one account and the credit to another account happen together. If anything fails—insufficient funds, network errors, database crashes—the entire transaction rolls back, preventing money from disappearing or appearing from nowhere.

However, atomicity alone doesn’t prevent race conditions. Two transactions can simultaneously read the same initial state and both successfully commit updates based on that stale data. We need additional coordination mechanisms to prevent concurrent transactions from interfering with each other.

Pessimistic Locking: This approach prevents conflicts by acquiring locks before performing operations. The name reflects a “pessimistic” assumption that conflicts will occur, so we prevent them proactively. In databases, explicit row locks can be acquired using SELECT FOR UPDATE statements. When one transaction locks a row, other transactions attempting to lock the same row will block until the first transaction completes. This guarantees that only one transaction can check and modify the resource at a time, completely eliminating race conditions.

The critical consideration with pessimistic locking is performance. Locks should be as narrow as possible in scope and as brief as possible in duration. Locking entire tables destroys concurrency and creates bottlenecks. Holding locks for seconds instead of milliseconds can cause entire systems to grind to a halt. The art of pessimistic locking lies in identifying precisely which resources need coordination and holding locks only during the critical section where conflicts could occur.

Isolation Levels: Rather than explicitly managing locks, database isolation levels allow the database to automatically handle conflicts. Isolation levels control how much concurrent transactions can observe each other’s uncommitted changes. The SERIALIZABLE isolation level provides the strongest guarantees by making transactions appear to execute sequentially, even though they actually run concurrently. The database automatically detects when transactions would conflict and aborts one of them, which must then retry. While this provides strong consistency with less explicit lock management, SERIALIZABLE isolation is more expensive than explicit locks because the database must track all reads and writes to detect potential conflicts.

Optimistic Concurrency Control: Instead of preventing conflicts upfront, optimistic concurrency control detects conflicts after they occur and retries failed operations. This approach assumes conflicts are rare and allows all operations to proceed without blocking. Each record includes a version number that increments with every update. When updating, the operation specifies both the new value and the expected current version. If another transaction has modified the record since it was read, the version number will have changed and the update fails, signaling a conflict that requires retry.

The performance benefits are significant under low contention. Transactions don’t block waiting for locks, eliminating that overhead entirely. However, the version check doesn’t need to be a separate column—any field that naturally changes when the record is updated can serve as the version indicator. For inventory systems, the stock count itself works perfectly. For auction systems, the current high bid serves as the version. When updates fail due to version mismatches, applications can re-read the current state and retry with the new version.

Distributed Coordination: When data spans multiple databases, contention management becomes significantly more complex. The key principle is to exhaust single-database solutions before moving to distributed coordination, as modern databases can handle enormous scale—tens of terabytes and thousands of concurrent connections cover the vast majority of applications.

Two-Phase Commit (2PC): The classic distributed transaction protocol, two-phase commit ensures atomicity across multiple databases through a coordinator that manages the transaction. In the prepare phase, the coordinator asks all participating databases to prepare the transaction—perform all work except the final commit. Each database verifies it can complete its portion, makes the changes, but leaves the transaction open. If all participants successfully prepare, the coordinator sends commit instructions in the second phase. If any participant fails to prepare, the coordinator sends abort instructions to all participants.

The critical requirement is that the coordinator must write transaction state to persistent logs before sending any commit or abort decisions. Without these logs, coordinator crashes create unrecoverable situations where participants don’t know whether to commit or abort their prepared transactions. Additionally, keeping transactions open across network calls is extremely dangerous because those open transactions hold locks on resources, potentially blocking other operations. If the coordinator crashes with transactions in the prepared state, those locks can persist indefinitely, effectively freezing affected resources.

Distributed Locks: For simpler coordination needs, distributed locks ensure only one process can work on a resource at a time across the entire distributed system. Rather than coordinating complex multi-phase transactions, distributed locks provide mutual exclusion—at most one process holds the lock for a given resource at any time. These can be implemented using Redis with time-to-live for automatic expiration, database status columns with background cleanup jobs, or purpose-built coordination services like ZooKeeper or etcd that provide strong consistency guarantees even during failures.

Beyond technical coordination, distributed locks dramatically improve user experience by preventing contention before it happens. When users select concert seats, the system doesn’t immediately mark them as sold. Instead, seats transition to a “reserved” state that gives users time to complete payment while preventing others from selecting the same seat. This pattern appears everywhere: ride-sharing apps set driver status to “pending request,” e-commerce sites place items “on hold” in shopping carts, and scheduling systems create temporary reservations. By creating these intermediate states, the contention window shrinks from the entire transaction (potentially minutes) to just the reservation step (milliseconds).

Saga Pattern: Rather than trying to coordinate everything atomically like two-phase commit, the saga pattern breaks operations into sequences of independent steps that can each be undone through compensation if something fails. For a bank transfer across different database shards, instead of holding both accounts locked while coordinating, the saga executes operations sequentially. First, debit the source account and commit that transaction immediately. Then credit the destination account and commit that transaction. If the second step fails, compensate by crediting the source account back.

Each saga step is a complete, committed transaction. There are no long-running open transactions and no risk of coordinator crashes leaving databases in limbo. The tradeoff is that during saga execution, the system is temporarily inconsistent. After debiting the source account but before crediting the destination, observers might see the source balance decreased but the destination balance unchanged. This eventual consistency is what makes sagas practical—accepting brief inconsistency to avoid the fragility and blocking behavior of distributed transactions. Applications handle this by designing for intermediate states, perhaps showing transfers as “pending” until all steps complete.

Choosing the Right Approach: The decision tree for contention management follows a natural progression. Start by asking whether all contended data can reside in a single database. If yes, choose between pessimistic locking for high-contention scenarios with predictable performance, or optimistic concurrency for low-contention scenarios with better throughput. Pessimistic locking provides simple reasoning about correctness and handles worst-case scenarios gracefully, while optimistic concurrency delivers superior performance when conflicts are rare.

Only move to distributed coordination when you’ve genuinely outgrown vertical scaling or need geographic distribution. For scenarios requiring atomicity across multiple systems where availability can be sacrificed for consistency, distributed transactions through two-phase commit provide strong guarantees at the cost of complexity and fragility. When resilience matters more than strict consistency, the saga pattern offers a pragmatic middle ground through eventual consistency and compensation. For user-facing competitive flows like ticketing or e-commerce, distributed locks with reservation states prevent users from entering contention scenarios, dramatically improving experience.

Common Pitfalls and Edge Cases: Several subtle issues can undermine even well-designed contention management. Deadlocks occur when pessimistic locking transactions wait in cycles—Transaction A holds Lock X and waits for Lock Y while Transaction B holds Lock Y and waits for Lock X. The solution is ordered locking: always acquire locks in a consistent order regardless of business logic flow. Sort resources by a deterministic key like user ID or database primary key, ensuring all transactions follow the same acquisition sequence to prevent circular waiting.

The ABA problem affects optimistic concurrency when values change from A to B and back to A between read and write operations. The version check sees the same value and incorrectly assumes nothing changed, missing important state transitions. For a restaurant review system, if two reviews arrive simultaneously—one five stars, one three stars—the average might remain unchanged at four stars, but the review count must increase. Using the review count as the version rather than the average rating ensures detection of all updates, since the count monotonically increases. When natural monotonic fields aren’t available, explicit version columns that increment on every update regardless of data changes prevent ABA issues.

The celebrity problem, or hot partition scenario, occurs when everyone wants the same resource simultaneously. Normal scaling strategies break down because you can’t shard a single resource across databases, load balancers can’t distribute requests competing for the same row, and read replicas don’t help write bottlenecks. The solution often requires rethinking the problem itself: can the resource be split into multiple identical units, or can consistency requirements be relaxed to eventual consistency? When strong consistency is truly necessary, queue-based serialization—funneling all requests for a resource through a dedicated queue processed by a single worker—eliminates contention by making operations sequential, trading latency for reliability.

Practical Applications: Contention management appears across countless real-world scenarios. Online auction systems demonstrate optimistic concurrency perfectly, using the current high bid as the version and only accepting new bids higher than the expected current value. Ticketmaster-style event booking showcases distributed locks with seat reservations, preventing the terrible user experience of filling out payment information only to discover the seat was taken. Banking and payment systems require distributed transactions for account transfers between different institutions, typically using the saga pattern for resilience unless strict consistency absolutely requires two-phase commit.

Ride-sharing dispatch systems leverage distributed locks to set driver status to “pending request” when offering rides, preventing multiple simultaneous requests to the same driver. Flash sales and inventory systems combine optimistic concurrency for stock updates with distributed locks for shopping cart holds, reducing checkout contention while maintaining inventory accuracy. Review platforms like Yelp use optimistic concurrency for rating updates, ensuring concurrent reviews don’t corrupt average ratings when they arrive simultaneously.

Performance and Scalability Considerations: The performance characteristics of different contention strategies vary dramatically. Pessimistic locking provides predictable, consistent latency with low overhead for simple database queries, but throughput suffers under high contention as transactions block waiting for locks. Optimistic concurrency delivers excellent performance when conflicts are rare, but performance degrades as contention increases and more transactions must retry. Distributed transactions incur substantial latency from network coordination between databases, making them suitable only when atomicity across systems is absolutely required.

Scalability also differs by approach. Pessimistic locking and optimistic concurrency within a single database can handle thousands of transactions per second. Vertical scaling through larger database instances extends this further. Distributed locks typically use fast cache operations for lock acquisition but introduce a coordination service as a potential single point of failure. Distributed transactions are inherently complex and fragile, requiring careful attention to coordinator availability, transaction timeout management, and recovery from partial failures.

When to Avoid Overengineering: Not every scenario requires sophisticated contention management. Single-user operations like personal to-do lists or private documents have no contention by definition. Read-heavy workloads with occasional writes can use simple optimistic concurrency with retry logic to handle rare conflicts without impacting read performance. Low-contention scenarios where conflicts happen infrequently—like administrative updates to product descriptions—don’t justify elaborate locking schemes when basic retries handle occasional conflicts.

The most common mistake is reaching for distributed locks or complex coordination when database transactions with row locking or optimistic concurrency suffice. Adding components introduces failure modes and operational complexity. Successful systems start with the simplest approach that meets requirements and only increase complexity when clearly necessary based on measured contention levels and demonstrated performance issues.

Testing and Monitoring: Proper testing of contention management requires simulating concurrent access patterns that might not occur during normal development. Race conditions are notoriously difficult to reproduce consistently because they depend on precise timing. Stress testing with many concurrent clients attempting the same operations helps expose contention issues before production deployment. Monitoring should track metrics like transaction retry rates, lock wait times, deadlock frequency, and saga compensation invocations to understand contention patterns and identify bottlenecks.

Contention management represents a fundamental challenge in building reliable distributed systems. The key insight is that simpler solutions are almost always preferable when they meet requirements. Exhaust single-database approaches with pessimistic locking or optimistic concurrency before considering distributed coordination. When distributed coordination becomes necessary, prefer the saga pattern’s resilience over two-phase commit’s complexity unless strict consistency is absolutely required. Design for the contention levels you actually observe rather than theoretical worst cases, and always fight to keep related data together in single databases as long as possible. Mastering these patterns enables building systems that maintain consistency and reliability even under extreme concurrent load.