15 Distributed Systems Concepts You Should Understand Beyond the Buzzwords
15 Concepts That Expose Whether You Actually Understand Distributed Systems or Just Know the Vocabulary
Distributed systems are full of ideas that are easy to name and hard to understand.
A candidate can learn the vocabulary in a weekend, memorize a few one-line definitions, and walk into an interview sounding prepared. For the first few minutes, this often works.
Then the interviewer asks a follow-up, and the surface cracks. The candidate who said the system is “CP” cannot explain what happens during a network partition.
The one who said they would “make it idempotent” cannot say how.
The one who claimed “exactly-once delivery” does not realize they have promised something that is famously impossible to guarantee in the general case. The buzzword was correct, but the understanding behind it was not there.
This is not an accident.
Experienced interviewers know exactly which concepts tend to be memorized rather than understood, and they probe those concepts deliberately.
These are the ideas where a shallow answer and a deep answer start out sounding identical and then diverge sharply the moment a second question arrives. They are, in effect, traps, and they are how interviewers separate candidates who truly understand distributed systems from those who have only learned to talk about them.
The good news is that these concepts are knowable. Each one has a shallow version that gets caught and a deeper version that holds up, and the gap between them is learnable.
Once you know where the traps are and what the deep answer looks like, the same questions that expose other candidates become the questions where you demonstrate real depth.
This article walks through fifteen of these concepts. For each one, it explains the idea, the shallow answer that fails, the deeper understanding that succeeds, and what the interviewer is actually probing for.
The goal is to make sure that when these questions come, you are on the right side of the gap.
1. The CAP Theorem
The CAP theorem states that during a network partition, a distributed system must choose between consistency and availability. It is one of the most cited ideas in distributed systems and one of the most misunderstood.
The shallow answer treats CAP as a free choice among three properties, saying a system “picks two of three” from consistency, availability, and partition tolerance. This phrasing reveals the misunderstanding immediately, because partition tolerance is not optional.
Networks fail, and any real distributed system must tolerate partitions, so the actual choice is only between consistency and availability when a partition occurs.
The deep answer is precise. It says that the trade-off only applies during a partition, and that the real decision is what the system does when nodes cannot communicate.
A system can either reject requests to preserve consistency, or keep responding and risk returning stale data to preserve availability.
When there is no partition, a system can be both consistent and available, so CAP is a statement about behavior under failure, not a permanent label.
The deeper answer goes further and mentions PACELC, which extends CAP by noting that even when there is no partition, a system still trades off between latency and consistency. This shows that the candidate understands the trade-off exists continuously, not only during failures.
What the interviewer probes is whether you understand that the choice is conditional and that partition tolerance is mandatory. Saying “pick two of three” is the classic shallow tell.
2. Consistency Is Not One Thing
Many candidates treat consistency as a single property a system either has or lacks.
In reality, consistency is a spectrum of models, and treating it as binary is one of the most common shallow tells.
The shallow answer says a system is “consistent” or “eventually consistent” and stops there, as if those were the only two options and as if “consistent” had a single clear meaning. This misses the rich middle ground.
The deep answer distinguishes among the models.
Strong consistency means every read returns the most recent write.
Eventual consistency means replicas converge over time but may differ briefly. Between them lie models like read-your-writes consistency, where a user always sees their own updates even if others do not yet, monotonic reads, where a user never sees data go backward in time, and causal consistency, where operations that depend on each other are seen in the right order.
A strong candidate picks the weakest model that still satisfies the requirements, since weaker models are cheaper and scale better.
The deeper point is that different data in the same system can use different consistency models, with critical data strongly consistent and tolerant data eventually consistent.
What the interviewer probes is whether you know that “consistency” is a dial with many settings, not a switch. Naming a specific model for specific data is the deep signal.