A file disappears from a laptop. Somewhere else, a little machine on somebody's shelf still has enough information to bring it back. That is the appealing version of distributed storage. The interesting version begins when the shelf machine is asleep, another machine lies about what it has, and your laptop loses its connection just after asking whether the backup succeeded.

This book is about that interesting version.

You will build a small storage network in Go. You will decide what a successful write means, turn a byte stream into a protocol, scatter recoverable pieces across machines, encrypt data before it leaves the owner, and teach the network to repair itself. The network is the project. Learning how to think about a system whose parts can fail independently is the lasting result.

What you will build

This course teaches Go and distributed systems through one practical project: a storage network built from first principles. You write the core yourself, guided by behavioural contracts, explanations, diagrams and pseudocode. The page does not hand you a finished function and ask you to retype it. Optional hints and worked designs sit behind disclosure controls. A small companion workshop contains runnable Go exercises and reference solutions to selected mechanisms. The final chapters deliberately leave more of the design to you.

The target is a personal network of cooperative, admitted machines. Start with separate processes on one computer; later use your own machines connected through Tailscale. Storage nodes hold other people's encrypted bytes. Each owner keeps their own recovery material. The book develops a 2-data + 1-parity erasure-coded core, then explains how a general Reed–Solomon configuration changes the trade-offs. This is enough to make repair and failure-domain placement real without hiding them behind a giant library.

There is also an existing project, MeshVault, with Mac and iPhone interfaces. Its supplied source is a separate comparison implementation: it uses full encrypted mirroring and an HTTP protocol. It is not the answer key for this course's TCP and erasure-coded system. Chapter 22 gives you the precise integration exercise. The book does not assume that a UI written for one protocol will magically work with another.

Read the archipelago

The map grows with the lessons. Its five settlements have different roles:

These are teaching roles. Three storage destinations illustrate a 2+1 stripe; five settlements on the map do not mean five independent storage nodes. The code examples keep generic node names because a real deployment can have different participants and failure domains.

Ships carry pieces of a file, harbour signals suggest communication, and fog represents lost contact. A pirate's failed reading attempt after encryption means the cargo remains confidential; encryption alone does not prevent theft, deletion or disruption. The market is an accounting experiment, not proof of a trustworthy open economy.

How to spend an evening here

Read a chapter once without opening the solutions. When you reach a prediction, actually make one: on paper, in a note, or aloud. A mistaken prediction that you can explain afterwards is unusually valuable. Then sketch your approach before opening your editor.

Each chapter has a build contract, acceptance scenarios and an exit question. The contract says what must become true. It is deliberately more precise than “add networking” and less prescriptive than a list of functions to copy. The acceptance scenarios are tests you can implement through your own public boundary. Where the workshop provides executable tests, the chapter says so explicitly; the other scenarios are specifications, not a claim that a hidden test runner already exists.

Reading estimates count explanatory text, including hidden material, at a leisurely pace. Building takes longer: a short chapter may lead to several evenings of experiments. Stop at a checkpoint with one sentence describing the next observable behaviour you want. Returning tomorrow to “make duplicate uploads preserve quota” is much easier than returning to “finish distributed systems”.

The route through the book

The ordering is intentional. You encounter a small, concrete problem before its larger theory. We discuss consistency after there are two plausible versions of a file. We discuss backpressure after there is a queue that can grow. You will revisit earlier decisions; a design that made sense for one process may become a bottleneck for six machines. Revising it is part of the work.

What you need to know already

You should be comfortable writing small programs: functions, loops, records or structs, and simple tests. The book assumes you can look up a Go syntax detail. It explains the Go habits that matter when the project grows: small interfaces, slice ownership, error chains, cancellation, bounded concurrency and package boundaries. Appendix A is a compact language bridge if you are coming from C# or another object-oriented language.

Reading requires only a browser. This website has no remote fonts, runtime libraries, account, server or internet dependency. Double-click index.html, then bookmark the address in the browser's address bar. It will start with file:///. A local path belongs in that address bar, not in a Google search. You can also bookmark a chapter: its address ends with a readable fragment such as #erasure.

Keep the course folder together if you move it. The HTML contains the lessons, diagrams and reading controls; the assets and companion folders contain the illustrations and workshop. A new location needs a new bookmark. Reading preferences last for the current page session; the browser bookmark is your reliable place marker. The chapter menu can record chapters you mark as read in this browser when local storage is available. The islands follow the lesson and illustrate its ideas; they do not inspect your code or mark exercises as passing. “Print chapter” prints the current lesson; “Print book” prints the complete book with solutions closed.

Building the Go project does require a Go installation and, eventually, running your own node processes. That requirement belongs to the programming exercises. It has nothing to do with opening or reading this book. The companion targets Go 1.24 or later; the separately supplied MeshVault prototype declares its own Go version.

Your working arrangement

Use Download starter files to get the tests and unfinished exercise functions, then extract the ZIP into a new folder such as my-storage-network. The download contains no completed solutions, app source or Git history. Initialise your own repository in that folder and keep the course repository as reading material. Appendix B explains the exercises and test commands. Version control is useful even when you are the only person working. Commit a passing milestone, then let the next chapter challenge it.

Do not begin by copying the whole reference architecture. Chapter 1 needs a tiny command, not a package for every noun in the table of contents. New boundaries should arrive with a reason. A local byte store, a network client and an erasure codec are useful distinctions because their responsibilities and failure modes differ.

By the end, you should be able to explain the consistency you offer, identify the recovery root, calculate the failure budget, trace a cancelled request, and design a test that demonstrates a specific limitation. You will also have a foundation for future features: phone photo backup, richer discovery, capacity credits and more advanced placement. You will not have learned every corner of distributed systems. You will have learned how to keep learning without somebody choosing every step for you.

Turn the page. Our first network has no network in it.

Imagine you hand a photograph to an archivist. They glance at it, put it on their desk, and say, “Safely stored.” A minute later somebody opens a window. The photograph flies away. You might reasonably feel that “stored” was doing rather too much work in that sentence.

Software makes the same mistake with less dramatic stationery. A server receives bytes, puts them in a buffer, and returns success. The bytes exist. The program is happy. None of this tells you what will happen when the process exits.

Our first task is to make a small promise and keep it. We will store an immutable object on one machine, stop the program, start a new instance, and retrieve the same bytes. No sockets. No encryption. No ambitious architecture. A small honest system is a more useful beginning than a large collection of misleading status messages.

What you will learn

By the end, you should be able to distinguish an object from a filename, state a storage contract in observable terms, and explain why an acknowledgement is a claim about the future. You will also have a Go module, a command that can fail usefully, and a test that crosses a process lifetime.

An object is a sequence of bytes with an identifier. A filename is a human-facing label that might change. Today we can use deliberately boring identifiers such as sample-001. Later identifiers will be derived from content. Keep the two ideas separate now: renaming a holiday photograph should not require the storage layer to understand holidays.

Your first invariant

An invariant is a property that must remain true across all the operations your system permits. “The upload usually works” is not one. Try this:

Once a node acknowledges storing an object, a later successful read of that object returns exactly the acknowledged bytes. It never returns a partial object as if it were complete.

Notice the qualification “successful read”. A disk might fail. A later read may legitimately return an error. The invariant prohibits silently substituting different bytes or treating an incomplete write as the original object. Availability is another promise, one we will strengthen with multiple machines.

Two further rules make the exercise tractable. An identifier cannot be reused for different bytes, and object bytes do not change after publication. If the caller repeats the same request, it should observe the same logical object. If it reuses the identifier for different content, it should receive a conflict. Immutability turns an awkward question about competing updates into a simpler question about whether an object already exists.

This is not how every storage system must work. It is a choice that makes the network we are about to build easier to reason about.

Start small in Go

Create your own project directory and initialise a module. This is an exercise command, not a requirement for reading the website:

mkdir my-storage-network
cd my-storage-network
go mod init example.com/my-storage-network

Choose a command boundary with two operations: put an object from a local input file, and get an object into a new output file. For this chapter, a command accepting a data-directory argument is enough. Put the storage behaviour in ordinary functions or a small concrete type. Do not add an interface until something needs to consume it.

Your command should translate intent into a call, then translate the result into an exit status and a useful message. The storage code should not call os.Exit. If it does, tests and future callers lose the chance to respond. Error handling belongs at different levels: the storage layer knows that an object is missing; the command knows how to explain that fact to a person.

Choose a restricted identifier grammar while paths are still simple: lowercase letters, digits and hyphens, with a fixed maximum length. Reject separators, empty identifiers and traversal components. Later the grammar becomes a fixed-size hexadecimal digest. Never let an arbitrary object identifier become an unchecked path below your data directory.

Build contract

Implement a local object store that accepts a data directory, stores a byte sequence under an identifier and returns it later. The directory survives process exit. Missing objects produce a distinguishable error. Duplicate writes of identical bytes are successful without creating a second logical object. Different bytes under an existing identifier produce a conflict. The command refuses to overwrite an existing restore destination.

For this milestone, you may keep one bounded object in memory. Set the bound explicitly, such as 8 MiB, and reject larger input. Chapter 6 replaces this temporary convenience with streaming. It is perfectly respectable to choose a limit. It is less respectable to discover the limit when somebody's laptop runs out of memory.

Do not yet claim survival of a sudden power loss. A clean close and a process restart are a weaker test than a storage-device crash. Chapter 3 will examine the stronger promise. Write this limitation beside your current contract; limitations that are recorded can be improved deliberately.

Acceptance scenarios

1. Put an empty object, a short text object and a binary object containing zero bytes. Get each into a fresh destination and compare bytes.
2. Stop the program completely. Start a new invocation with the same data directory. The objects remain readable.
3. Put the same identifier and bytes twice. There is one logical object, and both calls report success.
4. Put different bytes under that identifier. The call fails and the original bytes remain unchanged.
5. Get an unknown identifier. Receive “not found”, not a successful empty file.
6. Try ../outside as an identifier. The operation is rejected before touching a path outside the store.

The empty-object case is important. An empty file is a real object with zero content. “Nothing was returned” cannot tell you whether the file was empty, missing, or unreadable. Your API needs separate information for those possibilities.

A small experiment in honesty

Add an intentional pause just before your code reports success. Kill the process during that pause. Restart it and look at the store. Now move the pause to the middle of writing and repeat. Do you see the same externally visible state?

You have just discovered why storage implementations care about intermediate states. The final file can look correct on every successful run and still be dangerously misleading after an interrupted run. Do not patch the problem by making the pause shorter. A crash has no obligation to wait for a convenient moment.

Before you close the laptop

Explain your success message without using the word “safe”. Does it mean the bytes were accepted by your function, written through a file descriptor, visible to another process, or confirmed at a chosen persistence boundary? If you cannot finish the sentence precisely, the message is ahead of the implementation.

Your milestone is pleasantly small: a program can stop existing without taking its objects with it. Next we will give that program boundaries that survive growth.

The first version of a program often fits comfortably in your head. Then you add a command, a test helper, a special case for an existing file, and a second way to report errors. Nothing seems individually unreasonable. A week later, changing the storage directory means editing six unrelated functions.

Good boundaries are a way of keeping decisions local. The storage layer should know how an identifier maps to disk. A network handler should know how a request becomes an operation. Neither needs to know how the other prints an error. We are going to separate these responsibilities before a socket makes their mistakes harder to see.

What you will learn

You will design a small consuming interface, distinguish a capability from its implementation, make resource ownership explicit, and preserve error causes while adding context. You will also practise resisting a very tempting mistake: translating a familiar class hierarchy into Go one type at a time.

A package is a unit of naming and dependency. It is not automatically a service, an architectural layer, or a miniature organisation chart. Begin with a command package and an internal store package. Add more when a responsibility becomes distinct. A package called utils postpones that decision; it rarely resolves it.

Interfaces describe what the caller needs

Suppose an upload coordinator needs to put an object and retrieve it. That caller can define the behaviour it consumes. The concrete disk store can implement it without declaring membership in a hierarchy. Another implementation might deliberately fail after receiving a certain number of bytes. The coordinator should not care which concrete type it receives.

Keep the interface small. If your upload tests need a fake implementing eighteen unrelated methods, the upload code probably knows too much. Splitting a capability is useful when different callers need genuinely different operations; creating a one-method interface for every private helper is just additional paperwork.

Think carefully about the shape of retrieval. Returning []byte is convenient for bounded objects. Returning io.ReadCloser allows streaming but creates a new obligation: somebody must close it. Neither shape is inherently more idiomatic. Their costs differ. This book will eventually need streaming, so the contract should state who owns the returned reader and when it must be closed.

The language's standard library uses small interfaces such as io.Reader and io.Writer extensively. The important design lesson is that behaviour can be composed without inheriting a large base type. See the official io package documentation when you need exact method contracts.

Ownership is more than memory management

Go's garbage collector does not close a file, stop a worker, unlock a mutex, or decide who may mutate a slice. You still need ownership rules. “Who is responsible for ending this resource's useful life?” is often a better question than “Who allocated it?”

For a put operation, decide whether the store consumes a reader but leaves it open, or takes responsibility for closing it. Consuming an io.Reader normally does not imply ownership of the underlying resource. The caller that opened the input file can close it after put returns. For a get operation returning an io.ReadCloser, the caller owns the closer. Write these rules in the public documentation.

Slices deserve special attention. Passing a slice copies its descriptor, not its underlying bytes. If a store remembers a caller's buffer and the caller reuses that buffer, the stored object changes behind the store's back. An in-memory fake that accidentally shares slices may behave differently from a disk implementation. Copy at ownership boundaries when independent lifetime is required. A fake should preserve the real contract, including isolation of returned bytes.

Errors need identities and context

A missing object, a corrupt object and a full disk are different facts. A user may want a short message, while a repair scheduler needs to decide whether another peer might help. Avoid making either consumer inspect a sentence with a string search.

Use a small set of stable error identities or types where callers need decisions. Wrap lower-level errors with useful operation context, preserving the cause. At the command boundary, log or display once. Logging the same error in every layer turns one failed operation into a chorus of apparently unrelated incidents.

A useful error chain might communicate: “restore holiday photo: read object abc: permission denied”. The outer context explains why the operation mattered; the inner cause explains what failed. Do not include secrets, plaintext filenames on untrusted nodes, or whole request bodies simply because a log line is easy to write.

One subtle Go trap is the typed nil inside an interface. An interface value can have a concrete pointer type whose pointer is nil while the interface itself is not nil. Avoid returning such values as successful resources or errors. Return a plain nil interface when there is no value. The language specification and Effective Go provide the underlying rules; the project lesson is to make absence unambiguous.

Build contract

Move disk-specific behaviour behind a small store boundary. Keep the command responsible for parsing and presentation. Make not-found and conflict distinguishable without comparing error strings. Document input limits and resource ownership. Add an in-memory test double that satisfies the same observable contract, including immutable stored bytes.

Use that boundary to write a reusable contract test: a function that accepts a factory creating a fresh store and runs the same behavioural scenarios against each implementation. Each test gets its own directory or memory map. Tests should not depend on their order, global configuration, or your home directory.

Do not abstract clocks, random generators, filesystems and every helper in advance. Introduce those seams when a test needs control over a real source of nondeterminism. A boundary earns its place by isolating a meaningful decision.

Acceptance scenarios

1. Run the Chapter 1 object scenarios against both disk and memory implementations.
2. Modify the input slice after a successful put into the memory implementation. A subsequent get still returns the original bytes.
3. Modify bytes returned by a get. Another get is unchanged.
4. Wrap a not-found error with additional context. The caller can still identify it with an error-aware check.
5. Cause an output write to fail. The command returns a failing exit status and does not announce a successful restore.
6. Review every open resource: its owner and closing point are apparent without reading the entire program.

The fourth scenario checks the contract, not the spelling of a sentence. The fifth catches an especially embarrassing storage bug: correctly reading a backup, failing to write the restored file, and congratulating the user anyway.

Before you close the laptop

Pick one boundary and explain what change it protects the rest of the program from. “It makes testing easier” is a useful start, but finish the thought: which failure can you now cause deliberately, and which implementation detail can you now replace?

Your code has gained a little structure. Next we make sure that structure does not dissolve the moment a write is interrupted.

Your store passes its restart test. You write a file, stop the command and run it again. The bytes are there. It is tempting to conclude that persistence is finished and move on to the glamorous network part.

A clean process exit is a polite guest. A crash leaves halfway through a sentence. A power failure may take the operating system's unwritten buffers with it. We need to stop treating these events as equivalent.

This chapter is about the gap between “another reader can see the object” and “the object has passed the persistence boundary we promised”. The details are filesystem-specific, so our course makes a limited claim: a local filesystem with documented file and directory synchronization behaviour, one daemon owning its data directory, and hardware that honours that behaviour. A network filesystem or unusual storage device requires fresh investigation.

What you will learn

You will distinguish atomic publication from durability, handle errors throughout the write path, reason about concurrent duplicate puts, and enumerate the recovery state after a crash at each boundary. You will also learn why a little state table can be more useful than another hundred lines of code.

Atomic publication means a reader sees either the old state or the new complete state, not an intermediate mixture. Durability means an acknowledged change survives the failures included in your contract. A rename may give useful atomic visibility without, by itself, guaranteeing that a power loss preserves the new directory entry. Those are separate dimensions.

Stage before you publish

Write new data to a unique temporary file in the same filesystem as the final destination. The same-filesystem condition matters because publication operations do not generally retain the same semantics across filesystem boundaries. Ordinary object reads must ignore temporary names.

While staging, enforce the size limit and calculate the digest you expect to store. Check short writes and errors. A successful allocation of a file descriptor says nothing about the success of the bytes that follow it. Once the staged content is complete and validated, synchronize the file using the platform's appropriate operation. Check that result too.

Then publish the object using an operation whose replacement semantics match your contract. A plain rename that replaces an existing destination can violate immutability if two writers race. For the initial course node, serializing publication under a process-wide lock and preventing a second daemon from opening the same directory is one tractable design. Another is an atomic no-replace publication primitive, such as a carefully designed same-filesystem link operation where supported. Neither option should be selected merely because its name sounds atomic.

Finally, synchronize the containing directory as required by the chosen filesystem model. Only then return the stronger acknowledgement. Temporary-file cleanup is housekeeping; a cleanup failure should not turn a successfully committed object into a claim that the object never existed.

The error after the effect

Suppose publication succeeds, but directory synchronization fails. Is the object stored?

It may be visible. You cannot truthfully promise the requested crash durability. Returning an error is appropriate, but the error does not mean that the operation had no effect. This distinction will soon reappear in network requests. A result can be uncertain: some effects happened, while the promised completion could not be established.

A retry must therefore inspect the existing object. If it contains the expected complete bytes, the retry can validate it and re-establish the required sync boundary before acknowledging. Do not simply see an existing filename and assume that a previous successful durable acknowledgement occurred. The previous attempt might have failed immediately before it.

This is also why a final object should be self-validating once we adopt content-derived identifiers. Existence is weak evidence. Existence plus the expected digest is stronger evidence about the bytes, although it still does not tell you whether every persistence operation succeeded.

Draw the crash table first

Before implementing the stronger write, make a table of stages and restart observations. Before staging, no new object exists. During staging, an ignored temporary file may exist. After publication but before directory sync, a complete final object may be visible with uncertain crash persistence. After the successful durability boundary, the object should survive the failures included in your model.

A reader must never treat the temporary file as a committed object. A startup scan may remove abandoned temporary files when it knows no writer is using them. With one daemon per data directory, that decision is simpler. Without such ownership, a “cleanup” process can destroy another writer's active work.

You should also distinguish a torn or corrupted final object from an abandoned temporary file. Corruption belongs in quarantine or an explicit repair path. Automatically deleting every object you do not understand can convert a software-version mismatch into data loss.

Build contract

Strengthen put so that it reports success only after the selected durability sequence completes. Preserve immutable object semantics under concurrent requests within the daemon. Document and enforce the single-daemon ownership assumption, or implement a publication strategy that safely handles multiple processes. Readers must observe complete objects only.

Add an injected failure seam at meaningful stages: after staging starts, after content validation, after file sync, after publication and after directory sync. The seam can be a test callback; it need not become part of your public API. Separate an injected error from an actual process kill. Both are useful, but they test different things.

The companion workshop's Store.Put example demonstrates a constrained single-process durable publication pattern. Read its assumptions before adapting it. It is a mechanism reference, not a universal filesystem library.

Acceptance scenarios

1. Interrupt staging after half the bytes. A restart never returns the partial content as a successful object.
2. Make the staging write fail. No success is reported; the previous complete object is unchanged.
3. Make file synchronization fail. The request fails before publication.
4. Make directory synchronization fail after publication. Report uncertainty; a retry can validate and complete the durability boundary.
5. Issue two puts for one identifier concurrently. Identical content is idempotent; conflicting content never replaces a committed object.
6. Restart with an abandoned temporary file and a valid final file. Recovery preserves the valid object and handles the temporary file according to your stated policy.

Do not claim that mocked sync failures prove behaviour under a real power cut. They prove that your control flow handles the error paths. Stronger durability claims need testing on the actual storage environment, and even that evidence has limits.

package workshop

import (
	"bytes"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"fmt"
	"io"
	"os"
	"path/filepath"
	"sync"
)

var ErrIntegrity = errors.New("object integrity failure")

func Digest(b []byte) string { h := sha256.Sum256(b); return hex.EncodeToString(h[:]) }
func validID(id string) bool {
	if len(id) != 64 {
		return false
	}
	for _, c := range id {
		if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
			return false
		}
	}
	return true
}

// Store is a bounded teaching store for a trusted private local directory.
// Exactly one Store instance/process may write a directory at a time.
// It does not implement inter-process locking, symlink confinement, quotas,
// adversarial local-user isolation, or a universal filesystem durability layer.
// Sync semantics must be verified for the actual filesystem/device in use.
type Store struct {
	dir string
	mu  sync.Mutex
	// Stage is an optional test-only failure hook; set it before concurrent use.
	Stage func(string) error
}

func NewStore(dir string) (*Store, error) {
	if err := os.MkdirAll(dir, 0700); err != nil {
		return nil, err
	}
	return &Store{dir: dir}, nil
}
func (s *Store) stage(name string) error {
	if s.Stage != nil {
		return s.Stage(name)
	}
	return nil
}
func syncDir(dir string) error {
	f, err := os.Open(dir)
	if err != nil {
		return err
	}
	err = f.Sync()
	closeErr := f.Close()
	if err != nil {
		return err
	}
	return closeErr
}
func (s *Store) Put(id string, b []byte) error {
	if !validID(id) || Digest(b) != id {
		return ErrIntegrity
	}
	if len(b) > MaxFrame {
		return ErrLimit
	}
	s.mu.Lock()
	defer s.mu.Unlock()
	dst := filepath.Join(s.dir, id)
	if old, err := s.Get(id); err == nil {
		if !bytes.Equal(old, b) {
			return ErrIntegrity
		}
		f, e := os.OpenFile(dst, os.O_RDWR, 0600)
		if e != nil {
			return e
		}
		e = f.Sync()
		closeErr := f.Close()
		if e != nil {
			return e
		}
		if closeErr != nil {
			return closeErr
		}
		return syncDir(s.dir)
	} else if !errors.Is(err, os.ErrNotExist) {
		return err
	}
	f, err := os.CreateTemp(s.dir, ".pending-")
	if err != nil {
		return err
	}
	name := f.Name()
	defer os.Remove(name)
	if err = writeAll(f, b); err == nil {
		err = s.stage("staged")
	}
	if err == nil {
		err = f.Sync()
	}
	if err == nil {
		err = s.stage("file-synced")
	}
	closeErr := f.Close()
	if err != nil {
		return err
	}
	if closeErr != nil {
		return closeErr
	}
	// Safe against this Store's writers under its mutex. Not a no-replace
	// primitive against other processes; exclusive ownership is mandatory.
	if err = os.Rename(name, dst); err != nil {
		return err
	}
	if err = s.stage("published"); err != nil {
		return fmt.Errorf("published; durability uncertain: %w", err)
	}
	if err = syncDir(s.dir); err != nil {
		return fmt.Errorf("published; durability uncertain: %w", err)
	}
	return s.stage("directory-synced")
}
func (s *Store) Get(id string) ([]byte, error) {
	if !validID(id) {
		return nil, ErrIntegrity
	}
	f, err := os.Open(filepath.Join(s.dir, id))
	if err != nil {
		return nil, err
	}
	defer f.Close()
	b, err := io.ReadAll(io.LimitReader(f, MaxFrame+1))
	if err != nil {
		return nil, err
	}
	if len(b) > MaxFrame {
		return nil, ErrLimit
	}
	if Digest(b) != id {
		return nil, ErrIntegrity
	}
	return b, nil
}

Before you close the laptop

Point to the earliest stage at which another process can see a complete object. Then point to the earliest stage at which you acknowledge the durability contract. If those stages differ, explain why. You are now ready to send that acknowledgement across a network, where it can acquire an entirely new way to go missing.

You write a header to a socket, then write a file. On your first test, the server reads the header in one call and the file in another. Everything appears beautifully arranged. Unfortunately, the arrangement exists in your test, not in TCP.

A stream transports an ordered sequence of bytes. It does not preserve the boundaries of your writes. One write may arrive through many reads; several writes may be available in one read. If your protocol works only when a packet arrives in the shape you imagined, you have built a coincidence.

What you will learn

You will design a framed request grammar, handle partial reads and writes, put limits before allocations, and distinguish clean end-of-stream from a truncated message. The result is a parser you can test without opening a real network port.

Let us choose a course protocol with a small binary prefix followed by a JSON header and an optional binary body. JSON keeps the metadata inspectable. A binary body avoids base64 expansion for shard bytes. This is not the only sensible protocol; HTTP would also work. Building framing yourself here makes the stream problem visible. Later you can judge when an existing protocol is the better engineering choice.

Write a grammar before a parser

Our frame starts with a four-byte unsigned big-endian integer declaring the number of header bytes. The header is UTF-8 JSON with a protocol version, operation, request identifier and body length. Depending on the operation, it also carries an object identifier. The body follows immediately and has exactly the declared length.

For the learning protocol, cap the header at 16 KiB and the body at 8 MiB. These values are course design choices, not universal networking constants. Validate the prefix before allocating the header. Validate the parsed body length before allocating or streaming the body. Reject unknown protocol versions and unsupported operations. Decide whether unknown JSON fields are tolerated for future compatibility; apply that decision consistently.

The parser has four distinct outcomes: a complete valid frame, a clean EOF before a new frame begins, a truncated frame, or a malformed frame. If two prefix bytes arrive and then the connection ends, that is not a clean EOF. Treating it as such can hide broken senders and incomplete operations.

Reads report progress and a condition

In Go, a reader can return some bytes and an error in the same call. Process the returned bytes according to the interface contract before interpreting the terminal condition. For fixed-size regions, io.ReadFull expresses the intent clearly: acquire exactly this many bytes or report why that could not happen. Review its exact EOF behaviour in the official documentation.

For writes, the io.Writer contract requires a non-nil error when fewer bytes than requested are written. Defensive protocol code should still reject an unexpected short write. A helper can loop over remaining bytes while checking for zero progress and impossible counts. Do not let a malformed writer double count progress or spin forever.

A bufio.Reader can make many small reads efficient, but ownership matters. Once it has read ahead from a connection, the buffered reader owns those extra bytes. If you parse the next body directly from the raw connection, you may skip bytes already sitting in the buffer. Keep one reader abstraction responsible for the stream.

Parsing is a resource boundary

Imagine a peer sending a header length of four billion bytes. The interesting question is not whether that header is valid JSON. The interesting question is whether you allocate four billion bytes before discovering anything else.

Length checks are admission decisions. Check integer conversions as well as numeric limits; converting an untrusted large unsigned value to a smaller signed type can change its meaning. A request should not be able to reserve memory, disk, or a worker indefinitely just by declaring an impressive number.

A second attack is patience. A peer sends one header byte every minute. Your parser is technically making progress, but the connection occupies resources for an unreasonable time. Deadlines and concurrency budgets will address this later. Record the requirement now, even while your first parser runs against byte buffers.

Build contract

Implement frame encoding and decoding against io.Writer and io.Reader. Keep the codec independent of net.Conn. A successful decode returns one complete frame and leaves the next frame unread. Enforce the header and body limits, reject invalid versions and inconsistent fields, and make truncation distinguishable from clean EOF.

For this chapter, collecting a bounded body into memory is acceptable. Keep that bound in one named place. Later you can separate header parsing from body streaming without changing the wire grammar. The companion WriteFrame and ReadFrame workshop uses a simpler length-prefixed payload to isolate the hardest I/O rules; your full header grammar is a further implementation task.

Acceptance scenarios

1. Encode two frames back to back and decode exactly two frames in order.
2. Supply a reader that returns one byte per call. Both frames still decode.
3. Split the stream at every byte position and truncate there. Only boundaries between complete frames count as clean completion.
4. Announce a header larger than the maximum. Reject it without attempting a corresponding allocation.
5. Use an empty body and a body containing every possible byte value. Neither case is treated as a text special case.
6. Use a writer that accepts only a few bytes per call, and one that returns an error after partial progress. Encoding must either complete correctly or return an error; it must never silently claim success.

The tiny-reader test is more valuable than sending a thousand requests over your fast loopback interface. It removes the accidental generosity of local scheduling and forces the parser to obey its abstraction.

read exactly 4 prefix bytes
if no bytes at a frame boundary: clean EOF
if incomplete prefix: truncated frame
validate header length against limit
read exactly that many header bytes
parse and validate header fields
validate body length against limit
read exactly that many body bytes
return one frame

package workshop

import (
	"encoding/binary"
	"errors"
	"io"
)

const MaxFrame = 8 << 20

var ErrLimit = errors.New("size limit exceeded")

// ReadFrame reads one length-prefixed payload. Clean EOF is possible only
// before a prefix starts. This is the framing kernel, not the book's JSON protocol.
func ReadFrame(r io.Reader) ([]byte, error) {
	var prefix [4]byte
	if _, err := io.ReadFull(r, prefix[:]); err != nil {
		return nil, err
	}
	n := binary.BigEndian.Uint32(prefix[:])
	if n > MaxFrame {
		return nil, ErrLimit
	}
	body := make([]byte, int(n))
	if _, err := io.ReadFull(r, body); err != nil {
		if errors.Is(err, io.EOF) {
			err = io.ErrUnexpectedEOF
		}
		return nil, err
	}
	return body, nil
}

func writeAll(w io.Writer, p []byte) error {
	for len(p) > 0 {
		n, err := w.Write(p)
		if n < 0 || n > len(p) {
			return errors.New("invalid writer count")
		}
		p = p[n:]
		if err != nil {
			return err
		}
		if n == 0 {
			return io.ErrShortWrite
		}
	}
	return nil
}

func WriteFrame(w io.Writer, body []byte) error {
	if len(body) > MaxFrame {
		return ErrLimit
	}
	var prefix [4]byte
	binary.BigEndian.PutUint32(prefix[:], uint32(len(body)))
	if err := writeAll(w, prefix[:]); err != nil {
		return err
	}
	return writeAll(w, body)
}

Before you close the laptop

Explain why one successful Read is not proof that you have one request. Then explain why receiving all the declared bytes is still not proof that those bytes represent an authorised or valid operation. Framing solves boundaries. Meaning is next.

The server has stored your object. It sends a success response. At precisely that moment, the connection breaks. From the server's perspective, the operation is complete. From the client's perspective, nothing conclusive happened.

This is your first proper distributed-systems problem. No amount of careful local locking can make a response that never arrived tell the client what it contained. The two processes have different knowledge because information takes time to travel and can fail to arrive.

We will build a deliberately modest request-response service. One connection carries one in-flight request at a time. There is no multiplexing, streaming dashboard or clever connection pool. That constraint leaves us room to get the meaning of the exchange right.

What you will learn

You will separate transport failure from operation failure, correlate responses, make duplicate requests safe, and write an integration test using real independent processes. You will also identify which state belongs to the client and which belongs to the storage node.

The node owns durable object bytes. The client owns the source it is uploading, the request it is attempting, and the decision about what to do after an uncertain result. A request identifier helps correlate logs and responses. It does not, by its mere presence, make an operation idempotent.

Choose a small protocol vocabulary

Start with PUT, GET and STAT. PUT carries an object identifier and bytes. GET returns the bytes or a not-found error. STAT returns the object's size and integrity status according to the check actually performed. A “present” flag based only on directory enumeration is not an integrity check.

Responses include the protocol version, request identifier, status and a bounded body length. Choose stable machine-readable status codes such as ok, not_found, conflict, invalid and busy. Keep a short diagnostic message for people, but do not require clients to parse it.

A malformed frame should close the connection if continuing would leave the stream boundary uncertain. A well-framed but invalid operation can receive an error response if the body has been handled safely. If the server rejects a request before consuming its declared body, it must not parse those body bytes as a new header. Closing that connection is a simple, honest policy for the first implementation.

Exactly what does PUT acknowledge?

Carry Chapter 3's local storage promise across the network. A successful PUT response means the addressed node has committed that immutable object at its specified local durability boundary. It does not yet mean any other node has it. It does not mean the client possesses a durable manifest. It does not mean the network will remain available tomorrow.

There is value in this narrow statement. Later, a client can assemble stronger guarantees from several such acknowledgements. If the local acknowledgement is vague, the larger guarantee will be vague too.

An identical duplicate PUT should preserve one logical object. Different bytes under an existing identifier must fail. With content-derived identifiers, introduced next, the server can verify that the claimed identity matches the bytes. Until then, conflict checking still matters.

Do not make GET acknowledge a successful restore. GET only delivers bytes. The client must verify them and publish a destination file. This separation lets a future phone client or desktop app use the same protocol without inheriting your command-line assumptions.

Deadlines belong to operations

A dial can stall. A header read can stall. A response write can stall because the client stopped reading. Put finite time budgets around the first service, even before you build sophisticated cancellation.

A connection deadline governs actual socket operations. A context is a way to carry cancellation intent through your code. You need to connect those two mechanisms: merely creating a context does not make an arbitrary blocking read observe it. For now, one request per connection makes ownership straightforward: the request owns the connection and closes it on completion or cancellation.

Keep the timeout configurable for tests. A loopback success should not require sleeping for the entire timeout, and a failed request should not make the test wait several minutes. Later chapters add a fake clock for health logic; socket tests can use short, bounded real deadlines with explicit synchronisation.

Build contract

Run a storage node process with an explicit data directory and listen address. Run a separate client command that puts, gets and checks an object. The handler calls the store boundary, not filesystem helpers scattered throughout the networking code. Responses echo the request identifier and preserve the distinction between an application error and a broken transport.

Bind initial exercises to loopback. When you extend to your tailnet, add admission and authentication deliberately; the unprotected teaching listener should not become an accidental public storage service. This is a scope boundary for the implementation, not a reason to delay learning how the request works.

The protocol should state what happens after the client times out. At this stage, the client may report “outcome unknown; retry or check this object”. That is more accurate than “upload failed” when the server may have committed it.

Acceptance scenarios

1. Start a node and a separate client process. Upload a binary fixture, download it, compare bytes, restart the node and repeat the download.
2. Ask for a missing object. Receive the correct application status while the server remains usable.
3. Disconnect halfway through the body. No partial object becomes visible.
4. Commit the object but deliberately drop the response. A retry of the same PUT succeeds without replacing or duplicating the object.
5. Send a response with the wrong request identifier from a fake server. The client rejects it.
6. Connect a client that never finishes the header. The deadline releases its resources and other requests still make progress.

Scenario four is the one to savour. It turns a philosophical problem about knowledge into a test you can repeat on your desk.

Before you close the laptop

A client sees a timeout. Name three possible server states consistent with that observation. Your answer should include “never received the request”, “partway through the operation” and “committed successfully”. Once those all feel natural, you are thinking beyond the boundary of one process.

The first photograph fits comfortably in memory. A video does not. You could increase the size limit, buy more memory, and postpone the problem until a larger video arrives. Or you could stop requiring the whole file to be present at once.

Chunking turns one unbounded job into many bounded jobs. It creates units for hashing, retrying, distributing and repairing. It also creates bookkeeping. A bag of correct pieces is not yet a file; somebody must remember their order and exact lengths.

What you will learn

You will implement a streaming chunker, derive identifiers from exact bytes, reason about slice lifetime, and calculate memory use independently of total file size. You will also distinguish chunking from erasure coding. Dividing a file into pieces does not, by itself, add any redundancy.

For this course, use fixed-size plaintext chunks of at most 4 MiB. The final chunk can be shorter. A zero-byte file has no data chunks and still needs a file record. These are format decisions. Record them before writing tests so the meaning of “empty” does not drift between the uploader and restorer.

Hashes give bytes a name

A cryptographic digest such as SHA-256 produces a fixed-size value from arbitrary bytes. Use the lowercase hexadecimal digest of the stored byte sequence as its object identifier. The node can then verify the relationship between name and content rather than trusting the client's claim.

The important phrase is stored byte sequence. Before encryption, that may be plaintext in an early local exercise. Once encryption is introduced, object identifiers name ciphertext shards. Do not quietly keep a plaintext digest in an unencrypted public index: it can reveal whether a peer holds a guessed file.

A digest supports integrity when the expected digest comes from a trusted source. If an attacker can replace both the bytes and the expected digest, the comparison proves little about authenticity. Later, an authenticated manifest anchors these expectations. For now, distinguish “these bytes match this identifier” from “these are the bytes the owner intended”.

Content addressing makes repeated storage of identical bytes naturally converge on one identifier. It does not automatically solve ownership, quota, deletion or privacy. A node with two users referencing the same object still needs a policy for who may retrieve it and how resource use is charged.

Short reads do not define chunks

A chunk should normally end because it reached the configured size or because the input ended. It should not end merely because one call to Read returned fewer bytes than you requested. Readers can make partial progress for many reasons.

Fill a chunk across as many reads as necessary. Handle a final partial chunk exactly once. If a reader returns bytes and EOF together, those bytes still belong to the input. If it returns a non-EOF error after bytes, decide how your API reports incomplete work; the file upload must not commit a complete manifest after an input error.

The companion chunking exercise supplies a reader that returns tiny pieces. It is designed to expose implementations that confuse read boundaries with logical chunk boundaries. For exact helper behaviour, consult io.ReadFull; do not guess at how it distinguishes a short final block.

The reusable-buffer trap

A common chunker allocates one 4 MiB buffer and fills it repeatedly. That is efficient if each consumer finishes before the buffer is reused. It is disastrous if the chunker puts slices referencing that buffer into a queue and immediately overwrites it for the next chunk.

You may see an especially confusing failure: every uploaded chunk contains bytes from the final part of the file. The queue contains several slice descriptors, but they all point to the same underlying storage.

There are three reasonable policies. The consumer processes a borrowed buffer synchronously and must not retain it. The producer transfers ownership of a newly allocated buffer. Or a carefully managed pool returns buffers only after consumers finish. Start with one of the first two. A pool is an optimisation with a lifetime protocol attached, not free memory.

State the policy in your chunker contract. A callback named visit does not tell its caller whether retaining the slice is legal. Document that fact and test the chosen behaviour.

Fixed boundaries are a choice

Fixed-size chunking is easy to reason about and easy to seek through. It has a cost: inserting one byte near the beginning shifts subsequent boundaries, so many chunk digests change even though most file content is similar.

Content-defined chunking chooses boundaries based on the byte content, often using a rolling fingerprint. It can improve reuse after insertions, but adds parameters, variable chunk sizes and new performance considerations. You do not need it to understand distributed storage. Put it on the V2 list with a clear motivation instead of adding it because the name sounds sophisticated.

Our priority is a format you can reconstruct without ambiguity. The manifest will record ordered chunk descriptions, their lengths, and eventually the coding and encryption information needed to reverse the pipeline.

Build contract

Stream an input into fixed-size chunks with bounded memory. Emit each chunk's index, length and content digest. Preserve order independently of when later uploads complete. Define buffer ownership explicitly. Reject invalid chunk sizes and propagate source errors without claiming a complete file.

For now, you can produce a temporary local list of chunk descriptors. Chapter 9 makes it a durable manifest. Do not discard that list until you have demonstrated reconstruction; it is already essential state, even if it still lives in a plain local file.

Acceptance scenarios

1. Exercise lengths 0, 1, chunk-size minus 1, exactly chunk-size, chunk-size plus 1, and several chunks plus a tail.
2. Feed the same bytes through readers with different fragment sizes. Chunk boundaries and digests are identical.
3. Concatenate emitted chunks in index order. The result exactly matches the input.
4. Reuse or modify the producer's buffer according to the documented policy. No retained chunk unexpectedly changes.
5. Inject a non-EOF source error. The operation fails and never reports a complete file.
6. Process a generated large stream without holding the complete stream in memory. Explain your maximum number of simultaneously live chunk buffers.

The final check is a budget, not a request for a particular benchmark number. A bound that grows with worker count is acceptable. A bound that grows with file size defeats the purpose of streaming.

index = 0
repeat:
    allocate or borrow one chunk buffer
    fill until full, end of input, or source error
    if bytes were read: emit index, exact length, exact bytes
    if source error other than EOF: fail the upload
    if end of input: finish
    index = index + 1

package workshop

import (
	"errors"
	"io"
)

// Chunks transfers independently owned buffers to emit and leaves r open.
// A non-EOF source error fails the upload, even if it accompanies bytes.
func Chunks(r io.Reader, size int, emit func(int, []byte) error) error {
	if size <= 0 || size > MaxFrame {
		return ErrLimit
	}
	for index := 0; ; index++ {
		b := make([]byte, size)
		filled, emptyReads := 0, 0
		ended := false
		for filled < size {
			n, err := r.Read(b[filled:])
			if n < 0 || n > size-filled {
				return errors.New("invalid reader count")
			}
			filled += n
			if err != nil {
				if !errors.Is(err, io.EOF) {
					return err
				}
				ended = true
				break
			}
			if n == 0 {
				emptyReads++
				if emptyReads >= 100 {
					return io.ErrNoProgress
				}
			} else {
				emptyReads = 0
			}
		}
		if filled > 0 {
			if err := emit(index, b[:filled]); err != nil {
				return err
			}
		}
		if ended {
			return nil
		}
	}
}

Before you close the laptop

If you scatter ten ordinary chunks across ten nodes and lose one node, can you reconstruct the file? No. You have organised the information, not increased it. The next chapter adds information that initially looks redundant and becomes extremely valuable at exactly the wrong moment.

Suppose two archivists each hold half a photograph. You add a third archivist and ask them to hold something that will let either original archivist disappear. Giving the third person a full copy works, but it costs as much space as the original photograph. Can the extra information be smaller?

For a two-part stripe, yes. An XOR parity shard the size of either part is enough to recover either missing part. The trick is not predicting which machine will fail. It is arranging the information so that either of two possible losses can be reversed.

What you will learn

You will derive a 2+1 erasure code, preserve padding information, distinguish a missing shard from a corrupt shard, and explain why a general k+m code needs independent information. You will implement the simple codec yourself. The field arithmetic of a general Reed–Solomon implementation is a separate engineering task; we will learn its contract without casually inventing a production codec.

Take a chunk of length L. Split it into two equal-length data shards A and B, each of length ceil(L/2), padding the tail with zero bytes if necessary. Create parity P by XORing corresponding bytes of A and B. Record L so restoration can remove padding exactly.

Why XOR can bring a shard back

XOR has two useful properties: a value XOR itself is zero, and a value XOR zero is unchanged. If P = A XOR B, then P XOR B = A. Similarly, P XOR A = B. If parity alone disappears, compute it again from A and B.

Use tiny bytes before using megabytes. Let A be hexadecimal 3C and B be A5. Their XOR is 99. If A disappears, 99 XOR A5 gives 3C. This is not compression. Across the three shards you store more information than the original data, deliberately arranged to tolerate one missing component.

The original chunk occupies roughly 2S bytes for shard size S; the encoded stripe occupies 3S. That is approximately 1.5 times the original size, ignoring metadata, encryption overhead and small-file padding. Three complete replicas occupy 3 times the original size but can survive two replica losses if any remaining copy is intact. These are different failure budgets. Equal node counts do not imply equal protection.

One stripe, three distinct indexes

Call this configuration 2 data + 1 parity, or 2+1. A stripe is the group of related shards produced by one encoding operation. Each shard has an index: 0, 1 or 2. Recovery requires two distinct valid indexes from the same stripe and generation.

Two copies of shard 0 do not substitute for shard 1. They improve availability of index 0 but do not supply new independent information. A downloader that counts network responses instead of distinct verified indexes can cheerfully announce a quorum of duplicates and then fail to reconstruct anything.

Likewise, shards from two versions cannot be mixed. The fact that two shards have the same length or the same filename does not make them members of the same stripe. The manifest will bind each index to its expected digest and coding configuration.

Corruption must become an erasure

A simple erasure codec assumes it knows which pieces are missing. A corrupted byte presented as valid input can contaminate reconstruction. Therefore, verify each candidate shard against its expected digest before admitting it to the decoder. Treat a failed verification as an absent shard and try another candidate.

With 2+1, one missing or rejected shard is tolerable. Two are not. A parity consistency check can detect that a complete set does not agree, but that alone does not tell you which shard is wrong. Trusted per-shard digests supply the location of corruption. Later, authenticated metadata protects those digests from replacement.

The distinction between erasures and arbitrary errors is central. A phrase such as “survives one failure” must specify whether the failure is a known missing shard, an undetected corrupted shard, a malicious response, or a whole machine containing several indexes.

From XOR to Reed–Solomon

A suitable Reed–Solomon erasure code produces k data and m parity shards so that any k valid, distinct shards can reconstruct the stripe. It uses arithmetic over a finite field to build independent combinations. Repeating the same XOR parity twice does not produce a 2+2 code capable of tolerating arbitrary two-shard loss; the second copy contains no new equation.

For 4+2, the nominal expansion is 6/4, also 1.5 times. It can tolerate any two shard erasures when the assumptions hold, but needs at least four valid shards and spreads the stripe across more locations. Metadata, per-shard request overhead, repair fan-in and small-stripe padding all affect whether that is a good choice.

The maintained klauspost/reedsolomon project documents its split, encode, reconstruct and join operations. If you use it for the optional generalisation, pin a version in your own module, read its buffer-mutation and missing-shard conventions, and put your format contract around it. A library should not get to define an undocumented storage format by accident.

Build contract

Implement a pure 2+1 codec. Encoding accepts arbitrary bytes and returns three independent shard buffers plus original length. Decoding accepts indexed optional shards, validates their shapes and requires at least two distinct valid indexes. It returns the original bytes without padding. The codec owns no sockets, filenames, encryption keys or peer list.

Treat an empty chunk explicitly. In the course file format, an empty file emits no stripes, so ordinary upload never calls the codec for it. Your standalone codec may support a zero-length round trip or reject it consistently; the companion supports it to exercise the boundary. Never let zero-length “present” and nil “missing” silently become the same state.

Acceptance scenarios

1. Round-trip odd, even, one-byte, zero-byte and multi-megabyte inputs through the codec's documented empty-input policy.
2. Remove index 0, then 1, then 2 in separate trials. Each remaining pair reconstructs the original bytes.
3. Remove every pair of indexes. Recovery fails with insufficient information.
4. Supply unequal shard lengths or an impossible original length. Reject the input.
5. Corrupt one shard before the verification layer. The verifier rejects it; the other two reconstruct successfully.
6. Supply the same index twice. It counts once, even if fetched from different nodes.

The companion includes runnable XOR codec tests, including all single-loss cases and malformed lengths. They are a useful fixed target before networking enters the picture.

validate available shard indexes and equal lengths
require at least two distinct available indexes
if A is missing: A = B XOR P
if B is missing: B = A XOR P
concatenate A then B
return exactly original_length bytes

package workshop

import "errors"

var ErrShards = errors.New("invalid or insufficient shards")

// Encode2 returns two data shards and XOR parity. Every returned slice has
// independent storage. Empty non-nil shards represent a present empty stripe.
func Encode2(data []byte) ([3][]byte, int) {
	size := (len(data) + 1) / 2
	var out [3][]byte
	for i := range out {
		out[i] = make([]byte, size)
	}
	copy(out[0], data[:min(size, len(data))])
	copy(out[1], data[min(size, len(data)):])
	for i := 0; i < size; i++ {
		out[2][i] = out[0][i] ^ out[1][i]
	}
	return out, len(data)
}

// Recover2 requires hash-validated inputs from one stripe. Nil means missing.
// The codec does not authenticate inputs or locate unknown corruption.
func Recover2(shards [3][]byte, original int) ([]byte, error) {
	count, size := 0, -1
	for _, b := range shards {
		if b == nil {
			continue
		}
		count++
		if size == -1 {
			size = len(b)
		} else if size != len(b) {
			return nil, ErrShards
		}
	}
	if count < 2 || original < 0 || original > 2*size || (original+1)/2 != size {
		return nil, ErrShards
	}
	a, b := make([]byte, size), make([]byte, size)
	if shards[0] != nil {
		copy(a, shards[0])
	} else {
		for i := range a {
			a[i] = shards[1][i] ^ shards[2][i]
		}
	}
	if shards[1] != nil {
		copy(b, shards[1])
	} else {
		for i := range b {
			b[i] = shards[0][i] ^ shards[2][i]
		}
	}
	return append(a, b...)[:original], nil
}

Before you close the laptop

Explain why a three-node, 2+1 stripe survives one arbitrary node loss only when those nodes hold distinct indexes and represent suitable independent failure domains. That last phrase is the subject of the next chapter.

You start three storage nodes on your Mac. Each has a different port, data directory and shard. You stop one process and recovery succeeds. This is a useful test. It is not evidence that your backup survives the Mac falling into a pond.

All three processes share a machine, probably a disk, a power supply and a location. A single event can remove all of them. Distributed storage becomes reliable only when its placement decisions reflect the failures it is meant to survive.

What you will learn

You will distinguish node identity from network address, model failure domains, design deterministic placement, and calculate what an upload acknowledgement actually guarantees. You will also learn to treat capacity claims and placement plans as proposals rather than facts.

A node identity should persist across restarts. A listen address can change. If the identity is regenerated on every start, one physical machine can accidentally appear as several independent storage providers. If the identity is just an address, changing an address looks like replacing a machine. Store identity durably and keep it separate from contact information.

Choose the failure you are designing for

For the first network milestone, require three distinct nodes for a 2+1 stripe. For a deployment intended to survive one machine failure, place each index on a different machine. If surviving a building failure matters, machine independence is not enough; place data across buildings. Each stronger requirement adds operational costs and constraints.

You cannot infer physical independence merely from different public keys. A single operator can create many identities. In a cooperative personal tailnet, you may maintain trusted domain labels in configuration: machine, household, perhaps storage device. In an open network, verifying such claims becomes a separate trust and incentive problem.

The scheduler should use these labels as constraints, not decorations in a dashboard. If no valid placement exists, return an explicit under-protected state or reject the fully protected upload. Do not silently put two indexes on one machine and continue displaying “one-machine failure tolerant”.

A placement plan is not an acknowledgement

The client selects nodes A, B and C for indexes 0, 1 and 2. That plan is intent. Only successful durable PUT acknowledgements establish that the nodes accepted their assigned bytes at that moment.

Uploading to two nodes already makes a 2+1 stripe mathematically recoverable. It does not give the intended spare shard. Report the difference: recoverable means at least k distinct valid shards can be obtained; fully protected means the target placement and redundancy requirements have been satisfied by the evidence you require. An upload can be recoverable while awaiting protection.

For the first core, choose a strict policy: report “fully protected” only after all three shard acknowledgements and the required manifest acknowledgements. A failed attempt may leave useful orphan shards; immutable objects make later reconciliation possible. The user can receive a precise incomplete result rather than a false success.

Determinism makes retries calmer

A naive scheduler picks three random peers on every retry. If a response is lost, the retry may place the same stripe on three entirely different peers. This is not necessarily incorrect, but it can consume needless bandwidth and leave many untracked copies.

A deterministic ranking helps. One option is rendezvous hashing: score each eligible node using a stable hash of the object or stripe identity and node identity, sort by score, then choose nodes subject to failure-domain constraints. A membership change affects the rankings predictably without requiring a central table of every placement decision.

This course does not require a particular ranking function. It requires that the selection be testable and that retries reconcile previous successes. Keep dynamic capacity checks distinct from stable ranking. A node with a high rank can still be full, draining or unreachable. Record the successful placements, not just the preferred ones.

Do not use the byte digest alone to assign all indexes of one stripe to the same top node. Selection needs awareness of the stripe as a group and the requirement for distinct destinations. A placement policy that is excellent for independent replicated objects may be wrong for erasure-coded siblings.

Capacity has to be reserved somewhere

A health response says a node has 10 GiB free. Ten clients each decide to send 2 GiB. Their arithmetic was locally reasonable and globally impossible.

Treat free-space reports as advisory. The node that owns the disk must enforce actual capacity during admission and commit. It may reserve bytes for in-flight requests, reject excess work, and release reservations on failure. The placement client handles rejection by choosing another eligible node within a bounded attempt budget.

Quota accounting must be atomic with the state it protects. Checking free space and incrementing usage without coordination allows concurrent requests to pass the same check. We revisit this in the concurrency and credits chapters; for now, serialize local admission and commit decisions under a clear owner.

Build contract

Run three independent storage nodes. Give each a persisted identity and explicit failure-domain label. Implement a placement policy that returns one destination per shard index while respecting domain constraints. Upload a stripe, record only successful durable placements, and distinguish incomplete, recoverable and fully protected outcomes.

Make the policy a pure decision over a snapshot of candidate metadata where practical. The network code obtains observations; the placement function ranks and filters them. This separation lets you test awkward membership sets without opening sockets.

Acceptance scenarios

1. Shuffle the same eligible peer set. A deterministic policy produces the same placement.
2. Give two processes the same node identity. They never count as two independent nodes.
3. Give different identities the same machine label. The machine-failure policy never places two sibling indexes there.
4. Provide too few independent domains. Return an explicit inability to meet the protection target.
5. Make the preferred node reject storage as full. Select an eligible replacement without claiming the original plan succeeded.
6. Lose one node after a fully protected 2+1 upload. Restore using the remaining two indexes. Explain why repeating this with three local processes tests protocol behaviour rather than physical disaster tolerance.

Before you close the laptop

Draw your actual machines, disks and locations. Circle everything that can disappear in one event. Now compare those circles with the promise your UI or command prints. The diagram is less flattering than a node count, and much more useful.

The network contains every piece of your photograph. You have verified their hashes. Each piece exists on a healthy machine. Then you delete the only local list of shard identifiers.

The bytes are still there. The file, from your point of view, may be gone.

A manifest is the map that turns stored objects back into a file. It carries the information that the bytes alone do not reliably tell you: order, length, coding parameters, and later encryption context. Metadata is often small, which makes it easy to underestimate. Small is not the same as unimportant.

What you will learn

You will design a versioned immutable manifest, identify a recoverable root, separate content identity from mutable placement, and commit metadata only after its prerequisites. You will also learn to distinguish a recoverable object graph from a pile of durable files.

For each uploaded file version, create a fresh immutable manifest. Give it a format version, an owner identifier, a file identity, a version identity, original file size and an ordered list of stripes. Each stripe includes its position, original chunk length, encoded length where needed, codec name and parameters, and the expected digest for each shard index.

A name is a different kind of state

The human name photos/holiday.jpg may refer to different versions over time. Do not make that mutable name the identity of the stored bytes. Store immutable versions and represent the relationship between a name and versions separately.

A source identity is also useful. Two devices may both have a folder called Pictures, and two users may use the same relative path. Namespacing by owner, source and file identity avoids accidental collisions. It does not resolve concurrent edits; it merely stops unrelated files from masquerading as the same file.

For now, a file identity can be an owner-scoped random identifier and each version can reference its parent. Later, version graphs let you detect competing descendants. A timestamp is useful for display but does not establish a trustworthy global order between machines.

Keep content and location separate

A manifest's description of shard content should remain valid when repair moves a shard to a new node. If every location change requires rewriting the authenticated content manifest, a storage-only repair worker may need authority it should not possess.

Use two concepts. The immutable manifest binds a stripe and shard index to expected bytes. A placement catalogue or availability record says where those bytes were observed. Placement observations are hints: they expire, can be stale, and must be verified during retrieval. The digest in the trusted manifest decides whether a retrieved candidate is acceptable.

A small first implementation may store initial locations in the manifest as hints, then discover additional copies by inventory. Make their non-authoritative status explicit. Do not let an untrusted location update change the expected digest, codec or original length.

Publication has prerequisites

An upload can leave shards behind without ever publishing a file manifest. That is an orphan problem, not necessarily data loss. The opposite ordering is more dangerous: publishing a complete-looking manifest before its required shards have been acknowledged invites a reader to believe a file is protected when it is not.

Choose the commit sequence: generate and stage the immutable content description; upload the required shards; gather acknowledgements; durably publish the manifest to the chosen number of metadata locations; only then report the corresponding file-level success. If the final response is lost, a retry needs the version identity or operation record to discover whether publication happened.

There is no single atomic transaction spanning these ordinary nodes. The sequence must tolerate partial effects. Immutability makes it possible to retry and reconcile. A later journal will make the retry state survive client crashes.

Follow the chain all the way back

Suppose the manifest is replicated three times. How do you find it after losing your laptop? Perhaps you have an owner catalogue replicated on the network. How do you find that catalogue? Perhaps configured seed nodes can enumerate a signed owner namespace. How do you prove it belongs to you? Your recovery key anchors ownership.

This chain ends in a recovery root: the information that must be available independently of the failed device. In our course design, the owner retains recovery key material and enough bootstrap contact information to reach the admitted network. Nodes expose bounded inventories of signed or authenticated owner records according to the admission policy. The client verifies records and rebuilds its catalogue.

The key alone cannot conjure the address of an unknown network. A seed address alone cannot decrypt encrypted metadata. Write down both requirements. Also decide how to detect an incomplete or stale catalogue; a peer can serve an old valid record. Authenticity and freshness are different properties.

Build contract

Define and document manifest version 1. Serialize deterministically where identity or signatures depend on exact bytes. Enforce limits on stripe counts, lengths, nesting and total decoded metadata size. Validate all shard indexes and coding parameters before attempting a restore.

Persist immutable manifests after the required shard acknowledgements. Replicate the manifest itself; for the course target, store three complete copies across distinct nodes. Maintain a way to enumerate or otherwise recover known versions using owner-scoped bootstrap information. Make the recovery procedure work after removing the client's local catalogue.

The initial manifest may be plaintext during this milestone's local fixture tests. Chapter 11 encrypts the private description before real personal data enters the network. Treat this temporary format as a teaching stage with an explicit migration boundary.

Acceptance scenarios

1. Upload, terminate the client, discard its in-memory state, and restore using the durable manifest.
2. Remove the client's catalogue. Recover known manifests through your documented bootstrap route.
3. Remove one metadata replica. Restore still finds a valid manifest and enough shards.
4. Reject duplicate shard indexes, impossible lengths, unknown codec versions and excessive stripe counts.
5. Change a placement hint. The content identity and expected shard digests remain unchanged.
6. Fail the last required shard upload. The operation must not advertise a fully protected file version.

Before you close the laptop

Pretend every local cache has vanished. Describe restore from the first surviving piece of information to the final byte. Do not skip “find the manifest” as if it were a built-in law of nature. The map is part of the thing you are protecting.

Uploading is optimistic. It starts with the original file and a set of machines you hope will keep it. Restoring is an audit. It asks whether the promises made earlier still have enough evidence behind them.

A happy restore path can be surprisingly short: fetch pieces, reconstruct, concatenate, write. A trustworthy restore path needs to be more sceptical. Are these the right pieces? Are their indexes distinct? Does the padding length make sense? Did an error occur after some bytes were already written? Is the destination an existing file the user cares about?

What you will learn

You will build a staged restore pipeline, turn corrupted responses into erasures, publish output only after validation, and preserve useful failure explanations. You will also see how redundancy, integrity and availability cooperate without becoming the same thing.

Start from a validated manifest. The manifest is an instruction set supplied across a trust boundary, so validate its structure before allocating buffers or creating paths. A claimed file size of several exabytes is not an invitation to preallocate disk. A relative filename is not automatically safe to join to a restore directory.

Admit only verified candidates

For each stripe, enumerate candidate locations for the expected shard indexes. Fetch a candidate with a bounded request. Check its exact length and digest against the manifest. Only then mark that index available to the decoder.

If a node returns corrupt bytes, record the observation and try another location for that index or another required index. Do not feed the corruption into parity reconstruction and hope that a later whole-file hash will fix things. A final hash can tell you something went wrong; it cannot recover the good information you discarded.

Count each index once. Preserve its relationship to the stripe and file version. Once you have k valid distinct shards, reconstruction may proceed. Additional responses can still arrive, so the concurrent implementation will need to cancel or drain work without leaking resources. For this chapter, a sequential downloader is fine.

Check the reconstructed object too

After reconstruction, validate the reconstructed shard hashes when you have expected values for them. Reassemble the chunk and trim using the original length from the trusted description. Once encryption arrives, this reconstructed object will be ciphertext; authenticated decryption must succeed before plaintext becomes output.

Maintain a whole-file digest as you write the staged destination. Compare final size and digest to the manifest before publication. This catches ordering mistakes, repeated chunks and truncation that per-shard checks alone may not catch. In the encrypted format, keep that plaintext digest inside the encrypted manifest rather than advertising it publicly.

The whole-file check is not a substitute for the earlier checks. Each check guards a different boundary. Per-shard digests help identify usable reconstruction inputs. The final file check verifies the assembled result. Authenticated encryption and signatures establish additional properties about origin and context.

Never make recovery destructive

Restoring over an existing file is a separate operation with its own user intent. For the course, refuse to overwrite. Write to a unique temporary file under the chosen destination and publish only after all validation succeeds using a no-clobber strategy.

Confinement is equally important. A manifest may contain ../../somewhere or a path that traverses a symlink. String cleanup alone is not a complete defence against filesystem changes between checking and opening. Go's traversal-resistant file APIs explain the motivation for os.Root. If your target Go version supports the operations you need, use a rooted approach; otherwise design and test equivalent platform-specific confinement rather than assuming a cleaned string is sufficient.

For the early CLI, an even narrower boundary is useful: the user supplies one explicit output filename, and the remote manifest does not choose it. Directory-tree restoration can be a later extension with a separate path policy.

Make failures useful to a person

“Restore failed” gives no next action. “Only one of two required shard indexes is currently reachable for stripe 18” explains the immediate obstacle. Include whether candidates were missing, timed out, failed integrity checks or were rejected because metadata was invalid.

Avoid overstating permanence. A timeout means a node was not reachable within this request's budget. It does not prove permanent loss. Report the current inability to reconstruct and preserve enough information for another attempt. On the other hand, do not display an endless spinner when the current attempt has exhausted all eligible candidates. A bounded failure is an honest result.

An output file that contains half a photograph must not occupy the final destination name. Keeping a partial staged file for resumability is possible, but that becomes durable workflow state and requires validation when resumed. The first version can delete its own failed staging file and retry cleanly.

Build contract

Implement end-to-end restore from a persisted manifest. Fetch and validate distinct shards, reconstruct each stripe, maintain order, verify whole-file length and digest, then publish a new destination. Treat bad candidates as unavailable and continue while recovery remains possible. Return a precise failure when it does not.

The first full milestone is now within reach: three node processes, a file uploaded into 2+1 stripes, one stopped node, and a byte-identical restore from the remaining two. Keep the metadata copies in the test too. A test that secretly retains the original manifest in a global variable skips part of the recovery system.

Acceptance scenarios

1. Restore a multi-stripe binary file after stopping each node in turn. Compare the final bytes and size.
2. Corrupt one stored shard. Verify that it is excluded and the other two indexes recover the stripe.
3. Remove one shard and corrupt another in the same stripe. Fail clearly and leave no final destination.
4. Deliver correct shards in a deliberately shuffled order. The reconstructed file still follows manifest order.
5. Use an existing destination file. Its contents remain unchanged.
6. Supply an invalid manifest path or length. Reject it before escaping the destination boundary or making an unreasonable allocation.
7. Fail the final destination write. Do not report a successful restore even if every network read succeeded.

validate manifest and destination policy
create private staging output
for each stripe in manifest order:
    collect k distinct hash-valid shard indexes
    if impossible within budget: fail
    reconstruct and validate the encoded chunk
    decode / decrypt according to the format
    write exact plaintext bytes to staging output
verify total size and whole-file digest
publish output without replacing an existing file

Before you close the laptop

You can now deliberately remove a node and recover a file. That is satisfying. It is also time to notice that the surviving nodes can currently read the data they hold. Next we move the privacy boundary to the owner device.

Your friend offers spare disk space. You trust them to keep the machine running. You would still prefer that their storage daemon could not browse your photographs. This is a perfectly reasonable separation of trust.

Transport encryption protects bytes while they travel between endpoints. Once a storage node receives plaintext, transport encryption has completed its job. If the requirement is that storage operators cannot read your files, encryption must happen before the file reaches those operators.

We are going to encrypt on the client, then erasure-code the encrypted chunk. That ordering lets storage-only nodes reconstruct lost ciphertext shards without learning the plaintext or receiving the owner's key.

What you will learn

You will define a threat model, use authenticated encryption through a standard library, manage nonces and associated data, and separate repair authority from decryption authority. The aim is sound use of established primitives, not the invention of a new cryptographic scheme.

Our model trusts the owner's device while it is unlocked. It does not trust storage peers with plaintext or user secrets. Peers may inspect stored bytes and metadata, return bad data, replay old records, refuse service or delete objects. Encryption helps with confidentiality and authenticated decryption; it cannot force a peer to answer or restore a key that every trusted copy has lost.

Choose a complete authenticated-encryption format

Use an AEAD construction such as AES-GCM through Go's standard cryptographic APIs. AEAD combines encryption with an authentication tag. Decryption either authenticates the ciphertext and associated context or fails. Do not expose unauthenticated plaintext as a partially successful result.

For this course's explicit-nonce format, each fresh encryption uses a fresh random 96-bit nonce with AES-GCM. Never reuse a nonce under the same key. Random selection has a collision probability. Go's random-nonce GCM helper specifies at most 2^32 encryptions per key; treat that as a ceiling, not a target, and choose a documented lower operating budget where appropriate. Rotate or derive appropriately scoped keys before exceeding your limit. The Go cipher documentation and random-nonce helper documentation describe API-specific requirements; different helpers have different framing conventions. Pick one convention and record it.

A self-describing envelope needs a format version and algorithm identifier, the nonce, ciphertext and tag. The companion example is deliberately narrower: a known AES-GCM format with nonce prepended. In your network format, validate the algorithm and lengths before use. Algorithm agility means explicit versioned parsing, not accepting arbitrary cryptographic choices proposed by a peer.

Bind the ciphertext to its intended context

AEAD associated data is authenticated but not encrypted. It can bind a chunk to an owner, file version, stripe index and format version. A valid ciphertext from stripe 3 should not be accepted as stripe 7 merely because the key is the same.

Encode this context unambiguously. Concatenating arbitrary strings without lengths or separators can make different field tuples produce the same bytes. Fixed-width binary fields, length-prefixed values or another precisely specified encoding are reasonable options. Both encryptor and decryptor must construct exactly the same associated data.

Choose the file-version identity before encrypting. Do not create a circular definition in which associated data includes a manifest digest that itself depends on the resulting ciphertext. A random version identifier or a separately defined stable context avoids that loop.

Keep private filenames, plaintext digests and detailed timestamps inside the encrypted manifest. Storage peers may still observe ciphertext sizes, owner namespaces, object counts, associations and traffic timing. “Encrypted” does not mean “nothing can be learned”. State the metadata leakage you accept.

The order matters for repair

For each plaintext chunk, seal it into an authenticated ciphertext envelope. Record the envelope length. Encode that envelope into 2+1 shards and hash the exact stored shard bytes. The manifest describes the codec, original plaintext length, ciphertext-envelope length, associated-data context and expected shard digests.

On restore, verify shard digests, reconstruct the envelope, trim coding padding to its exact recorded length, authenticate and decrypt, then verify the assembled file. On repair, stop after reconstructing and verifying the missing ciphertext shard. No decryption is required.

If you instead encrypt each already-coded shard independently, repair must deal with a different set of keys, nonces and transformations. That design can work, but it is not interchangeable with this one. Choose one pipeline and make the format describe it. The word “encrypted” is not a sufficient interoperability specification.

Keys need a recovery story

Generate high-entropy key material with the operating system's cryptographic randomness. A human-chosen phrase is not equivalent to 32 random bytes. If you later support passwords, use an appropriate password-based key derivation design with documented parameters; do not hash a password once and call it an encryption key.

Separate purposes. Encryption and signing should use independent keys or keys derived with distinct domain-separated contexts from a high-entropy root. Store the root only on owner-controlled devices and in the owner's recovery backup. Storage nodes do not need it.

An encrypted backup with no surviving decryption key is unrecoverable. This is not an implementation error that a reset button can bypass. Practise recovery with synthetic data and a separately retained key before trusting the workflow. The system's bootstrap instructions must say which key material and network contact information the owner needs.

Build contract

Add a versioned client-side encryption envelope and place it before erasure coding. Encrypt the private manifest too. Storage-only nodes receive ciphertext shards and the minimum authenticated information needed for verification and repair. They never receive the owner's root or decryption key.

Keep the earlier plaintext format a distinct teaching version. Do not silently reinterpret old objects. For the course, re-upload synthetic fixtures into the new format; a real migration would need to preserve the old restore path until verification and retention requirements are met.

Acceptance scenarios

1. Encrypt and decrypt empty, short and multi-chunk data with the correct key and context.
2. Encrypt the same plaintext twice with fresh nonces. The resulting envelopes differ and both decrypt correctly.
3. Flip a bit in ciphertext, nonce, tag or associated data. Authentication fails; no successful plaintext result is exposed.
4. Use the wrong key or stripe index. Decryption fails.
5. Repair a lost ciphertext shard on a node that has no user key, then restore successfully on the owner device.
6. Inspect a storage-only node's files and protocol records. No plaintext filename, plaintext digest or user key is present in the public storage representation.

package workshop

import (
	"crypto/aes"
	"crypto/cipher"
	"crypto/rand"
	"errors"
	"io"
)

func aeadFor(key []byte) (cipher.AEAD, error) {
	if len(key) != 32 {
		return nil, errors.New("expected a 32-byte key")
	}
	block, err := aes.NewCipher(key)
	if err != nil {
		return nil, err
	}
	return cipher.NewGCM(block)
}

// Seal uses the fixed workshop format nonce || ciphertext || tag.
// A real versioned format must name this scheme. Enforce per-key encryption
// limits externally; never reuse a nonce with the same key.
func Seal(key, plaintext, aad []byte) ([]byte, error) {
	a, err := aeadFor(key)
	if err != nil {
		return nil, err
	}
	nonce := make([]byte, a.NonceSize())
	if _, err = io.ReadFull(rand.Reader, nonce); err != nil {
		return nil, err
	}
	return a.Seal(nonce, nonce, plaintext, aad), nil
}

func Open(key, envelope, aad []byte) ([]byte, error) {
	a, err := aeadFor(key)
	if err != nil {
		return nil, err
	}
	n := a.NonceSize()
	if len(envelope) < n+a.Overhead() {
		return nil, errors.New("truncated envelope")
	}
	plain, err := a.Open(nil, envelope[:n], envelope[n:], aad)
	if err != nil {
		return nil, err
	}
	return plain, nil
}

Before you close the laptop

A peer can repair your backup but cannot decrypt it. Explain exactly which information makes that possible. Then name two things encryption still cannot guarantee. Availability and freshness should be on your list.

A node returns a manifest with a perfectly valid digest. The digest matches the bytes. Unfortunately, the entire manifest was invented by the node. Your integrity checks have verified an internally consistent story told by the wrong author.

We need an anchor outside the storage peer's control. A public-key signature can let the client verify that a record was authorised by the owner identity it expected. But signatures are precise tools. They do not make a statement fresh, generous, truthful or useful merely because somebody signed it.

What you will learn

You will separate identities and capabilities, design signed immutable records, prevent ambiguous signed encodings, and reason about replay and key loss. You will also define what an admitted storage node is allowed to do without giving it the owner's authority.

Use different identities for different roles. A node identity represents a storage process or machine under your admission policy. An owner identity represents control of user key material. A network membership credential grants access to the cooperative storage network. A person can own several clients and operate several nodes; those relationships do not collapse the roles into one key.

Sign a statement with exact bytes

Choose a standard signature implementation such as Ed25519. Define exactly what is signed: a domain/version tag, owner identity, encrypted private manifest bytes and a public repair descriptor containing stripe structure and expected ciphertext-shard digests.

Signing a JSON object's abstract meaning is underspecified unless canonicalization is defined. Field order, whitespace, number formatting and escaping can vary. One straightforward design signs the exact serialized payload bytes and transmits those bytes unchanged inside an envelope. The verifier authenticates the exact payload, then parses and validates it. Alternatively use a carefully specified canonical format. Do not serialize independently on two platforms and assume the bytes will match.

A domain tag prevents a valid signature for one kind of statement from being reused as another. “Owner authorises file record version 1” and “Owner authorises deletion” should not accidentally have overlapping signed representations.

Signatures cover integrity and origin relative to a trusted public key. They do not encrypt public fields. If you sign a plaintext filename and send it to every peer, the filename is still public.

Two views of the same record

The private manifest holds filenames, plaintext sizes and digests, timestamps and client-specific metadata. Its encryption protects those details. The public repair descriptor exposes only what storage nodes need: owner namespace, record identity, codec parameters, stripe grouping, ciphertext lengths and expected shard hashes.

The owner signature binds the public descriptor to the encrypted private manifest. After decrypting, the client checks that repeated fields agree. A malicious peer must not be able to attach a different repair graph to a legitimate encrypted manifest. A repair worker can validate the signature and reconstruct ciphertext using the public descriptor without learning the private fields.

This leaks relationships: peers can see which opaque shards belong together and approximately how much data an owner stores. That is a conscious trade-off for repair without user keys. More private repair protocols are possible, but they belong to a different threat model and design budget.

Permission is a separate check

Knowing an owner's public key does not grant permission to fill the network with arbitrary data. An admitted node may be allowed to relay valid owner-signed records and ciphertext. It should not be allowed to create owner records, decrypt data, or issue destructive actions on the owner's behalf.

Likewise, possession of the network admission key should not become equivalent to possession of every user's recovery key. The existing MeshVault prototype uses these separate roles; its protocol notes are included with the companion comparison source. Treat that as an example to inspect, not a substitute for deciding your own course protocol.

For the course's small tailnet, use explicit membership admission and per-owner authenticated operations. Bind authorisation to operation, resource, body digest and relevant context. Limit request sizes and quotas even for authenticated users. A valid identity can still make an expensive or abusive request.

Replay is not forgery

A storage peer serves yesterday's correctly signed manifest and hides today's. The signature verifies. The record may be authentic and stale.

Immutable version IDs make replay of an individual PUT harmless at the storage-object level. They do not prove that a returned catalogue is complete or current. Preserve parent relationships, remember the newest observations you have durably seen, and compare independent peers where possible. Detecting that somebody withholds an unseen version is harder than detecting a forged version. Document that limitation.

For request authentication, timestamps and nonces can limit replay windows, but bring their own state and clock assumptions. A nonce cache that disappears at restart cannot support claims of replay protection across restarts. Design destructive effects to require stronger durable replay handling, or keep destructive operations out of the first core. Our baseline retains immutable versions and does not remotely erase owner history.

Build contract

Create owner-signed immutable record envelopes with a documented byte representation. Verify expected owner identity, signature, format and bounds before accepting a record. Separate public repair data from encrypted private metadata and validate agreement after decryption. Keep network membership distinct from owner authority.

Add a recovery-key export/import path for your synthetic test owner. The key is not a username and cannot be reset without an additional recovery design. Never put private keys in logs or signed public payloads. Public test vectors should use deliberately synthetic keys only.

Acceptance scenarios

1. Verify a valid signed record, then alter each signed field in turn. Every alteration fails verification.
2. Supply a correctly signed record from a different owner. It is rejected when the caller expects the original owner.
3. Re-encode a payload differently without regenerating the signature. Verification uses the transmitted signed bytes, not a guessed equivalent representation.
4. Attach a conflicting public descriptor to an encrypted private manifest. Signature or cross-check validation rejects it.
5. Replay an immutable record. It produces one logical version, while the system makes no unsupported claim that the catalogue is current.
6. Import the owner's recovery material into a fresh client and restore without giving that material to a storage-only node.

Before you close the laptop

Finish these sentences separately: “The transport is encrypted, therefore …”; “The signature verifies, therefore …”; “The record is the newest I have seen, therefore …”. If any answer says “therefore everything is safe”, try again with a narrower claim.

Your uploader sends one shard, waits, then sends the next. Most of its time is spent waiting for somebody else's disk or network. This is an excellent place for concurrency. It is also an excellent place to create thousands of goroutines that retain thousands of buffers while waiting for the same overloaded node.

The goal is not “use goroutines”. The goal is to overlap independent work while preserving a comprehensible limit on memory, connections and outstanding requests.

What you will learn

You will design a bounded worker pipeline, choose between channels and mutexes based on ownership, preserve ordered results from out-of-order completion, and calculate a memory budget. You will also learn why a race-free program can still be logically wrong.

Start by identifying stages: read a plaintext chunk, encrypt it, encode shards, upload them and collect placement results. These stages have different resource costs. More network workers may hide latency, but more simultaneous encoders can increase memory pressure. A useful design makes those costs visible.

Bound the thing that consumes memory

A semaphore limiting active socket writes is not enough if you create a goroutine and allocate a 4 MiB chunk for every future upload before acquiring it. The expensive work has already happened.

Acquire capacity before allocating or retaining the resources it is meant to bound. Alternatively, use a bounded job channel and a fixed number of workers, with the producer blocking when the queue is full. That blocking is backpressure: downstream capacity limits upstream production.

Calculate a first approximation. With a 4 MiB plaintext chunk and 2+1 encoding, retaining both the input and all encoded shards requires about 10 MiB before encryption overhead and other buffers. Four such workers need about 40 MiB. A queue holding ten additional complete chunk jobs can add another large allocation. The Go runtime, network buffers, manifests and temporary outputs add more.

The numbers are not a benchmark. They are a way of refusing to pretend that “goroutines are cheap” means their retained data is free. Measure your implementation after you have a model; a model tells you what a surprising measurement might mean.

Channels transfer work; locks protect shared state

A channel is useful when one stage hands a job or result to another. A mutex is useful when several goroutines need coordinated access to a shared map or counter. Neither mechanism is morally superior.

For example, a single result-collector goroutine can own the manifest under construction. Workers send results containing stripe index, shard index, node identity and error. The collector updates its private state and eventually emits a complete result. That avoids concurrent writes to the manifest and makes completion accounting local.

A node's small quota ledger might be simpler under a mutex. The lock protects the relationship between committed bytes, reserved bytes and the capacity limit. Holding the lock across a slow network call would make unrelated work wait, so reserve under the lock, release it during transfer, then settle or release the reservation under the lock.

The official Go memory model defines the synchronization relationships that make shared data access meaningful. The course rule is concrete: if multiple goroutines access mutable state and at least one writes, identify the synchronization or exclusive owner. Do not rely on “this usually runs first”.

Completion order is not file order

Stripe 7 may upload before stripe 2. A result channel reports completion order, not original file order. Preserve explicit indexes and assemble the manifest by those indexes.

The same problem occurs on download. If you write each completed chunk directly to the output stream, network timing becomes file layout. You can either buffer a bounded reorder window, schedule only a bounded number of sequential chunks, or write verified chunks at known offsets in a staged file with carefully checked lengths. Choose a strategy that preserves both order and your resource budget.

This is a logical concurrency bug, even if every map access is perfectly locked. The race detector will not tell you that you assembled a photograph in the order the network happened to finish it.

Closing a channel is a lifecycle decision

The goroutine that knows no more values will be sent should arrange closure. With multiple workers, that is often a coordinator that waits for all senders. A receiver should not close a shared input channel simply because it has enough results; other senders may still be active and panic on their next send.

A WaitGroup waits for a known group of work to finish. It does not cancel that work, carry its errors or make shared state safe. Add work to the group before the goroutine can race with waiting, and make every started worker account for its completion. Avoid copying synchronization primitives after use.

The next chapter handles cancellation. For now, ensure ordinary completion has one clear owner and no sender can outlive the channel it writes to. The Go team's pipeline article is useful further reading for channel lifecycle patterns; adapt its ideas to your actual budgets and failure paths.

Build contract

Introduce bounded concurrency for uploads. Configure maximum active workers and queue capacity. Ensure memory retention is bounded by those settings and chunk size, not total file size. Collect results under a clear owner and preserve stripe/index order. All workers finish before the operation returns normally.

Keep a sequential mode with one worker. It is useful for debugging and establishes that the concurrent version preserves the same observable results. Do not add concurrency to every stage at once; choose the slowest independent work first and measure whether the change helps.

Acceptance scenarios

1. Use a fake uploader that records active calls. The observed maximum never exceeds the configured worker limit.
2. Block every upload at a test gate. The producer eventually stops because the queue is full; it does not continue reading the complete file.
3. Complete results in reverse stripe order. The final manifest and restored file remain correctly ordered.
4. Run the relevant tests with go test -race ./.... Investigate every reported race.
5. Repeat with one worker and several workers. Both produce equivalent content descriptions, allowing expected differences such as fresh encryption nonces between separately created versions.
6. Verify that duplicate results cannot satisfy missing indexes or increment completion twice.

Before you close the laptop

Draw a box around each mutable value and write its owner or lock beside it. Then account for every live chunk buffer. If either exercise produces “shared by everyone”, you have found tomorrow's first improvement.

You cancel an upload. The command returns immediately. In the background, several goroutines keep waiting on sockets, one keeps trying to send a result nobody will read, and a temporary file remains open. The user thinks the work stopped. The process has merely stopped talking about it.

Cancellation is not a magic exception that unwinds every stack in Go. It is information that code must observe and act on. A correct operation has a path from cancellation intent to every resource that could prevent completion.

What you will learn

You will propagate contexts, connect cancellation to blocking I/O, make channel operations cancellable, join workers, and define the meaning of partial remote effects. You will also distinguish ending the caller's wait from ending the work itself.

Give each top-level upload or restore an operation context. Pass it explicitly through the call tree, normally as the first parameter. Derive child deadlines where a sub-operation needs a shorter budget, and call their cancel functions when the scope ends. Keep contexts out of long-lived structs unless there is a carefully justified ownership reason.

The official context documentation describes the API and cancellation propagation. It does not promise that arbitrary code using a reader or a file descriptor will automatically stop when the context is done.

Follow every blocking edge

List the places a worker can wait: receiving a job, sending a result, acquiring a concurrency slot, dialing a peer, reading a response, writing a body, or waiting for another worker. Each needs a defined exit when the operation is cancelled.

A channel send can select between delivering a value and the context's done signal. A context-aware dial can stop connection establishment. An established socket may need a deadline or closure by the owner to interrupt a blocked read. A callback that ignores cancellation may prevent the worker group from joining. You cannot fix that by surrounding the callback with one more goroutine and forgetting it.

For a sequential request-per-connection design, closing the operation-owned connection on cancellation is manageable. If you later share a connection between unrelated operations, closing it becomes a broader effect. That is one reason multiplexing is an architectural decision, not merely a performance toggle.

Early success also needs cleanup

During a 2+1 restore, the first two valid distinct shards may be enough. The third request can now be unnecessary. Returning immediately without handling it leaves work running.

You can cancel the remaining requests, wait for their workers to finish, and then continue. Or you can design a bounded result channel large enough for all guaranteed final responses and have a coordinator drain them. The exact strategy depends on the operation, but its proof must not rely on “the third response normally arrives quickly”.

A useful invariant is: when the operation returns, every goroutine it started has either finished or transferred ownership to an explicitly longer-lived component. For this course, prefer finishing. Ownership transfer is a more advanced lifecycle contract and should not happen accidentally.

Cancellation does not reverse a committed write

If a node committed a shard just before the client cancelled, the shard remains. Cancellation means stop further work as promptly as the contract allows; it does not mean undo every remote effect.

This matters for status reporting. An upload can be cancelled with some objects durably stored and no committed manifest. Keep the operation identity and provisional state if you intend to resume. Otherwise, leave immutable orphan objects for a later conservative cleanup policy. Do not send speculative deletes to every node merely to make the cancelled operation look as though it never existed.

The same principle applies to timeouts. A deadline ends the caller's willingness to wait. It does not establish a universal moment when every remote participant stopped acting.

Choose the error the caller should see

Concurrent failures can arrive together: one node times out, another returns corruption, and the user cancels. Decide which result represents the operation's outcome. You may preserve multiple diagnostic causes while returning a stable top-level category.

Avoid a race where whichever goroutine writes an error variable last wins. A single result owner can record the first fatal cause, initiate cancellation, collect cleanup results, and return a structured summary. The “first” error is a policy, not necessarily the most causally important event. If a user's explicit cancellation is the dominant outcome, state that while retaining relevant partial-effect information.

Cleanup failures should not erase the primary error. They may still matter: a temporary file that could not be removed consumes space, while a worker that could not be joined violates a stronger lifecycle expectation. Keep these distinctions visible in tests and diagnostics.

Build contract

Make upload and restore cancellation-aware from the public API to the actual network operations. Every worker has an exit path. The operation cancels outstanding work after a fatal error or sufficient successful restore inputs, then joins what it started before returning. Temporary resources are closed and handled according to policy.

Expose cancellation in the CLI through an interrupt signal without putting signal handling inside the storage package. The command translates the operating-system signal into context cancellation. A future Mac button can use the same engine boundary.

The companion worker helper demonstrates a bounded group whose callback is required to cooperate with context cancellation. Its documentation explicitly limits what it can guarantee if a callback ignores the context. That limitation is part of the contract, not an implementation embarrassment.

Acceptance scenarios

1. Cancel while workers are waiting for jobs, while the producer is blocked by a full queue, and while results are waiting for a consumer. Every group terminates.
2. Use a fake peer that accepts a connection but never responds. Cancellation interrupts the blocked operation within your documented bound.
3. Complete two valid restore indexes while the third request stalls. The operation cancels and joins the third request without waiting for its full ordinary timeout.
4. Cancel immediately after a remote commit. The result reports cancellation or uncertainty without claiming that no remote data exists.
5. Repeatedly start and cancel operations. Check explicit worker counters, open resources and temporary files rather than relying only on a noisy global goroutine count.
6. Run the cancellation tests under the race detector.

package workshop

import (
	"context"
	"errors"
	"sync"
)

// Run processes jobs with bounded workers, cancels after an error and joins
// every worker. fn must return when ctx is cancelled; this helper cannot
// forcibly stop a callback that ignores cancellation or blocks forever.
func Run(parent context.Context, workers int, jobs []int, fn func(context.Context, int) error) error {
	if workers <= 0 {
		return errors.New("workers must be positive")
	}
	ctx, cancel := context.WithCancel(parent)
	defer cancel()
	queue := make(chan int)
	var wg sync.WaitGroup
	var once sync.Once
	var first error
	for i := 0; i < workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for {
				select {
				case <-ctx.Done():
					return
				case job, ok := <-queue:
					if !ok {
						return
					}
					if ctx.Err() != nil {
						return
					}
					if err := fn(ctx, job); err != nil {
						once.Do(func() { first = err; cancel() })
						return
					}
				}
			}
		}()
	}
produce:
	for _, job := range jobs {
		select {
		case <-ctx.Done():
			break produce
		case queue <- job:
		}
	}
	close(queue)
	wg.Wait()
	if first != nil {
		return first
	}
	return parent.Err()
}

Before you close the laptop

Choose the slowest possible peer in your design. The user presses Stop. Trace the next events until every owned goroutine has exited. If your explanation includes “eventually the peer replies”, the stop path is unfinished.

You send a heartbeat to a node. Nothing returns. Perhaps the process crashed. Perhaps the machine is asleep. Perhaps your own network is broken. Perhaps the response is waiting behind a large transfer. The timeout does not tell you which story is true.

A failure detector is a decision mechanism built from uncertain observations. Its job is to help the system act despite that uncertainty. Its job is not to transform silence into proof of death.

What you will learn

You will model expiring health observations, separate suspicion from data loss, introduce deterministic time in tests, and choose a repair trigger that balances urgency against false positives. You will also distinguish liveness evidence from storage evidence.

A node that answers a health request proves that a particular endpoint responded at a particular time. It does not prove that every advertised shard is intact. A node can have a healthy process and a damaged disk. Conversely, a node that cannot currently answer may still have perfectly good shards that become useful when connectivity returns.

Observations have an age

Store when you last successfully contacted a node, what you observed and how long that observation is considered useful. Do not let a green status remain green forever because a successful response happened last week.

Use a monotonic elapsed-time basis within a running process. Wall-clock adjustments should not suddenly make a recent heartbeat appear years old or in the future. Across restarts, monotonic timestamps do not transfer as ordinary durable values. Start with unknown health after a restart and probe again, or use a carefully defined persisted wall-time observation with conservative expiry.

A simple detector can have unknown, alive, suspect and unavailable states. After one missed observation, mark suspect. After a longer threshold or several failed probes, treat the node as unavailable for current placement. A later successful probe can restore liveness. These are operational states, not permanent facts about the hardware.

The precise thresholds depend on your environment. A personal network with sleeping laptops behaves differently from a data centre. Make the policy configurable and test its transitions with a fake clock rather than waiting through real minutes.

Suspicion has two costs

An aggressive detector reacts quickly to real failures, reducing the time data remains under-protected. It also makes more false suspicions, potentially triggering needless repair traffic and consuming spare capacity.

A slow detector avoids some unnecessary work but leaves a longer exposure window after an actual loss. If another shard disappears before repair completes, recovery may become impossible. The choice is an engineering trade-off, not a hunt for the one universally correct timeout.

You can separate read routing from repair. Stop waiting on a slow node quickly during an interactive restore, while allowing a longer suspicion window before creating a replacement shard. That keeps the user experience responsive without treating every transient delay as a reason to move data.

Health is not an inventory

A heartbeat can include node identity, protocol version, capacity hints and recent load. Integrity requires a different kind of evidence: a valid object read, a verified inventory scan, or a more sophisticated proof scheme with stated assumptions.

An inventory also ages. A node may advertise a shard and lose it immediately afterwards. Store observations with timestamps and source identity. Use them to choose candidates, then verify bytes on retrieval. For repair, prefer recent verified evidence while accepting that no observation makes future loss impossible.

Avoid loading an entire enormous inventory into every heartbeat. Separate a small liveness signal from paginated or incremental inventory exchange. At personal-network scale a full inventory scan may be acceptable, but measure its disk and bandwidth cost and state the intended scale.

Give time a testable boundary

Health transition logic can be a pure function of the current state, an observation and a time value. The scheduler that performs probes uses real timers; the state reducer does not need to sleep.

This separation produces readable tests: at time zero the node is alive; advance to just before the threshold and it remains alive; advance beyond the threshold and it becomes suspect; deliver a successful observation and it returns to alive. Test boundaries precisely, including whether a threshold is inclusive.

Do not fake time everywhere simply because a clock interface exists. Use the seam where elapsed time determines state. Socket timeouts still need a small number of bounded integration tests to demonstrate actual I/O interruption.

Build contract

Implement an expiring health model with explicit unknown and unavailable states. Probe nodes under a bounded concurrency budget. Preserve the distinction between liveness, claimed inventory and verified object availability. Placement avoids currently unsuitable nodes; restore tries alternative candidates; no observation directly authorises deleting old data.

Add a health display or CLI report that includes observation age. “Last response 40 seconds ago; currently suspect” is more informative than a timeless red dot. The course's repair policy will consume this state in Chapter 18.

Acceptance scenarios

1. Advance a fake clock across each threshold. States change at the documented boundaries.
2. Deliver a successful probe after unavailability. The node can rejoin without losing its identity.
3. Restart the detector. Stale in-memory health is not presented as current evidence.
4. Make the process answer health while returning corrupt shard bytes. Health and integrity reports remain distinct.
5. Delay all probes simultaneously. The detector respects its concurrency budget and does not create an unbounded queue.
6. Mark a node unavailable. Its old placement record is not destructively erased merely because it is currently unreachable.

package workshop

import (
	"errors"
	"time"
)

// Health is a pure age classifier for observations within one clock domain.
// A restart should discard process-local monotonic observations and set known=false.
func Health(now, last time.Time, known bool, suspect, unavailable time.Duration) (string, error) {
	if suspect <= 0 || unavailable <= suspect {
		return "", errors.New("invalid thresholds")
	}
	if !known {
		return "unknown", nil
	}
	age := now.Sub(last)
	if age < 0 {
		return "", errors.New("observation is in the future")
	}
	if age >= unavailable {
		return "unavailable", nil
	}
	if age >= suspect {
		return "suspect", nil
	}
	return "alive", nil
}

Before you close the laptop

A node misses three heartbeats and then returns with all its data. Describe which decisions your system made while it was absent and why each was safe to reverse. You are preparing to build a network that survives ordinary uncertainty instead of panicking at it.

Your client currently has three addresses in a configuration file. It knows exactly where to begin. This is sometimes dismissed as insufficiently decentralised, as though a system becomes better whenever an address is harder to find.

An explicit peer list is a reasonable bootstrap for a small personal network. The interesting questions are what happens when addresses change, nodes join, views disagree or a seed disappears. We will make those questions visible before considering a distributed hash table.

What you will learn

You will separate bootstrap, discovery, membership and admission; exchange peer candidates safely; reason about gossip convergence; and explain what a DHT adds without confusing routing with storage durability.

Bootstrap gives a new participant its first contacts. Discovery finds possible contacts. Membership is the local view of participating nodes. Admission decides whether a candidate is authorised for a role. A discovered address should not automatically become a trusted storage provider.

Identity survives an address change

A peer record should contain a stable node identity, one or more contact endpoints, a protocol version and enough freshness information to compare updates under your chosen scheme. Authentication binds the record to the identity or to a trusted admission authority.

A monotonically increasing incarnation or record sequence can help distinguish a node's new advertisement from its old one, but only if its ownership and persistence are defined. A restart that resets the counter can make new records look stale. A node with a new disk and lost identity is a replacement node, not proof that the previous node's stored objects still exist.

For a cooperative tailnet, configuration can explicitly bind identities to allowed endpoints and failure-domain labels. For a more dynamic system, signed advertisements can carry candidates. In both cases, verify the endpoint and node identity during contact. A hostname is a route to a claim, not the claim's proof.

Tailscale solves a lower layer

Tailscale provides a private network built around encrypted connectivity and device access controls. It can make personal machines reachable without you implementing general internet NAT traversal. Its official overview describes the connectivity model.

It does not choose where your shards belong, authenticate owner manifests, decide how many replicas are healthy, or repair your application metadata. Keep those responsibilities in your storage system. Also keep application admission explicit: being reachable inside a tailnet does not necessarily mean every device should be allowed to consume storage.

Your initial configuration can therefore list several seed nodes by tailnet address or configured name. Retain more than one seed so a single unavailable machine does not prevent joining an otherwise healthy network. A recovery pack should include enough bootstrap information to find the network again.

Gossip spreads observations, not certainty

In a basic gossip-style membership exchange, a node periodically contacts a few peers, shares bounded records and merges newer valid records into its local view. Repeated exchanges can spread information through an eventually connected graph.

Different nodes may temporarily know different peer sets. That is expected. It becomes dangerous only when code assumes that its local set is the complete current network. Placement, repair and health decisions must tolerate stale or incomplete views.

Convergence depends on assumptions: contact opportunities continue, the network eventually allows communication, accepted updates have a deterministic merge rule, and records are not lost faster than they spread. “Eventually consistent” is not a spell that removes those requirements.

Bound the exchange. Limit record count, endpoint count, record size and rate of newly learned candidates. Otherwise, an admitted but broken peer can turn discovery into an unbounded memory or connection workload. Never dial arbitrary schemes or addresses from an unvalidated advertisement. In this personal-network design, endpoint policy can confine traffic to explicitly allowed tailnet or test addresses.

What a DHT would change

A distributed hash table maps keys through a distributed routing structure so a participant can find responsible nodes without knowing every node. Kademlia, for example, uses an XOR distance between identifiers and structured routing buckets. Its original paper is a useful later reading exercise.

A DHT can help locate providers or metadata at larger scale. It does not automatically keep the data, establish provider honesty, preserve a recovery key or solve catalogue conflicts. Routing to the node responsible for a key is different from proving that node holds the expected shard.

Do not implement a DHT simply to replace a peer list containing five machines. First state the limitation you need to remove: perhaps clients cannot hold complete membership, or provider lookup needs to scale beyond broadcast. That is a meaningful motivation for a V2 design.

Build contract

Support multiple explicit seed nodes and a bounded peer-candidate exchange. Keep discovered candidates separate from admitted members. Verify stable identity and protocol compatibility before using a node for storage. Merge updates deterministically and preserve useful peers when one seed is offline.

Implement the small version fully. The DHT discussion is an extension design, not a requirement to write a complete internet-scale routing layer in this chapter. You should finish with a reliable personal-network membership model and a clear explanation of its scale limits.

Acceptance scenarios

1. Start a node knowing only one of several seeds. After exchanges, it learns other admitted participants.
2. Make the first seed unavailable. A second seed still enables bootstrap.
3. Advertise the same identity at a newer endpoint according to your update policy. Membership changes without creating a duplicate independent node.
4. Advertise an unauthorised identity or disallowed endpoint. Discovery records it only as rejected or pending, never as trusted storage capacity.
5. Feed records in different orders. The same valid set converges to the same membership view under the merge rule.
6. Partition the membership graph and reconnect it. Valid updates spread again without erasing all peers merely because they were temporarily unseen.

Before you close the laptop

Explain what would continue working if the first seed disappeared after all nodes had joined. Then explain what a brand-new client with only that dead seed would need. Decentralisation is best understood by tracing dependencies, one concrete operation at a time.

A request times out, so you retry. The retry times out, so you retry faster. Other clients make the same decision. The overloaded node now receives more work precisely because it was struggling to handle the original work.

Retries are necessary in a network where responses can disappear. They are also a feedback mechanism. Designed poorly, they turn a brief delay into a prolonged outage.

What you will learn

You will separate retryable failures from permanent ones, preserve operation identity, use bounded exponential backoff with jitter, and reason about side effects beyond object bytes. The emphasis is now on your policy and its evidence. There is no single retry loop to copy into every layer.

The immutable store has already done some difficult work for you. Repeating a PUT for the same content-addressed object can converge on one stored object. That property is idempotency at the object layer: repetition does not change the intended logical result after the first successful effect.

Idempotent bytes do not imply idempotent everything

Suppose a duplicate PUT does not create another object but increments “bytes supplied” twice in a credit ledger. Storage is idempotent; accounting is not. Or suppose every attempt emits a new version record, creating ten apparent backups from one logical upload. The repeated byte write may be harmless while its surrounding workflow changes meaning.

Give a logical upload a durable operation or version identity before performing remote effects. Attempts can have separate trace identifiers, but they refer to the same logical work. Reconcile acknowledged shards and manifest publication against that stable identity.

A deduplication table also needs a lifetime. If it forgets an operation while a delayed retry can still arrive, duplicate effects become possible again. Immutable content identity avoids some of that problem for stored bytes; mutable charges, deletions and name updates need their own durable semantics.

Classify before repeating

A temporary connection failure, a busy response and an expired observation may justify another attempt. An invalid signature, unsupported protocol version, malformed manifest or deterministic content mismatch usually requires correction, not repetition against the same peer with the same input.

A full node may be a placement problem rather than a reason to hammer that node again. A corrupt response may justify another provider and a recorded integrity failure. A cancelled parent operation should stop retries entirely.

Classification should use stable error categories. A string containing “timeout” is not a reliable policy boundary. Preserve transport and application causes so the retry owner can make a deliberate choice.

One budget across the attempt tree

A top-level upload retries three times. Each stripe operation retries three times. Each peer request retries three times. Your nominal “three retries” can expand into a much larger number of network attempts.

Choose where retry authority lives. Low-level transport helpers can expose enough information for the operation coordinator to decide. If a lower layer does retry, it must consume the same overall deadline and attempt budget or have a clearly bounded sub-budget. Cancellation propagates through all attempts.

Backoff increases the interval between attempts, commonly up to a cap. Jitter randomises the exact delay so many clients do not synchronise into repeated bursts. Seed or inject the delay source in tests so a failure schedule is reproducible. Randomness is useful in production policy and inconvenient when a test cannot explain why it failed.

A reasonable course policy might have a total operation deadline, a maximum attempt count and full jitter within a capped exponential window. The exact values are yours to justify. State how a caller can distinguish exhausted budget from a permanent validation failure.

Retry the same encryption operation carefully

A logical upload should reuse its already-created ciphertext objects when retransmitting them. Re-running encryption with a fresh nonce produces different ciphertext and object identifiers, which can leave additional unreferenced objects and complicate reconciliation.

Never solve this by reusing a nonce to encrypt changed plaintext. Preserve the staged ciphertext and its manifest identity for a retry of the same operation. A genuinely new version gets a new encryption context and fresh nonces according to the key policy.

This is where a durable upload journal starts to earn its place. It can remember the planned immutable objects, acknowledged placements and publication status across a client restart. We will build that explicitly in Chapter 20.

Build contract

Design one retry policy for the client operation boundary. It must classify errors, honour parent cancellation, enforce a total budget and avoid synchronized retry bursts. Retries preserve immutable object and logical version identities. Duplicate delivery does not inflate local quota or create extra logical versions.

Write a short decision table covering at least timeout after possible commit, busy node, full node, invalid signature, corrupt shard response and user cancellation. The implementation should follow that table, and the tests should challenge its most expensive branch.

Acceptance scenarios

1. Drop a successful PUT response. The retry converges on the same object and logical version.
2. Return a permanent validation error. The client does not spend the retry budget repeating it.
3. Return busy for several attempts, then success. Attempts obey the configured total budget and delay bounds.
4. Cancel during backoff. The operation exits promptly without starting the next attempt.
5. Run many clients with the same initial failure. Their jittered retry times are distributed within the policy's window rather than aligned to one instant.
6. Nest operations that can fail. The total attempt count remains within the budget you documented, instead of multiplying invisibly across layers.
7. Retransmit staged encrypted objects. Their identifiers remain unchanged; a new file version uses fresh encryption material as required.

Before you close the laptop

A peer committed the object, your client retried twice, and the third response finally arrived. How many objects, versions and accounting events should exist? Explain each answer independently. If the answer depends on how many responses got lost, the effect boundary needs another look.

Your network can survive one missing shard. That is a window of opportunity, not a permanent achievement. If you leave the stripe with only its two required pieces and another one disappears, the file becomes unrecoverable.

Repair uses the remaining information to restore spare redundancy before the next loss. It is the difference between a system that tolerates one incident in its lifetime and a system that can repeatedly return to a protected state.

This is the first chapter where you should propose the architecture before opening the worked design. You now know enough to decide who observes the deficit, who authorises repair, which bytes move and what happens when two workers notice the same problem.

What you will learn

You will design a reconciliation loop, distinguish safety from liveness, make duplicate repair attempts harmless, and budget repair traffic. You will also explain why changing desired placement is not evidence that redundancy has been restored.

A safety property prevents a bad thing: repair must not change the expected content of an immutable shard. A liveness property promises progress under stated conditions: when sufficient valid source shards and eligible destination capacity remain reachable, repair eventually restores the target. The second statement needs assumptions about scheduling, connectivity and resources.

Reconcile desired and observed state

The desired state is a protection policy: for a 2+1 stripe, three distinct indexes on eligible independent failure domains. The observed state is a set of recent, sufficiently verified placements. The deficit is the difference between them.

Observation is imperfect. A node may be temporarily unreachable or may have lost a shard since its last inventory. Repair therefore acts on a policy threshold rather than a proof that old data is gone. Keep old valid placement hints until they expire or are superseded by evidence; creating an additional copy is safer than prematurely deleting the last recoverable one.

For the baseline, a repair cycle may enumerate signed repair descriptors and recent inventories, select a bounded set of deficits, fetch k valid distinct shard indexes, reconstruct the missing ciphertext shard, verify its expected digest, upload it to an eligible destination, and only then record the new placement observation.

The owner key is unnecessary because the repair descriptor already authenticates the expected ciphertext structure. The worker may reproduce existing authorised bytes; it may not invent a new file version or alter the expected digest.

Duplicate repair is normal

Nodes A and B both notice the same missing index. Both begin reconstruction. This is plausible in a decentralised system with delayed observations. Your design must be correct even if an optimisation intended to avoid the duplicate fails.

A deterministic preferred repair coordinator can reduce repeated work. A lease or task claim can help too, but introduces its own expiry and authority rules. Neither should become the only thing preventing corruption. Content-addressed immutable PUTs make duplicate publication of the same shard safe. Destination quota must count the object once. Placement records must deduplicate equivalent observations.

Two workers may choose different replacement nodes, leaving an extra copy. That costs resources but does not have to violate safety. Conservative cleanup can come later after the system proves enough independent valid copies exist. The initial core should prefer retained excess ciphertext over an eager deletion race.

Do not repair from a lie

A source node claims it has shard 0. The worker fetches it and receives corrupted bytes. That candidate does not count toward k. Repair must apply the same verification discipline as restore, using the expected digests from the authenticated descriptor.

A reconstructed shard should also match its expected digest before publication. If it does not, stop and report an integrity or metadata problem. Never “repair” the manifest by changing its expected digest to match whatever bytes happened to emerge. That would replace evidence with convenience.

There may be fewer than k reachable valid indexes. In that case, the current repair attempt cannot recover the stripe. Record the deficit and the reason, keep trying under a bounded policy, and distinguish current unavailability from proven permanent loss. No algorithm can manufacture missing independent information from two copies of the same index.

Repair competes with ordinary use

After a machine disappears, many stripes may need repair simultaneously. If repair saturates every disk and network link, user restores become slower and other nodes may start timing out. An overly enthusiastic recovery system can worsen the incident it is trying to resolve.

Give repair its own worker limit, bandwidth or byte budget, and scheduling priority. Prefer the most vulnerable stripes: those with exactly k known valid indexes have no spare margin. Pause or slow low-priority rebalancing when the network is under pressure. Use jitter so every node does not begin a full inventory scan at the same instant.

Estimate repair amplification. To recreate one missing shard, a simple decoder may read k shards and write one. For a 2+1 stripe of shard size S, that is roughly 2S read plus S written, before metadata and protocol costs. General codes and optimized repair methods change the details, but rebuilding is not free merely because the lost piece was small.

Build contract

The network must restore the configured redundancy after a node becomes unavailable, provided sufficient valid source shards and independent destination capacity remain reachable. Repair must work without user decryption keys, tolerate concurrent attempts, respect resource budgets and advertise only acknowledged replacements.

Write your design first: state owner, trigger, candidate selection, verification, commit order and retry behaviour. Include the case where the missing node returns during repair. Then implement the smallest loop that satisfies your design. Do not add destructive garbage collection as part of this milestone.

Acceptance scenarios

1. Upload a fully protected file, stop one node and introduce an empty eligible replacement. Repair restores the missing indexes. Then stop a different original node and restore successfully.
2. Start two repair workers with the same stale observations. Both may attempt work; the final content and quota remain correct.
3. Corrupt a source shard. Repair excludes it and succeeds only if k other valid indexes remain.
4. Fail the destination upload before its durable acknowledgement. The new placement is not advertised as healthy.
5. Return the original node while repair is in progress. Extra valid copies are tolerated without destructive conflict.
6. Create many deficits. Repair stays within its budget while an ordinary restore still receives service.

Before you close the laptop

You have crossed a significant boundary. The network can now recover some protection without a person telling it which file to copy. Explain the assumptions under which it eventually succeeds, and the conditions under which it must honestly stop short.

Your laptop and desktop both know version V0 of a document. The network splits. You edit the document on the laptop, creating V1. Somebody edits it on the desktop, creating V2. Both writes are locally valid. When the machines reconnect, which version is the latest?

“Whichever has the newer timestamp” is an answer, but it is a conflict policy with consequences. It is not a discovery of an objective global truth. Clocks can disagree, edits can be concurrent, and a later timestamp does not make the other person's work disposable.

What you will learn

You will reason about partitions, immutable version sets, causal relationships and explicit conflict policies. You will distinguish quorum intersection from a complete consistency protocol and state the limits of the baseline network without hiding behind the word “eventual”.

A partition prevents some participants from communicating with others. The affected nodes may continue running normally. Each side can serve reads and accept operations that do not require the other side, depending on your policy. The difficulty comes when an operation promises a single shared answer that both sides cannot coordinate.

Preserve versions before choosing a winner

Our core stores immutable versions. A new version references one or more parent version IDs. If V1 and V2 both reference V0 and neither descends from the other, they are concurrent branches. Preserve both.

A catalogue represented as a set of authenticated immutable records can merge by set union. Union is associative, commutative and idempotent: grouping, arrival order and repeated delivery do not change the final set. Under eventual communication and continued exchange, peers can converge on the same known records.

That does not make a mutable name resolve to one universally correct version. The set can converge while the application still needs to display a conflict or apply a documented policy. Separating those problems prevents an innocent synchronization routine from silently making a destructive editorial decision.

Causality is more useful than a wall clock

If V2 explicitly names V1 as its parent, the metadata records that V2 was made with knowledge of V1. A larger wall-clock timestamp alone does not establish that relationship.

Lamport's work on time and ordering in distributed systems formalizes the distinction between causal ordering and physical time. For this course, parent links are a concrete starting point. A version graph can answer whether one known version descends from another. Vector clocks or other causal summaries can compress certain relationships in more advanced designs, with their own membership and growth trade-offs.

Keep timestamps for display and human context. If you choose last-writer-wins, explain what “last” means, how ties are broken, and what information is retained. A deterministic winner can make replicas agree while still discarding a user's intended edit. Convergence and good product behaviour are separate goals.

What a partition forces you to choose

Suppose the API promises a linearizable single latest value: each completed operation must fit a single order consistent with real-time precedence. During a partition, both sides cannot always keep answering every request successfully while preserving that guarantee.

The formal consistency/availability trade-off concerns specific definitions. It is not “choose any two letters forever”, and it does not mean that a system with partitions must stop doing all useful work. Immutable object PUTs, reads of known versions and local operation journaling can remain useful even when a globally coordinated mutable name cannot be updated.

The original Gilbert and Lynch paper is further reading for the formal model. In your design, state the guarantee per operation. “The system is eventually consistent” is too broad if some operations are immutable writes, some are cached health observations and some are attempts to update a latest pointer.

Quorums need a protocol around the arithmetic

You may encounter the rule R + W > N: read and write sets intersect when chosen from the same fixed replica set of size N. Intersection is useful. It does not by itself specify how versions are ordered, how concurrent writers behave, how membership changes, or how failed partial writes are handled.

Erasure coding adds another distinction: k is the number of distinct valid shard indexes needed to reconstruct one stripe. That threshold is not automatically a quorum for a mutable metadata register. A 2+1 decoder and a three-replica consensus group may both involve three machines while solving different problems.

For the baseline, choose an explicit limited model: immutable versions can be accepted and exchanged during partitions; conflicts are retained; a local “preferred version” is a display decision, not a globally linearizable register. If V2 needs a strongly consistent latest pointer, design or adopt a suitable coordination protocol and accept its availability and membership requirements.

Build contract

Represent version parent relationships and preserve concurrent branches. Merge authenticated immutable catalogue records deterministically. Detect and expose multiple heads for a file identity. Document which operations remain available during a partition and what a latest-version read means.

No longer begin with the worked design. Write two competing policies, choose one, and give a concrete user-visible example of its trade-off. Then implement the chosen policy with a test history that includes concurrent writes and reconnection.

Acceptance scenarios

1. Partition two clients that share V0. Create V1 and V2 independently. After exchange, both versions survive and are identified as concurrent heads.
2. Create V3 with both V1 and V2 as parents after a deliberate merge. The graph recognises the merged head while retaining history.
3. Deliver duplicate records and every ordering of a small record set. The final known set and head calculation agree.
4. Skew one device's wall clock far ahead. It cannot silently erase another branch merely by presenting a larger timestamp.
5. Lose a response after version publication. Retrying the same version does not create another branch.
6. State how a temporarily stale reader is informed, or document that it may return only the locally known set without claiming completeness.

Before you close the laptop

Describe a history in which every stored byte is intact, every signature is valid, and two correct machines still return different answers. If you can do that without blaming a bug, the consistency discussion has become practical.

Your upload has three stripes. Two are fully placed, the third is partially placed, and the manifest has not been published. The client crashes. After restart, it has no idea which ciphertext objects it created or where they went.

The remote nodes may have done everything correctly. The client has lost the story that connected their effects into one operation. We need to persist intent and progress at the workflow level, not just at the object level.

What you will learn

You will design an upload journal, define restart states, reconcile uncertain remote effects and separate safe retention from dangerous cleanup. You will also learn to use an ordinary local transaction where it helps without pretending it spans the network.

A journal need not be an elaborate database at first. It can be an append-only, checksummed sequence of bounded records with a recovery rule for a torn tail, or a local transactional database with a documented durability configuration. The important property is that a restart can reconstruct the last committed workflow state and recognise incomplete progress.

Persist intent before irreversible assumptions

Allocate the logical operation and version identity. Record the source identity or immutable input snapshot policy, encryption context, staged ciphertext object identities and intended manifest before relying on remote effects that would be hard to rediscover.

A simple state model might include prepared, uploading, ready_to_publish, published and complete. The labels are less important than the transitions. Each transition must say what durable evidence exists and what a restart should do next.

For example, published should mean the required manifest publication acknowledgements were recorded durably, or that a reconciliation step has independently established them. It should not mean “we sent a request that probably worked”. The remote response could have been lost before the local journal recorded it.

Reconcile instead of guessing

After restart, inspect the journal, verify staged objects and query relevant nodes. An object may exist remotely even if the journal lacks its acknowledgement. Because object identities are immutable, the client can verify or safely repeat the PUT. A manifest may already be published under the stable version identity. Find it and compare the expected authenticated bytes before creating anything new.

Do not simply restart encryption from the current source file. The file may have changed, and fresh nonces produce different object identities. An operation should refer to a stable input snapshot or to staged ciphertext that preserves the intended version. If the input can no longer be reproduced, fail or abandon that operation explicitly and create a new version under a new identity.

This is where source-folder backup semantics become subtle. Reading a file while another program edits it can produce an inconsistent byte sequence. Detect changes using appropriate file metadata and final checks, retry from a stable source, or use a filesystem snapshot where available. A collection of individually stable files is still not necessarily a cross-file transactional snapshot.

A database is a local tool

A local database can atomically update the operation state, acknowledgement set and quota metadata. That is valuable. It cannot atomically commit a shard on a remote node in the same ordinary transaction.

Keep network calls outside long-held local write transactions. Persist intent, perform the remote action, then record the result in another local transaction. The gap between remote effect and local result remains; your idempotent reconciliation handles it.

If using an append-only journal, define record length limits, checksums, sequence numbers and the policy for an incomplete final record. A corrupted middle record is not equivalent to a harmless torn tail. Silently skipping arbitrary corruption can invent a workflow history that never existed.

The same discipline applies to metadata snapshots and compaction. Do not discard the old recoverable representation until the new one is durably published and verified. Maintenance code is part of the storage system's correctness, not an exemption from it.

Deletion is harder than retention

An abandoned upload can leave unreferenced ciphertext objects. Keeping them costs space. Deleting them too early can destroy work whose manifest publication was merely delayed or temporarily unseen.

For the course core, retain objects conservatively and report their storage cost. Design garbage collection as a later protocol with a reference graph, grace periods, handling for offline clients and journals, and protection against stale catalogues. A mark-and-sweep algorithm is only as trustworthy as the roots and snapshot it marks from.

A tombstone is an immutable record of a deletion decision. It does not by itself prove that every old version can be physically erased. Offline peers may still carry older state; retention and recovery policies may require history. If you add pruning, specify who authorises it and how a returning peer learns that resurrection is forbidden.

Build contract

Persist enough upload state to resume or explicitly abandon an interrupted logical operation without inventing extra versions or changing its ciphertext identity. Restart recovery verifies local journal integrity, reconciles remote objects and publication, and reports remaining work accurately.

Write a crash-state table before implementation. Cover every transition and the gap between each remote acknowledgement and its local recording. Choose a conservative orphan policy. Automatic destructive cleanup is outside the baseline milestone unless you separately prove its safety conditions.

Acceptance scenarios

1. Crash after journaling intent but before any upload. Restart resumes the same logical operation.
2. Crash after a remote shard commit but before recording its acknowledgement. Restart discovers or safely repeats the same object.
3. Crash after manifest publication but before local completion. Restart recognises the existing version rather than creating another.
4. Change the source file after a crash. The resumed operation uses its preserved snapshot/staged bytes or reports that it cannot resume; it does not silently mix versions.
5. Truncate the final journal record. Recovery follows the documented torn-tail policy while preserving prior committed records.
6. Corrupt a middle journal record. Recovery reports corruption instead of guessing a convenient state.
7. Leave unreferenced objects after an abandoned operation. The system reports retained space and does not delete potentially live data based on one stale catalogue.

Before you close the laptop

Choose any line in your upload coordinator and imagine the process stops there. Can you describe what the next invocation does using only durable information? You do not need to make every outcome successful. You do need to make every outcome explainable.

A test uploads a file and downloads it. Green. A second test starts three nodes and stops one. Green. A third test adds a random sleep and sometimes fails on Thursdays.

The third test may be exploring something real, but it is not yet a useful instrument. A failure you cannot reproduce is a clue. A recorded history with a stable seed and a violated invariant is a tool.

You now have enough moving parts to test the system as a system. This chapter asks you to build evidence, not another feature.

What you will learn

You will combine contract tests, deterministic fault injection and real-process integration tests; distinguish crash, omission and corruption failures; and use race detection and fuzzing for the problems they actually detect. You will also learn to state what a passing test does not prove.

Start with your invariant list. A successful restore returns exact bytes. An immutable object never changes identity. A fully protected 2+1 stripe has the required distinct indexes and failure-domain evidence at acknowledgement. A cancelled operation joins its owned workers. Repair never authenticates invented content. A restart never treats an incomplete journal transition as completed work.

Test at the cheapest honest boundary

Pure codec tests should use byte slices and table-driven cases. Framing tests should use hostile readers and writers. Health logic should use a controlled clock. Placement tests should use synthetic peer snapshots. None of these needs three real processes.

Integration tests earn their cost when the boundary matters: process restart, socket framing, real filesystem publication, signal cancellation, and independent node failure. Use temporary data directories and ephemeral test addresses. Ensure cleanup runs even when the test fails. A failed test that leaves a storage daemon running can contaminate the next run and make the evidence difficult to trust.

An in-memory fake is useful only if it obeys the same relevant contract as the real dependency. A fake store that cannot fail, shares mutable slices, or acknowledges before the point your real store promises can hide exactly the bug you need to find.

Faults should have names and positions

Create explicit fault points: before a write, after a durable commit, before a response, after N body bytes, during manifest publication, and between journal transitions. A test schedule can say, “On the first PUT for shard index 1, commit successfully and drop the response.” That is much clearer than “sleep a bit and kill something”.

Model different failure classes separately. A crash stops a process. An omission drops a message or response. A delay postpones progress. Corruption changes bytes. A malicious peer may send internally plausible but unauthorised records. A single random “failure” boolean rarely exercises all these contracts well.

Control interleavings with gates or channels. Wait until the test has observed the target stage, then release or interrupt it. Use timeouts only to bound the test and detect deadlock, not as the primary mechanism for reaching a state.

Record a history worth reading

For each operation, log a test-safe identity, stage, node, object or stripe index, result category and sequence number. Do not log private keys or user plaintext. A deterministic test can reconstruct ordering from its scheduler even when production logs use wall time.

When a test fails, preserve the seed, fault schedule and relevant temporary evidence. Minimise the history: can the failure still occur with one stripe, two operations or one dropped response? Smaller counterexamples improve both debugging and understanding.

A model-based test can compare your implementation with a simpler abstract state machine. For example, an immutable store model maps IDs to exact bytes and rejects conflicting reuse. Generate operation sequences and compare observable results. The model should omit implementation details so it can disagree meaningfully with the code.

Race detection and fuzzing have boundaries

Run concurrent tests with Go's race detector. It detects races exercised during the run. It does not prove every possible schedule is race-free, and it does not detect logical errors such as counting the same shard index twice under a perfectly good mutex.

Use Go's built-in fuzzing support for parsers and pure codecs. Useful properties include round-trip identity, bounded rejection of malformed frames, and reconstruction from every allowed loss set. Seed the corpus with empty inputs, boundary lengths, invalid indexes and truncated headers.

Do not fuzz a live personal data directory. Keep the target isolated and deterministic enough that a saved input reproduces the problem. Put allocation bounds in the code first so the fuzzer explores semantics rather than spending the run asking for impossible amounts of memory.

The capstone failure story

Create a multi-stripe synthetic file and an owner recovery pack. Start three nodes and one empty replacement candidate. Upload with the configured protection policy. Stop the client and discard its local catalogue. Stop one storage node. Bootstrap a fresh client, recover the manifest, restore and compare bytes.

Then let repair populate the replacement. Verify its actual shards. Stop another original node. Restore again. Finally, inject corruption into one candidate and show the system either recovers from sufficient remaining valid indexes or fails without publishing a corrupt destination.

This story crosses the recovery root, metadata, erasure coding, networking, repair and destination publication. It is a much stronger milestone than a screenshot with three green circles.

Build contract

Create a reproducible system test harness with explicit process lifecycle management, bounded waits, fault schedules and invariant checks. Include the capstone story and at least one lost-acknowledgement history, one concurrent repair history and one client restart during publication.

Write a verification report recording what ran, what passed and what remains outside the evidence. Three processes on one laptop test software behaviour; they do not simulate independent disks or a household power failure. Mocked sync errors test control flow; they do not certify hardware durability.

Acceptance scenarios

1. A failing seeded run can be repeated with the same seed and schedule.
2. Every spawned process and goroutine owned by the harness is cleaned up after both pass and failure.
3. The harness verifies restored bytes, distinct shard indexes and metadata recovery, not just command exit status.
4. Parser fuzz tests reject oversized and malformed input without panic or unbounded allocation.
5. Race-enabled tests cover concurrent upload, repair and cancellation.
6. A deliberately introduced bug—such as counting duplicate indexes—causes the intended invariant test to fail.

Before you close the laptop

Pick the strongest claim you make about the network. Point to the test that challenges it, then name one assumption that test cannot establish. This is how confidence becomes evidence instead of optimism.

A Mac button says “Back up now”. Underneath it are chunking, encryption, placement, retries, journaling and cancellation. The button should not need to know the order of every one of those operations.

A stable client boundary turns the storage engine into something several interfaces can use. The CLI becomes one client. A desktop app becomes another. A phone app can make different lifecycle choices while still relying on the same operation semantics.

The existing MeshVault project already has native Mac and iPhone source. This chapter uses it as a comparison and integration target. It is important to be precise: those apps speak the prototype's APIs, which differ from the course engine. An adapter is work you must design, not a missing filename you can fix by renaming your binary.

What you will learn

You will design an operation-oriented client API, keep UI state separate from durable workflow state, map the existing prototype's responsibilities to your course packages, and plan mobile upload behaviour around interruptions. You will also decide which features belong after the book's core milestone.

Expose operations, not internal steps

A client API can offer start upload, inspect operation, cancel operation, list versions and restore version. Starting an upload returns a stable operation identity. Progress reports facts the engine can substantiate: bytes read, shards acknowledged, metadata published and remaining protection deficits.

A percentage alone is ambiguous. “100% uploaded” might mean bytes left the client, one node acknowledged them, or the protection target was met. Use explicit states such as preparing, transferring, awaiting protection, publishing, complete, cancelled and failed. A UI can simplify the presentation without changing their meaning.

Progress observation should not own the operation. Closing a window or disconnecting a progress stream should not necessarily cancel a durable backup. Define that lifecycle separately. An explicit cancel action can signal the engine, while the engine's journal determines what survives restart.

Keep the control boundary private

A local management API can carry sensitive capabilities: unlocking an owner key, selecting source folders, initiating restores and changing network membership. Do not expose it on the same unauthenticated listener as peer storage traffic.

The comparison MeshVault prototype uses a loopback management API with a per-process bearer token, separate from its peer API. Its Mac interface starts or reconnects to the service. The course can adopt a similar separation, but your implementation must document token discovery, process ownership, origin policy and which operations are permitted.

A browser page is not automatically a trusted local client merely because it runs on the same machine. Your offline course website does not talk to the storage daemon. Keep the teaching material and the management control plane separate.

Understand the existing implementation honestly

The companion comparison source contains the existing MeshVault Go service, Swift app source and protocol notes. Its Go packages have concrete responsibilities: internal/store handles local objects and atomic publication; internal/vault handles cryptographic material; internal/network handles peer exchange; internal/syncer handles file records and backup work; internal/control exposes local management operations.

The prototype uses HTTP over the configured Tailscale network and full encrypted mirroring. Its repair loop exchanges opaque inventories and copies missing objects. It does not implement the course's 2+1 stripes, raw TCP frame format or selective placement policy.

Consequently, you have two sensible extension paths. Preserve the native interfaces and implement an adapter that exposes the operations they expect while delegating to your engine. Or define a new client API and update the native clients to use it. The first path reduces UI work but requires careful semantic mapping. The second gives a cleaner API at the cost of more client changes.

Do not claim compatibility based on matching method names. Compare request fields, authentication, progress semantics, recovery identities, cancellation and version behaviour. A “replica count” in the prototype cannot simply display your erasure-code shard count as if the protection guarantees were identical.

Folder backup is a workflow

Selecting a folder introduces discovery of local changes, source identity, stable reads and retention. File watchers can provide useful hints but should not be your only source of truth. Events can be coalesced or missed; a periodic reconciliation scan can discover changes that the event stream did not preserve.

Decide what “sync” means. The prototype performs continuous backup with explicit restore and retained history. It does not automatically apply every remote edit into the local folder. Bidirectional synchronization introduces conflict resolution, deletion propagation and overwrite policy. Those are additional product decisions, not a checkbox on the upload engine.

Use source identities to distinguish independent folders that happen to share a name. Treat symlinks, special files, metadata, permissions and changing files according to an explicit scope. A first backup tool can skip unsupported objects and report them; silent omission disguised as completeness is the problem.

A phone is an intermittently available client

An iPhone may lose connectivity, suspend the app, have limited Photos permission or need to download an original asset from iCloud. Background execution is scheduled by the operating system; it is not a promise that an arbitrary daemon can run continuously. Apple's Background Tasks documentation is the authoritative starting point for the current platform model.

Design a resumable upload lifecycle with durable asset identifiers and checkpoints. Encrypt on the phone before sending data to storage nodes. Keep the recovery key in appropriate owner-device secret storage. Treat Live Photo resources and videos as explicit asset components if preserving them is part of the product.

The phone should not become the only holder of the manifest needed to recover its own lost photos. The recovery-root reasoning from Chapter 9 applies just as strongly to a camera roll.

Build contract

Define and implement a stable CLI-facing engine boundary for start, status, cancel, list and restore. Document its state transitions and error categories. Then produce an adapter design for the existing Mac app, including the mismatch between mirrored copies and erasure-coded protection.

The required core milestone is the engine boundary and an executable CLI demonstration. Native Mac/iPhone integration is a post-core extension, so you finish with the knowledge to extend the system independently. The supplied native source gives you a concrete starting point, not a claim that your newly written engine is already connected to it.

Acceptance scenarios

1. Start an upload through the CLI boundary, observe status and cancel through the same public API.
2. Disconnect a progress observer. The operation follows the documented lifecycle rather than crashing or leaking a sender.
3. Restart the engine. Status is rebuilt from durable operation state.
4. Attempt a control operation without its required local credential. It is rejected.
5. Demonstrate how a fully protected 2+1 upload is presented differently from three complete replicas.
6. Write one integration contract test for the proposed native adapter using synthetic requests and responses.

Before you close the laptop

Describe the work required to add a phone client without mentioning the internal names of your codec functions. If you can speak in terms of authenticated operations, resumable work and durable results, the engine has a useful door.

You would like people who contribute reliable storage to earn the right to consume storage elsewhere. The first sketch is wonderfully simple: supplied bytes increase a balance, consumed bytes decrease it. Then one operator creates a hundred node identities, reports the same disk through all of them and asks for a hundred times the credit.

The counter was easy. The meaning of the number was the system.

This chapter is a design investigation rather than a requirement to invent a cryptocurrency. A small cooperative network may use a trusted local accounting authority or an explicit administrative policy. That can be a sensible choice if you name the trust it introduces.

What you will learn

You will define accounting units, separate claims from evidence, make ledger events idempotent, identify Sybil and collusion risks, and design a bounded experiment without presenting it as an abuse-resistant economic system.

Start by asking what you want to reward. Advertised free space? Bytes actually stored? Time those bytes remain retrievable? Independent failure domains? Successful repair service? Each measures something different and encourages different behaviour.

Define the unit before the balance

“Storage supplied” needs dimensions. A byte is capacity. A byte-hour includes time. An available verified byte-hour includes an observation policy. An independently hosted verified byte-hour includes a failure-domain trust assumption.

Suppose a node holds 10 GiB for 24 hours. That is 240 GiB-hours under a simple retained-capacity model. It does not prove every byte was retrievable throughout the interval. Sampling observations can support an estimate, but the ledger should not pretend sampled evidence is continuous certainty.

Decide how redundancy affects charges. A user's 1 GiB file encoded at roughly 1.5 times expansion consumes about 1.5 GiB of shard bytes before metadata and overhead. Three complete replicas consume about 3 GiB. Repair bandwidth and retained history consume additional resources. A fair policy must say which costs are charged and which are shared.

Claims, receipts and independent evidence

A node can sign “I stored shard X”. The signature establishes who made the claim. It does not prove that the node still has X, that X occupies unique physical space, or that the claimant is independent of the uploader.

An owner receipt after a successful retrieval is stronger evidence of a particular interaction. It may still be vulnerable to collusion: two identities controlled by one operator can manufacture activity. Periodic challenges can make some forms of dishonesty harder, but proof-of-retrievability and proof-of-storage systems have precise cryptographic assumptions and cost models. A casual random-byte challenge should not be marketed as a complete substitute.

For the personal network, choose a deliberately modest experiment: a trusted accounting component records unique, verified storage events and periodically sampled availability under configured node identities. Label the balance as an administrative estimate. That gives you a useful ledger exercise without pretending to solve open participation.

Events are easier to audit than mutable balances

Represent each charge or credit as an immutable event with a unique event identity, subject, resource, interval or operation, amount and evidence reference. Derive balances from the event set or maintain a materialized balance transactionally with it.

Duplicate delivery of the same event must not change the balance twice. Corrections should be explicit reversal or adjustment events with their own identities, not silent edits to history. This makes disagreement inspectable: two nodes can compare the events behind their totals.

A distributed event set still needs authority rules. Who may issue credits? Who may reverse them? Can an event be accepted while disconnected? If two authorities spend the same balance during a partition, what prevents overspending? The consistency chapter has followed you into accounting.

A single trusted ledger service simplifies these questions by centralising authority. It also becomes a dependency for fresh balance decisions. You can allow cached reads or bounded offline consumption, but those are policies with risk limits. Do not call the system fully decentralised merely because the storage bytes are distributed.

Admission controls the attack surface

A Sybil attack uses many identities controlled by one actor to gain influence or rewards. Cheap public keys do not imply independent participants. Reputation tied only to an easily replaced identity can be reset. Capacity claims tied only to software-reported disk size can be fabricated.

In a small network of known people, admission can be social and administrative. In an open network, you need stronger economic or verification mechanisms. The course does not require you to solve them all. It requires you to notice when a proposed reward scheme assumes away the incentives it creates.

Also consider denial of service through legitimate operations. A user may fill their allocation with tiny objects that cost much more in metadata and requests than their byte count suggests. Quotas may need object count, request rate, bandwidth and in-flight reservation limits as well as stored bytes.

Design contract

Write a two-page accounting proposal. Define the unit, authoritative issuer, evidence, event identity, duplicate policy, correction policy, partition behaviour and abuse assumptions. Include one worked example for a 1 GiB file under both 2+1 coding and three-way mirroring.

If you implement a ledger experiment, keep it local or explicitly trusted. Use unique event IDs and atomic insertion with balance updates. It should be useful for exploring semantics, while making no claim to be an open-network credit currency.

Acceptance scenarios

1. Apply the same event repeatedly. The balance changes once.
2. Reorder independent events. The final balance agrees under the chosen model.
3. Reverse an incorrect event. The audit trail preserves both the original and correction.
4. Create two identities on one physical node. Explain why they do or do not earn independent credit under your admission policy.
5. Partition two potential spenders. Demonstrate the chosen overspend prevention or bounded-risk policy.
6. Compare logical file size, encoded stored bytes, retained versions and repair traffic in your accounting example.

package workshop

import (
	"errors"
	"math"
)

type Event struct {
	ID    string
	Delta int64
}

// Balance demonstrates duplicate-safe event interpretation, not a trusted
// distributed currency. Issuer authority and persistence are separate tasks.
// Intermediate int64 overflow is rejected. Applications requiring order-independent
// totals even near numeric bounds should accumulate exactly and range-check at the end.
func Balance(events []Event) (int64, error) {
	seen := map[string]int64{}
	var total int64
	for _, event := range events {
		if event.ID == "" {
			return 0, errors.New("empty event ID")
		}
		if old, ok := seen[event.ID]; ok {
			if old != event.Delta {
				return 0, errors.New("conflicting event ID")
			}
			continue
		}
		d := event.Delta
		if d > 0 && total > math.MaxInt64-d || d < 0 && total < math.MinInt64-d {
			return 0, errors.New("balance overflow")
		}
		total += d
		seen[event.ID] = d
	}
	return total, nil
}

Before you close the laptop

You should now be able to look at a proposed feature and find its state ownership, persistence, trust, retry and consistency questions before writing the first function. That skill is the point of this book. One chapter remains, and it belongs mostly to you.

At the beginning, storage meant putting bytes in a file and hoping they were still there after restart. Now you have a vocabulary for the parts that hope was concealing: acknowledgement boundaries, immutable identities, recovery roots, failure domains, cancellation, partial effects, repair and conflicting histories.

You also have code that embodies your choices. Some choices will look sensible. Others will look like things you would do differently now. That is not a disappointing ending. It is evidence that your judgement has changed.

The final exercise is to design V2. There is no prescribed architecture to reproduce. There is a standard of explanation and evidence to meet.

What you should now be able to do

Given a proposed feature, identify its state owner and persistence boundary. Describe what happens if the operation is delivered twice, interrupted halfway through or attempted from two sides of a partition. Explain what evidence establishes integrity, authenticity, availability and freshness, without treating those properties as interchangeable.

In Go, you should be able to choose small package boundaries, pass capabilities explicitly, manage reader and slice ownership, preserve meaningful errors, bound concurrency, propagate cancellation and test failure paths deliberately. You should also know when a standard library or well-maintained protocol is a better tool than a custom mechanism.

This does not make you an expert in every distributed system. It gives you a practical foundation and a way to investigate the next unfamiliar problem. The measure is whether you can reason independently, not whether you remember every name in the glossary.

Audit the current system

Write a concise architecture record for the system you actually implemented. Include a diagram of the upload, restore and repair paths. Put trust boundaries and durable stores on the diagram. State how an owner recovers after losing their primary device.

Then write the guarantees. Be specific: “A fully protected version has three acknowledged ciphertext-shard indexes per stripe on three configured machine domains, plus three manifest copies at acknowledgement time” is assessable. “Highly resilient decentralized storage” is not.

List the limits with equal precision. Perhaps membership uses administratively configured seeds. Perhaps catalogues preserve concurrent versions but do not provide globally fresh latest-name reads. Perhaps the journal is durable only on the client disk, and orphan retention is unbounded. Perhaps repair reads too much data during full inventory scans. A limitation you can locate is a design input.

Choose one extension for a reason

Several extensions now have natural places to live. Content-defined chunking changes the chunker and manifest format, with effects on reuse and memory. General Reed–Solomon coding changes the codec and placement requirements. A phone client changes the client lifecycle and authentication boundary. A DHT changes discovery and provider lookup. Credits add an accounting authority and evidence model.

Choose one based on a real problem you can demonstrate. Do not redesign all of them at once. A focused V2 lets you compare before and after under the same workload and failure model.

Write the user-visible trigger and outcome. For example: “When the phone loses connectivity halfway through a video upload, reopening the app resumes the same logical version without retransmitting already acknowledged ciphertext.” That sentence implies durable operation identity, staged encryption, asset lifecycle and reconciliation. It is much more actionable than “add iOS”.

Compare two viable designs

For your chosen extension, propose at least two designs that could plausibly work. Compare state ownership, failure behaviour, complexity, migration and operational cost. Include what becomes harder, not just what becomes faster.

If you choose 4+2 coding instead of 2+1, compare shard count, minimum recovery fan-in, failure-domain requirements, small-file overhead and repair traffic. If you choose a strongly consistent latest pointer, compare a coordinator or consensus-backed service with the current conflict-preserving catalogue. If you choose native integration, compare adapting the existing MeshVault API with updating clients to a new operation API.

An architecture decision record should be readable six months later. Record the context, chosen decision, alternatives and consequences. Avoid writing a defence of a decision you made before considering the problem. The purpose is to make future revision informed, not embarrassing.

Migration is part of the feature

A new manifest format does not erase old backups. Decide how a new client reads old versions, how capabilities are negotiated, and whether old nodes can store opaque new objects without understanding them. If old clients cannot interpret the new format, return a clear unsupported-version result rather than a misleading corruption error.

A migration plan needs a rollback story. Perhaps new writes use the new format while old reads remain supported. Perhaps a background process re-encodes versions and verifies them before switching references. Never delete the last readable representation merely because a conversion command returned success once.

Preserve owner authority and recovery roots across key or format changes. Key rotation is not just changing a constant in the encryptor. Old data may need old keys or rewrapping, clients need a way to discover the correct key version, and revocation does not undo plaintext already learned by a compromised endpoint.

Build the smallest proof

Choose one vertical slice through the new behaviour. It should start at a public boundary and end in an observable result. Write the failure test before broadening the implementation.

For resumable phone-style uploads, you can begin with a command-line simulator of a suspended client. For general coding, begin with a pure codec contract and all tolerated loss combinations before involving six machines. For improved placement, begin with synthetic membership snapshots and a documented failure-domain model.

This is not avoiding the real product. It is isolating the claim that makes the product worthwhile. Once the proof holds, integrate it through the established boundaries and measure the costs you predicted.

Your final brief

Produce four artifacts in your own repository: an architecture diagram, a guarantee-and-limit table, a V2 decision record and a verification report for one implemented slice. Add a short roadmap with no more than three next milestones, each expressed as observable behaviour.

A useful self-assessment has four levels. Describe: you can name the mechanism. Explain: you can say why it works and where it fails. Demonstrate: you can show a test or trace supporting the claim. Extend: you can change the design while preserving or deliberately revising the guarantee. Aim for demonstrate on the core and extend on your chosen V2 slice.

The final acceptance conversation

Explain your system to an imaginary collaborator who has not read this book. They ask:

1. What exactly does a successful backup promise?
2. What must I keep outside the device I am backing up?
3. Which combinations of loss can this configuration recover from?
4. Can a storage operator read or forge my files?
5. What happens if an upload finishes but its response is lost?
6. What happens when two devices edit while disconnected?
7. How does repair avoid making the incident worse?
8. Where is the next bottleneck, and what evidence says so?

Answer from your implementation and tests. If one answer is “I have not implemented that guarantee”, say so and explain the current behaviour. Clear limits are part of ownership.

One last return to the photograph

A file disappears from a laptop. Somewhere else, enough information remains to bring it back. You can now trace that sentence through a real system: find the recovery root, authenticate the map, retrieve verified indexes, reconstruct ciphertext, decrypt, check the assembled file and publish it without destroying something else.

More importantly, when one of those steps fails, you have a way to investigate. You can ask which promise was made, what evidence survived and which next action is safe. That is a much more durable outcome than a tutorial's final screenshot.

The next chapter is whatever you decide to build.

The course is about building a storage system, so syntax should not become a second mystery. These notes collect the Go habits that repeatedly matter in the chapters. They are a companion to the language documentation, not a replacement for it.

Values, pointers and method sets

A struct value contains its fields. Assigning or passing it copies the value. A pointer refers to an existing value and can make shared mutation possible. Choose based on the type's meaning, size and ownership, not a rule that every object must be heap-shaped because it would have been a class elsewhere.

Methods can have value or pointer receivers. Receiver choice affects which method sets satisfy an interface. Be especially careful with types containing mutexes or other synchronization state: copying them after use can break the coordination you thought you had. If a type owns mutable shared state, make its non-copying usage clear.

Go interfaces describe behaviour and are satisfied implicitly. A function that needs to read bytes can accept io.Reader without knowing whether those bytes come from a file, socket, byte buffer or test fault injector. An interface belongs where that abstraction is useful; not every type needs an interface-shaped twin.

Slices are views over storage

A slice includes a pointer to an underlying array, a length and a capacity. Copying the slice does not copy the elements. Two slices can overlap the same array. Appending may reuse that array or allocate a new one depending on capacity, so apparent independence can change with input size.

For a chunk pipeline, define whether a slice is borrowed, transferred or copied. A synchronous consumer can borrow a buffer until it returns. An asynchronous consumer usually needs ownership that outlives the producer's next read. Tests should deliberately retain chunks and check that subsequent reads do not mutate earlier ones.

A nil slice and an empty non-nil slice often behave similarly for length and iteration. They can carry different semantic meaning in an API. The workshop erasure codec uses nil for a missing shard and an empty non-nil slice for a present zero-length shard. That distinction is deliberate.

Errors are values with consequences

Return errors to a caller that can decide what they mean. Use stable error identities or types where callers need branching, and add context without destroying the cause. errors.Is and errors.As let a caller inspect wrapped errors through the appropriate abstraction.

Do not log and return the same error at every layer. Choose the boundary that has enough context to explain the operation. A low-level store should not terminate the process; a command can choose an exit status. A background repair loop might record a classified failure and continue with other work.

An interface containing a typed nil pointer is not the same as a nil interface. This matters for errors and returned resources. Make absence explicit, and test the paths that return no value.

Defer closes scopes, not intentions

A deferred call runs when its surrounding function returns. It is useful for unlocks, cancellations and closing resources, but it does not know when an individual loop iteration is logically complete. Deferring thousands of file closes inside one long loop keeps those files open until the whole function returns.

Use a helper function for one iteration or close each resource explicitly when its lifetime ends. Check errors that affect the operation's promise. A deferred close whose error is ignored may be acceptable for some read-only resources; ignoring the final write or synchronization error is not acceptable when claiming durable storage.

Arguments to a deferred call are evaluated when the defer statement executes. That can be helpful, but it surprises people who expect a variable's future value to be looked up automatically. Keep cleanup code simple enough that the lifetime is evident.

Context, channels and synchronization

A context carries cancellation and deadlines across an operation. It should not become a bag of ordinary function parameters. Pass business inputs explicitly. A context's cancellation signal does not automatically interrupt arbitrary blocking I/O; the implementation has to connect it to the resource.

Channels are typed communication paths. Closing a channel says that no more values will be sent. It does not mean “please cancel every sender”. Receivers can observe closure after buffered values drain. The sender side, or a coordinator that knows all senders have ended, normally owns closure.

A mutex protects a relationship between shared values. A WaitGroup tracks completion. A channel hands work or ownership between stages. Choose the mechanism that expresses the relationship you need. The Go memory model is the reference for synchronization guarantees.

Tests that help you think

Use table-driven tests for families of cases with the same contract: lengths around a chunk boundary, each missing shard index, or each health threshold. Name subtests so a failure tells you which case matters. Give each filesystem test its own temporary directory.

Keep fakes faithful to the contract and simple enough to inspect. A fake clock controls time decisions. A tiny reader forces partial progress. A blocked peer exposes cancellation. A deliberately corrupting store tests integrity handling. These are more useful than mocking every private helper.

Run ordinary tests often while implementing. Use go test -race ./... for concurrent paths, and fuzz pure parsers and codecs when their invariants are clear. A passing race run covers the schedules exercised; it does not prove the whole system correct. The official testing package describes the current test and fuzz APIs.

A practical command shelf

These commands are for your programming project. None is needed to open the book.

gofmt -w .
go test ./...
go test -race ./...
go vet ./...
go test -run TestErasure -v ./...

Formatting makes code consistent without spending design energy on whitespace. Tests check behaviour you have specified. Race detection checks exercised unsynchronised accesses. Vet flags selected suspicious constructs. They complement code review and failure modelling; none replaces them.

When to add an abstraction

Add a boundary when you can name the decision it isolates. A codec should not know network addresses. A transport should not decide which file version wins. A UI should not reconstruct parity. These separations make failures and future changes local.

Delay abstractions whose only justification is that a larger application might need them someday. You can extract a useful interface from two real consumers later. It is harder to remove a framework that has already taught every package to depend on its assumptions.

For language details beyond these notes, start with the Go specification, A Tour of Go and the package documentation linked from each chapter. Effective Go remains useful for core idioms, but its own introduction notes that it does not cover all newer language and ecosystem features.

The book is complete without opening an editor. When you do want to build, the companion provides a small exercise pad and a separate comparison project. You can use the tests directly or adapt the behaviours to your own architecture.

What is included

The companion/workshop/student folder contains a Go module with deliberately unfinished functions and the tests they must satisfy. The companion/workshop/reference folder contains working solutions to those same focused mechanisms. They use the standard library and declare Go 1.24 or later. They are not a complete storage-network implementation.

The companion/meshvault-reference folder contains a source-only snapshot of the existing MeshVault prototype, including the Go service, Mac and iPhone sources, build scripts and protocol notes. It is a full-mirroring comparison implementation. Generated app binaries, build caches, local configuration and user data are excluded. Its own README states its toolchain requirements and known boundaries.

The workshop and prototype have different purposes. The workshop is a set of small, testable learning problems. The prototype lets you inspect a larger working arrangement and gives you native client source for later integration. Neither dictates the architecture of your course project.

Begin with one exercise

Download and extract storage-network-starter.zip into a new folder outside the course repository. The archive contains a Go module, the test file, unfinished exercise functions, a README and a .gitignore. Open a terminal in the extracted storage-network-starter directory and initialise your own Git repository there. Running all tests immediately will fail because the functions are intentionally unimplemented. Select the relevant group while you work.

The encryption tests also call the erasure codec. Complete Chapter 7's workshop before running that group. The store exercise needs Digest, a lowercase SHA-256 helper; implementing it early is a small preview of Chapter 6, not a hidden dependency on the full chunk pipeline.

The framing workshop deliberately uses a simpler payload grammar than Chapter 4's full JSON-header protocol: four big-endian length bytes followed by that many payload bytes, capped at 8 MiB. It isolates partial I/O, truncation and allocation limits. You still design the full request header and operation validation in your main project.

git init
git add .
git commit -m "Start storage-network exercises"

The starter has no remote and no connection to the course repository. Work in this extracted folder; the book and completed example remain unchanged. Add your own Git remote when you are ready.

Read the contracts before the tests

The store exercise assumes one Store instance owns a trusted private directory. Its mutex serializes writers within that instance. It does not provide multi-process locking, hostile-local-user isolation or universal filesystem semantics. Its Stage callback is a test seam; configure it before concurrent use. The injected-error tests exercise control flow around publication, not physical power-failure behaviour.

The worker exercise requires callbacks to observe their context and return. Go cannot forcibly stop an arbitrary function that ignores cancellation. The helper's promise is to cancel cooperative work and join every worker it starts.

The ledger exercise demonstrates immutable event identities and duplicate handling. It does not establish who may issue credit, verify real disk capacity or prevent collusion. Those are the design questions in Chapter 23.

These limits keep the examples understandable. If you adapt a mechanism into your network, carry its assumptions with it or strengthen the implementation before relying on a broader guarantee.

Hints before answers

Try the chapter's weakest hint first. If the shape of the algorithm remains unclear, open the pseudocode or worked design. For selected chapters, the last disclosure contains the exact companion reference source. You can inspect it offline in the page without opening another application.

Do not judge your solution by whether its line count resembles the reference. Compare the contract, failure behaviour and resource ownership. Two different designs can both be correct. Two visually similar designs can differ at the one error path that matters.

After solving an exercise, add one test the supplied suite did not include. A useful addition comes from a question about the contract, not a private implementation detail. For example, what happens when a reader returns bytes and a terminal error together? Which stage owns a buffer after cancellation? How does a retry treat a visible object whose previous durability check failed?

Run the reference deliberately

From the reference directory:

go test -race ./...
go vet ./...
go test -fuzz=FuzzFrames -fuzztime=5s

The first command includes the ordinary tests and race instrumentation. The second checks selected suspicious constructs. The third runs a short exploratory fuzz session beyond the saved seed cases. The website never runs these commands on your behalf when you open a chapter.

The verification report records checks performed for this edition. It distinguishes the workshop tests from the existing prototype and from the learner's future end-to-end network. A test suite for eight mechanisms is not evidence that an unwritten integration already works.

Use the comparison source as a reading exercise

The full comparison app is available separately as MeshVault source; it is not part of the starter download. In the course repository, the original application lives in Completed Example.

Open MeshVault's README and protocol notes. Trace one upload from the native client boundary into the Go service. Identify where encryption happens, when a record is published, and how repair discovers missing objects.

Then compare with your course implementation. Which guarantees are shared? Which differ because the prototype mirrors complete encrypted objects while your system codes stripes? Which native UI labels would need to change? These questions are more educational than copying the complete project into your own directory.

The snapshot preserves existing prototype source and documentation. It has not been converted into the course's erasure-coded engine, and the book does not claim that its iPhone background behaviour was newly verified on a physical device. Chapter 22 gives you a route to that work when you choose it.

Keep your own evidence beside your code

Create a short decision log and a verification file in your own repository. Record the contract at each milestone and the tests that challenge it. When the next chapter changes a guarantee, revise the record.

By the final chapter, that record should tell a coherent story: what you built, why it behaves as it does, what it has survived in tests and what remains to be designed. That is the beginning of engineering ownership.

Use this as a navigation aid. A definition is useful when it helps you ask a sharper question about your implementation.

Bytes, storage and recovery

Object. An immutable byte sequence with an identifier. A human filename is a separate reference to content or versions. Chapter 1.

Invariant. A property that must hold across every operation allowed by the system, such as never returning partial bytes as a successful immutable object. Chapter 1.

Atomic publication. A visibility transition in which readers see an old state or a complete new state, rather than an intermediate mixture. It does not automatically imply crash durability. Chapter 3.

Durability. Survival of an acknowledged change across the failures included in a stated storage model. The acknowledgement boundary and environment matter. Chapter 3.

Content addressing. Naming exact stored bytes using a digest derived from those bytes. Authenticity still depends on where the expected identifier came from. Chapter 6.

Chunk. A bounded segment of the input stream. Chunking alone provides no redundancy. Chapter 6.

Stripe. A group of related data and parity shards produced by one coding operation. Shards from different stripes or versions cannot be mixed. Chapter 7.

Shard index. A shard's position within its stripe. Multiple copies of one index do not provide multiple independent pieces of information. Chapter 7.

Erasure. A piece known to be missing or rejected. A decoder can recover specified erasure patterns when enough valid information remains. Unknown corruption is a different problem. Chapter 7.

Parity. Redundant information computed from data that allows certain missing pieces to be reconstructed. Repeating the same parity does not create a new independent equation. Chapter 7.

Failure domain. A set of components that can disappear together because of one event, such as a disk, machine or building failure. Chapter 8.

Recoverable. Enough valid distinct shard indexes are currently obtainable to reconstruct the data. This can be true while spare redundancy is missing. Chapter 8.

Fully protected. The configured redundancy and placement policy has been satisfied by the required acknowledgements or observations. The evidence has an age and a scope. Chapter 8.

Manifest. A structured description that binds file order, lengths, coding information and expected shard identities into a recoverable version. Chapter 9.

Recovery root. The independently retained information from which recovery begins: in this design, owner key material plus sufficient network bootstrap information. Chapter 9.

Communication and trust

Frame. A message boundary imposed by the application on a byte stream. TCP does not preserve the boundaries of application writes. Chapter 4.

Partial failure. A situation where some components or communication paths fail while others continue working. Different participants can observe different outcomes. Chapter 5.

Idempotency. Repetition preserves the intended logical effect after the first successful application. Idempotent object storage does not automatically make billing or version creation idempotent. Chapter 17.

AEAD. Authenticated encryption with associated data. It encrypts plaintext and authenticates ciphertext plus a chosen unencrypted context. Chapter 11.

Nonce. A per-encryption value with uniqueness requirements under a given key and construction. It need not be secret; reuse can be catastrophic for schemes such as GCM. Chapter 11.

Associated data. Context authenticated by AEAD but not encrypted, such as format, owner, version and stripe identity. Its byte encoding must be unambiguous. Chapter 11.

Signature. Evidence that exact signed bytes were authorised by possession of a corresponding private key, assuming the key and algorithm model. It does not prove freshness or truthfulness of a claim. Chapter 12.

Admission. The policy deciding whether an identity may join or perform a role. Discovering an address is not admission. Chapter 16.

Replay. Reuse of an earlier valid message or record. It may be harmless for immutable storage while remaining dangerous for mutable side effects or freshness assumptions. Chapter 12.

Sybil attack. Gaining influence through multiple identities controlled by one actor. Distinct keys are not evidence of independent machines or people. Chapter 23.

Concurrency and distributed state

Backpressure. Downstream resource limits cause upstream production to slow or wait, preventing unlimited queued work. Chapter 13.

Ownership. Responsibility for mutating state or ending a resource's lifetime. Garbage collection does not decide who closes a socket or may reuse a buffer. Chapter 2.

Cancellation. A signal requesting that work stop. Implementations must connect it to blocking operations and clean up owned resources. It does not undo completed remote effects. Chapter 14.

Failure detector. A policy that turns incomplete observations into operational suspicions or routing decisions. Silence is not proof that stored bytes are permanently gone. Chapter 15.

Gossip. Repeated bounded exchanges that spread information through a network under stated connectivity and merge assumptions. It does not make every local view instantly complete. Chapter 16.

Reconciliation. Comparing desired state with observed state and making safe, repeatable changes toward the target. Repair and restart recovery are both forms of reconciliation. Chapter 18.

Safety. A property excluding a bad outcome, such as publishing a shard under an incorrect digest. Chapter 18.

Liveness. A progress property under stated assumptions, such as eventually restoring redundancy when enough sources and capacity remain reachable. Chapter 18.

Partition. A communication failure separating participants that may each remain otherwise functional. Chapter 19.

Causality. A relationship in which one event can influence another, represented in the course by version parent links. A larger physical timestamp alone does not establish it. Chapter 19.

Linearizability. A consistency guarantee in which completed operations can be placed in one order consistent with real-time precedence and the object's sequential behaviour. Chapter 19.

Quorum intersection. Overlap between selected read and write sets under specified membership assumptions. Useful arithmetic that still needs a complete protocol around it. Chapter 19.

Journal. Durable records of intent and progress used to recover an interrupted workflow. It does not make remote effects part of a local atomic transaction. Chapter 20.

Tombstone. An explicit deletion record. Its existence does not alone establish that every older physical object can be safely erased. Chapter 20.

Fault injection. Deliberately causing a named failure at a controlled stage to test an invariant. Chapter 21.

Four questions that keep terms honest

For any guarantee, ask: what evidence establishes it, when was that evidence observed, which failures does it cover, and what can invalidate it later? These questions turn vocabulary back into design work.

The explanations, exercises, diagrams and workshop in this book are original teaching material. These primary sources support specific API details and provide deeper treatments of the theory. The links require internet access; no lesson depends on loading them at runtime.

Go and the operating-system boundary

The Go language specification is the reference for language rules. Use it when a question concerns what the language guarantees rather than what one example happened to do.

Effective Go explains core idioms. Its own notice says that it does not cover all later language and ecosystem developments, so pair it with current package documentation.

The io package, context package and testing package specify contracts that matter throughout the course. Pay particular attention to partial progress, cancellation responsibilities and testing entry points.

The Go memory model is the source for synchronization guarantees. Pipelines and cancellation illustrates communication and lifecycle patterns. Read them after building a bounded pipeline so the rules connect to a problem you have experienced.

Traversal-resistant file APIs explains why path checking alone can be insufficient and introduces rooted filesystem operations. Consult your target platform and Go-version documentation before relying on a particular API.

The race detector guide and Go fuzzing documentation explain the tools used in the verification chapters. Their results complement, rather than replace, explicit invariants.

Redundancy and cryptography

The klauspost/reedsolomon repository documents a Go erasure-coding implementation. Use it for the optional general-code extension after you can explain the small XOR codec and the distinction between erasures and corruption.

The crypto/cipher package, crypto/ed25519 and crypto/hkdf are the API references for authenticated encryption, signatures and key derivation. The course's compact examples intentionally expose only a small part of their contracts. Check current key, nonce, input and format requirements when implementing a real protocol.

Cryptographic primitives do not specify your whole application. Your format still needs exact byte encodings, bounds, domain separation, recovery procedures and a clear trust model. The relevant chapters focus on those joins.

Distributed-systems papers

Time, Clocks, and the Ordering of Events in a Distributed System — Leslie Lamport develops a formal account of event ordering. Read it after the concurrent-version exercise and connect its ordering ideas to your parent links.

Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services — Seth Gilbert and Nancy Lynch defines the consistency and availability model behind the familiar CAP discussion. Its definitions are more useful than the slogan.

Kademlia: A Peer-to-peer Information System Based on the XOR Metric — Petar Maymounkov and David Mazières is a starting point for structured peer routing. Ask which lookup problem it could solve in your V2 before adopting the architecture.

SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol is a useful next step beyond simple heartbeat membership. Compare its failure-detection and dissemination responsibilities with the separation in Chapters 15 and 16.

These papers are further reading, not evidence that the baseline implements their full protocols. The course uses a small explicit membership design, an immutable version catalogue and conservative repair.

Platform boundaries

Tailscale's overview describes the private networking layer used for the intended personal deployment. Application storage policy remains your responsibility.

Apple Background Tasks is the starting point for current mobile background-execution behaviour. Use it when planning the phone extension, alongside the relevant Photos and security documentation for your chosen platform version.

The supplied MeshVault protocol notes document the existing comparison implementation. They are local project material, with their own limitations and verification record.

About this edition

This first edition was prepared in September 2026. The approach is practical: build the core in Go, learn through behavioural contracts and failure cases, keep implementation guidance optional, and gradually move toward independent design.

The book contains 24 main chapters, a prologue and four appendices. The technical figures use original vector diagrams embedded in the HTML. Two original editorial illustrations were created with the built-in image generation tool; the illustration record includes their prompts and purpose.

The course uses original writing and page design. Its incremental, project-based teaching approach is shared by many good programming books; it does not reproduce another author's prose, artwork or chapter text.

The workshop reference code was checked as recorded in the verification report. The exercises deliberately ask you to build beyond those focused kernels. Your completed network should earn its own verification report before you rely on it for data you cannot replace.

If you find an error, record the chapter, the claim, a minimal counterexample and the behaviour you believe is correct. That is the same habit the book has been teaching all along: turn uncertainty into something that can be checked.