Posted By
Concurrency is Go’s superpower — but raw goroutines and channels aren’t enough to build clean, reliable systems. Go’s concurrency primitives (goroutines, channels, select, etc.) can be composed into higher-level patterns that make concurrent programs more robust and maintainable. In this guide, we’ll explore these patterns with practical code examples and real-world use cases — from confinement and for-select loops to pipelines, fan-out/fan-in, and worker pools. You’ll also see modern updates like using context.Context instead of ad hoc “done” channels and learn when each pattern fits best.
The Go Concurrency Model
Go uses goroutines to run functions concurrently and channels to communicate safely between them. Together, they enable developers to structure concurrent systems without relying on traditional locking mechanisms.
Goroutines
A goroutine is a lightweight thread managed by the Go runtime. You can start one simply by adding the “go” keyword before a function call:
go doWork()
The Go runtime manages thousands of goroutines efficiently using an internal scheduler that maps many goroutines onto a smaller number of operating system threads.
Channels
Channels are Go’s way of communicating safely between goroutines — no shared memory, no explicit locking.
ch := make(chan int)
go func() {
ch <- 42 // send
}()
fmt.Println(<-ch) // receive
Channels allow values to be passed between goroutines safely. This eliminates the need for explicit locks or shared-memory coordination.
Select Statement
The select statement allows waiting on multiple channel operations simultaneously — it’s like a switch for channels.
select {
case msg := <-ch1:
fmt.Println("Received", msg)
case <-time.After(time.Second):
fmt.Println("Timeout")
}
Using select gives you flexible control over concurrent operations, such as timeouts, cancellation, or prioritizing one channel over another.
Go Concurrency patterns
Confinement
Confinement is a simple pattern: keep data accessible to only one goroutine. By confining ownership of data, you avoid race conditions without any locks. In Go, this often means that one goroutine owns a channel or data structure and no other goroutines write to it. For example, an “owner” goroutine can generate values and send them on a channel, while other goroutines only read from that channel. Because only the owner writes or closes the channel, there is no concurrent access to worry about.
func producer() <-chan int {
out := make(chan int, 5)
go func() {
defer close(out)
for i := 0; i < 5; i++ {
out <- i
}
}()
return out
}
func main() {
// The main goroutine is the consumer; the producer confines the writes.
for v := range producer() {
fmt.Println(v) // safely receives 0,1,2,3,4
}
}
In this example, the producer goroutine writes to its own channel and then closes it. No other goroutine writes to that channel, so no locks are needed. Real-world use case: a single goroutine feeding data (from a sensor, network, or batch) into a channel, where worker goroutines safely receive from that channel. Confinement also extends to arrays or maps: each goroutine may be given its own slot or data structure (as in an index in a slice) and never share it with others. This makes code simpler and lock-free, while ensuring safe sequential access.
The For-Select Loop
A for-select loop is an idiomatic way in Go to wait on channels and optionally do work until a stop condition. It typically looks like this (often with a done or quit channel to break out):
func worker(done <-chan struct{}) {
for {
select {
case <-done:
// Stop the loop and exit.
return
default:
}
// Perform non-blocking work here.
fmt.Println("working")
time.Sleep(100 * time.Millisecond)
}
}
func main() {
done := make(chan struct{})
go worker(done)
time.Sleep(500 * time.Millisecond)
close(done) // signal worker to exit
}
Here, the goroutine loops indefinitely but uses a select to check <-done each time. If a value (or a close) is received on done, the goroutine returns. This pattern shows up throughout Go. You’ll see two common variants:
- Minimal select with default: keep the select as short as possible (case <-done and default), then do long-running work after. Example above.
- Select around work: put the work inside select cases (e.g. case msg := <-workCh: vs case <-done:).
Either way, the for-select loop cleanly handles waiting on multiple channels or an exit signal. It’s fundamental for patterns like rate-limited loops, worker goroutines, and pipelines.
Preventing Goroutine Leaks
A goroutine leak happens when a goroutine never terminates because it’s stuck waiting (perhaps on a channel that no one will ever send on). To prevent leaks, always ensure goroutines can exit when they should. A common approach is to use a done channel (or better, a context.Context) that signals cancellation. For example:
func doWork(done <-chan struct{}) {
for {
select {
case <-done:
fmt.Println("doWork canceled")
return
default:
}
// Simulate work
time.Sleep(200 * time.Millisecond)
}
}
func main() {
done := make(chan struct{})
go doWork(done)
// Cancel after 1 second to avoid leak
time.Sleep(1 * time.Second)
close(done)
// Wait a bit for doWork to print cancelation message
time.Sleep(100 * time.Millisecond)
}
Here, the doWork loop checks <-done and exits when done is closed. The key insight is always arrange for your goroutines to terminate. Using a cancellation channel (or context) ensures you “join” goroutines instead of leaving them running forever, avoiding resource leaks.
Or-Channel
An or-channel (often called Or or orDone pattern) takes multiple “done” channels and produces a single channel that closes if any input closes. It’s like a logical OR for cancellation signals. This is useful when you have several cancellation signals but want one unified way to wait. For example:
// OrChannel combines multiple channels into one that closes when any input closes.
func OrChannel(channels ...<-chan interface{}) <-chan interface{} {
switch len(channels) {
case 0:
return nil
case 1:
return channels[0]
}
orDone := make(chan interface{})
go func() {
defer close(orDone)
switch len(channels) {
case 2:
select {
case <-channels[0]:
case <-channels[1]:
}
default:
// Recursively wait on others
select {
case <-channels[0]:
case <-channels[1]:
case <-channels[2]:
case <-OrChannel(append(channels[3:], orDone)...):
}
}
}()
return orDone
}
This function starts a goroutine that waits on any of the input channels. As soon as any channel receives or closes, it closes orDone, which signals all consumers. For instance, you might use it to wait for the first of several timeouts or events. This “closes if any of its component channels close”. A real-world use case: waiting for the first response from multiple servers or cancelling a task if any of several error signals occur.
Error Handling
In concurrent programs, parent goroutines often need to handle errors from child goroutines. A common pattern is to send a result struct over a channel that includes both the value and an error. For example:
type Result struct {
Err error
Response *http.Response
}
// checkStatus queries multiple URLs and reports each Response or error.
func checkStatus(done <-chan interface{}, urls ...string) <-chan Result {
results := make(chan Result)
go func() {
defer close(results)
for _, url := range urls {
resp, err := http.Get(url)
select {
case <-done:
return
case results <- Result{Err: err, Response: resp}:
}
}
}()
return results
}
func main() {
done := make(chan interface{})
defer close(done)
urls := []string{"https://example.com", "https://golang.org", "https://doesnot.exist"}
errCount := 0
for res := range checkStatus(done, urls...) {
if res.Err != nil {
fmt.Printf("error: %v\n", res.Err)
errCount++
if errCount >= 2 {
fmt.Println("too many errors, exiting")
break
}
continue
}
fmt.Printf("Status: %d\n", res.Response.StatusCode)
}
}
Here, the Result struct carries both the HTTP response and any error. The parent loop reads from the channel and inspects res.Err. Real-world scenarios: contacting multiple services and handling whichever fail, or aggregating errors from parallel tasks. The parent can decide when to abort (as shown by breaking out after too many errors) while child goroutines simply send back their results via the shared channel.
Pipelines
A pipeline is a series of stages connected by channels, where each stage runs in its own goroutine (or set of goroutines) and passes data to the next stage. Think of it like Unix pipes but in Go. Each stage receives from an inbound channel, does work (filtering, transformation, etc.), and sends results downstream. When done, each stage closes its outbound channel.
For example, a simple pipeline with two stages – “generator” and “squarer” – can be written as follows (in idiomatic Go using range loops):
// gen emits the list of ints onto a channel and then closes it.
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for _, n := range nums {
out <- n
}
}()
return out
}
// sq reads ints from 'in', squares them, and sends them on its returned channel.
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for n := range in {
out <- n * n
}
}()
return out
}
func main() {
// Set up the pipeline: gen -> sq -> sq
for n := range sq(sq(gen(2, 3))) {
fmt.Println(n) // prints 16 then 81
}
}
Each stage closes its channel when complete. Real-world usage: data processing pipelines, streaming transformations, log processing stages, etc.
Best practices: Always have a defer close(channel) in each stage. Also, it’s important to allow early exit. In practice, we include a “done” or context.Context to cancel a pipeline if some stage fails. For instance, we can modify stages to take a done <-chan struct{} or ctx context.Context and return early if canceled. This avoids goroutine leaks when downstream stages stop reading. For brevity, the example above omits cancellation, but in production code you’d typically pass a context to each stage to allow terminating the pipeline early.
Handy Generators
Generators are helpers for pipelines, producing streams of data. Two common ones are Repeat and Take:
- Repeat: repeatedly emits a set of values (e.g. 1,2,3,1,2,3,…) on a channel until stopped.
- Take: reads the first N values from a channel and then closes.
Example using a done channel to stop:
// repeat sends the given values endlessly on a channel until done is closed.
func repeat(done <-chan struct{}, values ...int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for {
for _, v := range values {
select {
case <-done:
return
case out <- v:
}
}
}
}()
return out
}
// take reads up to 'n' values from 'in' and then closes.
func take(done <-chan struct{}, in <-chan int, n int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for i := 0; i < n; i++ {
select {
case <-done:
return
case out <- <-in:
}
}
}()
return out
}
func main() {
done := make(chan struct{})
defer close(done)
// Repeat 1,2,3 until taken 5 values.
for v := range take(done, repeat(done, 1, 2, 3), 5) {
fmt.Print(v, " ") // output: 1 2 3 1 2
}
}
It’s handy for testing examples: you can generate infinite streams of data and then constrain them. Updated idiomatic tip: in modern Go you might use a context.Context instead of a done channel, and you could use type parameters (generics) to avoid int vs interface{} issues.
Fan-Out, Fan-In
Fan-Out means starting multiple goroutines (workers) to process items from the same channel. Fan-In means merging results from multiple channels back into one. This is used to parallelize work. For example, we might send the same input to several workers (fan-out) and then merge their outputs (fan-in).
The Go blog demonstrates this by running two sq stages in parallel:
in := gen(2, 3)
// Two parallel workers for sq (fan-out).
c1 := sq(in)
c2 := sq(in)
// fan-in: merge results from c1 and c2.
out := merge(c1, c2)
for n := range out {
fmt.Println(n) // prints 4 and 9 (order not guaranteed)
}
The merge function (aka fan-in) uses a sync.WaitGroup to copy from each input channel to one output channel, then closes the output when done:
func merge(chs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int)
output := func(c <-chan int) {
defer wg.Done()
for v := range c {
out <- v
}
}
wg.Add(len(chs))
for _, c := range chs {
go output(c)
}
// Close out when all copies are done
go func() {
wg.Wait()
close(out)
}()
return out
}
Fan-out is useful when you have CPU-bound or I/O-bound tasks and want to spread work across cores or handle multiple requests. Fan-in (merge) is essential to collect all results into one place. The pattern is widely used in pipelines for parallelism.
Worker Pool
A worker pool limits concurrent work to a fixed number of goroutines. Instead of spawning a goroutine per task (which can flood memory/FDs/CPUs), you create N workers that pull tasks from a queue channel, process them, and send results back. This yields steady throughput with predictable resource usage and natural backpressure (the task queue fills when producers outpace workers).
Where it shines
- Hitting an external API with rate or connection limits
- CPU-bound jobs (hashing, image transforms) where you want to match GOMAXPROCS
- Disk/network scraping where too much parallelism hurts more than it helps
Minimal Worker Pool: Tasks, results, done, and fan-In
Below is a self-contained, production-ready pattern using:
- tasks <-chan T as the input queue (read-only to workers)
- results chan<- R as the output sink
- done <-chan struct{} to cancel/stop
- sync.WaitGroup to join workers
- fan-in close: only close results after all workers finish
Example scenario
We’ll “process” integers by squaring them, but you can imagine each task being an HTTP call, DB query, or CPU job.
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
// Task and Result types (can be any domain-specific payloads)
type Task struct {
ID int
Val int
}
type Result struct {
TaskID int
Out int
Err error
}
// worker consumes tasks and produces results.
// It must: respect 'done', never panic, and never leak.
func worker(
id int,
done <-chan struct{},
tasks <-chan Task,
results chan<- Result,
) {
for {
select {
case <-done:
// Cooperative cancellation.
return
case t, ok := <-tasks:
if !ok {
// No more tasks; clean exit.
return
}
// Simulate variable work (I/O/CPU/etc.)
time.Sleep(time.Duration(rand.Intn(150)+50) * time.Millisecond)
// Example "processing":
out := t.Val * t.Val
// Send the result OR honor cancellation.
select {
case <-done:
return
case results <- Result{TaskID: t.ID, Out: out, Err: nil}:
}
}
}
}
// startWorkerPool launches 'n' workers and returns a function to wait for them.
// The caller is responsible for: closing 'tasks' when done submitting, and
// closing 'results' AFTER all workers finish (we do that here via a helper goroutine).
func startWorkerPool(
n int,
done <-chan struct{},
tasks <-chan Task,
) (<-chan Result, *sync.WaitGroup) {
results := make(chan Result)
var wg sync.WaitGroup
wg.Add(n)
for i := 0; i < n; i++ {
go func(id int) {
defer wg.Done()
worker(id, done, tasks, results)
}(i + 1)
}
// Close results after all workers exit.
go func() {
wg.Wait()
close(results)
}()
return results, &wg
}
func main() {
rand.Seed(time.Now().UnixNano())
done := make(chan struct{}) // signal to stop early
tasks := make(chan Task, 8) // buffered queue -> backpressure control
const workers = 4 // pool size
results, _ := startWorkerPool(workers, done, tasks)
// Producer: enqueue tasks (could be another goroutine reading a file/DB/etc.)
go func() {
defer close(tasks) // signal "no more tasks"
for i := 1; i <= 20; i++ {
select {
case <-done:
return
case tasks <- Task{ID: i, Val: i}:
// enqueued
}
}
}()
// Consumer: drain results. In real systems, this might be writing to a DB, another stage, etc.
for r := range results {
if r.Err != nil {
fmt.Printf("task %d error: %v\n", r.TaskID, r.Err)
continue
}
fmt.Printf("task %02d -> %d\n", r.TaskID, r.Out)
// Example early-stop condition (optional):
// if r.TaskID == 10 { close(done) }
}
fmt.Println("all done")
}
Why this works
- Backpressure: The buffered tasks channel controls enqueue rate. When full, producers block—preventing unbounded memory growth.
- Bounded concurrency: Exactly workers goroutines execute tasks at once.
- Clean shutdown: Closing tasks lets workers exit naturally; closing done cancels immediately.
- No leaks: Workers select on done and exit; results is closed only after workers finish.
Tee Channel
A tee-channel duplicates a stream: it reads from one input channel and sends each item to two output channels. Each output gets every value from the input (like teeing a garden hose). This allows, for instance, logging and processing the same events in parallel. Example:
func tee(done <-chan interface{}, in <-chan interface{}) (<-chan interface{}, <-chan interface{}) {
out1 := make(chan interface{})
out2 := make(chan interface{})
go func() {
defer close(out1)
defer close(out2)
for val := range orDone(done, in) {
// Try sending to out1 and out2, but if one sends successfully,
// disable that channel to avoid double-sending to it.
c1, c2 := out1, out2
for i := 0; i < 2; i++ {
select {
case c1 <- val:
c1 = nil
case c2 <- val:
c2 = nil
}
}
}
}()
return out1, out2
}
// Example usage:
done := make(chan struct{})
in := make(chan interface{})
go func() {
defer close(in)
for i := 0; i < 5; i++ {
in <- i
}
}()
c1, c2 := tee(done, in)
for x := range c1 {
fmt.Println("c1:", x)
if x == 2 {
close(done) // stop tee after some values
}
}
for x := range c2 {
fmt.Println("c2:", x)
}
In this code, each value from in is sent once to both out1 and out2. The trick of setting c1 = nil or c2 = nil ensures each value is sent exactly once to each channel. Use case: splitting a stream of events to two consumers (e.g. one for logging, one for business logic).
Bridge Channel
A bridge channel flattens a stream of channels into a single stream of values. Imagine you have a channel that emits other channels (like <-chan <-chan T). The bridge will take each inner channel in turn and emit its values on one output channel. This is useful when you have dynamic pipelines that produce more channels. For example:
func bridge(done <-chan interface{}, chanStream <-chan (<-chan interface{})) <-chan interface{} {
valStream := make(chan interface{})
go func() {
defer close(valStream)
for {
var stream <-chan interface{}
select {
case maybeStream, ok := <-chanStream:
if !ok {
return
}
stream = maybeStream
case <-done:
return
}
// Drain values from the current inner channel.
for v := range orDone(done, stream) {
select {
case valStream <- v:
case <-done:
}
}
}
}()
return valStream
}
// Usage:
// chanStream := make(chan (<-chan interface{}))
// ... send some channels on chanStream ...
// The bridge will output all values from those channels in order.
This code takes a chanStream (a channel of channels). Each time a new channel is received, it consumes that channel fully (or until done) and emits all its values on valStream. Use case: You might use this if an API returns multiple channels of results and you want to process them in one loop.
Queuing
The Queuing pattern uses a buffered channel as a semaphore to limit the number of concurrent tasks. This decouples task submission from execution and throttles concurrency. For example, to process up to N tasks in parallel (say N workers), you can use a channel of size N:
func main() {
const numWorkers = 3
const numTasks = 5
var wg sync.WaitGroup
sem := make(chan struct{}, numWorkers)
for i := 1; i <= numTasks; i++ {
wg.Add(1)
// Acquire a slot.
sem <- struct{}{}
go func(taskID int) {
defer wg.Done()
// Do the work.
fmt.Println("Start task", taskID)
time.Sleep(500 * time.Millisecond)
fmt.Println("Done task", taskID)
// Release the slot.
<-sem
}(i)
}
wg.Wait()
}
Here, sem is a buffered channel of capacity numWorkers. Each goroutine sends a dummy struct to sem before working (blocking if sem is full) and receives from sem (releasing) when done. This ensures at most numWorkers tasks run concurrently. The Go Patterns site describes this: “A buffered channel is commonly used as a semaphore to throttle the number of active goroutines,” providing backpressure and preventing resource exhaustion. Use case: limiting concurrent network calls, database queries, or CPU-heavy jobs in a server.
The context Package
In modern Go, many cancellation and timeout patterns use context.Context instead of raw channels. The context package provides deadlines, cancellation signals, and value propagation across API boundaries. For example:
func work(ctx context.Context) {
for i := 0; i < 5; i++ {
select {
case <-time.After(1 * time.Second):
fmt.Println("Work step", i)
case <-ctx.Done():
fmt.Println("Work canceled at step", i)
return
}
}
}
func main() {
// Create a context that is canceled after 2 seconds.
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
go work(ctx)
// Wait for work to finish or be canceled.
time.Sleep(3 * time.Second)
}
This example uses context.WithTimeout to automatically cancel the context after 2 seconds. The work function checks ctx.Done() (like a done channel) to know when to stop. Cox-Buday’s chapter emphasizes that context can be used “to add timeouts and cancellations” to patterns. In short, use context to manage lifecycles instead of custom channels when possible: it’s idiomatic, integrates with many libraries, and makes cancellation explicit.
Choosing the right pattern
With so many patterns, how do you pick? Some guidelines:
- Confinement: Use when you can give each goroutine exclusive ownership of the data it needs (no sharing). Great for simple pipelines and one-writer scenarios.
- For-Select Loop: Use this as the basic structure for any long-lived goroutine that needs to wait on channels or implement timeouts (select on <-time.After, <-done, etc.).
- Done / Context: Always incorporate a cancellation mechanism (a done channel or, better, context.Context) in long-running goroutines and pipelines, to avoid leaks.
- Pipelines: Use when you have staged processing (e.g. transforming a stream of data step-by-step). They allow composability and parallelism.
- Fan-Out/Fan-In: Use when you need to parallelize independent work (fan-out) and then combine results (fan-in). Good for parallelizing CPU-bound tasks or doing multiple I/O calls concurrently.
- Generators: Handy for creating test data or streaming values when you need infinite or repeatable inputs.
- Queuing: Use when you must limit concurrency (e.g. to N workers). A buffered channel as semaphore is a simple and effective strategy.
- Advanced Patterns (or-channel, or-done, tee, bridge): Use when combining or splitting streams in complex ways. For most everyday code, these are less common, but they can simplify code for broadcasting to multiple consumers or merging streams of streams.
In summary, Go’s CSP-style primitives let you pick the right abstraction for each problem. Simpler patterns (single loop with select, one goroutine per channel) are usually sufficient. The key takeaway is to design so that goroutines can run concurrently without race conditions or leaks – whether by sharing nothing (confinement) or communicating properly (channels and select) – and always allow cancellation.
Related Blogs
