7 Ruby Fiber Patterns That Make Asynchronous Data Processing Clean and Fast

I remember the first time I tried to process a massive CSV file in Ruby. My program chugged along, reading the whole thing into memory, and then my laptop fan sounded like a jet engine. I had to find a better way. That’s when I discovered Ruby fibers.

Fibers are light. They are not threads. You can pause a fiber, let something else run, and then come back exactly where you left off. The magic is that you control when to pause and resume. There is no hidden switching. It’s all explicit.

In my early days, I thought fibers were just a curiosity. Then I started using them for reading streams, generating data lazily, and even building simple state machines. They changed how I write asynchronous code. Instead of fighting with locks or callbacks, I use fibers to write clean, sequential-looking code that actually runs in chunks.

Below are seven patterns I rely on for asynchronous data processing. Each pattern comes with code you can run yourself. I’ve kept the examples simple so you can see the core idea without distractions. Try them. Modify them. You’ll start seeing places where fibers fit perfectly.

---

Pattern 1: Generator – Produce Items One at a Time

A generator is the simplest fiber pattern. You create a fiber that produces values one by one. Each time you call Fiber.yield, the fiber stops and hands back a value. When you call fiber.resume, the fiber wakes up and continues from where it stopped.

I use this when I need to produce a sequence of numbers, lines from a file, or any data that takes time to compute. The fiber does the work lazily, only when I ask for the next piece.

# A generator that yields a new number each time
numbers = Fiber.new do
  n = 1
  loop do
    Fiber.yield(n)
    n += 1
  end
end

# Get the first five numbers
5.times { puts numbers.resume }
# Output: 1, 2, 3, 4, 5

This pattern is perfect for reading a file line by line without loading the entire file into memory. Let me show you a real example.

def line_generator(file_path)
  Fiber.new do
    File.foreach(file_path) do |line|
      Fiber.yield(line.chomp)
    end
    nil # signal end
  end
end

lines = line_generator('large_sales_data.csv')

# Process each line on demand
while (line = lines.resume)
  process_line(line)
end

The generator gives me one line, I process it, then ask for the next. The file stays open, but only one line is in memory at a time. This pattern saved my laptop from that jet engine sound.

---

Pattern 2: Enumerator with Fibers – Lazy Infinite Sequences

Ruby’s Enumerator already uses fibers under the hood. You can create an enumerator that generates an infinite sequence lazily. This is useful for simulation, test data, or any scenario where you don’t know how many items you will need.

fibonacci = Enumerator.new do |y|
  a, b = 0, 1
  loop do
    y.yield(a)
    a, b = b, a + b
  end
end

# Take the first 10 Fibonacci numbers
puts fibonacci.take(10).inspect
# Output: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

The enumerator itself is a fiber. Every time you call next or take, it resumes the fiber until the next yield. The sequence is infinite, but we only compute as many values as we need.

I once built a test data generator for a billing system. I needed thousands of random invoices. Using an enumerator fiber let me generate them on the fly, without pre‑allocating an array.

random_invoice = Enumerator.new do |y|
  loop do
    y.yield({
      id: SecureRandom.uuid,
      amount: rand(100..10000),
      date: Date.today - rand(365)
    })
  end
end

# Generate 5 sample invoices
5.times { puts random_invoice.next }

This pattern makes your code memory efficient and expressive. You describe an infinite source, then consume only what you need.

---

Pattern 3: Streaming Data Processing – Handle Chunks as They Arrive

Sometimes you get data in chunks – from a network socket, a slow API, or a file that comes in parts. You want to process each chunk as it arrives, without waiting for the whole thing. Fibers let you suspend processing until the next chunk is available.

I use a simple reactor loop that calls Fiber.yield to wait for data. The main loop feeds chunks into the fiber.

# A fiber that consumes chunks one by one
consumer = Fiber.new do
  buffer = ""
  loop do
    chunk = Fiber.yield(buffer)  # resume gives next chunk
    buffer << chunk
    # Process any complete lines inside buffer
    while (line = buffer.slice!(/(.*\n)/))
      puts "Processed line: #{line.chomp}"
    end
  end
end

# Start the fiber, get initial buffer (empty)
consumer.resume

# Simulate receiving chunks
['Hello ', "World\n", "Second line\n"].each do |chunk|
  consumer.resume(chunk)
end

The pattern works like a pipeline. The main loop feeds data into the fiber. The fiber processes what it can, then yields control back. No threads, no callbacks. Just two pieces of code talking through a fiber.

I built a log parser this way. The log came over a TCP socket in unpredictable chunks. The fiber kept a partial line buffer. Each time a new chunk arrived, the fiber attempted to extract complete lines. The main loop just kept reading the socket and handing bytes to the fiber.

---

Pattern 4: State Machine – Manage Steps with Fiber Suspension

A state machine can be written as a fiber that pauses at each state. When you resume the fiber, you pass an event, and the fiber moves to the next state. This makes the state logic linear and easy to follow.

I once had to implement a multi‑step signup wizard. Each step had its own validations and data. Instead of a messy switch‑case, I used a fiber.

def signup_wizard
  Fiber.new do
    # State 1: collect name
    name = Fiber.yield(:request_name)
    return :invalid_name if name.strip.empty?

    # State 2: collect email
    email = Fiber.yield(:request_email, name)
    return :invalid_email unless email.match?(/\A[\w.+-]+@\w+\.\w+\z/)

    # State 3: collect password
    password = Fiber.yield(:request_password, email)
    return :weak_password if password.length < 8

    # Done
    Fiber.yield(:complete, { name: name, email: email, password: password })
  end
end

wizard = signup_wizard
state = wizard.resume  # => :request_name

# Simulate user input
state = wizard.resume("Alice")  # => :request_email
state = wizard.resume("[email protected]")  # => :request_password
state = wizard.resume("secure123")  # => :complete

Each Fiber.yield returns a symbol that tells the caller what input is needed next. The caller passes the user’s response via resume. The fiber continues exactly where it paused. No callback nesting, no global state variables.

This pattern also works for network protocols. I’ve used it to implement a simple HTTP client that pauses when waiting for headers, then resumes when the body arrives.

---

Pattern 5: Cooperative Multitasking – Run Multiple Tasks Side by Side

Threads are heavy. Fibers are cheap. You can create hundreds or thousands of fibers and switch between them manually. This is cooperative multitasking. Each fiber does a little work and then yields to the scheduler.

Here is a simple scheduler that runs multiple fibers in round‑robin fashion.

class Scheduler
  def initialize
    @fibers = []
  end

  def add(fiber)
    @fibers << fiber
  end

  def run
    while @fibers.any?
      @fibers.reject! do |f|
        next true unless f.alive?
        f.resume
        false
      rescue FiberError
        true
      end
    end
  end
end

# Create two fibers that interleave
f1 = Fiber.new do
  3.times do |i|
    puts "Fiber 1: step #{i}"
    Fiber.yield
  end
end

f2 = Fiber.new do
  3.times do |i|
    puts "Fiber 2: step #{i}"
    Fiber.yield
  end
end

scheduler = Scheduler.new
scheduler.add(f1)
scheduler.add(f2)
scheduler.run

The output alternates between the two fibers. Each fiber runs only as long as it wants, then yields control. This is perfect for I/O‑bound tasks. If one fiber is waiting for a database query, it can yield so another fiber processes a previous result.

I built a small web scraper this way. I had a list of 200 URLs. I created a fiber for each URL. Each fiber made an HTTP request (using a non‑blocking library) and yielded while waiting for the response. The scheduler advanced all fibers, and the total time was much less than sequential requests.

---

Pattern 6: Pipeline – Chain Fibers for Stream Transformation

A pipeline connects fibers like assembly line stages. Data enters the first fiber, gets transformed, yielded to the next, and so on. Each stage runs in its own fiber. You control when data moves from one stage to the next.

I use this pattern for ETL (extract, transform, load) processes. Let me show you a simple pipeline that reads lines, filters them, and transforms them.

def source(data)
  Fiber.new do
    data.each { |item| Fiber.yield(item) }
    nil
  end
end

def filter(upstream, condition)
  Fiber.new do
    while (item = upstream.resume)
      Fiber.yield(item) if condition.call(item)
    end
    nil
  end
end

def transform(upstream, transformation)
  Fiber.new do
    while (item = upstream.resume)
      Fiber.yield(transformation.call(item))
    end
    nil
  end
end

def sink(upstream)
  while (item = upstream.resume)
    puts "Result: #{item}"
  end
end

# Build the pipeline
data = [1, 2, 3, 4, 5, 6]

source_fiber = source(data)
filter_fiber = filter(source_fiber, ->(x) { x.even? })
transform_fiber = transform(filter_fiber, ->(x) { x * 10 })

sink(transform_fiber)
# Output:
# Result: 20
# Result: 40
# Result: 60

Each fiber pulls from an upstream fiber. The pipeline is lazy: data flows only as fast as the sink consumes it. You can add or remove stages without rewriting the whole chain.

I’ve used this to process CSV rows: first stage reads lines, second stage parses them, third stage validates, fourth stage inserts into a database. Each stage runs as a fiber, and the database writes happen while the next row is being parsed.

---

Pattern 7: Async I/O with Fibers and a Reactor – Non‑blocking, Sequential Code

Ruby 3.0 introduced Fiber.scheduler, which lets you write asynchronous I/O that looks synchronous. You create a scheduler that handles select/poll/epoll, and your fibers just read or write normally. When a fiber does an I/O operation that would block, the scheduler suspends it and runs another fiber.

This pattern is the most powerful. It gives you the performance of event‑driven code with the readability of sequential code.

Here is a minimal example using Fiber.scheduler (available in Ruby 3.0+). I’ll use a simple scheduler that I wrote, or you can use a gem like async.

require 'socket'
require 'fiber'

# A minimal scheduler using IO.select
class SimpleScheduler
  def initialize
    @readable = {}
    @waiting = []
  end

  def kernel_sleep(duration = nil)
    # Not fully implemented; just yield
    @waiting << Fiber.yield
  end

  def io_wait(io, events, duration = nil)
    @readable[io] = Fiber.current
    Fiber.yield
  end

  def run
    while @readable.any? || @waiting.any?
      # Check for ready I/O
      ready, _, _ = IO.select(@readable.keys, [], [], 0.1)
      if ready
        ready.each do |io|
          fiber = @readable.delete(io)
          fiber.resume
        end
      end
      # Resume waiting fibers (simplified)
      @waiting.each { |f| f.resume if f.alive? }
      @waiting.clear
    end
  end
end

# Use the scheduler
scheduler = SimpleScheduler.new
Fiber.set_scheduler(scheduler)

fiber1 = Fiber.new do
  sock = TCPSocket.new("example.com", 80)
  sock.write("GET / HTTP/1.0\r\n\r\n")
  # This read would block, but scheduler suspends fiber
  response = sock.read
  puts "Got #{response.length} bytes"
end

fiber1.resume
scheduler.run

In real code, you’d use a production scheduler like Async::Scheduler. But the idea is the same. You write plain Ruby – sock.read, File.read, sleep – and the scheduler makes them non‑blocking by suspending the fiber when necessary.

I rewrote a batch image downloader using this pattern. Before, it was sequential and slow. After, I launched 50 fibers, each downloading one image. The scheduler ran them concurrently on a single thread. The code looked like a simple loop, but the execution was concurrent.

---

Bringing It All Together

Fibers give you control. You decide when to pause and when to resume. This lets you write code that is efficient, readable, and testable. Each pattern above solves a common async problem: generating data lazily, handling chunks, managing state, multitasking, building pipelines, or doing non‑blocking I/O.

I started with simple generators and ended up writing a small Ruby web framework based on fibers. The learning curve is gentle. Start with Pattern 1. Add a line generator to your next CSV parser. Then try the state machine for a wizard. You’ll soon see fibers everywhere.

The best part is that you never need to think about threads or locks. Fibers are cooperative. If you don’t call Fiber.yield, the fiber runs to completion. No race conditions from preemption. That simplicity is what made fibers my go‑to tool for asynchronous data processing.

Now go open your editor. Create a fiber that yields the first 100 squares. Or write a pipeline that reads a file, filters lines, and writes to another file. The code will be clean, and your laptop will stay quiet.