Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Interactive 3D/Ray Distributed Computing Architecture

Ray Train

Distributed model training across workers

from ray.train.torch import TorchTrainer

Ray Cluster (5 nodes)

Head Node

GCS (Global Control Store) · Scheduler · Dashboard

GCS

Sched

API

raylet + object store on each worker

Worker 1

Tasks: 1

GPU: 100%

Object Store

Worker 2

Tasks: 1

GPU: 100%

Object Store

Worker 3

Tasks: 1

GPU: 100%

Object Store

Worker 4

Tasks: 1

GPU: 100%

Object Store

Ray Train - Example

@ray.remote(num_gpus=1)

def train_fn(config):

    model = build_model(config)

    for batch in data_loader:

        loss = model(batch)

        loss.backward()

Ray Controls

Use Case

Cluster Size4 workers

216

Show Object StoresFractional GPU AllocationShow Actor vs Task

Object Store: each node has a shared-memory object store (Plasma). When a task returns a large tensor, it stays in object store memory - no serialization copy needed for local tasks.

GCS: Global Control Store - the source of truth for cluster membership, resource availability, and actor locations. Runs on the head node.

Ray Distributed Computing Architecture - Interactive Visualization

Ray is a general-purpose distributed computing framework built on a simple primitive: any Python function or class can be made distributed with a single decorator. The Ray cluster consists of a Head Node (running the Global Control Store scheduler and metadata service) and Worker Nodes (each running a Raylet for local scheduling and a Plasma Object Store for zero-copy data sharing). Ray Train wraps PyTorch/TensorFlow training loops to scale across workers. Ray Tune runs hundreds of hyperparameter trials in parallel. Ray Serve builds production serving DAGs with per-deployment autoscaling and fractional GPU allocation. All three share the same cluster and resource model, making Ray a unified runtime for the full ML lifecycle.

Ray @ray.remote: one decorator turns any Python function into a distributed task submitted to the cluster scheduler
Object store: results live in shared memory - local tasks read tensors without copying or serializing
Fractional GPU: 4 small models can share one GPU (0.25 each), maximizing utilization for inference workloads
Ray Serve DAG: compose preprocessing, embedding, LLM, and formatting into a typed deployment graph

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.