GPUlayer: Train PyTorch on every GPU you can reach-faster than PyTorch Lightning

Train PyTorch on every GPU you can reach.

One Trainer, from your laptop to many machines. GPUlayer picks the machines and GPUs, starts every process and brings back logs, checkpoints and metrics.

Documentation

Fine-tune on 8 GPUsYour GPU fleetSweep

ResNet-50 on 2 machines x 4 A100s, from a laptopsimulated replay

a100-1

4x A100 80GB (NVLink)

a100-2

4x A100 80GB (NVLink)

Trains onNVIDIA CUDAApple MetalCPU

Benchmarks

Same model, data and optimizer, each trainer's defaults. Faster in 13 of 17 GPU cases and every CPU case; slower on a CNN on one GPU of one host and on a CNN across machines. Every number has a script and a log.

A complete training API

Module, DataModule, hooks, callbacks, loggers, profilers and a CLI, with DDP, FSDP, DeepSpeed and tensor parallelism behind one Trainer.

One Trainer, any scale

The same script runs in your process, as DDP on every GPU of a machine, or across machines with Trainer(num_nodes=N). FSDP is experimental.

Placed where it trains fastest

A run goes only to machines that can train it, on their best-connected GPUs, in one rack or zone when possible. Otherwise it is rejected with a reason per machine.

Every GPU you can reach

Laptops, workstations and servers that join with one command, cloud GPUs on AWS and Vast.ai that join the same way, and Google Cloud VMs created for a run. NVIDIA CUDA and Apple Metal are validated.

Logs, checkpoints, metrics

Logs stream while training runs. Checkpoints and metrics come back to your machine, and a dashboard lists every run.

Your own loop works too

Any training script runs anywhere with two launch() lines; Runtime handles device, DDP and precision in a custom loop.

The Trainer asks the control plane for machines that can run it; workers on those machines start every process and send back logs, checkpoints and metrics. One command turns a machine into a server the others join.

Run a server (gpulayer up)

Before and after

ssh, copy code, rebuild the environment, copy checkpoints backpython train.py --target remoteSeveral GPUs in one machinetorchrun with the right process count and visible devicesTrainer(devices=4)Several machinestorchrun on every node with matching ranks, address and portTrainer(num_nodes=2, devices=4)An existing PyTorch DDP scripta cluster scheduler or manual torchrun to leave one machineglt.launch() at the top, the rest unchanged

With one GPU on the machine you are using, a plain training loop is simpler. Jobs are not sandboxed, so GPUlayer is for people who trust each other. When to use it

Install with curl -fsSL https://gpulayer.dev/install.sh | sh and train one file in your process, with DDP and on two nodes, in about ten minutes.