Show HN: Claude optimizes mel-band roformer to run 4x faster on Mac

Fast voice separation on Apple Silicon. This runs a Mel-Band RoFormer source separation model on the Neural Engine and the GPU, at about 4x real time on an 8 GB M3 MacBook Air.

It uses an unmodified public checkpoint (vocals_mel_band_roformer.ckpt, 228M parameters). Over a 3 minute clip the output matches the fp32 run at 56.5 dB SDR, which you can't hear. A 45 minute TV episode takes about 11 minutes instead of 42.

Measured end to end on the same 3 minute clip:

The biggest surprise was the rotary embeddings. They ran in fp32 and rebuilt their angle tables on every call, which ate 30 to 46% of attention time on MPS. Fixing that is a small patch (fastmbr/patch.py) that helps any Mac user of this model.

Quantization would not have helped speed. 88% of the weights sit in the mask estimators, and those take only 4% of the time.

NOTES.md has every measurement, including the ideas that failed.

You need an Apple Silicon Mac, macOS 15 and uv.

git clone https://github.com/aipdv/fast-melroformer && cd fast-melroformer uv sync .venv/bin/audio-separator --download_model_only -m vocals_mel_band_roformer.ckpt --model_file_dir ~/models/melroformer bench/guard.sh .venv/bin/python bench/ane_stack.py convert # builds the Neural Engine model, once .venv/bin/python -m fastmbr.separate episode.mp4 -o out/

You get episode_vocals.wav and episode_other.wav. The first run compiles the model for the Neural Engine, which takes about 5 minutes. After that it loads in 2 seconds.

--engine gpu uses only the GPU path (2.6x). --engine ref16 and --engine ref32 run the unmodified model for comparison.