返回资讯中心
外部精选
软件工程
#开发者实时流

Decoding base64 in Python at 10 gigabytes per second

We sometimes need to convert binary data to text. For example, your emails must be stored as text files. Naïvely, you could just use a string of 0 and 1, but it would mean using one character per bit. We can do better b…

Hacker News 最新smilelamp2026/10/17 · 2026/10/1712 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

We sometimes need to convert binary data to text. For example, your emails must be stored as text files. Naïvely, you could just use a string of 0 and 1, but it would mean using one character per bit. We can do better by using 64 ASCII characters: all letters, all numbers, plus a few extra characters. Base64 represents three bytes with four ASCII characters.

Base64 is how email carries attachments, how certificates are stored, how images get embedded in web pages (data URLs), and how binary data is encoded inside JSON.

Python comes with a base64 module. It is written in C, but it processes the data one character at a time. There is a popular alternative, pybase64.

We have been working on base64 for years. With Wojciech Muła, we showed how to encode and decode base64 with SIMD instructions. More recently, I worked on forgiving base64 decoding: the variant used by web browsers, which skips spaces and line breaks. Our simdutf library implements these ideas. Mainstream browsers rely on it.

But simdutf had no Python binding. So we wrote one: fastpybase64.

pip install fastpybase64

import fastpybase64 as base64 base64.b64encode(b"hello") # b'aGVsbG8=' base64.b64decode("aGVsbG8=") # b'hello'

It is meant as a drop-in replacement for the standard base64 module.

How fast is it? I use an Intel Xeon Gold 6548N (Emerald Rapids) pinned to one core, with Python 3.14.6. I installed fastpybase64 0.1.0 and pybase64 1.5.1 from PyPI. I decode the base64 encoding of random bytes, and I report the speed in gigabytes of decoded data per second. A lot of real base64 has line breaks. MIME, the format of email attachments, breaks the base64 text into lines of 76 characters. So that’s what I use as an example. We use a single thread.

The fastpybase64 module is four times faster than pybase64 and 27 times faster than the standard library.

The fastpybase64 module is available on PyPI for Linux, macOS and Windows, for Python 3.10 to 3.14, including free-threaded Python 3.14. My benchmark is available.

Daniel Lemire, "Decoding base64 in Python at 10 gigabytes per second," in Daniel Lemire's blog, October 17, 2026, https://lemire.me/blog/2026/10/17/decoding-base64-in-python-at-10-gigabytes-per-second/.

https://lemire.me/blog/2026/10/17/decoding-base64-in-python-at-10-gigabytes-per-second/'> [BibTeX]

Copy

`;

modal.addEventListener('click', function(e) { if (e.target === modal) modal.close(); });

modal.querySelector('#bibtex-copy-btn').addEventListener('click', function() { const text = modal.querySelector('#bibtex-target').textContent; navigator.clipboard.writeText(text).then(() => { const origText = this.innerText; this.innerText = "Copied!"; setTimeout(() => this.innerText = origText, 1500); }); });

document.body.appendChild(modal);

const style = document.createElement('style'); style.innerHTML = dialog::backdrop { background: rgba(0, 0, 0, 0.5); }; document.head.appendChild(style); }                         // 1. Extract the URL             const fullLinkHtml = el.dataset.fullLink;              const tempDiv = document.createElement('div');             tempDiv.innerHTML = fullLinkHtml;              const linkElement = tempDiv.querySelector('a');             const rawUrl = linkElement ? linkElement.href : '';                           // 2. Compute the current access date             const accessedDate = this.getCurrentAccessedDate(); 

            // 3. --- NEW LOGIC: Extract ONLY the year (YYYY) ---             // Gets the full date string, e.g., "November 23, 2025"             const fullDateString = el.dataset.year;             // Use regex to find the four-digit year at the end of the string             const match = fullDateString.match(/(\d{4})$/);             const publicationYear = match ? match[0] : '????'; // e.g., '2025'                          // 4. Generate BibTeX Data with the corrected year             const safeTitle = el.dataset.title.replace(/[^a-zA-Z0-9]/g, '').substring(0, 15);             // Use the clean year for the BibKey             const bibKey = (publicationYear + safeTitle);

            const content = @misc{${bibKey}, author = {${el.dataset.author}}, title = {{${el.dataset.title}}}, year = {${publicationYear}}, howpublished = {\\url{${rawUrl}}}, note = {Accessed: ${accessedDate}} };                          // 5. Show Modal             document.getElementById('bibtex-target').textContent = content;             modal.showModal();         }     }; })();

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ). View all posts by Daniel Lemire

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →