How Firebird QA works and why it needs to be stronger

October 9, 2026 · Alexey Kovyazin, Firebird Foundation

Firebird is raising money to reform its QA. The flow of code changes grows faster than the project can check it. Without new resources, the quality of releases will suffer.

Firebird is an open-source relational database. Its community develops it and uses it. The project estimates that about one million software developers use Firebird in their applications. Four branches are supported at the same time: 3.0, 4.0, 5.0 and 6.0 (in development). Every bug fix and every improvement must pass manual and automated checks before release.

In 2025, the Firebird repository received 190 pull requests (PRs). This is 3.1 times more than in 2020, and 2026 is already ahead of 2025. AI tools are a big reason: writing code became cheaper, but checking it did not. Complexity also grows. Firebird 6.0 is in development, and in the first half of 2026 there were almost twice as many PRs with more than 200 lines as one year earlier: 40 against 21.

The main idea of the reform is to run QA more often. Today the test suite runs once a day and only after merge. For PRs it does not run at all. After the reform, every approved PR will pass a mandatory QA subset before merge, on every update. Large PRs will also pass the full suite and a performance test.

For this we need QA engineers, cloud machines, an AI agent for the first analysis of PRs, and new tests. We ask Firebird users, and companies whose business depends on Firebird, to support the reform. The first step is done: on October 8, 2026, the Firebird Foundation started a crowdfunding campaign to raise US$20,000 for QA.

Firebird has been developed as an open-source project for 26 years. Today it has four branches, 12 platforms in release 5.0.4 and about 763 thousand lines of code. A suite of 3,138 automated tests checks this code.

We counted the code with the cloc tool in the src directory of release 5.0.4, without blank lines and comments. About 288 thousand of the 763 thousand lines are national character set tables. The C/C++ code of the engine, the SQL layer, the network layer and the utilities is about 453 thousand lines. The platforms are taken from the files of release 5.0.4 from April 17, 2026: Windows (x86, x64), Linux (x86, x64, ARM32, ARM64), macOS (x64, ARM64) and Android (x86, x64, ARM32, ARM64).

Since 2020, the number of pull requests in Firebird has grown 3.1 times. In the first nine months of 2026 there were almost as many PRs as in all of 2025: 184 against 190.

Besides PRs, the tracker receives bug reports and feature requests. On October 9, 2026, about 1,800 issues and 92 PRs were open. The number of authors also grows: 19 people opened PRs in 2020, 30 in 2025, and already 32 in 2026. Every bug report must be reproduced. Every PR must be checked and tested on its own branch and on the branches where the fix is ported.

Bug reports arrive at a stable rate: from 235 to 368 per year, with no clear growth. What grows is the flow of code changes: in 2026, PRs passed issues for the first time, 184 against 173 by October 9. Together this is about 440 work items per year, and each one must be checked.

Today QA runs only after merge. CI on GitHub builds the code but does not test it. The test suite runs once a day on servers provided by IBSurgeon, and only for branches that had commits. People do the review, the checks and the analysis of results, and these steps do not scale with the flow of changes.

The rule is the same for all branches: if there was a change, there is a QA run. Older branches change less often, so they have fewer runs. The run time grows faster than the number of tests:

Run time is the median of the last 35 runs on firebirdtest.com as of October 9, 2026. It is the sum of two runs, in SuperServer and Classic modes. “Tests” is the number of tests that run for this branch. All branches use the same machines: the Linux server has 2 CPUs and 7 GB, the Windows server has 8 CPUs and 127 GB. Branch 6.0 has 7% more tests than 5.0, but its Linux run takes 86% longer. The time depends not only on the number of tests, but also on what these tests do.

Today QA does not run for PRs at all. A regression becomes visible only the next morning after merge, together with all commits of that day. Then someone still has to find the exact change that caused it.

Verifying one fix is a small investigation. A test is published only if the QA engineer caught the bug on a build before the fix and confirmed that the bug is gone on a build with the fix.

Each step takes engineer time, and often a separate snapshot build. So the bottleneck is not writing code, but checking it. This is exactly where the reform adds people and cloud machines.

AI tools made writing code and bug reports much cheaper, but they did not make checking cheaper. For a database engine this is especially dangerous. A patch that looks correct can break concurrency, crash recovery or performance, and this will show only under load or on another platform.

The flow in Firebird grew in the same way: 134 PRs in 2024, 190 in 2025 (+42%) and 184 by October 9, 2026, which is about 238 at the yearly rate. We welcome new contributors and do not want to limit contributions. The goal is to scale checking, not to close the door.

But complexity matters more than numbers. Firebird 6.0 is in development, and more and more PRs bring large features. In the first half of 2025, 21 of 104 PRs had more than 200 lines; in the first half of 2026, 40 of 106. The total number of PRs almost did not change, but the share of large PRs grew from 20% to 38%.

Such changes need checks on several levels: functional tests on Linux and Windows in both server modes, performance under load, and porting to other branches. And not only once: a large PR is updated after review comments, and PRs with more than 1,000 lines have about eight such iterations on average. Each iteration must be checked again.

The main idea of the reform is to run QA more often: not once a day after merge, but for every approved PR before merge, on every update. The second part is more tests, including a fast mandatory QA subset for PRs.

All four directions build on what already works: the test suite, nightly runs on IBSurgeon servers, and CI. Nightly runs stay: direct commits of core developers also go through them, and in master these are about three quarters of all changes.

How the path of a PR changes after the reform: analysis and approval come before QA, and tests move to the time before merge.

The Firebird Foundation crowdfunding campaign for US$20,000 is announced for QA engineers' time and virtual machines. The collected money also pays for the first-analysis agent and the setup of QA for PRs.

A QA run starts only after a person approves it. The run executes the PR code on our machines, so the PR must be reviewed first. It can be rejected right away as a duplicate, nonsense, an unconfirmed problem or an intentionally destructive change.

In the cloud, four VMs in parallel cost the same as one machine in sequence. But the result is ready when the longest branch finishes: after 4 h 39 min instead of 11 hours.

Take the nightly run of the full suite on Linux for all four branches: 3.0 takes 1 h 31 min, 4.0 takes 2 h 19 min, 5.0 takes 2 h 30 min, and 6.0 takes 4 h 39 min. On one server this is 11 hours in a row; on four VMs it is 4 h 39 min. The cloud charges for actual working time, so in both cases we pay for the same 11 machine-hours.

PRs work the same way: the full suite on Linux and Windows and two OLTP-EMUL measurements run at the same time, and a large PR gets an answer in about 5 hours, not 11.

Since January 1, 2026, DigitalOcean bills droplets per second, with a minimum of 60 seconds, so we pay exactly for the run time. Prices are from the DigitalOcean pricing page as of October 9, 2026.

For performance tests we use General Purpose: dedicated cores are not shared with neighbors on the server, so the measurements are repeatable. Full 6.0 suite on Linux: $0.1875/h × 4.65 h ≈ $0.87. The IBSurgeon Linux server is weaker (2 CPUs and 7 GB), so in the cloud the run will probably be faster; in our calculations we use the measured time as is.

DigitalOcean does not offer Windows droplets, so Windows runs go to Azure. Prices are for the West Europe region, pay-as-you-go, from the Azure Retail Prices API as of October 9, 2026. The Windows license is included in the price; disks and traffic are not included.

Full 6.0 suite on Windows: $0.414/h × 2.47 h ≈ $1.02. The IBSurgeon Windows server is more powerful than the cloud VM (8 CPUs and 127 GB), so in the cloud the run may take longer; the reserve for repeated runs covers this. Spot VMs are more than five times cheaper, but Azure can take them back at any moment. They are fine for functional tests, but not for performance measurements.

QA for every approved PR will cost about $477 for six months. This uses 2.8 times less machine time than the full suite with OLTP-EMUL on every push. The savings come from the mandatory subset and the complexity factor.

The forecast is 127 PRs in the next six months: 114 PRs in the last six months with 25% growth per year, the average since 2020: 114 × 1.25^0.5 ≈ 127. Issues are not part of cloud runs: a QA engineer reproduces a bug report on snapshots, and the fix comes as a PR or a commit.

There are three types of runs. All are calculated for branch 6.0, where most PRs go and where the suite is the longest. We add 15 minutes of preparation to each VM start:

What to run and how many times depends on the PR. A 5-line PR and a 5,000-line PR differ both in the number of updates and in the depth of checks. So we use a complexity factor: the machine time per PR compared with the simplest class.

Class shares and push counts come from 214 PRs of the last 12 months. We estimated pushes by the number of days with new commits in a PR: on average 1.2, 1.6, 2.1 and 8.0 by class. This is a lower bound, because force-push hides some iterations, so the values in the table are rounded up.

Of the 127 forecast PRs, 53 are in class S, 30 in M, 28 in L and 16 in XL. One PR costs $1.13, $1.50, $4.30 and $10.76 by class. We add another 20% for repeated runs because of unstable tests and VM failures.

Compare this with today and with the option “everything on every push”:

Today's machine time is calculated from the last 30 days: 34 runs on Linux and 26 on Windows. In both scenarios the cost is small. The waiting time matters more: with the mandatory subset, the author gets an answer for every update in about 1 hour 15 minutes; with the full suite, only after 5 hours.

If PRs grow faster than the forecast, at the 2025 rate of +42% per year, there will be about 136 of them, and the runs will cost about $511. The reserve covers the difference.

Preparing the 6.0 release and running tests more intensively will add about $150 to cloud runs for six months, about 30% on top of QA for PRs. The main cost of the expansion is not machines, but the time of QA engineers for new tests.

According to the Firebird 6 roadmap, the alpha version comes in Q4 2026, the beta in Q2 2027, and the final release in Q4 2027. The suite grows with them: tests that run only on 6.0 grew from 93 to 225 in one year; 91 of them were added in the last six months and 49 in the last quarter. New features (schemas, tablespaces, JSON functions, the ROW type, the shared metadata cache) need new and often heavy tests.

Machine-hours are shown without repeats; the cost includes the 20% reserve for repeated runs, as above.

With the expansion, machine time grows about three times compared with today: 153 machine-hours of nightly runs, 232 for QA of PRs and another 82 for the expansion, about 467 per month in total. Nightly servers do not have this capacity. And after the beta, a separate 6.0 branch will appear, so there will be five nightly runs instead of four.

The main expansion is people's work. If the pace of the last quarter