BudHist Benchmark: Testing the Knowledge of Buddhist History in Open-Weight LLMs

This ongoing project aims to develop datasets and best practices to test the knowledge of language models in the domain of Buddhist history. The focus is on open-weights models. Local, open-weights models are important for researchers because they allow researchers full control over their experiments. Experiments are repeatable without restrictions and the models can be archived without constraints.

We use a growing dataset of four-options multiple choice questions regarding different periods of Buddhist history. Currently, Ollama is used as middle-ware for the local experiments, OpenRouter for the cloud-based tests.

The aim of the project is to:

track the improvement of open-weights models in the domain of Buddhist history

to understand to what degree knowledge in that field is encoded in language models and whether there is bias towards certain regions or periods

to identify the currently most knowledgeable open-weights models in the domain of Buddhist history

As of , among the tested local open-weight models qwen3.6:35b 'knows' most about Buddhist history. Among the tested cloud-based open-weights models glm-5.2 and minimax-m3 are best, but with deepseek-v4-pro and gemini-3.6-flash tied close behind. With that (~94%) the best open-weight, cloud-based models have basically saturated the benchmark. The closed, non-open-weights models we included as reference perform in the same ball park.