I get 215 GitHub notifications a day, so I built a bot to triage them

When your open source project gets popular, the real work starts.

I counted everything that happened in my repositories in the last 30 days, leaving out what I did myself:

That’s about 215 notifications a day, every day, on top of building the things people are filing issues about. Each one is an email, a badge on the bell, a push on my phone. I can’t read all of them, and the important ones get buried under the rest: a pull request waiting for my sign-off ends up under fifty comments about a bug that was fixed last week.

So I built Copilot Triage. It reads every issue, discussion, and pull request, answers the reporter when it has something useful to say, and puts everything on a board sorted by what it needs from me. I turned off GitHub’s email and open the board, a GitHub Project, instead.

It’s an independent open source project, not made by GitHub. The name comes from what it runs on: your GitHub Copilot allowance by default, or any model through RubyLLM.

Every issue and pull request lands on the board, in the column for what it needs from me:

Each card has a priority and a one-line next step, so I know what to do before I open it. Urgent issues get assigned to me, and my review is requested on the pull requests that need it.

When someone opens an issue, the agent investigates with six read-only tools:

It picks which tools to use and in what order, and then decides what would help. Usually that’s a label and nothing else. When it can help, it posts one reply: the answer from the docs, the release that fixed it, the existing issue it duplicates, the one question the report is missing, or a short summary of a long report.

Discussions get answered from the docs, and a discussion that’s really a bug report moves to an issue. For pull requests, it requests a Copilot code review when a change deserves one, reads the verdict when it arrives, and checks the contribution guidelines, like opening an issue first or attaching screenshots for UI changes. Merging is always up to me.

My first attempt used GitHub Agentic Workflows. For RubyLLM, the compiled workflow was 2,035 lines of YAML: tool gateways, agent jobs, a threat detector, safe-output jobs, and failure reporting. Then it started opening issues about its own failures and commenting about its detector failing, which was more email, not less.

The first versions of Copilot Triage went the other way: short, bounded prompts, separate routes for answering from files, looking things up, and comparing duplicates, and keyword ranking and regex rules guessing whether a follow-up mattered. Every new kind of report needed another route. In 0.6 I replaced all of that with one agent that chooses its own searches. Copilot Triage is now a RubyLLM agent with a system prompt and the six tools above:

class TriageAgent < RubyLLM::Agent instructions { system_prompt } tools do [SearchRepository, SearchIssues, ListReleases, SearchChanges, ReadEvidence, SubmitDecision] .map { |tool| tool.new(toolbox) } end end

The agent picks what to search and reads what it finds, with a budget of 12 searches and reads per report. Its last call submits a decision. The model can’t change anything on GitHub: there’s no shell, no file access, and no tool that writes. Ruby checks the decision, re-reads the report, verifies the evidence the model cited, and only then labels, comments, closes, or moves the card. If the report changed while the agent was working, it’s left alone.

When something goes wrong, the error goes to the GitHub Actions job summary. It doesn’t open an issue about it.

Before 0.18 I went through everything triage did across my repositories in one week. Labels were right 95% of the time. Replies were much worse: only a third of them actually helped anyone. The rest were things like “Thanks for the report!”, a second and third diagnostic question after the reporter had already answered the first, or a link to an issue that was already linked in the thread.

Each of those replies is another notification, for the reporter and for me. A wrong label costs me a click to fix, but a wrong reply costs the reporter’s time and some goodwill toward the project. A triage bot that adds noise defeats the point.

So it now replies much less. A thank-you gets no answer. When a reporter answers its question, it doesn’t ask another one unless the answer reveals something new. After I comment, it replies at most twice more, then stops and moves the card to Decide. A clear report gets labels and no comment.

Changing a prompt to make a bot quieter is easy. Knowing it still answers when it should is the hard part, and for that I needed evaluations.

The evaluation is a RubyLLM::Evaluation with 32 cases. Most are modeled on real reports from my repositories, and each one says what should happen:

A case passes only with the expected action, the text the reply must contain (and none it must not), at least the evidence reads it needs, and no more model calls than its budget.

It runs in two modes. Offline, it replays recorded tool calls and decisions through the real tools and publisher against fixture GitHub data. That costs nothing, runs on every change, and catches plumbing bugs, but it can’t tell me whether a model behaves. Live, the real model investigates the same fixtures, with publishing switched to dry run. I run that before every release and before changing the default model. Because the evaluation also shows up in the RSpec suite, each case fails like an ordinary test.

The last full comparison ran each case twice:

GPT-5 mini replies to almost everything. Claude Haiku stays quiet even when the docs have the answer. GPT-6 Luna is newer and cheaper, but misses more and replies more when it shouldn’t, so newer isn’t automatically better for a job like this. Gemini 3.6 Flash never finished a case within Copilot’s 90-second limit, and small local models failed most cases even with the same tools. You can run it against any model:

EVAL_LIVE=true TRIAGE_MODEL=gpt-5.6-luna bundle exec rake ruby_llm:eval

33 releases in five weeks

I released 33 versions between September 15 and October 17, almost all of them tuning. Running it on busy repositories found problems no evaluation had, and many of those reports became evaluation cases. Some of what I learned:

By default it runs on your Copilot allowance, at about a cent or less per report at GitHub’s listed prices. When the allowance runs out, it can switch to an API key and keep going.

You can also skip Copilot and set engine: rubyllm to run the same agent on any provider RubyLLM supports: your company’s OpenAI or Anthropic key, OpenRouter, Amazon Bedrock, Azure, or a local model on Ollama.

I also make Chat with Work, an AI agent for work, and you can build bots like this one there without writing code. A Chat with Work bot can start when an issue or pull request is opened, commented on, or labeled, and keep one conversation going per issue. Once an admin installs its GitHub App on your repositories, it reads the issue and the repository’s code, then labels, comments, or closes. You decide which of those it does on its own and which wait for your approval. The issue triage recipe sets it up in a few clicks. It can also open draft pull requests with fixes, and the same bots can run on schedules, emails, Slack messages, and more.

Here’s the triage bot on an example issue, set to ask first for everything: it read the issue, checked the labels, searched for duplicates, and is waiting for me to approve the labels and its reply.

It’s a GitHub Action. You need a token, a triage.yml file with your labels and project rules, and a short workflow file. It runs on all my repositories, from RubyLLM to Spotifast.