
How to Profile Rust Performance Using Grafana and Prometheus
With agentic AI writing more code, production observability becomes critical to understanding what your app is actually doing. In this tutorial, I’ll show how to use the hotpath-rs Grafana & Prometheus integration to co…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
With agentic AI writing more code, production observability becomes critical to understanding what your app is actually doing. In this tutorial, I’ll show how to use the hotpath-rs Grafana & Prometheus integration to configure actionable performance metrics for your Rust application. We’ll discuss which signals are worth observing and how to ensure that profiling overhead does not impact your production deployment.
In this article I won’t describe how to get started with Grafana itself. There are numerous tutorials and AI skills dedicated to that. I assume you already have Grafana and Prometheus up and running.
Instead, we’ll focus on extracting metrics that provide useful insights without overwhelming you with distracting information. One issue I had when I started using Grafana was its almost infinite flexibility in how metrics can be queried and displayed. It makes deciding what’s actually worth monitoring difficult.
Consider two functions: A has a p95 of 10μs, while B takes over 5 seconds on average. So A is fast and B is slow? What if A is a low-level processing hotpath called millions of times, while B represents a long-running background job?
It all depends on CONTEXT. Raw function execution times or allocation metrics provide limited value without understanding how they relate to the application’s behavior. With AI writing increasingly more code, that context can be difficult to maintain even when you’re a supposed “author” of the feature.
That’s why in this article I’ll show how to instrument a Rust application in a way that provides useful insights even with limited knowledge of the underlying codebase. And when you do know the codebase well, the same metrics become even more useful.
hotpath-rs makes it extremely easy to expose raw performance metrics through a Prometheus-compatible endpoint. See the Grafana & Prometheus docs for detailed instructions and configuration options. If your Rust app is already instrumented with hotpath, all you need to do is enable the hotpath-prometheus feature flag:
[dependencies] hotpath = "0.28"
[features] hotpath = ["hotpath/hotpath"] hotpath-alloc = ["hotpath/hotpath-alloc"] hotpath-prometheus = ["hotpath/hotpath-prometheus"]
run your process with this flag:
cargo run --features='hotpath,hotpath-alloc,hotpath-prometheus'
and your metrics will be exposed at the default 6772 port:
curl http://127.0.0.1:6772/metrics | head
# TYPE hotpath_build_info gauge
hotpath_build_info{hotpath_version="0.28.3",rustc_version="1.98.0",profile="debug",os="macos-aarch64"} 1
# HELP hotpath_uptime_seconds Seconds since profiling started.
# TYPE hotpath_uptime_seconds gauge
hotpath_uptime_seconds 3.309838291
# HELP hotpath_function_calls_total Total calls of each instrumented function, including calls skipped by time sampling.
# TYPE hotpath_function_calls_total counter
hotpath_function_calls_total{function="hotpath::cmd::console::app::App::new"} 1
hotpath_function_calls_total{function="hotpath::cmd::console::app::data::App::handle_data_response"} 8
Now point your Prometheus scraper at the endpoint and you’re good to go!
This could be the end of the tutorial. But read on if you’re interested in how hotpath-rs provides context-rich metrics that are easy to reason about.
I’ve described my approach to optimizing Rust applications in the Rust performance profiling guide, so check it out to improve your context window. The core idea is to apply a Pareto principle mindset to performance optimization: focusing on high-level layers can take 5% of the effort while accounting for 95% of the potential improvements.
Before migrating to Rust, I worked as a Ruby on Rails performance consultant for ~5 years and used the same approach when optimizing RoR backends. See My Recipe for Optimizing Performance of Rails Applications for more details.
This layered observability approach translates directly to how a useful Grafana dashboard can be structured. hotpath-rs lets you monitor different performance layers separately while using each one to provide context for the others.
That’s enough “performance philosophy” for now. Let’s look at some code and charts.
The sample metrics below come from the production deployment of hotpath-backend, the Axum app powering this blog and hotpath Cloud app.
Let’s start with SQL query performance. Database interactions are often among the biggest bottlenecks in backend applications. hotpath-rs offers SQL performance tracing for popular SQL libraries. The instrumentation is agnostic to the underlying database engine, so it works with PostgreSQL, SQLite, and others.
The following Grafana query shows where your app spends its database time. Each bar represents the percentage of total SQL execution time spent by a single query, accounting for both how slow the query is and how often it runs:
sort_desc( topk( 10, 100 * sum by (query) ( histogram_sum( increase(hotpath_sql_duration_seconds[$__range]) ) ) / on() group_left() sum( histogram_sum( increase(hotpath_sql_duration_seconds[$__range]) ) ) ) )
Alternatively, you can use the following query to show only queries that cross a defined slowness threshold, e.g. p95 > 500ms, making it easy to spot candidates for optimization:
sort_desc( histogram_quantile( 0.95, sum by (query) ( increase(hotpath_sql_duration_seconds[$__range]) ) ) > 0.500 )
Context-aware SQL performance metrics
The metrics above can also be extracted directly from the database, for example with tools like pg-extras-rs. You don’t need to run a dedicated profiler for that.
Where hotpath-rs really starts to show its value is in contextualizing those metrics. If you’re using Axum for your backend, you can analyze each SQL query in the context of the route that executed it.
Let’s start with a chart showing which Axum endpoints contribute most to the application’s total processing time:
sort_desc( topk( 10, 100 * sum by (route) ( histogram_sum( increase(hotpath_server_duration_seconds[$__range]) ) ) / on() group_left() sum( histogram_sum( increase(hotpath_server_duration_seconds[$__range]) ) ) ) )
Now we can add a selector to inspect exactly which SQL queries are executed by each endpoint, and how much time each of them adds to a single request:
sort_desc(
topk(
10,
sum by (query) (
histogram_sum(
increase(hotpath_sql_duration_seconds{route="$route"}[$__range])
)
)
/
scalar(
sum(
increase(hotpath_server_scoped_requests_total{route="$route"}[$__range])
)
)
)
)
$route is a Grafana query variable populated with label_values(hotpath_server_requests_total, route).
With this setup in place, you can easily discover which SQL queries meaningfully contribute to response time and which can be safely ignored, even without an in-depth knowledge of the underlying codebase.
N+1 queries are one of the most common causes of slowdowns in backend applications. An endpoint loads a list of records and then runs a separate query for each one. This can be fast with a few rows in development but gets progressively slower as production data grows. Because each individual query is fast, it may never show up in a slow query report.
Since hotpath-rs knows which Axum route executed each query, we can calculate how many SQL queries a single request triggers on average:
sort_desc( topk( 10, sum by (route) ( increase(hotpath_server_sql_calls_total[$__range]) ) / ( sum by (route) ( increase(hotpath_server_scoped_requests_total[$__range]) ) > 0 ) ) )
Endpoints with an unexpectedly high number of queries per request are prime N+1 candidates. You can add a threshold, e.g. > 5, to show only the endpoints that cross it.
Next to SQL queries, HTTP calls are often another major contributor to a backend application’s response time.
A useful graph can show how many HTTP calls each endpoint triggers on average. Multiple blocking HTTP calls can cause a problem similar to N+1 queries, with response times quickly growing as the number of calls increases:
sort_desc( topk( 10, ( sum by (route) ( increase(hotpath_server_http_calls_total[$__range]) ) > 0 ) / ( sum by (route) ( increase(hotpath_server_scoped_requests_total[$__range]) ) > 0 ) ) )
The next chart shows how much time a request to each endpoint spends waiting on blocking HTTP calls. A route that makes only one outbound call per request can still be the slowest if that call hits a slow external API:
sort_desc( topk( 10, sum by (route) ( histogram_sum( increase(hotpath_http_duration_seconds{route!=""}[$__range]) ) ) / ( sum by (route) ( increase(hotpath_server_scoped_requests_total[$__range]) ) > 0 ) ) )
route!="" filters out HTTP calls made outside of a request, e.g. by background jobs.
You can compare this HTTP overhead with the slowest endpoint response times to identify which blocking calls are worth optimizing or moving to the background.
With the hotpath-alloc feature enabled, hotpath-rs also tracks how many bytes are allocated while serving each request. The following chart shows how much memory a request to each endpoint allocates on average:
sort_desc( topk( 10, sum by (route) ( increase(hotpath_server_alloc_bytes_total[$__range]) ) / ( sum by (route) ( increase(hotpath_server_scoped_requests_total[$__range]) ) > 0 ) ) )
If your app is using more memory than expected, this overview lets you quickly spot potential culprits. Endpoints allocating megabytes per request, e.g. by loading whole tables or large payloads into memory, are the first place to look.
Once you’ve found a suspicious endpoint, you can narrow it down further. The following chart shows which instrumented functions contribute to its memory usage, measured as average bytes allocated per request:
sort_desc(
topk(
10,
sum by (function) (
increase(hotpath_function_route_alloc_bytes_total{route="$route"}[$__range])
)
/
scalar(
sum(
increase(hotpath_server_scoped_requests_total{route="$route"}[$__range])
)
)
)
)
By default, allocations are counted exclusively: each function reports only the memory it allocates itself, excluding nested instrumented calls. Here, comparisons_controller::show does not include the bytes allocated by comparisons::build and render_show, so each bar points directly to where the allocations happen. You can switch to cumulative accounting with HOTPATH_ALLOC_CUMULATIVE=true, but it double-counts recursive functions.
The same approach works for response times. Let’s start with the slowest endpoints, ranked by their p95 response time:
sort_desc( topk( 10, histogram_quantile( 0.95, sum by (route) ( increase(hotpath_server_duration_seconds[$__range]) ) ) ) )
And for a selected endpoint, we can check which instrumented functions contribute to its response time, measured as average time spent per request:
sort_desc(
topk(
10,
sum by (function) (
increase(hotpath_function_route_duration_seconds_total{route="$route"}[$__range])
)
/
(
sum by (function) (
increase(hotpath_function_route_timed_calls_total{route="$route"}[$__range])
) > 0
)
*
sum by (function) (
increase(hotpath_function_route_calls_total{route="$route"}[$__range])
)
/
scalar(
sum(
increase(hotpath_server_scoped_requests_total{route="$route"}[$__range])
)
)
)
)
Dividing by timed_calls and multiplying by calls keeps the numbers correct when time sampling is enabled.
Keep in mind that this timing data is inclusive. A function’s time includes time spent in all the instrumented functions it calls, so comparisons_controller::show above includes, for example, comparisons::build. Future releases of hotpath-rs will implement exclusive time attribution, showing only the time spent in each function’s own body.
Channels are one of the primary ways Rust programs communicate across threads and often represent important data flows in your application. The hotpath::channel! macro lets you extract detailed performance metrics for any channel:
Arguably one of the most us
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。