Instrumenting Your Golang App

Learn how to enhance your Golang application’s observability by integrating with Grafana Tempo.

Have you ever developed an application that suddenly slows down, and you can’t figure out why?

You can try by utilizing OpenTelemetry Protocol (OTLP), which collects and exports telemetry data. You may gain detailed visibility into your application’s workflow. This lets you trace requests across services, identify slow endpoints, and find performance bottlenecks.

This article delves into leveraging the OTLP exporter to seamlessly push traces, enabling detailed insights into your application’s performance using Go and Grafana Tempo Cloud.

Before we start, you’ll need:

1. Go environment, mine is 1.22.3
2. Grafana Cloud Account, you can make one here

In this example, we’ll create an app that has 3 main flow: create order, do payment and notify the payment. In each process, we‘ll create a trace point to monitor each process performance.

Every traces produced will be pushed into OTLP via HTTP Exporter from Grafana and pushed into Tempo

Create a new API Key for OpenTelemetry integration in Grafana:

Go to My Account -> OpenTelemetry -> Configure

There, You’ll see the Base URL (mine is otlp-gateway-prod-ap-southeast-0.grafana.net)

To get the Basic Auth token, you need to convert userId:apiToken to Base64. in Linux, for example, you can use this command:

echo -n "userId:apiToken" | basic64

or you can find the encoded Basic Auth token inside tab Environment Variables

Create Go project and add these 3 files

This Go code initializes OpenTelemetry for tracing in a service. Here’s a brief overview:

1. Function NewHttpExporter: Creates an HTTP exporter for sending traces to an OTLP endpoint. It includes configurations for the endpoint URL and Grafana Cloud basic authentication.
2. Function createNewSpan: Creates a new span within the provided context and tracer. If no tracer exists, it creates one. Then, it starts a new span with the specified service name.

Note: Replace placeholders like <OTLP_BASE_URL> (without https://) and <GRAFANA BASIC AUTH (username:api_key)> with actual token as instructed above.

Then, add this file as utility functions:

The logInfo function logs information with trace and span IDs, while the logError function logs errors.

Then, wire it all to the main function file:

Now run your app.

Explore the Trace Data in Grafana Tempo Cloud

For every 10s, the app will execute the dummy order creation process and will push the trace into Grafana Tempo Cloud, you can explore the traces in Grafana Cloud Dashboard -> Explore and choose grafanacloud-{username}-traces (by default)

There, you can see each process for each row, marked by Trace Id that generated by Go OTLP SDK. You can also see that the app log with Trace ID & Span ID.

You can try to pick one Trace Id from the app console log and search it in Grafana Tempo

Then, put the traceId in TraceQL field

There, you can see the detailed trace of every process (createOrderAndPay, pay & payNotify).

Try to explore it, add an internal OTLP Collector, add more function such as mark some process as error, add span attribute and further, propagate the trace into another external application.