Streamlining Log Management in Go with Grafana Loki Integration

Simple Log Management in Go for Beginners with Grafana Loki

Log management is vital for understanding application behavior and troubleshooting issues. In this article, we’ll explore how to streamline log management in Go applications using Grafana Loki.

Setting Up Your Environment

The things to prepare:

1. Go: The author will be working with Go version 1.20, but other versions are also suitable.
2. Grafana: Grafana is an open-source platform for monitoring and observability. You can use the Grafana Cloud version. The free tier provides ample resources to start learning.
3. Loki: It is included inside Grafana Cloud. Loki helps to organize and manage system logs from various sources. With its indexing capabilities, it simplifies log retrieval and analysis. Designed to handle multiple applications concurrently.
4. FluentBit: will act as the tool to transmit your logs to Grafana Loki.

Implementing Logging in Your Go Code

First, we will create a simple go app that will produce some line of logs for every second, the log will be stored into a file.

A simple logging app

you can try running it, it will produce a log file default.log inside folder log like this:

log/default.log

{"level":"info","trace":"aa2391b8-1dd9-11ef-8c17-628869e732c8","time":"2024-05-29T23:36:57+07:00","message":"Random number: 46"}
{"level":"info","trace":"aabc4cbe-1dd9-11ef-8c17-628869e732c8","time":"2024-05-29T23:36:58+07:00","message":"Random number: 40"}
{"level":"info","trace":"ab551002-1dd9-11ef-8c17-628869e732c8","time":"2024-05-29T23:36:59+07:00","message":"Random number: 35"}
{"level":"info","trace":"abeddbfc-1dd9-11ef-8c17-628869e732c8","time":"2024-05-29T23:37:00+07:00","message":"Random number: 22"}

Integrating Grafana Loki for Log Aggregation

It’s time to push the log to Grafana Loki, first, we set up Fluent Bit to read the log from our file, then pushes it to the Grafana cloud

Set up Grafana Loki Key
Go to Grafana web -> Log in -> My Account and navigate to Loki -> Send Logs

There, you will see the box containing your URL & UserID. Then obtain your API Key by clicking Genarate Now.

Now, you are ready to set up Fluent Bit config.

Set up fluent bit
You can install according to your setup. Ensure the installation was successful by checking the version from the terminal.

~ » fluent-bit --version 
Fluent Bit v3.0.5
Git commit:

Set up fluent bit config
Create a file named config.conf inside the folder where your main.go located.

[SERVICE]
    Flush        3
[INPUT]
    Name         tail
    Path         <YOUR_LOG_FILE_ABSOLUTE_PATH>
    Tag          golang-app
    Path_Key     filename
[OUTPUT]
    Name         loki
    Match        golang-app
    Host         <YOUR_GRAFANA_LOG_URL>
    URI          /loki/api/v1/push
    Port         443
    Labels       job=fluentbit,service_name=golang-app,filename=$filename
    tls          on
    tls.verify   on
    http_user    <YOUR_GRAFANA_USER>
    http_passwd  <YOUR_GRAFANA_API_KEY>

Time to Test

First, run the fluent bit by using this command:

~/golang-app » fluent-bit -c config.conf 

Fluent Bit v3.0.5
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
...

Run golang app, and ensure the log is running by tailing the log file at log/default.log

~/golang-app » tail -f log/default.log   
{"level":"info","trace":"f49b3234-1e0e-11ef-a4e4-628869e732c8","time":"2024-05-30T05:58:25+07:00","message":"Random number: 21"}
{"level":"info","trace":"f533f7da-1e0e-11ef-a4e4-628869e732c8","time":"2024-05-30T05:58:26+07:00","message":"Random number: 53"}
{"level":"info","trace":"f5ccc71c-1e0e-11ef-a4e4-628869e732c8","time":"2024-05-30T05:58:27+07:00","message":"Random number: 60"}
{"level":"info","trace":"f6658ede-1e0e-11ef-a4e4-628869e732c8","time":"2024-05-30T05:58:28+07:00","message":"Random number: 65"}

Now, go to Grafana Cloud Dashboard -> Explore and choose grafanacloud-<username>-log

Put the following query:

{job="fluentbit"}

You can see that your log is successfully pushed to Grafana Loki by seeing the log listed there.

Try exploring the query, you can learn further like for aggregating the value, filtering, extracting or others. For example:

{job="fluentbit"} | json | line_format `{{.log}}` | json