Grafana + Loki Local Setup

This guide explains how to use the local Grafana and Loki stack for log aggregation during development.

What Are Loki and Grafana?

Loki and Grafana are separate but complementary products from Grafana Labs:

Loki - Log Storage Engine

Purpose: Stores and indexes logs (like PostgreSQL for logs)
Does: Receives logs via HTTP, stores data on disk, serves LogQL queries
Port: 3100 (HTTP API)
Analogy: The database that holds your logs

Grafana - Visualization Layer

Purpose: Query and visualize data (like pgAdmin for logs)
Does: Web UI for querying, creating dashboards, alerting
Port: 3200 (Web UI)
Analogy: The admin panel where you view your logs

Why Two Separate Images?

Different responsibilities - Loki = backend storage, Grafana = frontend UI
Modular architecture - Grafana can query many data sources (Prometheus, databases, etc.)
Scalability - Run multiple Loki instances with one Grafana
Independent updates - Upgrade one without affecting the other

The Flow


Your App (pino-loki)
    │
    ├─► Console (pretty-printed)
    ├─► File (logs/api.log, rotated daily)
    └─► HTTP POST
         ▼
    ┌─────────┐
    │  Loki   │  ← Stores logs (like a database)
    │ :3100   │
    └────┬────┘
         │ LogQL queries
         ▼
    ┌─────────┐
    │ Grafana │  ← Visualizes logs (like pgAdmin)
    │ :3200   │
    └─────────┘
         ▼
    Your Browser

Quick Start

1. Start the Services


# Start Loki and Grafana (along with other infrastructure)
docker compose up -d loki grafana
 
# Or start everything:
docker compose up -d

2. Configure Your Application

Add to your .env file or environment:


LOKI_ENDPOINT=http://localhost:3100

For apps running inside Docker, use the service name:


LOKI_ENDPOINT=http://loki:3100

3. Access Grafana

URL: http://localhost:3200
Auth: Anonymous access enabled (no login required in dev)

Using the Logger

Basic Usage

The logger automatically sends logs to Loki when LOKI_ENDPOINT is set:


import { createLogger } from "@leadmetrics/logger";
 
const log = createLogger({ 
  service: "my-service",
  lokiEndpoint: process.env.LOKI_ENDPOINT,
});
 
log.info("Application started");
log.error({ err, userId: 123 }, "Failed to process request");

Log Levels


log.trace("Very detailed debugging");
log.debug("Debug information");
log.info("General information");
log.warn({ deprecated: true }, "Warning message");
log.error({ err }, "Error occurred");
log.fatal({ err }, "Critical failure");

Querying Logs in Grafana

1. Open Explore

Navigate to: http://localhost:3200/explore

2. Example LogQL Queries

All logs from a specific service:


{service="api"}

Error logs only:


{service="api"} |= "level\":50"

Logs containing specific text:


{service="api"} |~ "payment|subscription"

Logs by environment:


{env="development"}

Filter by multiple labels:


{service="api", env="production"}

Count rate of errors:


rate({service="api"} |= "level\":50" [5m])

3. Log Levels

trace: level 10
debug: level 20
info: level 30
warn: level 40
error: level 50
fatal: level 60

Filter by level:


{service="api"} |= "level\":50"  # Errors only
{service="api"} | json | level >= 40  # Warnings and above

Service Names

Your application uses these service names (configured via LOGGER_SERVICE env var):

api - Main API server
api-pgcallbacks - Payment gateway webhooks
server-notifications - Notification worker
server-billing - Billing worker

Docker Services

Both services are required and work together:


# docker-compose.yml
services:
  loki:
    image: grafana/loki:3.0.0      # Storage backend
    ports: ["3100:3100"]            # API for receiving logs
    
  grafana:
    image: grafana/grafana:10.4.0  # Visualization frontend
    ports: ["3200:3000"]            # Web UI
    depends_on: [loki]              # Needs Loki to query from

Why both images?

Loki = Database (stores the logs)
Grafana = UI (queries and displays the logs)
They’re separate so Grafana can also connect to other data sources (Prometheus metrics, PostgreSQL, etc.)

Log Outputs

With LOKI_ENDPOINT set, logs are written to three destinations:

Console (pretty-printed in dev, JSON in prod)
File (logs/{service}.log, rotated daily)
Loki (for centralized querying)

Troubleshooting

Logs not appearing in Grafana

Check Loki is running:


docker ps | Select-String loki
curl http://localhost:3100/ready

Check LOKI_ENDPOINT is set:
```
console.log(process.env.LOKI_ENDPOINT);
```
View Loki logs:
```
docker logs leadmetrics-loki
```

Test Loki directly:


curl http://localhost:3100/loki/api/v1/labels

Grafana shows “No data”

Wait a few seconds (pino-loki batches every 5 seconds)
Adjust time range in Grafana (top right)
Try broader query: {service=~".+"}

Configuration Files

Loki config: loki-config.yml
Grafana datasource: grafana-datasources.yml
Docker Compose: docker-compose.yml

Ports

Loki: 3100 (HTTP API)
Grafana: 3200 (Web UI, mapped from internal 3000)

Production Considerations

In production, you would:

Use managed Grafana Cloud or self-hosted cluster
Set GF_AUTH_ANONYMOUS_ENABLED=false (require login)
Configure retention policies in Loki
Use object storage (S3, GCS) instead of filesystem
Enable authentication on Loki endpoints
Use TLS/HTTPS for all connections

Stopping Services


# Stop just Loki and Grafana
docker compose stop loki grafana
 
# Stop and remove (data is preserved in volumes)
docker compose down
 
# Remove everything including data
docker compose down -v