Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(logs): add alloy + loki #451

Merged
merged 9 commits into from
Jan 6, 2025
Merged

Conversation

0xNineteen
Copy link
Contributor

@0xNineteen 0xNineteen commented Dec 18, 2024

  • mac support
  • linux support

note: alerting requires proper parsing of logs which requires some changes to our logger which ill put into a separate PR

  • logs are stored in logs/sig.log
  • alloy scrapes the logs and pushes to loki
  • loki is a log database which we query from grafana on the explore page (dashboards + alerting coming in future PR -- see note)

TL;DR

Added log aggregation capabilities to the metrics stack using Loki and Alloy.

What changed?

  • Added Loki for log storage and querying
  • Added Alloy for log collection and forwarding
  • Updated Grafana configuration to include Loki as a data source
  • Added /logs to gitignore
  • Consolidated message handle time metrics into the main Gossip dashboard
  • Updated documentation for running with logs

How to test?

  1. Run sig with log output: ./zig-out/bin/sig gossip -n testnet 2>&1 | tee -a logs/sig.log
  2. Start the metrics stack:
    • Mac: docker compose -f compose_mac.yaml up -d
    • Linux: docker compose -f compose_linux.yaml up -d
  3. Access Grafana at localhost:3000 and verify logs are visible in Explore view
  4. Verify metrics dashboards show handle time data correctly

Copy link
Contributor Author

0xNineteen commented Dec 18, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

@0xNineteen 0xNineteen marked this pull request as ready for review December 18, 2024 12:16
@0xNineteen 0xNineteen marked this pull request as draft December 18, 2024 12:16
@0xNineteen 0xNineteen force-pushed the 12-18-feat_logs_add_alloy_loki branch 4 times, most recently from 8eb43d7 to 55efe88 Compare December 18, 2024 22:21
@0xNineteen 0xNineteen force-pushed the 12-18-feat_logs_add_alloy_loki branch from 6ec7912 to f2a4f09 Compare January 2, 2025 19:25
@0xNineteen 0xNineteen marked this pull request as ready for review January 3, 2025 13:17
@0xNineteen 0xNineteen requested review from dnut and Sobeston January 3, 2025 13:17
@0xNineteen 0xNineteen self-assigned this Jan 3, 2025
Copy link
Contributor

@Sobeston Sobeston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a few hitches getting things up, but seems generally good.

metrics/compose_linux.yaml Outdated Show resolved Hide resolved
metrics/loki/loki.yml Show resolved Hide resolved
@Sobeston

This comment was marked as resolved.

@Sobeston

This comment was marked as resolved.

Sobeston and others added 2 commits January 6, 2025 06:40
* fix and clean up metrics

* allow prom to be accessed by external services

---------

Co-authored-by: x19 <[email protected]>
@0xNineteen
Copy link
Contributor Author

Copy link
Contributor

@dnut dnut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I just had a few minor nitpicks. I'm not very opinionated about any of this, so feel free to turn them down.

metrics/README.md Outdated Show resolved Hide resolved
metrics/docker-compose.yml Show resolved Hide resolved
metrics/docker-compose.yml Outdated Show resolved Hide resolved
metrics/docker-compose.yml Outdated Show resolved Hide resolved
src/prometheus/http.zig Show resolved Hide resolved
.gitignore Show resolved Hide resolved
@0xNineteen 0xNineteen requested review from Sobeston and dnut January 6, 2025 19:13
@0xNineteen 0xNineteen enabled auto-merge (squash) January 6, 2025 19:13
metrics/.env Outdated Show resolved Hide resolved
@0xNineteen 0xNineteen requested a review from Sobeston January 6, 2025 21:28
Copy link
Contributor

@Sobeston Sobeston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@0xNineteen 0xNineteen merged commit 74d3c43 into main Jan 6, 2025
8 checks passed
@0xNineteen 0xNineteen deleted the 12-18-feat_logs_add_alloy_loki branch January 6, 2025 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants