tools

DeepLM Insights

Free, open-source Grafana dashboards for SLURM GPU clusters.

DeepLM Insights is a free, open-source monitoring stack for SLURM GPU clusters. It gives teams job-level visibility into utilization, power and efficiency.

Dashboards

Five pre-built Grafana dashboards, backed by Prometheus and Cassandra:

  • Job Insights: per-job CPU and GPU utilization, memory, priority and power draw.
  • System Overview: cluster-wide power, GPU temperatures, fan speeds, and per-node CPU and memory.
  • Live Jobs: a real-time active job monitor with 5-second refresh.
  • Historical Jobs: duration analysis, completion rates and CPU hours by user.
  • Checkpoint Analysis: sync vs async checkpoint comparison, with stall time and overhead quantified.

How it works

Insights integrates with SLURM through prologue and epilogue hooks. When a job starts, the hooks push metadata and telemetry to a lightweight Flask API, which exposes Prometheus-formatted metrics to Grafana.

text
SLURM Hooks -> Flask API -> Cassandra + Prometheus -> Grafana

Quick start

bash
git clone https://github.com/DeepLM/Insights.git
cd Insights
cp .env.example .env
docker compose up -d

Grafana is available at localhost:3000. You need Docker and Docker Compose. NVIDIA Base Command Manager is optional but recommended for precise GPU power and utilization metrics. The full setup is in the repository.