Monitoring Concept — Executive Summary
Status: Draft · Author: Martin Riemann · Date: 2026-07-20 Full document: monitoring-concept.md
🤖 Generated with Claude Code
✅ Reviewed by Martin Riemann
Co-Authored-By: Claude (noreply@anthropic.com)
This proposal defines a single, shared approach to monitoring, alerting, and dashboarding for the TOGETHER platform. It replaces today's collection of independently-grown solutions with one model that every team applies the same way, so it is always clear where a service's signals go and when a person is actually notified.
The platform's applications run mostly on-premises (Windows/IIS and SQL Server) and send their telemetry to Azure Application Insights. Collecting data is therefore already in good shape. What this proposal addresses is everything built on top of that data: deciding what is worth an alert, where to look at the data, how people are notified, and how they respond.
The Problem
Monitoring capabilities were added over time, incident by incident, by whoever was closest to each problem. The result works in places but is inconsistent, and the gaps carry real cost:
- Notifications are sent through several unrelated channels. Alerts today reach people via a mix of Azure action groups, a Logic App, a Teams channel, and an in-house app that sends its own e-mail. Because no one can see the whole picture, it is hard to be sure an important alert reaches someone, hard to quieten a noisy one, and easy to end up with both duplicate alerts and blind spots.
- Alerts are created by hand in the Azure portal. They are not defined in version control, so they cannot be reviewed, copied to another environment, or restored if lost — and they are effectively invisible until they fire, or fail to. Hand-creation is also how the current sprawl accumulated: each one-off addition is untracked and unowned.
- A lot of collected data is never acted on. Windows event data, for example, is ingested but almost nothing raises an alert from it, so events such as a cleared audit log, an account lockout, or an expiring certificate can pass unnoticed — even though the data is already being collected, stored, and paid for.
- There is no shared standard. No common way to classify a signal, no shared severity levels, no standard dashboards, and no agreed definition of a healthy service. Each new service invents its own approach, and new team members have no template to follow.
Fachtest — the partner-facing test environment — is a specific example: its applications do send telemetry to Application Insights, so it is not unmonitored, but it currently has no alerting at all, so a failure that blocks a partner would notify no one. The fix is proportionate alerting, not more data collection.
The overall effect is missed or slowly-diagnosed incidents, monitoring that no one clearly owns, friction when onboarding new services, and difficulty evidencing managed monitoring for regulatory purposes (e.g. DORA).
The Approach
The concept rests on a few well-established operational principles, stated plainly so every team can apply them consistently:
- Separate alerting from diagnostics. Every signal has one of two jobs: to alert (notify a person because something needs action now) or to explain (sit on a dashboard you read once you are already investigating). Keeping these apart avoids both alert fatigue and blind spots.
- Measure the same four things everywhere. Each service is instrumented for the four "golden signals" — latency, traffic, errors, and saturation — giving one consistent dashboard shape and alerting posture across the platform.
- Alert on symptoms, not causes. Notify a person when something is actually affecting users (or predictably about to) — requests failing, a deadline missed. Underlying causes such as high CPU, a filling disk, or a slow dependency go on dashboards to explain an alert, rather than paging someone on their own.
- Keep database queries on-premises. Cloud services cannot reach the on-premises database directly, and this concept does not change that. Anything that needs to query the database runs on-premises and publishes its results outward, rather than exposing the database to the cloud.
Target Model
The concept adds a small number of layers on top of the existing data collection:
- Detect — Azure Monitor alerts (metric, log, and availability), later refined by service-level objectives.
- Visualise — one dashboard tool for the whole platform (choice discussed below).
- Notify — one small, named, version-controlled set of Azure Monitor action groups delivering to Teams and e-mail, replacing today's many ad-hoc paths.
- Respond — short runbooks linked from each alert, with clear ownership and a regular review of the alert set.
Severity and delivery
Severity (how serious) and delivery (how people are told) are kept separate. Delivery has three modes: Push (interrupts a person, via a Teams channel — Prod-Probleme for Sev1, Monitoring - PROD for Sev2 — and e-mail), Digest (a single daily summary, posted to Together-Standup), and Dashboard (looked at when chosen). Response is best-effort — there is no formal on-call today, and none is promised to customers.
| Severity | Meaning | Delivery |
|---|---|---|
| Sev1 — Critical | Customer-facing outage, or a security/data emergency | Push |
| Sev2 — High | Partial or degraded service, or an imminent Sev1 | Push |
| Sev3 — Medium | No customer impact yet; trends or recurring errors | Daily digest + dashboard |
| Sev4 — Low/Info | Informational or housekeeping | Dashboard only |
Production carries the full model; Fachtest is treated proportionately — it interrupts people only for a partner-blocking failure and otherwise uses the digest and dashboards.
Application health
Each service declares, in code, what "healthy" means for it — using standard ASP.NET Core health checks wired centrally, exposing a liveness, a readiness, and a detailed health endpoint. A set of simple rules keeps health checking from generating noise: trivial liveness checks, evaluation on a timer rather than per request, cached results, health traffic filtered out of telemetry, and a single published health signal.
Runbooks
Every alert links a short runbook — a half-page guide that tells whoever responds what the alert means, how to confirm it is real, what to do first, and when to escalate. This matters because response is best-effort: with no formal on-call, whoever is available picks up an alert, and a runbook lets someone who is not the original author act correctly and quickly. Runbooks are also the practical test of whether something should alert at all — if no short runbook can be written for a signal, it belongs on a dashboard, not in a notification. They are kept in version control next to the alert definition and written promptly: the first time an alert fires without one, writing it is part of closing that incident.
Key Decisions
Two choices shape the rollout and need a platform/cost decision.
Which dashboard tool
All options read the data that already exists (Azure Monitor metrics, Log Analytics, and Application Insights — all queryable with KQL). They differ mainly in cost, in how comfortably dashboards can be managed as code, and in whether they can reach the on-premises database.
| Option | Advantages | Disadvantages / cost |
|---|---|---|
| Azure Managed Grafana (cloud) | Full Grafana experience and plugins; strong dashboards-as-code; queries Azure Monitor, Log Analytics and App Insights | Cannot reach the on-premises database; ≈ $60 / month per instance plus ~$6 per active user / month, billed per subscription — so both are paid twice across the separate Prod and Fachtest subscriptions |
| Self-hosted Grafana (on a VM) | Same Grafana experience and dashboards-as-code; queries both the Azure data (KQL) and the on-premises database directly; no per-user licence | The team runs, patches, secures and backs up a VM |
| Azure Workbooks + Dashboards (built-in) | No added licence cost; native to Azure, so definitions deploy as Bicep alongside app code | Less polished for operational dashboards; the JSON is awkward to edit by hand |
Dashboards owned by each application. A goal is for each application to keep its dashboards in its own repository and deploy them through its own pipeline, rather than from a central store. Both tool families allow this — Grafana dashboards are JSON that can be provisioned from a repo, and Azure Workbooks/Dashboards are Azure resources whose JSON deploys via Bicep. Grafana offers the smoother as-code experience.
Lean: the pricing (≈ $60/month per instance plus ~$6 per active user, billed per subscription — so both doubled) makes Managed Grafana hard to justify in the current two-subscription setup. That leaves self-hosted Grafana — the best experience and the only option that can also read the on-premises database, at the cost of running a VM — or Azure Workbooks + Dashboards as the no-extra-cost, Azure-native option. A short trial building the same dashboard in each would settle it.
The role of the in-house monitoring app
An existing on-premises app (tis.monitoring.dashboard) runs a set of scheduled checks — some against the on-premises database, some against data that is already available in the cloud — and reports on them. Its future depends on the dashboard-tool choice, because the deciding factor is where each check has to run:
- Some of its checks read data the cloud can already see. Those can move into Azure Monitor and the chosen dashboard tool regardless, so the app no longer needs to do them.
- If we choose cloud (Managed) Grafana, the app's remaining role is to provide the information the cloud cannot obtain directly: it runs the database queries on-premises and publishes the results outward for the dashboards and alerts to use. In that shape it becomes a small, headless publisher rather than a full application with its own UI and e-mail.
- If we choose self-hosted Grafana, which can query the on-premises database itself, the app could be retired entirely and its checks re-created directly in Grafana.
In every case, its current habit of sending its own e-mail and holding credentials in source would be replaced by the shared notification path and proper secret storage.
Roadmap
| Phase | Focus |
|---|---|
| 0 — Discovery | Inventory every existing alert and notification path |
| 1 — Foundations | Agree the classification, severity, naming, and the standard action groups |
| 2 — Health standard | Add the health-check standard centrally and onboard a first service |
| 3 — Visualisation | Choose the dashboard tool and build a golden-signal dashboard template |
| 4 — Consolidation | Re-home existing alerts as code; bring Fachtest under proportionate alerting |
| 5 — SLOs | Define starter service-level objectives per service |
| 6 — As code + governance | Everything defined as code, with a regular review cadence |
Next Steps
- Review this proposal with the stakeholders (Platform/Operations, Architecture, Development Leads, Security).
- Decide the two open choices — the dashboard tool, and the future of the monitoring app.
- Run the discovery phase to produce a full inventory of today's alerts and notification paths.
- Instrument one pilot service end to end as the reference implementation.