Monitoring & Logging
🤖 Generated with Claude Code
⚠️ Not yet reviewed
Co-Authored-By: Claude (noreply@anthropic.com)
Gesamtbild der Monitoring- und Logging-Infrastruktur auf der TOGETHER-Plattform.
Uebersicht​
| Säule | Status | Ziel | Daten |
|---|---|---|---|
| Application Insights | aktiv (alle Apps) | bleibt | Telemetry, Metrics, Traces |
| Strukturierte App Logs | aktiv | CLEF + Shipping via Fluent Bit | TisApplicationLogs_CL |
| IIS Logs | aktiv | W3C via AMA | W3CIISLog |
| Windows Event Logs | aktiv | AMA/DCR | Event |
| Request Logs | aktiv (Reverse Proxies) | bleibt / wird erweitert | .http Dateien pro Cluster |
Application Insights​
Zentrales APM fĂĽr alle TOGETHER-Anwendungen. Integration ĂĽber Tis.Hosting.Extensions.
Registrierung​
builder.AddTisApplicationInsightsTelemetry("my-service-name");
Setzt CloudRoleName auf den ĂĽbergebenen String fĂĽr alle Telemetry-Items.
Automatisches Enrichment​
Der TisTelemetryInitializer ergänzt jedes Telemetry-Item:
| AI Property | Quelle |
|---|---|
tis-correlation-id | ITisCorrelationIdFeature |
benutzer-id | ClaimsPrincipal (gekĂĽrzt) |
makler-id | HttpContext |
oauth2-client-id | ClaimsPrincipal client_id |
tis-session-id | HttpContext |
trace-identifier | HttpContext.TraceIdentifier |
Zusätzlich: User.AuthenticatedUserId, User.AccountId, Session.Id im AI Context.
Serilog Sink​
Der Application Insights Sink ist auf Warning und höher beschränkt (TraceTelemetryConverter). Debug/Info Logs gehen nur in lokale Dateien.
Details zur Konfiguration:
Tis.Hosting.Extensions→src/Tis.Hosting.Extensions.Serilog/_examples/
Correlation IDs​
TisCorrelationId Middleware​
app.UseTisCorrelationId();
Generiert eine kurze ID (8 Zeichen) basierend auf dem W3C Trace Root ID (Activity.Current.RootId) — damit ist die Correlation ID an die Application Insights Operation ID gebunden.
Header: X-Correlation-Id (Default, konfigurierbar via TisCorrelationIdOptions)
Propagation: Die Middleware setzt ITisCorrelationIdFeature auf HttpContext.Features. Serilog-Enricher und TisTelemetryInitializer lesen daraus.
Serilog Properties​
Die Serilog-Variante von AddTisApplicationInsightsTelemetry registriert automatisch 5 Enricher:
| Serilog Property | Quelle |
|---|---|
TisCorrelationId | Correlation ID (8 Zeichen) |
TisBenutzerId | ClaimsPrincipal (gekĂĽrzt) |
TisClientId | ClaimsPrincipal client_id |
TisRequestId | Activity.Current.SpanId |
TisRequestUrl | HttpRequest.GetDisplayUrl() |
Diese Properties sind in allen Log-Sinks verfĂĽgbar und bilden die Grundlage fĂĽr die Cross-System-Korrelation.
Strukturierte Application Logs​
Format​
CLEF (Compact Log Event Format) via Serilog.Formatting.Compact.CompactJsonFormatter.
Dateinamen-Konvention​
logs/Debug-{CloudRoleName}-YYYYMMDD.json
logs/Application-YYYYMMDD.json
Der CloudRoleName im Debug-Dateinamen ist Pflicht — Fluent Bit extrahiert ihn per Lua Script für die CloudRoleName-Spalte in Log Analytics.
Log Shipping​
Fluent Bit (Windows Service) liest CLEF JSON und schickt an TisApplicationLogs_CL via Logs Ingestion API.
App (Serilog CLEF) ──► logs/*.json ──► Fluent Bit ──► TisApplicationLogs_CL
Details:
Tis.Hosting.Extensions/docs/fluent-bit.md
NLog (Legacy)​
Aeltere Apps nutzen Tis.Hosting.Extensions.NLog mit JsonLayout (kein CLEF). Gleiches Muster: Debug/Application Files + AI Sink (Warning+).
Details:
Tis.Hosting.Extensions→src/Tis.Hosting.Extensions.Nlog/_examples/NLog.config.Default
IIS Logs​
W3C IIS Logs werden via Azure Monitor Agent (AMA, Azure Arc Extension) an W3CIISLog in Log Analytics geliefert.
IIS ──► W3C Log Files ──► AMA (Arc Extension) ──► DCR ──► W3CIISLog
Kein Custom Parsing nötig — AMA versteht W3C nativ.
Details:
Tis.Hosting.Extensions/docs/fluent-bit.md→ Abschnitt "IIS Logs (AMA)"
Windows Event Logs​
Windows-Events werden via AMA/DCR an Log Analytics geliefert. Pro Serverrolle (Webserver, App-Server, DB-Server) eine eigene DCR mit konservativer Baseline.
Details und XPath-Beispiele: windows-event-log-baseline.md
Request Logging​
Alle Request Logs verwenden das .http-Format (kompatibel mit VS Code REST Client / JetBrains HTTP Client).
Reverse Proxy Request Logging​
Der DownstreamLoggingMessageHandler loggt HTTP Requests/Responses an Downstream-Services.
Modi​
| Modus | Verhalten | Body Buffering |
|---|---|---|
Never | Kein Logging | Keins |
OnError | Nur bei Non-Success Status oder Exception | FileBufferingReadStream |
Always | Jeder Request/Response | FileBufferingReadStream (32 KB Memory, dann Temp File) |
Konfiguration via YARP Cluster Metadata​
{
"Clusters": {
"my-cluster": {
"Metadata": {
"RequestLogging": "OnError"
}
}
}
}
Log-Datei Organisation​
Logger-Kategorie: RequestLogging.{ClusterId}. Serilog filtert per SourceContext in separate .http-Dateien:
logs/RequestLog-YYYYMMDD.http
Authorization-Header werden redaktiert (nur letzte 5 Zeichen sichtbar).
Registrierung​
builder.AddReverseProxy()
.AddDownstreamMessageLogging(); // Status aus Cluster Metadata
.AddDownstreamMessageLogging("MyName"); // Expliziter Name, Default: Always
Reverse Proxy Telemetry​
Der ReverseproxyTelemetryInitializer (via .AddApplicationInsights() auf IReverseProxyBuilder) ergänzt AI RequestTelemetry:
| Property | Quelle |
|---|---|
reverseproxy-cluster-id | YARP Route Feature |
reverseproxy-destination | Proxied Destination |
reverseproxy-error | IForwarderErrorFeature |
reverseproxy-response-origin | Response Header |
HttpClient Request Logging (Sink Pattern)​
FĂĽr ausgehende HTTP Requests (via IHttpClientFactory). Serialisiert Request/Response im .http-Format und dispatcht an registrierte Sinks.
| Sink | Zweck | Package |
|---|---|---|
FileRequestLogSink | Lokale .http Dateien | Tis.Hosting.Extensions.HttpClientLogging |
BlobRequestLogSink | Azure Blob Storage (WAL + Upload) | Tis.Hosting.Extensions.HttpClientLogging.AzureBlob |
Blob Upload: Write-Ahead Log → Upload mit Retry → Cleanup. Recovery von Leftovers bei Neustart.
SpanId-Korrelation​
Jeder geloggte Request bekommt eine eindeutige SpanId als Dateiname. Default: SpanIdSelector extrahiert die SpanId aus dem traceparent Header (W3C Trace Context), der vom HttpClient auf jeden ausgehenden Request gesetzt wird. Damit korreliert die Log-Datei exakt mit der Application Insights Dependency.
SpanIdSelector kann ĂĽberschrieben werden:
// Default: HttpClient Dependency SpanId (traceparent Header)
o.SpanIdSelector = HttpClientLoggingOptions.DefaultSpanIdSelector;
// Parent Activity (alle Calls in einem Scope teilen sich die SpanId):
o.SpanIdSelector = _ => Activity.Current?.SpanId.ToString() ?? Guid.NewGuid().ToString("N");
Registrierung​
// Logging aktivieren
services.AddHttpClientLogging(o =>
{
o.LogStatus = RequestLoggingStatus.OnError;
o.StorageTemplate = "https://{account}.blob.core.windows.net/{container}/{serviceName}/{spanId}.http";
});
// Sink(s) registrieren
services.AddRequestLogFileSink(o => o.RequestLogsDirectory = "logs/requests");
services.AddRequestLogBlobUpload(o =>
{
o.AccountUrl = "https://{account}.blob.core.windows.net";
o.ContainerName = "tis-request-logs";
o.ServiceName = "my-service";
o.RequestLogsDirectory = "logs/request-wal";
});
// Handler an HttpClient hängen
services.AddHttpClient("downstream")
.AddHttpClientLoggingHandler();
Blob Storage Zugriff​
StorageTemplate setzt ein request-log-url Activity Tag auf jeden Request:
https://{account}.blob.core.windows.net/{container}/{serviceName}/{spanId}.http
Zugriff via:
- az CLI:
az storage blob download --blob-url "<url>" --auth-mode login --file - - Storage Explorer: Deep Link (
storageexplorer://Protokoll)
Blob Storage Konfiguration​
| Setting | Wert |
|---|---|
| Default Access Tier | Cool |
| Lifecycle Policy | delete-after-90-days (90 Tage nach Modification) |
| Auth | Managed Identity (DefaultAzureCredential) |
| RBAC | Storage Blob Data Contributor |
Fachtest Environment​
| Resource | Name | Resource Group |
|---|---|---|
| Subscription | Together-Fachtest | — |
| Log Analytics Workspace | Fachtest-LogAnalyticsworkspace | LogAnalytics-Fachtest |
| Application Insights | Fachtest | LogAnalytics-Fachtest |
| Application Insights (BoaBot) | TIS-Fachtest-BoaBot-AI | DefaultResourceGroup-WEU |
| Application Insights (BoaApi) | BoaApi-Fachtest-ApplicationInsights | LogAnalytics-BoaApi-Fachtest |
| Blob Storage (Request Logs) | tisrequestlogsfachtest | TIS-Fachtest-OnPremises_Servers-WE-RG |
Request Log Container:
| Container | Zweck |
|---|---|
tis-request-logs | Allgemeine Request Logs |
boa-api-request-logs | BoaApi Request Logs (separater Zugriff fĂĽr externes Team) |
Workspace ID: 87f6d081-015f-436f-b58b-add840d4be7a
| Fluent Bit Resource | Name |
|---|---|
| DCE | dce-tis-fachtest-api-gateway |
| DCR | dcr-tis-fachtest-api-gateway (Immutable ID: dcr-ad876b7af565485b9ea7f741ad63d012) |
| Custom Table | TisApplicationLogs_CL |
CLI Zugriff​
az monitor log-analytics query \
--workspace "87f6d081-015f-436f-b58b-add840d4be7a" \
--analytics-query "<KQL Query>" \
-o table
Prod Environment​
| Resource | Name | Resource Group |
|---|---|---|
| Subscription | Together-Prod | — |
| Log Analytics Workspace | Together-Plattform-LogAnalyticsworkspace | LogAnalytics-Prod |
| Application Insights | Together-Plattform | LogAnalytics-Prod |
| Application Insights (CCA9) | CCA9-ApplicationInsights-Prod | TIS-PROD-CCA_Integration-WE-RG |
| Application Insights (BoaApi) | BoaApi-Prod-ApplicationInsights | LogAnalytics-BoaApi-Prod |
| Blob Storage (Request Logs) | tisrequestlogsprod | TIS-PROD-OnPremises_Servers-WE-RG |
Request Log Container:
| Container | Zweck |
|---|---|
tis-request-logs | Allgemeine Request Logs |
boa-api-request-logs | BoaApi Request Logs (separater Zugriff fĂĽr externes Team) |
Workspace ID: 9ee01003-d32e-4110-ab57-1f09be580fc3
| Fluent Bit Resource | Name |
|---|---|
| DCE | dce-tis-prod-app-logs |
| DCR | dcr-tis-prod-app-logs (Immutable ID: dcr-e5ac3e6b6f114941a8a8f8e2ccd8e818) |
| Custom Table | TisApplicationLogs_CL |
CLI Zugriff​
az monitor log-analytics query \
--workspace "9ee01003-d32e-4110-ab57-1f09be580fc3" \
--analytics-query "<KQL Query>" \
-o table
Tabellen-Referenz​
Log Analytics​
| Tabelle | Inhalt | Schema-Quelle |
|---|---|---|
TisApplicationLogs_CL | Strukturierte App Logs (CLEF) | Tis.Hosting.Extensions/docs/fluent-bit.md → Field Mapping |
W3CIISLog | IIS Access Logs | Standard W3C Schema (built-in) |
Application Insights Tabellen​
Standard-Tabellen (AppRequests, AppDependencies, AppTraces, AppExceptions, AppPageViews) folgen dem Azure Monitor Schema. TOGETHER-spezifische Custom Properties auf allen Telemetry-Items:
| Custom Property | Beschreibung |
|---|---|
tis-correlation-id | Korrelations-ID (8 Zeichen, aus W3C Trace Root) |
benutzer-id | Authentifizierter Benutzer (gekĂĽrzt) |
makler-id | Makler-ID aus HttpContext |
oauth2-client-id | OAuth2 Client ID |
tis-session-id | Session-ID |
Details siehe Abschnitt "Application Insights" und "Serilog Properties" oben.
Korrelation zwischen Tabellen​
| Korrelationspfad | SchlĂĽssel |
|---|---|
AI ↔ TisApplicationLogs_CL | OperationId = TraceId (W3C Trace ID) |
AI ↔ TisApplicationLogs_CL | TisCorrelationId (8 Zeichen) |
AI ↔ W3CIISLog | Zeitfenster + URL-Matching |
KQL Beispiele​
Application Logs nach CorrelationId​
TisApplicationLogs_CL
| where TisCorrelationId == "abcd-1234"
| order by TimeGenerated asc
Fehler pro Service (letzte 24h)​
TisApplicationLogs_CL
| where TimeGenerated > ago(24h)
| where Level == "Error"
| summarize count() by CloudRoleName
| order by count_ desc
Cross-Table: AI + IIS korrelieren via Zeitfenster​
let timeWindow = ago(1h);
let aiErrors = AppRequests
| where TimeGenerated > timeWindow
| where Success == false
| project TimeGenerated, OperationId = OperationId, AppRoleName = AppRoleName, Url = Url;
let iisRequests = W3CIISLog
| where TimeGenerated > timeWindow
| where scStatus >= 500
| project TimeGenerated, sSiteName, csUriStem, scStatus;
aiErrors
| join kind=inner iisRequests on $left.TimeGenerated == $right.TimeGenerated
Cross-Table: AI + Application Logs via Trace ID​
let traceId = "abcdef1234567890abcdef1234567890";
AppRequests
| where OperationId == traceId
| union (
TisApplicationLogs_CL
| where TraceId == traceId
)
| order by TimeGenerated asc