Monitoring and Logs

Operations monitoring is divided into three layers: infrastructure status, product runtime status, and security audit.

Services and Queues

Continuously check frontend, api, worker, sandbox-worker, beat, PostgreSQL, Redis, and Qdrant. Pay special attention to API 5xx, database connections, Redis queue backlog, Knowledge Base processing failures, Sandbox timeouts/disk limits, and whether Beat runs repeatedly.

Queues are split by responsibility, so identify which one is backed up before acting:

docker compose ps
docker compose logs -f api worker sandbox-worker beat frontend

Kubernetes uses commands such as kubectl get pods and kubectl logs -f deployment/api.

Database Health

PostgreSQL underpins statistics, the lexical retrieval channel, and audit, so beyond connection state, watch:

• The most expensive statements in pg_stat_statements (the supplied image preloads it with track=all) to find queries missing an index.
• Invalid indexes: a failed CONCURRENTLY build leaves an INVALID index behind and IF NOT EXISTS will not rebuild it.
• Dashboard and statistics endpoint latency: slowness with low CPU usually means a sequential scan instead of a covering index.

SELECT query, calls, mean_exec_time, total_exec_time
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

SELECT c.relname
FROM pg_class c
JOIN pg_index i ON i.indexrelid = c.oid
WHERE NOT i.indisvalid;

For index creation, cleanup, and verification steps, see Database Maintenance and Index Tuning.

Observability console

The built-in observability console lives under Monitoring > Observability (route /dashboard/observability) and is backed by GET /api/v1/admin/observability/*, which requires the admin:dashboard:access permission. There is no Prometheus-style /metrics endpoint and no built-in Prometheus, Grafana, Datadog, or ELK integration; external monitoring must collect logs and poll the health endpoint plus this API.

The endpoints fall into two groups with different parameter semantics:

/system/workers returns the Celery-side queue and worker state: status, worker_count, active_tasks, reserved_tasks, scheduled_tasks, plus the pending length of all five queues (default, knowledge, workflow, agent, sandbox) and a per-task pending breakdown. /system/health returns CPU, memory, disk, database, Redis, and that worker snapshot, and appends the result to a Redis snapshot list that /system/trend reads back. Every observability payload is cached for 30 seconds.

The basic health check GET /api/v1/health requires no authentication and is meant for container healthchecks and external liveness probes; it returns only {"status": "healthy"} with no dependency detail.

Product Metrics

The workspace dashboard and Agent/Workflow monitoring pages provide conversation counts, message counts, Tokens, response time, tool calls, Workflow run counts, success rate, failure counts, and average duration. Filter by team and resource permissions; different users may see different data scopes.

Audit and Alerts

The audit log records the operator, resource, action, time, and before/after snapshots. The notification system can send in-app or external notifications for document processing failures, Workflow failures, Agent publishing, quota near/over limits, abnormal logins, and account locks.

Log Security

Logs may contain URLs, model errors, user input summaries, tool parameters, and resource IDs. In production, never expose log endpoints unprotected; mask, retain, and archive according to organization policy.