System Architecture

Clouisle uses a layered architecture separating frontend, backend, and infrastructure: a Next.js frontend for UI and SSR, a FastAPI backend for Agents/Workflows/Knowledge Base, Celery for async tasks, and PostgreSQL, Redis, and Qdrant for business data, caching/queues, and vector retrieval.

Technology Stack

Request Flows & Execution Lifecycle

1. Basic Request Flow

Browser → (optional external reverse proxy / Ingress) → Next.js SSR (node server.js) → FastAPI API → PostgreSQL → Response

2. Chat Request Flow & Asynchronous AgentRun Lifecycle

Clouisle implements a durable, decoupled AgentRun execution architecture for conversational agents:

User Message ──► FastAPI (POST /api/v1/agents/{agent_id}/chat/runs) ──► Redis Queue (Celery agent)
                       │ (202 Accepted, run_id)                           │
                       ▼                                                  ▼
                  SSE Stream                                  Celery Worker (run_agent_task)
     (GET /api/v1/agents/{agent_id}/chat/runs/{id}/stream)                │
                       ▲                                         Acquires Conversation Lock
                       │                                                  │
              Redis Event Buffer ◄──────────────────────────── AgentLoop ReAct Cycle
              (SSE Event Pub/Sub)                                         │
                                                               ┌──────────┴──────────┐
                                                               ▼                     ▼
                                                        Tool Calling /        Context Compaction
                                                        RAG Retrieval         (Micro & Macro)
                                                               │                     │
                                                               ▼                     ▼
                                                        [ask_user pause]      Watermarked Summary
                                                        (Status: WAITING)     (Tokens kept in budget)

1. Durable AgentRun: Starting a run dispatches a Celery task and immediately returns 202 Accepted with a run_id. The task is routed to the agent queue and consumed by the main worker; if you rewrite the worker's -Q and drop agent, runs stay in queued forever without any error, and users only see "queued" (queue mapping: Deployment Architecture). The client establishes an SSE stream (/chat/runs/{run_id}/stream) and can disconnect or reconnect (from_sequence) without interrupting background tool execution. Tasks on the agent queue set no automatic retries — model calls and tool execution have side effects, so a retry would execute them twice; runs interrupted by a dead worker are reclaimed by the Beat-scheduled sweep-lost-agent-runs (every 2 minutes) and marked interrupted rather than replayed.
2. AgentLoop ReAct Engine: Manages iterative reasoning and tool-calling cycles (max_iterations, timeouts, and credential injection).
3. Three-Level Context Compaction:
   • Micro Compaction: Automatically truncates and summarizes oversized tool output payloads and raw base64 data to preserve token budgets.
   • Macro Compaction: When conversation tokens cross the configured trigger budget, the engine computes turn boundaries, runs an LLM summarization over older messages, inserts a context_summary message with watermark tracking, and retains recent turns in verbatim form.
   • Preflight & Recovery Guards: Token headroom reservations and automatic retries on context-length provider errors.
4. Human-in-the-Loop (ask_user) Suspension & Resumption: When ask_user is called, AgentLoop enters AgentRunStatus.WAITING with pending question metadata. Submitting answers via POST /chat/runs/{run_id}/answers takes an ORM row lock, validates payloads, and resumes the active run seamlessly.

3. Workflow Engine Execution Flow

Trigger Event / User Input ──► FastAPI (POST /workflows/{id}/runs) ──► Celery Queue (workflow)
                                                                              │
                                                                              ▼
                                                                   WorkflowOrchestrator
                                                                              │
                                                                   Parse DAG Graph Definition
                                                                   (ExecutionPlan, filter comments)
                                                                              │
                                                                   Execute Node Pipeline
                                                                   (LLM, Code, Condition,
                                                                    Iteration, Pause...)
                                                                              │
                                                      ┌───────────────────────┴───────────────────────┐
                                                      ▼                                               ▼
                                               Node Execution                                  Pause Request
                                            (ExecutionContext Flow)                          (Status: WAITING)
                                                      │                                               │
                                                      ▼                                               ▼
                                               Completed Result                              Resume with Feedback

1. Graph Topological Planner (ExecutionPlan): Validates DAG dependencies, resolves parallel executable batches, and strips non-executable canvas documentation nodes (NON_EXECUTABLE_NODE_TYPES = {"comment"}).
2. Modular Node Executors: Handlers for llm, condition, code (Sandbox), iteration/loop, template, variable_aggregator, knowledge_retrieval, tool, agent, and sub_workflow.
3. Pause & Resume Handlers: Nodes requiring human review (pause) persist a WorkflowPauseRequest and suspend execution until approved or filled via /runs/{id}/pause-requests/{req_id}/submit.

Scalability

The frontend and API are stateless and scale horizontally. The main worker (python main.py worker -c 4 -Q default,agent,knowledge,workflow) consumes all four of default, agent, knowledge, and workflow; to scale one queue independently, start an extra worker subscribed to a single queue (for example -Q agent), but a worker must consume default and agent at all times, otherwise notifications, cleanup tasks, and Agent conversations silently pile up in the queue. The code sandbox is a separate fifth queue, sandbox, consumed by the dedicated sandbox-worker; scaling it does not affect knowledge base or workflow processing. Celery Beat must run exactly one replica — it is a bare scheduler without a database lock, and multiple replicas would deliver scheduled tasks repeatedly. For the full queue mapping, periodic Beat tasks, and scaling commands, see Deployment Architecture.

Security and Multi-tenancy

• Authentication: password (bcrypt), SSO (OAuth2/OIDC/SAML/CAS), API Keys (clou_ prefix + Authorization: Bearer).
• Authorization: JWT validation + permission checks (resource:action / admin:resource:action); Super Admin bypasses.
• Data isolation: resources belong to a team, queries are filtered by team membership automatically; regular users only see their own conversations (see Teams & Security).
• Audit: records the actor, operation, and before/after snapshots, with retention days and archiving.

Performance and Caching

User sessions default to 30 days (controlled by the session_timeout_days site setting), with a 8-day JWT fallback token; site settings are not cached and are read directly from the database on every lookup.

There is no single global rate-limiting middleware: each protection is implemented per scenario, with its own boundary:

Health Checks

• /api/v1/health — public basic health check (used by the container HEALTHCHECK).
• /api/v1/admin/observability/system/health — admin observability health (CPU/memory/disk/database/Redis/worker), requires admin:dashboard:access.

See also:

• Deployment Architecture — 3 images / 5 services topology and volumes
• Agent, Workflow, Knowledge Base — core resource concepts
• Teams & Security — multi-tenancy and permission boundaries
• Environment Variables Reference — startup-level configuration