Fig. 01 · Professional · Tacbot.ai
Enterprise AI Platform
An LLM orchestrator, a self-hosted small model and a per-tenant vector index powering summaries, RCA, search and insights.
- Project
- Enterprise AI Platform
- Role
- Software Engineer
- Period
- 07/2025 - present
- Context
- Professional · Tacbot.ai
- Stack
- Python · FastAPI · LlamaIndex · Milvus · llama.cpp · AWS Bedrock · OpenLIT
- Outcomes
- LLM orchestrator routing each use case between a self-hosted small model and AWS Bedrock
- About 40% lower AI inference cost for backend operations
- Per-tenant vector search with structure-aware chunking and cross-encoder reranking
Problem
A case-management platform for enterprise IT operations receives a constant stream of vendor email, alerts and inventory changes. Making that useful means parsing messages, summarising long threads, drafting root-cause timelines and answering questions about past cases - for many tenants at once, without sending every task to an expensive hosted model and without one tenant’s data ever reaching another’s prompt.
Constraints
- Tenant isolation. Context for a prompt must come only from that tenant’s own records - in the database and in the vector index.
- Cost. Backend tasks run continuously, so the per-call cost of a hosted large model adds up quickly.
- Quality where it is visible. Insights shown to users still need a strong model.
- Case data is not prose. Timelines, numbered RCA items and email threads lose their meaning when cut at arbitrary lengths.
- Cold start. A new tenant should not start with an empty history.
Architecture
The figure follows an AI task from the case platform to the model that answers it.
- 01
Every AI task enters through one service
The case platform sends AI work - summaries, RCA timelines, AI search, form assist, SLA extraction, dashboard questions - to a single AI service, so prompts, context and results are handled in one place.
- 02
An orchestrator picks the model per use case
Each use case has its own model configuration. Routine work - email parsing, summarisation, RCA, embeddings and reranking - runs on a small model inside the platform's environment; case-intelligence features call a hosted large model on AWS Bedrock.
- 03
Fallback is a permission, not a reflex
A hosted model is used for a small-model use case only when that use case allows fallback. Every call passes a shared token-bucket rate limit and is recorded with tokens, latency and throughput, and monthly usage thresholds trigger notification emails before costs surprise anyone.
- 04
Retrieval built for case data
Cases, alerts and inventory are split by their structure - timelines, numbered RCA items, email threads - into overlapping chunks in a per-tenant collection. A search takes 20 candidates, drops those below a similarity threshold, reranks the rest with a cross-encoder on the small model and keeps the best five.
- 05
Years of email become cases
Historical onboarding reads a mailbox's history in bulk and turns past threads into cases, so a new tenant starts with its context already in place.
- 06
Every model call can be inspected
OpenLIT, deployed across environments, records OpenTelemetry traces of model calls alongside the orchestrator's own usage records, so latency, tokens and cost can be examined per environment and per use case.
Read the diagram as text
Components
- Mailbox history (external, not owned)
- Years of past vendor email in a customer's mailbox, read in bulk when a new tenant is onboarded.
- Case platform
- The multi-tenant case-management platform that requests AI work and shows the results to users.
- Tenant databases
- One database per tenant holding cases, email threads and history.
- AI service
- Python (FastAPI) service for summaries, RCA timelines, AI search, conversational form assist, SLA extraction, natural-language dashboard queries and historical onboarding.
- Vector index
- Milvus with one collection per tenant. Cases, alerts and inventory are chunked by structure and re-synchronised whenever they change.
- LLM orchestrator
- Resolves the model for each use case from configuration, prefers the small model, falls back to a hosted model only where allowed, and rate-limits and meters every call.
- Small language model
- A quantised open model on llama.cpp behind an OpenAI-compatible gateway, serving chat, embeddings and reranking inside the platform's own environment.
- Hosted LLM (AWS Bedrock) (external, not owned)
- A managed large model (Claude Sonnet on AWS Bedrock) used for case-intelligence features.
- LLM observability
- OpenLIT, the open-source OpenTelemetry-native LLM observability platform, deployed and operated across environments.
Flows
- Backend task on the small model: Case platform → AI service → LLM orchestrator → Small language model.
- Case insight from the hosted model: Case platform → AI service → LLM orchestrator → Hosted LLM (AWS Bedrock).
- Prompt grounded in case history: Tenant databases → Vector index → AI service → LLM orchestrator.
- Mailbox history becomes cases: Mailbox history → AI service → Case platform → Tenant databases.
- Model calls traced: LLM orchestrator → LLM observability.
Around this core sit the features built on it: AI search over cases, alerts and inventory; RCA timelines; SLA timestamp extraction; natural-language questions answered as read-only SQL for dashboard widgets; a conversational form assistant that fills difficult fields such as identifier regexes but never writes to the database itself; and an MCP server that lets AI assistants list, search, update, reply to and merge cases through the platform’s API with the user’s own OAuth token.
Key decisions & trade-offs
Route by use case through one orchestrator
Every model call - chat, tool use, embeddings, reranking and natural-language SQL - goes through one orchestrator that resolves the model from per-use-case configuration. Routine backend work runs on the small model; case insights use Claude Sonnet on AWS Bedrock. The alternative, one hosted model for everything, was simpler to build but made cost grow with every email and could not keep embeddings and reranking inside the platform.
Fallback only where a use case allows it
When a small-model use case fails, the orchestrator calls a hosted model only if that use case is configured to allow fallback. Silent automatic fallback would have hidden small-model problems and quietly moved cost back to the hosted model; making it explicit keeps both quality and spend visible. A shared token-bucket limit and per-call usage records (tokens, latency, throughput) apply to every call, and monthly thresholds trigger usage notices.
Self-host embeddings and reranking next to the small model
The same llama.cpp gateway that serves chat also serves embeddings and cross-encoder reranking through OpenAI-compatible endpoints, with CPU and GPU images and autoscaling on Kubernetes. Retrieval therefore has no per-call API cost, and tenant text never leaves the platform’s environment to be embedded.
Chunk by structure, then rerank
Instead of fixed-size splitting, each section is split by what it is - timeline entries, numbered RCA items, email threads - and only long pieces fall back to overlapping word windows (200 words, 40 overlap). Each tenant has its own collection, and every entity’s chunks are replaced as a whole when it changes. Search over-fetches 20 candidates, drops those below a similarity threshold and reranks the rest, trading an extra model call for better context than raw vector order gives the LLM.
Pick the summarisation strategy from the input size
Before summarising, the service estimates the prompt and document tokens. Inputs under 60% of the model’s input budget are summarised in one pass; larger ones use hierarchical tree summarisation. This keeps short cases fast while long email histories still fit the model.
Results
- One orchestrator routes every AI use case between a self-hosted small model and AWS Bedrock, with explicit fallback, rate limiting and usage metering.
- AI inference cost for backend operations fell by about 40%.
- Cases, alerts and inventory are searchable per tenant with structure-aware chunks and cross-encoder reranking.
- Mailbox history can be onboarded into cases in bulk.
What I’d do next
- Move more use cases to the small model, each gated by an evaluation set of real cases so a switch is decided by measured quality rather than by feel.
- Add keyword search alongside vectors for identifiers such as ticket numbers and hostnames, which embeddings match poorly.
- Feed the usage records into per-tenant cost dashboards so routing decisions are visible to the people paying for them.