■FIG. A.4 · Self project · Python Slack bot
Slack Data Copilot
Most data bots fail the same way: they answer a question they should have refused, or execute SQL they should have routed to docs. This one is built around a router that abstains on principle. The train-free confidence-abstention router scores 92% on a 150-query set (138/150); a fine-tuned SmolLM2-135M variant reaches 94% (141/150). The train-free scorer is deterministic, so the lab below is a verbatim port — the confidences you see are the real algorithm’s, not a mockup’s.
route
SQL
conf 0.990
sql_score = 9.0 · rag_score = 0.0
min_signal 1.5 · min_margin 0.8
reason: sql=9.0 rag=0.0
train-free (runs live above) · 92% (138/150)
| true→sql | 57 / 0 / 3 |
| true→rag | 1 / 51 / 8 |
| true→refuse | 0 / 0 / 30 |
fine-tuned SmolLM2-135M (LoRA) · 94% (141/150)
| true→sql | 57 / 0 / 3 |
| true→rag | 1 / 55 / 4 |
| true→refuse | 1 / 0 / 29 |
confusion cells ordered pred sql / rag / refuse · the fine-tuned path needs a server runtime; its numbers are quoted from the committed benchmark, not simulated here
DOWNSTREAM OF THE ROUTER
SQL-bound queries hit a self-healing text-to-SQL path: schema linking first, a read-only database role, statement timeouts, and at most three error-fed repair retries. Retrieval-bound queries fuse BM25 with dense retrieval (pgvector / BGE) via Reciprocal Rank Fusion, rerank with a cross-encoder, and answer with inline clause-level citations.
OPS IS PART OF THE SYSTEM
Golden-set evaluations block regressions in CI, Prometheus tracks p95 latency and cost per query, prompt-injection guardrails sit at the boundary, LangSmith traces every hop, and Ragas gates the retrieval quality. The router’s own benchmark — confusion matrices for both variants — is rendered above from the committed results file.
Prev: FIG. A.3 · Deaf Audio AssistantNext: FIG. A.5 · Language-Based Audio Retrieval