DMS AI — AI-Powered Document Management System
Enterprise DMS solving the 'keyword search problem' in document retrieval: users cannot find what they need because they don't know the exact words in the document. Semantic vector search via Qdrant + LLM embeddings enables natural-language document discovery at enterprise scale, built on a Go + Python + gRPC microservices architecture.
// Key Challenges
- Keyword Search Failure: Traditional search forces users to remember exact document terminology. For regulatory and policy documents, this results in critical documents being 'hidden in plain sight'.
- Concurrent Processing: Uploading and indexing large batches of documents (100+ files) blocked other users during indexing operations.
- Cross-Service Latency: AI embedding generation (Python/ML) and document storage (Go/Postgres) needed to communicate without introducing response time bottlenecks.
- Real-Time Collaboration: Multiple team members needed to see document status updates (indexed, processing, failed) in real time without polling.
// Approach & Solutions
1. Semantic Vector Search via Qdrant
Documents are chunked, embedded via OpenRouter LLM models, and stored as high-dimensional vectors in Qdrant. User queries are embedded at search time and matched by cosine similarity, returning semantically related documents regardless of exact keyword overlap.
2. gRPC Async Document Processing Pipeline
Document upload triggers an async gRPC stream to the Python AI service for non-blocking embedding generation. The Go gateway returns immediately to the user while background processing completes, eliminating indexing-induced UI blocking.
3. WebSocket Real-Time Status Updates
Persistent WebSocket connections push document processing status updates (uploaded → embedding → indexed → ready) to all connected team members in real time, eliminating the need to refresh or poll for document availability.
// Tech Stack
| Technology | Reason |
|---|---|
| Go (Golang) API Gateway | Go's goroutine concurrency model handles thousands of simultaneous document requests with minimal memory overhead — critical for enterprise environments with 50+ concurrent users. |
| Qdrant Vector Database | Qdrant's HNSW indexing enables sub-100ms semantic search across 100,000+ document vectors — performance that PostgreSQL full-text search cannot approach for semantic similarity queries. |
| gRPC (Go ↔ Python) | Binary protocol with Protocol Buffers reduces inter-service serialization overhead by 60-70% vs REST/JSON for high-frequency AI embedding generation calls. |
| Kubernetes | Horizontal pod autoscaling for the Python AI service handles document batch spikes without manual intervention, maintaining consistent embedding throughput during peak upload periods. |
// Results & Business Impact
- <100ms - Semantic Search Response (Qdrant HNSW index returns relevant documents in under 100ms across 100K+ vectors)
- 60-70% - Inter-Service Overhead Reduction (gRPC + Protobuf vs REST/JSON for AI embedding pipeline)
- 0ms - UI Blocking During Indexing (Async gRPC pipeline processes documents without blocking any user operations)
- Natural - Search Interface (Users describe what they need in plain language — no keyword memorization required)
// Project Gallery
