SANDI Solr

Abstract

1Integration Architecture

SANDI's AI functionality is delegated to three independent microservices, each exposing a lightweight HTTP API. The search and index modules communicate with these services via configurable URLs in the properties file. Any service can be swapped between local and cloud-backed implementations with a URL change — the interface contract is identical.

Search and indexing pipeline with OpenAI services

OpenAI adapter services

Embedding model options for emb5

2Configuration Reference

All OpenAI settings are controlled via environment variables. In the provided deployment, these are stored in a .env file in the deployment root directory, which docker-compose reads automatically at startup. Each variable has a documented default defined in docker-compose.yml; the .env file only needs to contain overrides and the required API key.

Environment variables

Minimal .env file

Full .env file with all options

3Advantages

No GPU required — minimal hardware

All three OpenAI adapter services are thin Flask/Gunicorn wrappers that forward requests to the OpenAI API and return results. They consume negligible CPU and RAM (under 100 MB each). Local alternatives — emb3, emb4, emb6, llm2, rer1 — each load multi-gigabyte neural network weights and require a CUDA-capable GPU with several GB of VRAM. With the OpenAI services, a complete SANDI stack runs comfortably on a laptop, a basic VPS, or any machine with internet access.

Ideal for learning, development, and testing

Developers can explore hybrid search, RAG pipelines, reranking, and query expansion without GPU hardware investment. Switching models, adjusting embedding dimensions, or comparing retrieval strategies is a configuration change followed by a container restart. The full SANDI stack — Solr, ZooKeeper, search, index, NLP, embedding, LLM, and reranker — starts with a single docker compose up command.

Access to the newest and most capable models

OpenAI continuously releases improved models. The embedding service can use text-embedding-3-large with state-of-the-art retrieval quality, the LLM service defaults to o4-mini (a reasoning model capable of precise structured output), and the reranker uses gpt-4o-mini. Upgrading to a newly released model requires only an environment variable change — no model downloads, no compatibility testing, no quantization decisions.

Unlimited and elastic capacity

OpenAI's infrastructure scales automatically. There is no need to manage memory pressure, batching queues, or model sharding. Batch embedding requests send multiple texts in a single API call, and throughput grows with OpenAI's rate limits rather than local hardware constraints. For small-to-medium collections this is effectively unconstrained.

Zero-maintenance deployment

No model downloads, no CUDA driver management, no quantization decisions, no out-of-memory crashes during indexing. The only prerequisite is an active OpenAI API key. Deployments start in seconds and require no ongoing model maintenance.

Azure OpenAI and compatible API support

All three services honour an OPENAI_BASE_URL override, enabling transparent switching to Azure OpenAI Service (which offers private endpoints and data residency options), a self-hosted OpenAI-compatible server (Ollama, vLLM, LM Studio), or another cloud provider. No SANDI application code changes are required — only the base URL and model name in .env.

4Disadvantages and Limitations

Network latency on every AI call

Each embedding, LLM completion, and reranking request crosses the public internet. Round-trip latency of 200–800 ms per call is typical. For interactive search, users notice slower response times compared to sub-millisecond local model inference. Bulk indexing of large document corpora is especially impacted: every text chunk requires a network round-trip to the embedding service, and the IndexingJobScheduler (which checks for work every second) will spend most of its time waiting on network I/O rather than processing.

Reduced throughput for batch indexing

Local GPU models can embed hundreds of text chunks per second. OpenAI's API, even with batch calls, is constrained by network bandwidth, serialization overhead, and per-account rate limits (tokens per minute and requests per minute). Indexing millions of documents will take significantly longer than with a local embedding service and may require explicit rate-limit handling, retry logic with exponential backoff, and throughput monitoring.

Data privacy and security risks

All document content, query text, and search results are transmitted to OpenAI's servers to obtain embeddings and LLM responses. Every text chunk from every indexed document is sent to the embedding API during indexing; every user search query is sent to the embedding API at search time; LLM calls include the full query and retrieved document passages for RAG generation. For use cases involving confidential, proprietary, legally privileged, or personally identifiable data (GDPR, HIPAA, financial, legal), this mode may be unacceptable or require explicit data processing agreements with OpenAI.

Ongoing API cost

Every embedding, completion, and reranking call is billed per token by OpenAI. Embedding costs scale with corpus size (every chunk is embedded once at index time) plus query volume (each search query is embedded at runtime). LLM and reranker costs scale with search query volume and result set size. For large-scale or high-traffic deployments, API costs can exceed the amortized hardware cost of running local models.

Requires continuous internet connectivity

The OpenAI services are non-functional without a live connection to api.openai.com. Any network outage will cause all AI-dependent features — embedding generation, semantic search, spell checking, query expansion, RAG answers, and reranking — to fail. SANDI's basic BM25 text search through Solr continues to function, but the hybrid and intelligent search features become unavailable.

Rate limits and service interruptions

OpenAI enforces per-account rate limits on tokens per minute and requests per minute. Under heavy indexing load, requests may be throttled (HTTP 429), causing SANDI's indexing jobs to stall or fail. OpenAI API outages and scheduled maintenance windows directly affect SANDI availability when using cloud services.

Vendor dependency and model deprecation

Relying on OpenAI introduces dependency on a third-party provider's roadmap and pricing. Models can be deprecated (e.g., text-embedding-ada-002 was succeeded by the text-embedding-3 family). When the embedding model used at index time is no longer available, the entire corpus must be re-indexed with the replacement model, since vector similarity scores are not comparable across different model families or dimension counts.

5OpenAI vs. Local Models — Comparison

6Recommendations by Use Case

7Conclusion

The OpenAI integration adapters make SANDI immediately accessible to any developer or team that wants to explore semantic search, RAG, and hybrid retrieval without GPU infrastructure. A single OPENAI_API_KEY in the .env file unlocks the full SANDI feature set — vector embeddings, hybrid KNN + BM25 search, spell checking, query expansion, RAG answer generation, and relevance reranking — on commodity hardware. The trade-offs are real: every AI operation crosses the network, all indexed content reaches OpenAI servers, and costs scale with usage. For production systems with sensitive data, large corpora, or strict latency requirements, the local GPU-backed services remain the appropriate choice. The two modes are interchangeable — transitioning from cloud to on-premises is a configuration change, not an architecture change.