Overview

SANDI Solr is a comprehensive search and indexing platform built on Apache Solr. It provides separate APIs for searching documents and indexing content with support for multiple document formats, scheduled processing, and advanced search features including semantic search, reranking, and RAG (Retrieval-Augmented Generation).

The system consists of two main components:

• Search API - Handles search queries and returns results
• Index API - Manages document indexing, scheduling, and administration

Search API

The Search API provides powerful search capabilities with support for various query types, result formatting, and advanced features.

Endpoints

POST /search

Performs a search query using JSON request body.

Request Body Parameters:

Example Request:

{
    "requestId": "2c38e64d-19ce-4db2-acde-8df40edbf447",
    "clientId": "TREC_001",
    "pageSize": 10,
    "pageNumber": 1,
    "searchQuery": "Case for \"Samsung Galaxy\" with mirror",
    "filterQuery": "",
    "resultFields": "_id,id,title,content,score,rscore,_chunks",
    "group": false,
    "groupFields": "id",
    "facet": false,
    "facetFields": "id",
    "sortFields": "",
    "precision": "medium",
    "legacy": false,
    "rerank": true,
    "rag": true,
    "synonyms": false,
    "dym": true,
    "collapse": false,
    "exact": false
}

GET /search

Performs a search query using URL parameters. All parameters from the POST endpoint can be passed as URL parameters.

Example Request:

Response Format

Successful Response:

{
    "requestId": "2c38e64d-19ce-4db2-acde-8df40edbf447",
    "status": "SUCCESS",
    "message": null,
    "dymQuery": "case for samsung galaxy with mirror",
    "ragAnswer": "Based on the search results, here are some cases for Samsung Galaxy phones with mirror features...",
    "foundResults": 1247,
    "start": 0,
    "took": 156,
    "results": [
        {
            "_id": "doc123",
            "id": "product_456",
            "title": "Samsung Galaxy S24 Mirror Case",
            "content": "Premium mirror case for Samsung Galaxy...",
            "score": 0.95,
            "rscore": 0.87,
            "_chunks": ["chunk1", "chunk2"]
        }
    ]
}

Error Response:

{
    "requestId": "abc123",
    "status": "ERROR",
    "message": "Client not found"
}

Search Features

Index API

The Index API manages document ingestion, processing, and scheduling of indexing jobs.

Indexing Interface

POST /index

Indexes documents directly via API.

Request Body:

{
    "requestId": "req123",
    "clientId": "CLIENT_001",
    "data": [
        {
            "id": "doc1",
            "title": "Document Title",
            "content": "Document content...",
            "metadata": {
                "category": "news",
                "date": "2025-01-20"
            }
        }
    ]
}

POST /index/json

Indexes JSON documents with flexible schema.

Request Body:

{
    "requestId": "req124",
    "clientId": "CLIENT_001",
    "data": [
        {
            "title": "Product Review",
            "description": "Excellent product...",
            "rating": 5,
            "tags": ["electronics", "mobile"]
        }
    ]
}

Scheduler Interface

The scheduler manages automated indexing jobs with support for various document sources and formats.

Job Types

POST /schedule/index

Schedules an indexing job.

Request Body:

{
    "requestId": "schedule123",
    "clientId": "CLIENT_001",
    "jobType": "JSONL",
    "directory": "/sandi/documents/data/",
    "fileExtensions": ".jsonl,.json",
    "forceReindexing": true,
    "scheduledTime": "2025-01-21T10:00:00",
    "cron": "0 0 2 * * ?",
    "jobId": "daily-import-001"
}

Parameters:

• jobType: One of the supported job types (see table above)
• directory: Source directory or URL
• fileExtensions: Comma-separated file extensions to process
• forceReindexing: Whether to reindex existing documents
• scheduledTime: When to start the job (ISO format)
• cron: Optional cron expression for recurring jobs
• jobId: Optional custom job identifier

Job Management

Available Endpoints:

• GET /schedule/jobs - Returns all indexing jobs
• GET /schedule/jobs/{jobId} - Returns specific job details
• GET /schedule/jobs/status/{status} - Returns jobs by status
• POST /schedule/jobs/{jobId}/cancel - Cancels a job
• GET /schedule/stats - Returns job statistics

Admin Interface

The admin interface provides management capabilities for clients, collections, and configurations.

Client Management

• GET /admin/clients - Lists all clients
• GET /admin/clients/{clientId} - Gets specific client details
• POST /admin/clients - Creates or updates a client
• DELETE /admin/clients/{clientId} - Deletes a client

Create Client Request:

{
    "clientId": "NEW_CLIENT",
    "name": "Client Name",
    "collection": "client_collection",
    "active": true,
    "createCollection": true,
    "configuration": "default_config"
}

Collection Management

• GET /admin/collections - Lists all Solr collections
• GET /admin/collections/{name} - Gets collection details
• POST /admin/collections - Creates a new collection
• DELETE /admin/collections/{name} - Deletes a collection

Configuration Management

• DELETE /admin/configuration/{name} - Deletes a configuration

Error Handling

Common HTTP Status Codes

• 200 OK: Request successful
• 400 Bad Request: Invalid request parameters
• 404 Not Found: Resource not found
• 500 Internal Server Error: Server error

Error Response Format

{
    "requestId": "req123",
    "status": "ERROR",
    "message": "Detailed error description"
}

Common Errors

• Client not found: Invalid clientId
• Client is inactive: Client exists but is disabled
• Invalid page size: pageSize exceeds maximum (10000)
• RequestId is required: Missing requestId parameter
• Invalid job type: Unsupported jobType in scheduler

Best Practices