Search Indexes

Status: Roadmap (Q3 2025)

Support for document stores and search indexes for ranking pipelines.

Planned Infrastructure

Document Stores

MongoDB:

from senren import DocumentStore

mongo = DocumentStore(
    name="product-catalog",
    type="mongodb",
    storage_gb=100,
    regions=["aws:us-east-1"],
)

DocumentDB (AWS-managed MongoDB):

documentdb = DocumentStore(
    name="catalog",
    type="documentdb",
    storage_gb=100,
    regions=["aws:us-east-1"],
)

Search Indexes

Elasticsearch:

from senren import SearchIndex

es = SearchIndex(
    name="product-search",
    type="elasticsearch",
    memory_gb=16,
    storage_gb=500,
    regions=["aws:us-east-1", "gcp:us-central1"],
)

OpenSearch (AWS-managed Elasticsearch fork):

opensearch = SearchIndex(
    name="search",
    type="opensearch",
    memory_gb=16,
    storage_gb=500,
    regions=["aws:us-east-1"],
)

Vector Search (Semantic Search)

Qdrant:

from senren import VectorDatabase

qdrant = VectorDatabase(
    name="embeddings",
    type="qdrant",
    memory_gb=32,
    dimensions=768,  # For BERT embeddings
    regions=["aws:us-east-1"],
)

Weaviate:

weaviate = VectorDatabase(
    name="semantic-search",
    type="weaviate",
    memory_gb=32,
    dimensions=1536,  # For OpenAI embeddings
    regions=["gcp:us-central1"],
)

Use Cases

Product Search and Ranking

1. Store product catalog in MongoDB
2. Index in Elasticsearch for full-text search
3. Store product embeddings in Qdrant for semantic search
4. Hybrid search: keyword + vector similarity

Document Retrieval (RAG)

1. Store documents in document store
2. Generate embeddings and store in vector database
3. Semantic search for retrieval-augmented generation

Current Workaround

For simple semantic search, you can store embeddings in Redis:

# Store embedding as binary blob
import numpy as np

embedding = np.array([0.1, 0.2, ...])  # 768-dim vector
r.set("embedding:doc123", embedding.tobytes())

# Retrieve and search (requires client-side similarity computation)

Limitations: No built-in similarity search, no indexing.

Timeline

Q3 2025: Document stores, search indexes, and vector databases.

See the roadmap for details.