Case Study
Research Paper Discovery & Semantic Search
AI-Powered Scientific Literature Search Engine
The Problem
Challenge & Background
Researchers struggle to discover relevant scientific literature across vast corpora. Keyword searches miss semantically related papers that use different terminology.
The Solution
System Approach
Built an end-to-end paper discovery platform combining arXiv scraping, PostgreSQL metadata storage, and 384-dimensional Sentence-Transformers embeddings stored in a FAISS index with Mistral 7B structured insights.
Architecture
System Architecture & Data Flow
arXiv Scraper
Paper ingestion
PostgreSQL
Relational storage
MiniLM-L6
384-dim embeddings
FAISS Index
Vector search
FastAPI
REST endpoints
Redis Cache
Query caching
Mistral 7B
Paper analysis
Search Results
Ranked output
Engineering Decisions
Trade-offs & Technical Choices
Why hybrid FAISS + PostgreSQL architecture?
→ PostgreSQL provides structured filtering (authors, categories, publication dates) while FAISS handles high-dimensional vector similarity — isolating search and metadata workloads.
Why Redis caching layer?
→ Semantic embeddings and vector similarity lookups are computationally intensive. Redis caching delivers sub-500ms response times for repeated queries.
Why MiniLM-L6 embeddings?
→ 384-dimensional dense vectors achieve exceptional semantic similarity performance with minimal memory footprint and fast inference.
Capabilities
Key Features & Engineering Scope
- ▹Automated arXiv paper ingestion and parsing
- ▹Semantic similarity search via FAISS
- ▹384-dimensional Sentence-Transformer embeddings
- ▹FastAPI REST backend with Pydantic schemas
- ▹Redis caching for sub-500ms repeated queries
- ▹Batch paper insight extraction via Mistral 7B
- ▹Relational schema indexing 13,000+ papers
Outcomes
Results & Impact
Research Papers Indexed
Vector Embeddings
Cached Query Latency
Full-stack architecture indexing 13,000+ research papers with sub-500ms cached query performance and automated LLM insights.