Omkara Extractor
Knowledge graph construction pipeline from unstructured text
An LLM-powered pipeline that extracts entities, relationships, and structured knowledge from unstructured documents, building domain-specific knowledge graphs with 93% precision.
Problem
Domain-specific knowledge remains trapped in unstructured documents — research papers, clinical notes, legal contracts. Manual knowledge graph construction doesn't scale (a single medical domain graph takes months of expert curation). Rule-based extraction fails on the linguistic variety and ambiguity of real-world text. Organizations have valuable knowledge locked in PDFs they can't query or reason over.
Solution
Omkara Extractor uses a staged LLM pipeline: entity extraction with few-shot prompting, co-reference resolution, relationship extraction with confidence scoring, and schema validation against domain ontologies. Each stage is independently configurable — medical entities need different extraction strategies than legal entities. The staged approach means each stage can be optimized and evaluated independently.
Architecture
Five-stage extraction pipeline: Entity Detection → Co-reference Resolution → Relationship Extraction → Schema Validation → Graph Construction. Built on Neo4j with REST API. Each stage uses models optimized for its specific task — spaCy for base NER, GPT-4 for domain-specific extraction, custom cross-encoders for validation.
Challenges
- Co-reference resolution across 50+ page documents with ambiguous pronouns spanning paragraph breaks
- Balancing extraction recall vs. precision — needed 95%+ precision for clinical use but didn't want to miss critical relationships
- Handling contradictory information from multiple source documents about the same entity
- Scaling LLM-based extraction to document corpora of 10,000+ papers without API costs exploding
Results
- 93% precision on entity extraction across medical and technical domains (evaluated against expert-annotated gold standard)
- Successfully constructed a 500K+ node medical knowledge graph from 12,000 clinical papers
- Reduced manual knowledge engineering time by 80% — what took experts 6 weeks now runs in 4 hours
- Published methodology in peer-reviewed workshop (EMNLP 2024 Findings)
Lessons Learned
- Multi-stage pipelines outperform single-pass extraction by 35% — letting each stage specialize on its subtask dramatically improves quality
- Few-shot prompting with carefully curated examples beats fine-tuning for extraction tasks when examples are selected by embedding similarity to the target domain
- Schema validation is non-negotiable — LLMs produce structurally valid but semantically wrong triples 8% of the time; validation catches 95% of these
System Architecture
┌──────────────────────────────────┐
│ Input Documents │
│ PDF · HTML · Plaintext · XML │
└──────────────┬───────────────────┘
│
▼
┌──────────────────────────────────┐
│ Stage 1: Entity Detection │
│ · spaCy NER (base entities) │
│ · LLM few-shot (domain entities) │
│ · Dictionary matching (known) │
└──────────────┬───────────────────┘
│
▼
┌──────────────────────────────────┐
│ Stage 2: Co-reference Resolution │
│ · Rule-based (clear cases) │
│ · LLM-based (ambiguous refs) │
│ · Distance-weighted scoring │
└──────────────┬───────────────────┘
│
▼
┌──────────────────────────────────┐
│ Stage 3: Relationship Extraction │
│ · Entity pair candidate gen │
│ · LLM relation classification │
│ · Confidence score per triple │
└──────────────┬───────────────────┘
│
▼
┌──────────────────────────────────┐
│ Stage 4: Schema Validation │
│ · Domain ontology check │
│ · Semantic plausibility filter │
│ · Conflict resolution │
└──────────────┬───────────────────┘
│
▼
┌──────────────────────────────────┐
│ Stage 5: Graph Construction │
│ · Neo4j insertion w/ provenance │
│ · Deduplication & merging │
│ · REST API for downstream apps │
└──────────────────────────────────┘
How Each Stage Works
Stage 1: Entity Detection
Combines three strategies: spaCy NER for common types (person, org, location), LLM-based few-shot extraction for domain-specific entities (drug names, gene names, legal concepts), and dictionary matching for known entity lists. Entities carry spans, types, and confidence scores.
Stage 2: Co-reference Resolution
Resolves pronouns and definite descriptions to their referents. "The drug" → "Metformin", "It" → "The clinical trial". Uses rule-based resolution for clear cases (nearest antecedent matching gender/number), LLM-based for ambiguous references spanning paragraph breaks, and distance-weighted scoring.
Stage 3: Relationship Extraction
For each co-occurring entity pair, the system classifies relationship type from the domain ontology. The LLM receives entity types, surrounding context, and valid relationship types. The key insight: extract all candidate pairs first, then classify — this separates the retrieval problem from the classification problem and lets each be optimized separately.
Stage 4: Schema Validation
Extracted triples are validated against the domain schema. Invalid triples are corrected or discarded. Both structural validation (does the relationship type exist between these entity types?) and semantic validation (does this make medical sense?). A cross-encoder model fine-tuned on validated/corrected triple pairs catches 95% of LLM errors.
Stage 5: Graph Construction
Valid triples are inserted into Neo4j with full provenance — source document, extraction confidence, timestamp. Graph supports Cypher queries for downstream applications including GraphRAG retrieval, clinical decision support, and research literature analysis.