From 29b544405a843eba622eb55c87c7a419d29cdaf5 Mon Sep 17 00:00:00 2001 From: Farouk Adeleke Date: Wed, 17 Dec 2025 10:57:10 -0800 Subject: [PATCH] Add MODAIC token support and fix init to accept kwargs --- README.md | 239 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 163 insertions(+), 76 deletions(-) diff --git a/README.md b/README.md index 30ded3a..6737ffa 100644 --- a/README.md +++ b/README.md @@ -1,92 +1,179 @@ -Modaic and Weaviate -Modaic is a hub to share and manage DSPy programs! +# CrossEncoderRanker -You can learn more about the modaic sdk here and check out programs on the hub here. +> **A DSPy-powered cross-encoder ranker for intelligent document retrieval and reranking** -This notebook will illustrate how to load the CrossEncoderRanker program from the Modaic Hub, as well as PromptToSignature. +CrossEncoderRanker combines hybrid search from Weaviate with LLM-based relevance assessment to deliver highly accurate document ranking. Built with [DSPy](https://github.com/stanfordnlp/dspy) and published on [Modaic Hub](https://modaic.ai). -Further, check out ce_ranker.py and pyproject.toml to see how to publish your programs on the Modaic Hub! +--- -CrossEncoderRanker +## Features + +- **Hybrid Search**: Leverages Weaviate's hybrid search (combining vector and keyword search) +- **Intelligent Reranking**: Uses LLM-based cross-encoder with Chain-of-Thought reasoning +- **Async Processing**: Concurrent document scoring for improved performance +- **Easy Integration**: Load pre-compiled programs directly from Modaic Hub +- **Flexible Configuration**: Customize collection, properties, and ranking parameters + +--- + +## Quick Start + +### Installation + +```bash +pip install modaic weaviate-client python-dotenv +``` + +### Environment Setup + +Create a `.env` file with your credentials: + +```env +MODAIC_TOKEN=your_modaic_token +WEAVIATE_URL=your_weaviate_cluster_url +WEAVIATE_API_KEY=your_weaviate_api_key +OPENAI_API_KEY=your_openai_api_key +``` + +### Usage + +```python from dotenv import load_dotenv -import os from modaic import AutoProgram load_dotenv() -# This is looking for MODAIC_TOKEN, WEAVIATE_URL, WEAVIATE_API_KEY, and OPENAI_API_KEY - +# Load the pre-compiled program from Modaic Hub ce_ranker = AutoProgram.from_precompiled( "connor/CrossEncoderRanker", - config_options={ # replace this with your collection name and other custom parameters - "collection_name": "IRPapersText_Default", - "return_properties": ["content"], - "k": 5 - } -) - -response = ce_ranker("What does HyDE stand for?"); - -for idx, ranked_doc in enumerate(response): - print(f"Rank {idx+1}: {ranked_doc[:300]}") -SSSS connor/CrossEncoderRanker -Rank 1: Figure 1: An illustration of the HyDE model. Document snippets are shown. HyDE serves all types of queries without changing the underlying InstructGPT and Contriever/mContriever models. - -GPT-3 (Brown et al., 2020) models can be aligned to human intents to follow instructions faithfully. - -With these -Rank 2: | | Scifact | FiQA | DBPedia | -|-----------|---------|-------|---------| -| Contriever | 64.9 | 24.5 | 29.2 | -| HyDE w/ InstructGPT | **69.1** | 27.3 | 36.8 | -| w/ GPT-3 | 65.9 | **27.9** | **40.5** | - -Table 5: nDCG@10 comparing InstructGPT vs. 3-shot GPT-3 on BEIR. Bes -Rank 3: 6 Conclusion - -In this paper, we introduce HyDE, a new approach for building effective dense retrievers in a completely unsupervised manner, without the need for any relevance labels. We demonstrate that some aspects of relevance modeling can be delegated to a more powerful, flexible, and general-pur -Rank 4: estimate Equation 5 by sampling N documents from g, [d̂1, d̂2, ..., d̂N]: - -v̂qij = 1/N ∑_(d̂k ∼ g(qij, INSTi)) f(d̂k) (6) - - = 1/N ∑_(k=1)^N f(d̂k) (7) - -We also consider the query as a possible hypothesis: - -v̂qij = 1/(N+1) [(∑_(k=1)^N f(d̂k)) + f(qij)] (8) - -Inner p -Rank 5: | | sw | wo | ko | ja | bn | -|-----------|-----|-----|------|------|------| -| **Unsupervised** | | | | | | -| BM25 | 38.9| 28.5| 21.2 | 41.8 | | -| mContriever| 38.3| 22.3| 19.5 | 35.3 | | -| HyDE | 41.7| 30.6| 30.7 | 41.3 | | -| -PromptToSignature -from dotenv import load_dotenv -import os -from modaic import AutoProgram - -load_dotenv() # This is looking for MODAIC_TOKEN and OPENROUTER_API_KEY - -agent = AutoProgram.from_precompiled( - "fadeleke/prompt-to-signature", config_options={ - "lm": "openrouter/anthropic/claude-sonnet-4.5", - "max_tokens": 32000, - "temperature": 0.7, + "collection_name": "IRPapersText_Default", # Your Weaviate collection + "return_properties": ["content"], # Properties to retrieve + "k": 5 # Number of results } ) -result = agent("Summarize a document and extract key entities") +# Query and get ranked results +response = ce_ranker("What does HyDE stand for?") -print(result) -SSSS fadeleke/prompt-to-signature -Signature generation took 0.00 seconds in inference. -Prediction( - signature_name='DocumentSummaryAndEntityExtraction', - task_description='Extract a concise summary of a document and identify key entities (people, organizations, locations, dates, etc.) mentioned within it.', - signature_fields=[GeneratedField(name='document', type=, role=, description='The document text to summarize and extract entities from', pydantic_model_schema=None, literal_values=None, default_value=None), GeneratedField(name='summary', type=, role=, description="A concise summary of the document's main points and content", pydantic_model_schema=None, literal_values=None, default_value=None), GeneratedField(name='key_entities', type=, role=, description='Structured extraction of key entities found in the document', pydantic_model_schema=PydanticModelSchema(model_name='KeyEntities', description='Container for extracted entities from the document', fields=[PydanticFieldDef(name='people', type=, description='Names of people mentioned in the document', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='organizations', type=, description='Names of organizations, companies, or institutions mentioned', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='locations', type=, description='Geographic locations, cities, countries, or regions mentioned', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='dates', type=, description='Important dates or time references mentioned in the document', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='topics', type=, description='Main topics or themes covered in the document', required=False, literal_values=None, nested_model=None)]), literal_values=None, default_value=None)], - reasoning=None -) \ No newline at end of file +# Display results +for idx, ranked_doc in enumerate(response, 1): + print(f"Rank {idx}: {ranked_doc[:300]}...\n") +``` + +### Example Output + +``` +Rank 1: Figure 1: An illustration of the HyDE model. Document snippets are shown. +HyDE serves all types of queries without changing the underlying InstructGPT and +Contriever/mContriever models... + +Rank 2: Table 5: nDCG@10 comparing InstructGPT vs. 3-shot GPT-3 on BEIR... + +Rank 3: In this paper, we introduce HyDE, a new approach for building effective +dense retrievers in a completely unsupervised manner... +``` + +--- + +## How It Works + +1. **Hybrid Search**: Performs initial retrieval from Weaviate using hybrid search +2. **Relevance Assessment**: Each document is scored using a DSPy ChainOfThought signature +3. **Concurrent Processing**: Documents are scored asynchronously for efficiency +4. **Reranking**: Results are sorted by relevance score and returned + +--- + +## Configuration Options + +| Parameter | Type | Description | Default | +|-----------|------|-------------|---------| +| `collection_name` | `str` | Weaviate collection name | Required | +| `return_properties` | `list[str]` | Properties to retrieve from Weaviate | Required | +| `k` | `int` | Number of results to return | Required | +| `lm` | `str` | Language model to use | `"openai/gpt-4.1-mini"` | + +--- + +## Publishing Your Own Program + +Want to publish your own DSPy program to Modaic Hub? Check out the source code: + +- **`ce_ranker.py`**: Main implementation with `CERankerAgent` class +- **`pyproject.toml`**: Project dependencies and metadata + +### Publishing Steps + +```python +from ce_ranker import CERankerAgent, CERankerConfig + +config = CERankerConfig( + collection_name="YourCollection", + return_properties=["content"], + k=5 +) + +agent = CERankerAgent(config) +agent.push_to_hub( + "username/ProgramName", + with_code=True, + commit_message="Initial release" +) +``` + +--- + +## Architecture + +``` +┌─────────────────────────────────────────────────┐ +│ User Query │ +└────────────────────┬────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ Weaviate Hybrid Search │ +│ (Vector Search + Keyword Search) │ +└────────────────────┬────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ Async Relevance Assessment │ +│ (DSPy ChainOfThought + LLM Scoring) │ +└────────────────────┬────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ Reranked Results (Top K) │ +└─────────────────────────────────────────────────┘ +``` + +--- + +## Resources + +- **Modaic SDK Documentation**: [Learn more about Modaic](https://docs.modaic.dev) +- **Modaic Hub**: [Browse programs](https://modaic.dev) +- **DSPy Framework**: [GitHub Repository](https://github.com/stanfordnlp/dspy) +- **Weaviate**: [Vector Database Documentation](https://weaviate.io/developers/weaviate) + +--- + +## Requirements + +- Python >= 3.11 +- `modaic >= 0.8.0` +- `weaviate-client >= 4.18.1` + +--- + +## License + +MIT License - See LICENSE file for details + +--- + +## Contributing + +Contributions are welcome! Please feel free to submit a Pull Request. \ No newline at end of file