connor/CrossEncoderRanker
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add MODAIC token support and fix init to accept kwargs

Browse Source
This commit is contained in:
Farouk Adeleke
2025-12-17 10:57:10 -08:00
parent 76b959d985
commit 29b544405a
1 changed files with 163 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files

239
README.md
View File

@@ -1,92 +1,179 @@
Modaic and Weaviate # CrossEncoderRanker
Modaic is a hub to share and manage DSPy programs!
You can learn more about the modaic sdk here and check out programs on the hub here. > **A DSPy-powered cross-encoder ranker for intelligent document retrieval and reranking**
This notebook will illustrate how to load the CrossEncoderRanker program from the Modaic Hub, as well as PromptToSignature. CrossEncoderRanker combines hybrid search from Weaviate with LLM-based relevance assessment to deliver highly accurate document ranking. Built with [DSPy](https://github.com/stanfordnlp/dspy) and published on [Modaic Hub](https://modaic.ai).
Further, check out ce_ranker.py and pyproject.toml to see how to publish your programs on the Modaic Hub! ---
CrossEncoderRanker ## Features
- **Hybrid Search**: Leverages Weaviate's hybrid search (combining vector and keyword search)
- **Intelligent Reranking**: Uses LLM-based cross-encoder with Chain-of-Thought reasoning
- **Async Processing**: Concurrent document scoring for improved performance
- **Easy Integration**: Load pre-compiled programs directly from Modaic Hub
- **Flexible Configuration**: Customize collection, properties, and ranking parameters
---
## Quick Start
### Installation
```bash
pip install modaic weaviate-client python-dotenv
```
### Environment Setup
Create a `.env` file with your credentials:
```env
MODAIC_TOKEN=your_modaic_token
WEAVIATE_URL=your_weaviate_cluster_url
WEAVIATE_API_KEY=your_weaviate_api_key
OPENAI_API_KEY=your_openai_api_key
```
### Usage
```python
from dotenv import load_dotenv from dotenv import load_dotenv
import os
from modaic import AutoProgram from modaic import AutoProgram
load_dotenv() load_dotenv()
# This is looking for MODAIC_TOKEN, WEAVIATE_URL, WEAVIATE_API_KEY, and OPENAI_API_KEY # Load the pre-compiled program from Modaic Hub
ce_ranker = AutoProgram.from_precompiled( ce_ranker = AutoProgram.from_precompiled(
"connor/CrossEncoderRanker", "connor/CrossEncoderRanker",
config_options={ # replace this with your collection name and other custom parameters
"collection_name": "IRPapersText_Default",
"return_properties": ["content"],
"k": 5
}
)
response = ce_ranker("What does HyDE stand for?");
for idx, ranked_doc in enumerate(response):
print(f"Rank {idx+1}: {ranked_doc[:300]}")
SSSS connor/CrossEncoderRanker
Rank 1: Figure 1: An illustration of the HyDE model. Document snippets are shown. HyDE serves all types of queries without changing the underlying InstructGPT and Contriever/mContriever models.
GPT-3 (Brown et al., 2020) models can be aligned to human intents to follow instructions faithfully.
With these
Rank 2: | | Scifact | FiQA | DBPedia |
|-----------|---------|-------|---------|
| Contriever | 64.9 | 24.5 | 29.2 |
| HyDE w/ InstructGPT | **69.1** | 27.3 | 36.8 |
| w/ GPT-3 | 65.9 | **27.9** | **40.5** |
Table 5: nDCG@10 comparing InstructGPT vs. 3-shot GPT-3 on BEIR. Bes
Rank 3: 6 Conclusion
In this paper, we introduce HyDE, a new approach for building effective dense retrievers in a completely unsupervised manner, without the need for any relevance labels. We demonstrate that some aspects of relevance modeling can be delegated to a more powerful, flexible, and general-pur
Rank 4: estimate Equation 5 by sampling N documents from g, [d̂1, d̂2, ..., d̂N]:
v̂qij = 1/N ∑_(d̂k ∼ g(qij, INSTi)) f(d̂k) (6)
= 1/N ∑_(k=1)^N f(d̂k) (7)
We also consider the query as a possible hypothesis:
v̂qij = 1/(N+1) [(∑_(k=1)^N f(d̂k)) + f(qij)] (8)
Inner p
Rank 5: | | sw | wo | ko | ja | bn |
|-----------|-----|-----|------|------|------|
| **Unsupervised** | | | | | |
| BM25 | 38.9| 28.5| 21.2 | 41.8 | |
| mContriever| 38.3| 22.3| 19.5 | 35.3 | |
| HyDE | 41.7| 30.6| 30.7 | 41.3 | |
|
PromptToSignature
from dotenv import load_dotenv
import os
from modaic import AutoProgram
load_dotenv() # This is looking for MODAIC_TOKEN and OPENROUTER_API_KEY
agent = AutoProgram.from_precompiled(
"fadeleke/prompt-to-signature",
config_options={ config_options={
"lm": "openrouter/anthropic/claude-sonnet-4.5", "collection_name": "IRPapersText_Default", # Your Weaviate collection
"max_tokens": 32000, "return_properties": ["content"], # Properties to retrieve
"temperature": 0.7, "k": 5 # Number of results
} }
) )
result = agent("Summarize a document and extract key entities") # Query and get ranked results
response = ce_ranker("What does HyDE stand for?")
print(result) # Display results
SSSS fadeleke/prompt-to-signature for idx, ranked_doc in enumerate(response, 1):
Signature generation took 0.00 seconds in inference. print(f"Rank {idx}: {ranked_doc[:300]}...\n")
Prediction( ```
signature_name='DocumentSummaryAndEntityExtraction',
task_description='Extract a concise summary of a document and identify key entities (people, organizations, locations, dates, etc.) mentioned within it.', ### Example Output
signature_fields=[GeneratedField(name='document', type=<FieldType.STRING: 'str'>, role=<FieldRole.INPUT: 'input'>, description='The document text to summarize and extract entities from', pydantic_model_schema=None, literal_values=None, default_value=None), GeneratedField(name='summary', type=<FieldType.STRING: 'str'>, role=<FieldRole.OUTPUT: 'output'>, description="A concise summary of the document's main points and content", pydantic_model_schema=None, literal_values=None, default_value=None), GeneratedField(name='key_entities', type=<FieldType.PYDANTIC_MODEL: 'pydantic'>, role=<FieldRole.OUTPUT: 'output'>, description='Structured extraction of key entities found in the document', pydantic_model_schema=PydanticModelSchema(model_name='KeyEntities', description='Container for extracted entities from the document', fields=[PydanticFieldDef(name='people', type=<FieldType.LIST_STRING: 'list[str]'>, description='Names of people mentioned in the document', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='organizations', type=<FieldType.LIST_STRING: 'list[str]'>, description='Names of organizations, companies, or institutions mentioned', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='locations', type=<FieldType.LIST_STRING: 'list[str]'>, description='Geographic locations, cities, countries, or regions mentioned', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='dates', type=<FieldType.LIST_STRING: 'list[str]'>, description='Important dates or time references mentioned in the document', required=False, literal_values=None, nested_model=None), PydanticFieldDef(name='topics', type=<FieldType.LIST_STRING: 'list[str]'>, description='Main topics or themes covered in the document', required=False, literal_values=None, nested_model=None)]), literal_values=None, default_value=None)],
reasoning=None ```
) Rank 1: Figure 1: An illustration of the HyDE model. Document snippets are shown.
HyDE serves all types of queries without changing the underlying InstructGPT and
Contriever/mContriever models...
Rank 2: Table 5: nDCG@10 comparing InstructGPT vs. 3-shot GPT-3 on BEIR...
Rank 3: In this paper, we introduce HyDE, a new approach for building effective
dense retrievers in a completely unsupervised manner...
```
---
## How It Works
1. **Hybrid Search**: Performs initial retrieval from Weaviate using hybrid search
2. **Relevance Assessment**: Each document is scored using a DSPy ChainOfThought signature
3. **Concurrent Processing**: Documents are scored asynchronously for efficiency
4. **Reranking**: Results are sorted by relevance score and returned
---
## Configuration Options
| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `collection_name` | `str` | Weaviate collection name | Required |
| `return_properties` | `list[str]` | Properties to retrieve from Weaviate | Required |
| `k` | `int` | Number of results to return | Required |
| `lm` | `str` | Language model to use | `"openai/gpt-4.1-mini"` |
---
## Publishing Your Own Program
Want to publish your own DSPy program to Modaic Hub? Check out the source code:
- **`ce_ranker.py`**: Main implementation with `CERankerAgent` class
- **`pyproject.toml`**: Project dependencies and metadata
### Publishing Steps
```python
from ce_ranker import CERankerAgent, CERankerConfig
config = CERankerConfig(
collection_name="YourCollection",
return_properties=["content"],
k=5
)
agent = CERankerAgent(config)
agent.push_to_hub(
"username/ProgramName",
with_code=True,
commit_message="Initial release"
)
```
---
## Architecture
```
┌─────────────────────────────────────────────────┐
│ User Query │
└────────────────────┬────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ Weaviate Hybrid Search │
│ (Vector Search + Keyword Search) │
└────────────────────┬────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ Async Relevance Assessment │
│ (DSPy ChainOfThought + LLM Scoring) │
└────────────────────┬────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ Reranked Results (Top K) │
└─────────────────────────────────────────────────┘
```
---
## Resources
- **Modaic SDK Documentation**: [Learn more about Modaic](https://docs.modaic.dev)
- **Modaic Hub**: [Browse programs](https://modaic.dev)
- **DSPy Framework**: [GitHub Repository](https://github.com/stanfordnlp/dspy)
- **Weaviate**: [Vector Database Documentation](https://weaviate.io/developers/weaviate)
---
## Requirements
- Python >= 3.11
- `modaic >= 0.8.0`
- `weaviate-client >= 4.18.1`
---
## License
MIT License - See LICENSE file for details
---
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.