SearchTweak

Self-hosted search relevance evaluation platform with AI Judges. Assess search quality by running keyword queries against your search APIs, collecting human and AI relevance judgments, and calculating industry-standard IR metrics.

Use it to benchmark search configurations, label training data for ML models, and track search quality over time.

Key Features

• AI Judges — automated relevance judging with OpenAI, Anthropic, Google, DeepSeek, xAI, Groq, Mistral, Ollama, and Custom OpenAI-compatible providers
• Human + AI Hybrid Flow — people and AI judges work in one feedback model with Single and Multiple (3) strategies
• Judge Operations — provider/model configuration, per-scale prompts, batch size control, logs, token usage, runtime statuses, and JSONL log export by active filters
• Search Evaluation — run keyword queries against any search API, collect relevance judgments, and compute metrics automatically
• IR Metrics — Precision, MAP, MRR, CG, DCG, nDCG with support for Binary, Graded, and Detail grading scales
• Metrics Over Time — track how search quality changes across evaluations with historical charts
• Feedback Management — assign grading tasks to team members, reuse judgments (human and AI) across evaluations, scoring guidelines
• SSO / OpenID Connect — enterprise authentication via Keycloak, Azure AD, Okta, or any OIDC provider with SSO Only Mode
• Team Collaboration — role-based access (Admin, Evaluator), team invitations, tag-based organization
• Real-time Updates — live progress via WebSockets as evaluations run and grades come in
• REST API — manage evaluations and models programmatically with token-based authentication
• Customizable Dashboard — configurable widgets for metrics, leaderboard, and feedback activity

How It Works

1. Configure a Search Endpoint (URL, method, headers, mapper).
2. Configure a Search Model (request template with #query#).
3. Create a Search Evaluation (keywords, scale, metrics, strategy).
4. Collect feedback from humans and/or AI judges.
5. Review metrics and compare quality over time.

More screenshots

Metrics Dashboard

Give Feedback

Search Models

Tech Stack

Layer	Technology
Backend	Laravel 11 (PHP 8.3)
Frontend	Livewire 3, Alpine.js, Tailwind CSS
Real-time	Laravel Reverb (WebSockets)
Queue	Laravel Horizon
Database	MySQL
Cache/Queue/Sessions	Redis
Infrastructure	Docker Compose, Traefik, Nginx, PHP-FPM
Auth	Jetstream + Sanctum + Fortify + Socialite (SSO/OIDC)

Requirements

• Docker and Docker Compose
• Make (pre-installed on macOS/Linux)

Getting Started

1. Clone the Repository

git clone https://github.com/afedukov/searchtweak.git
cd searchtweak/devops

2. Setup Environment

cp .env.dist .env

3. Configure Hosts File

Edit your /etc/hosts file (or C:\Windows\System32\drivers\etc\hosts on Windows):

127.0.0.1    searchtweak.local traefik.searchtweak.local db.searchtweak.local

4. Start the Application

make

This will start all containers and run make bootstrap (destructive bootstrap with fresh migrations + seed + assets/docs build).

5. Open in Browser

Service	URL
App	http://searchtweak.local
Docs (VitePress)	http://searchtweak.local/docs/overview
Traefik Dashboard	http://traefik.searchtweak.local
phpMyAdmin	http://db.searchtweak.local
MailHog	http://localhost:8025

Default Admin User

Log in at http://searchtweak.local with:

• Email: [email protected]
• Password: 12345678

Useful Commands

cd devops
make start        # Start the application
make stop         # Stop the application
make bootstrap    # Bootstrap (fresh DB reset + seed + assets/docs build)
make bootstrap-up # Bootstrap without DB reset (safe migrate + seed)
make bootstrap-fresh # Explicit destructive bootstrap (same as make bootstrap)
make jobs         # Start Horizon (queue worker)
make queue-reload # Reload Horizon workers in the running queue container
make reverb       # Start Reverb (WebSocket server)
make vite         # Start Vite development server
make vite-prod    # Build Vite for production
make seed-demo    # Reset DB + default seeders + DemoMarketplaceSeeder
make docs-install # Install docs dependencies
make docs         # Run docs dev server (http://localhost:3001/docs/overview)
make docs-build   # Build documentation
make docs-preview # Preview built documentation (http://localhost:3001/docs/overview)
make docs-publish # Publish built docs to public/docs

make bootstrap is intended for clean local setup. Use make bootstrap-up if you need to keep existing local data.

Running Tests

cd devops
make test                                                   # Run all tests
docker compose run --rm artisan test                        # Run all tests (alternative)
docker compose run --rm artisan test --testsuite=Unit       # Run unit tests only
docker compose run --rm artisan test --testsuite=Feature    # Run feature tests only
docker compose run --rm artisan test --filter TestName      # Run a specific test

Email Setup

By default, SearchTweak uses MailHog to capture outgoing emails in development. Access the MailHog UI at http://localhost:8025.

Configuring real email sending (SMTP / Amazon SES)

Remove the mailhog service from /devops/docker-compose.yml, then update your .env file:

SMTP

MAIL_MAILER=smtp
MAIL_HOST=smtp.your-email-provider.com
MAIL_PORT=587
MAIL_USERNAME=[email protected]
MAIL_PASSWORD=your-email-password
MAIL_ENCRYPTION=tls
MAIL_FROM_ADDRESS=[email protected]
MAIL_FROM_NAME="${APP_NAME}"

Amazon SES

MAIL_MAILER=ses
AWS_ACCESS_KEY_ID=your-aws-access-key
AWS_SECRET_ACCESS_KEY=your-aws-secret-key
AWS_SES_REGION=us-east-1
MAIL_FROM_ADDRESS=[email protected]
MAIL_FROM_NAME="${APP_NAME}"

Enforcing email verification

1. Update the User Model

In App\Models\User, implement the MustVerifyEmail interface:

class User extends Authenticatable implements TaggableInterface, MustVerifyEmail
{
    // ...
}

2. Enable Email Verification in Fortify

In config/fortify.php, uncomment:

'features' => [
    // ...
    Features::emailVerification(),
    // ...
],

Documentation

• Main docs: http://searchtweak.local/docs/overview
• API reference: http://searchtweak.local/docs/api/overview
• Project meta docs: DOCUMENTATION.md

Security

Endpoint URL Allowlist

To prevent SSRF attacks, restrict which hosts can be used in Search Endpoint URLs by setting:

ALLOWED_ENDPOINT_HOSTS=api.example.com,search.example.com

When set, only endpoints pointing to the listed hosts will be accepted. When empty (default), any URL is allowed — suitable for local development.

SSO / OpenID Connect

SearchTweak supports Single Sign-On via any OIDC-compatible identity provider (Keycloak, Azure AD, Okta, etc.) using Laravel Socialite with the Keycloak provider.

To enable SSO, configure the following environment variables in your .env file:

OIDC_CLIENT_ID=your-client-id
OIDC_CLIENT_SECRET=your-client-secret
OIDC_BASE_URL=https://your-idp.example.com
OIDC_REALM=your-realm
OIDC_BUTTON_LABEL="Sign in with SSO"

Then enable SSO in Admin > Settings > Single Sign-On.

SSO Only Mode hides the email/password login form, requiring all users to authenticate via SSO. Super admins can still access the login form via ?fallback=1 in case the IdP is unavailable.

SSO users are matched by email address. On first SSO login, if no matching account exists, a new user with a personal team is created automatically.

Contributing

Contributions are welcome! Please fork the repository and submit a pull request with your enhancements or bug fixes.

License

This project is licensed under the Functional Source License, Version 1.1 (FSL-1.1-Apache-2.0), with an irrevocable grant to the Apache License, Version 2.0 effective on the second anniversary of the software's release.

Copyright 2024-2026 Andrey Fedyukov

• You may use, modify, and redistribute the software for any purpose, except in products or services that compete with the software or any other product or service we offer.
• After two years, you may alternatively use the software under the Apache License, Version 2.0.

See the full LICENSE file for details.