feat: Add SQLite conversion, FTS5 search optimization, and compression pipeline for offline use for App

[IsmailHosenIsmailJames]

Overview

This PR introduces a complete offline-first pipeline that converts the Hadith JSON API into optimized SQLite databases. This enable mobile and web developers to integrate high-performance Hadith search and synchronization with minimal overhead.

Key Features

1. SQLite Conversion with FTS5 (json_to_sqlite.py)

• Relational Schema: Maps JSON data into book_info, sections, hadiths, and grades tables.
• Lightning-Fast Search: Implements FTS5 (Full-Text Search) virtual tables.
• Optimized Storage: Uses External Content Tables to prevent text duplication, keeping index overhead to only 20-40%.

2. Compression & Metadata Pipeline (compress_and_info.py)

• 70% Size Reduction: Automatically compresses SQLite databases into .zip files.
• Smart Organization: Groups assets by language (e.g., ara/, eng/, ben/).
• Data Integrity: Generates SHA256 checksums for every database to allow apps to verify integrity and detect updates.
• Global Metadata: Generates all_info.json and language-specific info.json for easy app integration.

3. Interactive Documentation

• Added a premium, interactive Integration Guide (index.html).
• Developers can input their hosting URL to get personalized code snippets and download links instantly.

Hosting & Resources

The assets generated by this pipeline are now publicly hosted:

• Compressed Assets Hosting: https://ismailhosenismailjames.github.io/compressed_hadith_sqlite/
• Public Resource Repo: IsmailHosenIsmailJames/compressed_hadith_sqlite
• Interactive Instruction Guide: https://ismailhosenismailjames.github.io/hadith-api/

How to Regenerate Data

To update the SQLite databases after JSON changes:

1. Run python3 json_to_sqlite.py
2. Run python3 compress_and_info.py
3. Push the contents of the compressed_hadith_sqlite/ folder to the resource repository.

[fawazahmed0]

Salam Alaykum, I have added link to your sqlite repo in README. It would be more efficient to keep these SQLite-related updates localized in that repository to avoid cluttering the main branch.

[IsmailHosenIsmailJames]

Okay. Maybe you are correct. You can close it. Jajakallhu Kairon