Skip to content

README.md

Latest commit

 

History

History
105 lines (75 loc) · 2.83 KB

README.md

File metadata and controls

105 lines (75 loc) · 2.83 KB

🧪 Crypto QA Framework

🧪 QA framework for testing LLM agents with cryptocurrency data support. Includes Mock LLM for offline testing, FastAPI endpoints, Bitcoin data validation, and PyTest test suite. Built with Python, LangChain, and FastAPI.

A lightweight QA framework for validating prompt-based LLM responses using Python, PyTest, and a simulated MockLLM agent.
Designed to enable fast and reliable testing of AI agents without requiring external API access (e.g., OpenAI).

🚀 Key Features

  • ✅ Mock-based testing without OpenAI API
  • 🧠 Prompt QA validation for factual, structured, and formatted responses
  • ⚙️ CI/CD friendly – run anywhere with Python and PyTest
  • 📦 Easy to extend for JSON, Markdown, logic or classification use cases

📁 Project Structure

crypto-qa-framework/
├── data/                       # Optional data assets (CSV, etc.)
├── llm_mocks/
│   └── mock_llm.py            # MockLLM simulating model behavior
├── tests/
│   └── prompt_test_cases.py   # PyTest unit tests for prompts
├── requirements.txt           # Dependencies
└── README.md                  # This file

🧩 Example: MockLLM

class MockLLM:
    """
    Simulated LLM for testing prompt-based responses without using OpenAI API.
    """
    def invoke(self, prompt: str) -> str:
        prompt_lower = prompt.lower()
        if "capital of canada" in prompt_lower:
            return "Ottawa is the capital of Canada."
        elif "return json" in prompt_lower:
            return '{"status": "ok", "country": "Canada"}'
        elif "markdown" in prompt_lower:
            return "# Welcome\nThis is a markdown response."
        return "Unknown"

✅ Tests Included

prompt_test_cases.py:

  • test_llm_factual_answer_canadian_capital()
    → Validates factual knowledge

  • test_llm_structured_json_output()
    → Validates structured response format

  • test_llm_markdown_formatting_response()
    → Validates Markdown-style formatting

$ pytest tests/prompt_test_cases.py
...
3 passed in 0.01s

💡 Use Cases

  • Testing prompt-response logic offline
  • Validating fallback agents in LLM-based systems
  • Educational environments where API cost/availability is limited
  • Preparing test scaffolding before model integration

📦 Installation

git clone https://github.com/yourusername/crypto-qa-framework.git
cd crypto-qa-framework
pip install -r requirements.txt
pytest

🔮 Roadmap

  • Add CLI test runner
  • Generate Markdown test reports
  • Extend test cases to handle hallucinations, edge cases, and multilingual prompts

🧑‍💻 Author

Aleksandr M. – QA Automation Engineer focused on AI systems testing, LangChain workflows, and test strategy for LLMs.