Inspiration
Manual E2E testing is tedious, time-consuming, and expensive. We wanted to build a tool that automates the entire QA pipeline—from understanding a website's structure to generating and running comprehensive tests—so developers can ship faster with confidence.
What it does
QAsmith automatically crawls any website, builds an intelligent graph database of pages and interactions, and uses Claude AI to generate production-ready Playwright tests. It handles smoke testing for quick validations, logic testing for forms and workflows, and load testing for performance—all through a visual dashboard with AI-powered failure analysis.
How we built it
We built a Python backend with FastAPI orchestrating a five-stage pipeline: BFS crawling with Playwright, Neo4j graph database for storing site structure, Claude AI for test generation, a TypeScript compiler for Playwright specs, and a test runner with artifact collection. The React frontend visualizes the crawl graph with D3.js and manages the entire workflow.
Challenges we ran into
Getting consistent test generation quality was hard—the AI initially hallucinated selectors or picked unstable ones. We solved this by capturing detailed selector metadata during crawling and engineering prompts to prioritize test IDs and ARIA labels.
Accomplishments that we're proud of
Achieving 90%+ test pass rates on real-world sites, building a fully working Neo4j graph layer that enables intelligent path-finding for test generation, and creating a complete end-to-end pipeline from URL to executable tests in under 2 minutes.
What we learned
LLM prompt engineering is critical for structured output: small changes in how we format crawl data dramatically affected test quality. We also learned that graph databases are perfect for representing website navigation, enabling shortest-path algorithms for efficient test generation.
What's next for QA-smith
Adding GraphRAG with vector embeddings for semantic test recommendations, implementing CI/CD integrations (GitHub Actions, Jenkins), supporting API testing with OpenAPI spec imports, and building collaborative features for teams to review and refine AI-generated tests.
Log in or sign up for Devpost to join the conversation.