Find Every Regression in your Pull Requests
Catch Regressions before merging code
Simple Developer Workflow
Powered by AI
Frequently Asked Questions
No. SmartTests is not a direct alternative to consumer-driven contract testing tools like Pact. SmartTests is a provider-focused validation tool that gives you a "risk score" for your PR by analyzing its runtime behavior, rather than coordinating contracts between consumer/provider teams.
The AI system performs two main functions: First, it analyzes API responses from both versions to identify structural, data type, and content differences. The models determine which differences actually impact compatibility rather than flagging every change.
Second, it reduces test flakiness by learning from historical test runs, distinguishing between expected variations (timestamps, random IDs) and genuine breaking changes. The system adapts to your specific APIs over time, improving its accuracy with continued use.
SmartTests detects:
- Schema changes: Removed fields, changed data types, altered object structures
- Behavioral changes: Different response codes, new error conditions, altered validation rules
Because SmartTests observes runtime behavior rather than just validating static schemas, it can identify subtle issues that traditional approaches miss.
Integration requires two components:
1) The Signadot CLI integrated into your pipeline to create sandboxes for each PR
2) Smart Test triggers configured to automatically run tests against those sandboxes
Most teams have their first SmartTests running in under an hour:
1) Install the Signadot operator on your Kubernetes cluster (10-15 minutes)
2) Write your first test in Starlark to call your API endpoints (5-10 minutes)
3) Configure triggers to run the test on your sandboxes (5 minutes)
4) Integrate sandbox creation with your CI/CD pipeline (15-30 minutes)
Since SmartTests runs against your existing staging/pre-production environment, you avoid the typically time-consuming process of setting up separate test environments or managing test data.
SmartTests minimizes maintenance when APIs evolve:
- When you make compatible API changes (adding fields, extending functionality in backward-compatible ways), SmartTests automatically recognizes these as non-breaking and doesn't require test updates.
- For intentional breaking changes, you typically only need to update the test code that calls the changed endpoints - not assertions or expectations, since those are handled by the AI comparison system.
