.png)
Strategy
How Startups Like Figma, Facebook, and Airbnb Found Sudden Growth Inflections
Learn more
Smoke Testing in Experimentation and Software Development
Why is smoke testing essential for successful experimentation?
Smoke testing is a critical step in the software development process. It ensures that your builds are stable and the core features are functional before you proceed to more detailed testing.
Without a solid smoke testing process, critical issues can slip through early, leading to costly fixes, delays, and bugs that impact user experience and stability.
In this guide, we'll explain smoke testing, when to run it, and how tools like Eppo can help automate and streamline your testing process for faster, more reliable releases.
What You'll Learn in This Blog:
- Why smoke testing is key to identifying issues early.
- The different types of smoke testing and their role in your testing pipelines.
- How to integrate smoke testing with sanity and regression testing.
- How to choose an automation tool to streamline the testing process.
What Exactly Is Smoke Testing?
Smoke testing is a quick, high-level build verification testing process that checks whether the basic features of a new software build are functioning properly. It's typically done after a new build or after code changes to ensure that core functionalities and key features are working as expected before diving into more detailed testing.
For example, when a new build is released, a smoke test might be run to check if the website loads properly, if the login feature works, and if critical tasks can be completed without issues. If any significant problems—aka "smoke"—are found, further testing is paused until those issues are addressed. It's a simple software quality assurance check to catch significant issues in the early stages of the process, which can save you time and resources before your development team dives deeper into the testing cycle.
Let's Look at the Different Types of Smoke Testing
Whether you choose to run all three or prioritize certain tests depends on the specific needs of your build, but generally, a comprehensive smoke test will incorporate each of these types of software testing to ensure the software's stability across various dimensions.
Functional Smoke Testing
Functional testing is usually the first testing phase, and it focuses on verifying that the core functionalities of the software are working properly. This is the first thing to check when a new build is released. For example, if you're testing a new version of a website, functional testing would check that basic actions like logging in, adding items to the cart, or completing a purchase are all working as expected. This type of testing ensures that the most critical user-facing functions are up and running before moving on to more in-depth testing.
Non-Functional Smoke Testing
Non-functional testing looks at the underlying, non-visible aspects of your software, like performance, security, and reliability, to ensure they all hold up under normal conditions. It's essential to check that the application can handle realistic traffic loads, that security features are intact, and that the system's stability isn't compromised when it's under stress. For example, load testing can simulate a high volume of users to see if the system can handle the demand. At the same time, basic security tests can tell you if any apparent vulnerabilities are present. This part of the smoke test is crucial for confirming that the software is robust enough for real-world scenarios.
Integration Smoke Testing
Integration testing strategies ensure that the front-end, back-end, databases, and third-party APIs integrate properly. For example, after integrating a new payment gateway into an e-commerce web application, integration testing ensures that the data flows correctly between the checkout page and the payment processor without causing errors.
When Should You Perform Smoke Testing?
After Every New Build
Run a smoke test right after each new build to ensure the core features are working. This gives you quick feedback on whether the new code is stable enough to proceed with more detailed testing, which can save you tons of time and help you avoid unnecessary work on broken builds.
After Integration Stages
Once different parts of your app are integrated, smoke testing can reveal if the system is stable and everything is working together. This helps catch any major issues early and ensures that new components don't break anything already in place.
Before User Acceptance Testing (UAT)
Smoke testing before UAT is your final check to confirm that new features are in place and functioning. If anything's broken, you'll catch it before the users do, saving time, money, and headaches down the line.
In Continuous Integration Pipelines
Automate your smoke tests in the CI pipeline to catch issues as they happen. You ensure new code doesn't disrupt the build by running tests with every new commit or code change.
What Metrics Should You Look at When You're Running These Smoke Tests?
Time to Execute: Track how long your smoke tests take to run. The faster they execute, the quicker you'll get feedback. Optimizing execution time keeps your testing workflows efficient and prevents delays in the development cycle.
Defects Detected: Your QA team should monitor how many critical issues appear during smoke testing. A higher number of defects indicates deeper problems with the build that need to be fixed before diving into more detailed testing.
Test Coverage: Measure the percentage of critical functionalities covered by your smoke tests. Too little coverage risks missing important issues, while too much can slow you down. Try to find that balance so you can focus on the most important aspects without overloading the process.
Test Pass Percentage: Test results should also show you the pass rate of your smoke tests. A high pass rate suggests that your build is stable and ready for further testing, while a low pass rate flags potential issues that need to be addressed before your testing team continues with the testing process.
Smoke Testing vs. Other Testing Methods
Sanity Testing vs. Smoke Testing
Both sanity and smoke testing are designed to catch issues early, but they differ in the level of detail they use to do that. As you've learned, smoke testing offers a 30,000-foot view of your app. It's a quick, broad check that ensures your app's basic features are functioning properly after a new build or code change.
Sanity testing zooms in and takes a more granular look with a more targeted scope. It focuses on specific recently updated areas or bug fixes to ensure those particular features thoroughly work as expected.
Typically, sanity testing follows smoke testing, validating that the detailed parts of the app are intact after the overall stability check.
Regression Testing vs. Smoke Testing
After smoke testing confirms that the build is stable, regression testing runs a full set of tests on the entire application to ensure no existing features are disrupted. This more comprehensive check catches any unintended side effects of the new code.
This comes after smoke testing. Smoke tests quickly confirm basic stability, and once that's clear, regression tests validate that everything else is still solid.
Manual vs. Automated Smoke Testing
Manual Smoke Testing
Manual smoke testing is exactly what it sounds like—running a series of checks manually on a new build to ensure that core features are functioning as expected. While this method allows for direct interaction with the software, it's time-consuming and prone to human error. Manual testing can be okay for small projects or one-off builds, but it's too inefficient and unreliable as your codebase grows and the number of builds increases.
Automated Smoke Testing
With automated tests, you can quickly verify your application's basic functionality every time a new build or code change occurs. This reduces the manual effort required and provides faster, more consistent feedback.
Automating your smoke tests offers:
- Faster Feedback: Automated tests can run faster than manual tests, allowing you to catch issues early and keep the development process moving forward.
- Consistency: Automated tests eliminate the risk of human error by running each test in the same way every time, without variations.
- Scalability: As your project grows, the number of tests required increases. Automation allows you to scale testing without dramatically increasing the time or effort involved.
- Continuous Integration (CI) Compatibility: Automated tests make sure that each commit is tested as soon as it's made.
Smoke Testing Tools to Help You Automate
With the right automation tools, you can significantly streamline the testing process, catch issues early, and save valuable time and resources.
Here are some key features to look for:
Integration with CI/CD Pipelines
A strong smoke testing tool should integrate seamlessly with your continuous integration and delivery pipelines. This allows smoke tests to run automatically with every new commit or code change, providing real-time feedback without manual intervention.
Real-Time Feedback on Build Stability
You want a tool that offers rapid feedback on whether your code is stable enough for more detailed testing. Look for tools that highlight critical failures immediately so you can address issues immediately.
Customizable and Scalable Testing
As your application grows, so will your testing needs. A good smoke testing tool should allow you to customize tests for your specific use cases and scale as your project evolves.
Comprehensive Metrics and Reporting
Tracking the results of your smoke tests helps you identify bottlenecks and improve your testing process over time. Look for tools that provide detailed metrics like test execution time, defect detection rates, and pass/fail percentages.
Support for Feature Flags and Gradual Rollouts
A smoke testing tool should allow you to easily manage feature flags to control the rollout of new functionality in your staging or production environment. This helps you limit the impact of any issues and ensures that only the right users see experimental features.
Ready to streamline your smoke testing process and catch issues earlier in your development cycle? Eppo's tools offer everything we mentioned above and more, making your testing process faster, more reliable, and more efficient.
Request a demo today to see how Eppo can help optimize your software testing workflow and produce smoother releases.
Table of contents
Ready to go from knowledge to action?
Talk to our team of experts and see why companies like Twitch, DraftKings, and Perplexity use Eppo to power experimentation for every team.
Get a demo
