Smoke Testing in Production With Synthetic Monitors

Imagine you’re sitting on a plane flying across the country when the captain gets on the intercom and says, “Thanks for flying with us today, we’re just going to run a few tests now that we’re airborne. If you see smoke, please call a flight attendant.” This scenario would make anyone nervous. While the stakes aren’t quite as high for software developers, testing in your production environment, if that’s your only testing, is very risky business. Just check out the many hilarious memes that illustrate the danger of testing in production. 

However, when paired with a robust pre-deployment testing process, testing in production, often called smoke testing, is a crucial part of a modern continuous delivery/continuous deployment (CI/CD) process. Continuous deployment is a big part of DevOps and modern web development. Integrating rapid and proactive testing can help teams perform more deployments with confidence. Smoke testing is a tactical tool in the DevOps toolbox to ensure core functionality is working and operational in just a few minutes following new deployments or feature releases.

Let’s take a closer look at smoke testing: why it’s used for web applications, the benefits smoke testing can give you, and how you can easily set up smoke testing using automated web-testing tools like synthetic monitors.

Ensure successful deployments with smoke testing

Smoke testing is about running a series of fast and basic tests on the core functionality of your production environment (as well as other environments), to ensure deployments are successful and that you catch any major issues before end-users run into them. Smoke testing should take just a few minutes and, ideally, your tests should run in parallel, making them even faster. They should run immediately after new code or features are deployed to production and cover a broad range of core functionality. 

Reporting on smoke testing over time can help teams understand whether they are pushing too hard to deploy new functionality and features at the expense of reliability and uptime, especially when combined with real user monitoring (RUM) and other observability analytics.

Automate smoke testing with synthetic monitors 

You can manually run smoke tests, but it’s easier and more reliable to automate the critical workflows you want to test. For example, if you have an eCommerce app, you might want to test that users can log in, check out, and connect to APIs via their mobile app. Fortunately, you can test this entire process with a synthetic monitor.

Synthetics is a powerful tool in the observability toolbox, but it’s easy to overlook when developers are focusing on application metrics, error rates, and logs. However, once you set up synthetic monitors, you can test on demand and on a regular basis, which improves your confidence in your deployments to production and, ultimately, improves your CI/CD pipelines.

Let’s take a look at several use cases where you can use synthetic monitors with New Relic Synthetics to perform smoke testing. You can also learn more about the different types of monitors from the video below.

Testing performance

First, you’ll probably want to test if your latest deployment changed the performance of your website. With the simple browser monitor, which is a pre-built, scripted browser monitor, you can quickly set up a monitor that makes a request to your site using an instance of Google Chrome by just entering the URL you want to test. 

It is always recommended to run these tests from at least three locations to ensure failed tests are not due to a localized connectivity issue. A fresh container runs simple browser tests each time, bypassing any caching and providing you with fast response times for benchmarking your RUM data.

The scripted browser monitor also tests performance, but it takes a bit more configuration to set up and is more useful when testing user flows, which is described in the next section. By testing performance and capturing core web vitals, you can catch any pain points your users might experience such as slow page loads. 

Testing functionality

After a deployment, your smoke tests should ensure that key functionality in your application is working correctly. This includes user flows like logging in and interacting with your site, the status of links on your page, and the availability of backend services. New Relic has several options available to perform these tests.

Scripted browser monitor: Scripted browser monitors are used for more sophisticated, customized monitoring. You can create a custom script that navigates your website, takes specific actions, and ensures specific resources are present, like the checkout button on your eCommerce site. The monitor uses Google Chrome browser. You can also use a variety of third-party modules to build your custom monitor.

Step monitor: This has the same functionality as the scripted browser monitor, but requires no code to set up. The monitor can be configured to:

• Navigate to a URL
• Type text
• Click an element
• Assert text
• Assert an element
• Secure a credential

Learn more about the step monitor here.

Scripted API monitor: API tests are used to monitor your API endpoints. This can ensure that your app server works in addition to your website. For example, if your mobile app makes calls to a backend service to pull product data, you would use this monitor. New Relic uses the http-request module internally to make HTTP calls to your endpoint and validate the results.

Broken links monitor: You can also check your site for broken links. Provide a URL and this monitor will test all the links on the page. If any links fail, you can view these links individually.

Testing for availability

Sometimes you just want to make sure that your app or site is available. With New Relic’s ping monitor, you can send a request to a URL and ensure that it returns a 200 response. You can run this type of monitor with high regularity and from multiple sites around the world, then view the difference in connection times based on the location. Note that this type of monitor is not an actual ping. The synthetic ping monitor uses a simple Java HTTP client to make requests to your site.

You can also use a monitor to check if your SSL certificate needs to be renewed. While monitoring your SSL certificate isn’t strictly related to smoke testing, you don’t want to scare users away with a “site is not secure” warning.

Use reports to fine-tune your deployments

One of the most powerful ways to improve DevOps processes is ensuring that the right teams and roles have access to reporting data in the form that makes the most sense for them. Tools like synthetic monitors immediately connect this test data with your other telemetry data and provide different ways to visualize that data.

Synthetics provide service-level agreement (SLA) reports at both the account level and at the individual test level. These reports show duration, uptime, and Apdex scores in daily, weekly, and monthly charts. These charts can easily be exported or shared directly with other platform users and external users that don’t have access to the platform. The image below demonstrates how you can share a Daily SLA report with others.

Build a custom synthetic dashboard

You can also set up synthetic dashboards to get deeper visibility and understanding of how your tests and deployments are performing. Once smoke tests are running on a regular cadence, you can build performance data over time. This helps you track deployments and determine which ones are successful. In addition, you can measure each deployment’s impact on performance or business KPIs. This way, you can better understand whether you are effectively balancing the rate of deployments with service availability and stability. 

To get started, use a Quickstart template (from the Apps catalog) to give you sample charts out of the box, or build your own dashboard with custom charts and NRQL queries. This allows your teams to see and share performance data in a more visual and interactive way.

If you build your own dashboard, you could plan it around two strands: success rates and performance. Here are some questions you can ask along with the NRQL queries that will give you the answers you need.

What is the success rate of individual monitors?

SELECT percentage(count(result), WHERE result = 'SUCCESS') AS 'Success Rate' FROM SyntheticCheck WHERE monitorName = 'Your Test Name' SINCE 1 week ago

Which monitors are failing and why? Which locations are failing and why?

SELECT count(result) FROM SyntheticCheck FACET error

SELECT count(result) FROM SyntheticCheck WHERE result != 'SUCCESS' FACET locationLabel, error

What’s the benchmark performance and timing of tests from different locations? 

SELECT average(duration) FROM SyntheticCheck FACET locationLabel 

Use alerts to quickly respond to deployment issues

Finally, you should add alert conditions to notify your teams before your users run into deployment issues. For example, you can set up alerts so you’re notified when any of your tests fail. You can also add alert conditions that trigger when the overall success rate of monitors drops below your SLO’s, or if the performance of any of your tests drops below acceptable baseline levels that you specify. Read more about New Relic alerts here.