Documentation

Measure Performance

TL;DR;
Establish a base metric before the switch to ImageEngine.

Use a third-party tool like WebPageTest to test from multiple locations around the globe, using real browsers and real consumer connection speeds.

User-centric metrics are ideal. These include:

  • "Largets Contentful Paint”
  • “Time to First Meaningful Paint"
  • Lighthouse “Performance”

Don’t forget to monitor the actual payload reduction related to images.

Metrics like “Time To First Byte” are not as important because they don’t reflect user experience.

Still not getting the results you expect? It may be time to investigate your CSS and JS.

There are many tools and services out there that can help to monitor and measure web performance. Performance is a moving target and different metrics serve different purposes. In the following we will look at a few legacy metrics in addition to user-centric metrics. Traditional performance metrics like load time or DOMContentLoaded time are extremely unreliable since when they occur may or may not correspond to when the user thinks the webpage is loaded. Unfortunately, these legacy metrics are still reported by many analytics tools and considered a KPI for many sites.

ImageEngine is all about improving the user experience. Improved user experience has direct impact on the bottom line. But how should you measure the impact of ImageEngine?

Let’s start by looking at some traditional metrics.

# Traditional Performance Metrics

Traditional metrics can be described as metrics that are easily collected programmatically based on events fired in the browser or http traffic timings. These metrics are dubbed “traditional” because the way a webpage is implemented has developed significantly the last couple of years. Additionally, the definition of performance has changed. For example a low TTFB, doesn’t necessarily mean that the key content is visible to the end user any sooner.

# Time To First Byte

Time to first byte (TTFB) may be defined as:

“Time spent waiting for the initial response, also known as the Time To First Byte. This time captures the latency of a round trip to the server in addition to the time spent waiting for the server to deliver the response.”

In the traditional CDN space, a low TTFB has always been an important metric. TTFB is definitely still very important, as a delay here will add directly to any other metric. Looking at this only will however give a skewed picture of how fast a webpage is from the users’ perspective.

# Page Load Time

Page load time is usually reported by the load event or DOMContentLoaded event.

The problem with page load time on today’s modern web is that these events can fire any time regardless of when content is shown or the page is usable for the user. Reasons for this may be asynchronous loading of resources, lazy loading, third-party dependencies or pages rendered client-side using JavaScript. Page Load Time related metrics are available by default in most tools, including Google Analytics, but should be used with care.

# Averages

Another problem with Google Analytics and other tools is that they report averages.

Google Analytics Traditional Timing Metrics

Because of the complexity of the modern web, and the vast number of variables in play when a page is loaded an average is heavily skewed by the long tail of outliers. Whenever possible, stick to percentiles.

# User Centric Performance Metrics

One way to define user-centric performance metrics is think of the characteristics of a page which from the users’ perspective is fast.

Performance Metrics Loading Timeline

The above image illustrates when some user-centric metrics occur during the page load process. Note that TTFB and page load time does not correlate well to any of these.

Google also talks about user centric metrics as “Web vitals”.

Web Vitals is an initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.

A complete definition of user-centric metrics can be found [elsewhere](https://web.dev/lighthouse-performance/#metrics. Google illustrate the most important, Core Web Vitals like this:

Performance Metrics Loading Timeline

# Largest Contentful Paint

Largest Contentful Paint

Most recent metric implemented by Google Lighthouse is Largest Contentful Paint (LCP), deprecating First Meaningful Paint, in Lighthouse 6.0.

“LCP marks the point in the page load timeline when the page’s main content has likely loaded—a fast LCP helps reassure the user that the page is useful.”
(from web.dev)

It is similar to the other paint-based metrics and it measures how soon content is visible to the user.

# Lighthouse and Page Speed Insights

Google, with its focus on web performance, have implemented user-centric metrics in Lighthouse and PageSpeed Insights. The webpage will get a “performance grade” composed of many different metrics, such as First Contentful Paint, Speed Index, Time to Interactive and more. How these metrics are weighted vary slightly across versions. Here is a neat tool that illustrates the weighting and how the score is computed.

PageSpeed Insights

The main takeaway is that improvements in user-centric metrics are more likely to impact your business than the traditional metrics.

# Metrics to Monitor ImageEngine Performance

When evaluating ImageEngine performance it is important to establish a current baseline or in other ways create a data set (like A/B tests) to which ImageEngine performance metrics can be compared. As explained in this document, user-centric metrics are recommended. If you don’t have these metrics implemented it is highly recommended to do so.

At the same time, you should stick to whatever datasets you have at hand and datasets that your business use on a daily basis. These sets can for example be whatever is available by default in Google Analytics or other analytics tools you may use. It is very likely that ImageEngine will have positive impact on all performance related metrics.

Make sure the ImageEngine cache is warm (few cache missses) before running performance tests! If not there will be a heigh degree of cache misses which will slow down delivery.

# Payload Reduction

Monitoring the reduction in payload generated by images is probably the most isolated metric to look at when evaluating ImageEngine. Comparing the byte size of images with- and without ImageEngine enabled clearly illustrate the core purpose of ImageEngine.

The easiest way to monitor this is either through the control panel or using tools like webpagetest.org to compare before and after data.

# Time To First (Last?) Byte

TTFB may vary more with ImageEngine than with other CDNs because ImageEngine does real time optimization. In the case when ImageEngine has to produce a new version of an image, a few extra milliseconds are spent server side before returning the payload. You’ll notice this as outliers in your datasets.

ImageEngine perform quite well compared to other CDNs with regards to TTFB when the cache is properly warmed:

ImageEngine TTFB
50th percentile 30ms
90th percentile 65ms
95th percentile 91ms

It is also important to remember that even if ImageEngine might sometimes be slower in TTFB, ImageEngine will send less data over the wire. Especially in slow mobile networks less data transfer will weigh up for a delayed TTFB because most of the time on slow networks is spent on downloading data to the browser. In these settings what really count is “Time to Last Byte”.

# Largest Contentful Paint and Images

How fast an image is loaded and painted on the screen has direct impact on LCP. From ImageEngine’s perspective this is then one of the more important metrics to keep an eye on.

In addition to blocking CSS or slow JavaScript rendered content, Server response time and asset/sub-resource (typically images) load time are the primary factors affecting LPC.

As currently specified in the Largest Contentful Paint API, the types of elements considered for Largest Contentful Paint are:

  • <img> elements
  • <image> elements inside an <svg> element
  • <video> elements (the poster image is used)
  • An element with a background image loaded via the url() function (as opposed to a CSS gradient)
  • Block-level elements containing text nodes or other inline-level text elements children.

# How to measure LCP

LCP can be monitored in Chrome User Experience Report or by using the Largest Contentful Paint API. The following example shows how to create a PerformanceObserver that listens for largest-contentful-paint entries and logs the LCP value to the console:

// Create a variable to hold the latest LCP value (since it can change).
let lcp;

// Create the PerformanceObserver instance.
const po = new PerformanceObserver((entryList) => {
  const entries = entryList.getEntries();
  const lastEntry = entries[entries.length - 1];

  // Update `lcp` to the latest value, using `renderTime` if it's available,
  // otherwise using `loadTime`. (Note: `renderTime` may not be available if
  // the element is an image and it's loaded cross-origin without the
  // `Timing-Allow-Origin` header.)
  lcp = lastEntry.renderTime || lastEntry.loadTime;
});

// Observe entries of type `largest-contentful-paint`, including buffered
// entries, i.e. entries that occurred before calling `observe()`.
po.observe({type: 'largest-contentful-paint', buffered: true});

// Send the latest LCP value to your analytics server once the user
// leaves the tab.
addEventListener('visibilitychange', function fn() {
  if (lcp && document.visibilityState === 'hidden') {
    console.log('LCP:', lcp);
    removeEventListener('visibilitychange', fn, true);
  }
}, true);

Instead of logging to console, one could report LCP in Google Analytics through custom dimensions.

Read more about measuring LCP

# Percentiles

Any metric on the web should be analyzed in percentiles rather than averages or single requests. Normal web traffic will always produce some outliers which will skew the average. Whenever possible, try to organize the data so that you can look at percentiles. Focus on for example the 50th, 75th and 95th percentile.

# Other Issues Affecting Performance Negatively

In some cases the datasets and test results does not show the expected improvement to web performance metrics even if ImageEngine reducing payload significantly. In these cases there are usually other resources blocking or slowing down the rendering (or painting) of content to the screen.

Typically these resources are:

  • JavaScript
    • Try to move as much JavaScript out from the <head> tag as you can. Defer loading whenever you can.
    • Be careful with lazy loading of images. When possible, do not lazy load images above the fold such as hero images. Do not use custom JavaScript lazy loaders that are breaking the browser’s pre-loader functions. A rule of thumb: Make sure the src attribute is present with a valid image url in your html.
    • Monitor the browsers main-thread workload. Do what reduce the time spent parsing, compiling and executing JS.
  • CSS
    • CSS is render blocking by nature because it contains information about how the page should be rendered to the screen. Try to inline CSS for critical above-the-fold content.

For JavaScript and CSS in head TTFB is critical because nothing, not even images already optimized by ImageEngine and downloaded by the browser, will be displayed on the screen until the JavaScript and CSS is downloaded and made sense of by the browser.

See web.dev for more tips on render blocking resources.

# Conclusion

Continue to monitor your current metrics even after switching to ImageEngine. However, it is strongly recommended to move to modern user-centric metrics if you haven’t done so already.

Look at percentiles, not averages.

When evaluating ImageEngine, isolate the delivery of images. Monitor things like image payload, image formats, image sizes, and network related performance metrics.

If the site has other performance issues caused by CSS or JavaScript, the improvement in image delivery may not be visible in some metrics. Make sure to eliminate render blocking CSS and JavaScript.