Mastering Tracking Technologies: Transcend Engineers Do the Legwork

Today, web teams rely on conversion funnels, which track a user’s journey from landing on a site to converting (signup, purchase, etc.). However, GDPR requires websites to get consent before collecting data about the user, which led to the all-too-common, nightmare user experience of pop-up cookie banners.

Our team set out to find a way to move consent management beyond pop ups by creating the ability for web teams to ask for consent later in the user journey. Borrowing the approach made popular by crash reports, we designed a solution that can capture tracking events locally but doesn’t upload the data until consent is given by the user.

Spoiler: the account below is full of our blood, sweat, and tears. But it was important that we get this right so that other engineering teams wouldn’t have to go through what we did. Now there’s a solution that everyone can use.

Our decision to use local tracking introduced significant engineering challenges because most analytics happen through imported third-party scripts, like Google Analytics. Once they’re imported, these scripts immediately begin tracking and uploading data about the user. Unfortunately for our case, it’s very difficult to alter the functionality of third-party scripts.

It was clear early on that we would need to build the capability to govern the data flows of these scripts without the ability to change the scripts themselves. Building Airgap.js as a firewall gave us the ability to set governance rules on all network traffic, allow essential traffic through, and block or allow other traffic based on the user’s stated consent preferences.

The firewall architecture also enables Airgap.js to quarantine local tracking events on the user’s device unless (or until) the user gives consent for that data to be shared. By default, it blocks scripts that do not map to consented tracking purposes and postpones their execution for later consent.

Before we could start building, we needed to understand how all the trackers worked to ensure we could correctly capture and replay all their data flows. Unlike previous consent managers, we wanted our customers to be able to govern all tracking technologies (not just cookies) on a website automatically. Every data emission must be locally assessed against the user’s consent preferences before being allowed to leave the user’s device.

Traditional consent managers assume cookies are the only method for tracking, that functional third-party scripts don’t have tracking capabilities, and that all tracking happens through third-parties. Yet, these assumptions don’t align with everything we know about tracking technologies.

First, there are dozens of methods to track users on a website, not only cookies (e.g. XHR, Fetch, Pixels, web beacons, iframe). We also know third-party widgets, fonts, payment processors, and customer data platforms (CDP) also include tracking functionalities which are often excluded from consent managers. Finally, because first-party tracking also happens, we built functionality to block certain locations from loading based on their URL rather than just their hostname.

To find the best engineering strategy, we painstakingly explored several potential ideas including sandboxing every script in its own locked down iframe and using dynamic content security policies. Long story short, those didn’t work for a variety of reasons, which we discussed during our presentation at PEPR 21 and as well as this previous post. Ultimately, we landed on a combination of patching, virtual documents, and some CSPs to cover everything.

One of the most challenging and complex aspects of managing consent in this way is the need to understand the behavior of all the various tracking technologies on your site in order to correctly capture and replay their data flows. There are many tracking technologies to cover, from JavaScript networking APIs and all the different types of HTML elements that can cause requests, to parts of the browser that can also perform requests.

We knew we needed to take the burden of understanding every possible tracking technology, so we began by cataloging and indexing various methods that browsers can create network requests. Because our engineers are already experienced with the DOM (Document Object Model) and other browser technologies, we were able to design a system which precisely patches request-causing triggers transparently without any site breakage.

The average web page receives dozens of data requests from trackers from cookies, super cookies, embedded scripts, and fingerprints. Keeping your consent manager updated with the latest iterations of each type of tracker can be exhausting, so we’re making it easy for you with Transcend’s Consent Manager. You don’t need to worry about manually updating your consent manager for each individual tracker, ongoing automatic updates are built into our product.

Another often overlooked aspect of effective consent management is efficiently synchronizing consent cross-domain. Some consent managers must wait for a consent sync before they can load tracker scripts, which poses performance problems and tracker collection gaps.

As a result, internal engineers teams often struggle maintaining performance and user session observability when paired with blunt consent management solutions. We built a solution for that directly into our consent manager.

Our quarantine and replay system allows site owners to load most tracker scripts immediately, quarantine subsequent emissions, and replay these events after a deferred consent sync. This solution improves site performance while at the same time closing these tracker collection gaps.

Today’s tracking technologies are nuanced and ever changing–and keeping your internal team across it all requires industry knowledge and ongoing engineering resources. Our team has engineered a solution that regulates across the expansive breadth of tracking technologies and does so in detail. Interested in seeing Transcend Consent in action? Sign up below to get started today.