Defeating cookie banners

Modern privacy laws like GDPR dictate that companies must secure consent from their users before tracking their personal data and actions on the site. This is particularly a problem for sites that rely on conversion funnels and track a user’s journey from landing on the website to a conversion (eg. sign up or purchase).

In order to get the full story, while complying with modern privacy laws, companies ask for consent as early as possible. At a time when UX engineers are focused on providing a clean user experience, the web is suddenly cluttered with distracting, annoying pop-ups.

If you’re familiar with Transcend, you know we’re all about building high-end user experiences surrounding privacy, and then making it simple for our customers to offer these experiences to their users. We agreed that we wouldn’t build a consent manager that was just another pop-up, rather, we should defeat the pop-up

This decision led to a core question: How can we ask for consent later in the journey, while preserving the full story for the conversion funnel?

Asking for consent with embedded UI

Our solution: let users land on the website without a banner, and allow tracking events to be retained locally on the user’s device. Then, at an appropriate moment, we could ask the user for permission to release the full story.

Asking users for consent later in the funnel

Building Airgap: a journey into the depths of browser APIs.

Pivoting to local tracking would be a challenge, since most analytics happen through third party scripts that begin tracking and uploading once they are imported. Imported code is difficult to change so altering the behavior of these third party scripts is challenging.

This technology, Airgap.js, could mimic a firewall on the browser level and regulate all network traffic on the user’s device. It would inspect all outgoing transmissions, classify each purpose {Essential, Functional, Analytics, Advertising}, and block or allow the transmission according to the user’s consent preferences. This firewall essentially converts all tracking into local tracking – it would allow arbitrary analytics on the browser, but capture and quarantine any attempt to transfer that data off of the device to a remote data store.

Additionally, Airgap.js can quarantine non-essential data transmissions locally on the visitor’s browser, Then, if and when consent is given, it can replay these transmissions so that the data can flow to their respective downstream systems, like Google Analytics.

First Idea: Sandboxed iframe documents

Initially, we sandboxed every script in its own locked-down iframe with a copy of the current document. We then re-dispatched parent document events into these iframes and attached mutation observers onto the iframe’s document context to capture any request-causing DOM nodes. We forwarded mutation data up to the parent document, which could then allow or deny specific mutations based on tracking consent criteria.

This solution benefited from a simple implementation, but didn’t scale well enough to keep up with our performance demands. As a site included more and more scripts, the memory usage and CPU requirements became untenable.

Our initial attempt could cause site functionality to break if DOM mutations were replayed back in the wrong order. This meant that we could not support script async and script defer in a high-performance manner without continuously synchronizing each sandboxed iframe document. We needed to find another way to allow scripts to run while intercepting all possible data emissions.

Second Idea: Dynamic Content Security Policies

Next, we attempted to generate dynamic Content Security Policies (CSPs) which provide total control over the network requests. Site owners can use CSPs to specify a list of URLs that the user is allowed to connect to. CSPs are usually specified directly by web servers through HTTP response headers, but they can also be set through meta tags generated at runtime.

Here we dynamically generate a meta tag CSP by analyzing a list of potential domains that a site may request to and specify the CSP’s “allow list” based on the purposes of each request and the user’s own tracking consent preferences.

The CSP method can block all tracking attempts, but it doesn’t offer an easy way to replay them once the user provides consent. CSP cannot capture HTTP POST body data from blocked requests, which means that it cannot reliably replay requests made from form submissions or JavaScript networking APIs.

Critically, on a given page-load, a CSP can only be updated to become more strict about where data can flow, so when consent is given, we cannot update the CSP to allow data to flow to a new destination, not without a page reload. The CSP method was excellent at blocking data emission, but we needed something that could reliably replay events, and handle consent changes seamlessly.

This method surfaced an important question though – What should the default behavior be for a network request that is missing from our purpose map? Allow it through, and guarantee we’re not breaking something on the site? Or block it, and guarantee we’re not accidentally letting an analytics event through? CSPs enforce the latter. While this is a rare corner-case, we think the default behavior should be up to the company.

The Solution: Patchers and Virtual DOM

We then tried to combine the best parts of both approaches. We detect JavaScript network API events by patching and overriding all of the global interfaces used to create network requests, such as XMLHttpRequest(), fetch(), and navigator.sendBeacon(). Our patched methods evaluate requests for any unconsented tracking purposes and quarantine the request for release on consent.

Airgap.js also detects pending DOM mutations that contain request-causing DOM nodes by directly patching the prototypes, methods, and accessors of all global element constructors and DOM tree construction utilities. We can write a patcher which overrides the constructor prototype, which allows us to check if the URL is allowed before the DOM mutation is performed. If it is, we’ll import the request and if not, we will quarantine it. This enables us to regulate mutations in real-time, before they can cause network requests or modify the document tree.

However, DOM APIs that take HTML strings as input do not use an image constructor. To address these exceptions, we patch these methods to insert nodes into a sandbox first using virtual documents. These are much faster than iframes and are unable to make network requests.

From Fetch, XHR, img and more, there are many parts of the browser stack that can cause network requests. By using patchers, virtual documents and sometimes CSP, we could cover everything.

This approach enables us to regulate page mutations in a highly-performant manner that scales from the smallest blogs to the most complex single-page web applications, without CPU strain or memory bloat. It governs every type of network request and allows for replay, giving site owners flexibility on when to ask for user consent. This technology is bundled into Airgap.js – a 30KB drop in script that acts as a browser firewall which allows for policy-based data flow governance.

Developing additional Airgap.js capabilities

As we explored these approaches, we discovered additional ways to make it easier for companies to build and manage user consent within modern day, complex privacy environments and site structures.

Our consent sync architecture allows users to privately set tracking preferences across multiple domains without sharing this consent data over the network. By integrating Airgap.js with a cross-domain postMessage interface, along with a list of known trusted domains, we can mediate and process consent sync events across every domain without sharing consent state over the network.

Airgap.js can also actually modify or overwrite requests (eg. a cookie or querystring) before the request is made. This allows us to actually offer an out-of-the-box adoption of Facebook’s Limited Data Use parameter or restricted data processing settings for Google publisher ad tags. If a user says “Do Not Sell” we can automatically rewrite all of their network requests to Facebook or Google’s APIs to include this parameter.

What’s next

Modern privacy laws don’t have to be anathema to UX. With Airgap.js, Transcend’s Consent Manager enables companies to collect data at an appropriate moment without losing the opening chapters of the user’s story. By synthesizing the best of our ideas, we engineered a solution that harmonizes user privacy, clean UX and a company’s business goals.

Experience Transcend Consent Manager for yourself. Learn more here or get a demo.