Frozen User Agent

The greatest gift you can give the wicked is your inaction.
                     - DaShanne Stokes

Wishing is nice but doing is better. Take action.
                     ― Akiroq Brost

Do not wait until regrets are your only option,
make the changes and act before it’s too late.
                     ― RJ Intindola


Outside of a narrow niche of experts, many people have a good opinion of Google. After all, they offer a ton of useful services free of charge to users, organizations and companies across the globe. Over the years, Google has conquered a dominant position in many fields and it was just a matter of time before that advantage was used to establish a monopoly and bully competitors out of the respective markets. Those impacted are mighty aware of what Google can do by exploiting its dominant position. The internet is an ecosystem in which a whole industry (Google included) has thrived since the inception of its mass adoption. With this website, we intend to draw attention to how Google is hellbent on redefining the technology stack that has enabled the ecosystem since the mid 90’s. Once the context is clear, we will also provide the tools to enable publishers to counteract Google's efforts on at least one front, the User-Agent freeze. Google is acting as if they are the internet, and everyone else has to either become an element in Google’s ecosystem or succumb. Accepting Google’s plan for world domination might be almost tempting for many, as resistance may appear futile, but things don’t have to be that way.

First, whenever Google single-handedly manages to redefine internet standards, it makes it harder for the rest of the ecosystem to resist its power in the future. Secondly, once a company is reduced to a well-behaved subject in Big G's kingdom, Google (essentially a private company, regardless of its being publicly traded) will turn its knobs and reduce other companies’ margins to make more money for itself. This is not ok, and we believe that many governments are increasingly aware of this.

In the meantime, there are many Google “initiatives” that are directed at redefining how the internet has always worked, such as AMP pages, the blocking of third-party cookies (from the comfort of a first-party data business model), FLoCs, push-back on the attempt to define alternative user IDs by other players in the Ad Tech industry and more.

Google’s narrative is powerful: they argue that they intend to protect user privacy. We would be moved by the nobility of their motivations if it wasn’t for a little detail: they are “first party”, they know everything about every user on the planet and every user has already agreed to let Google have its way with his/her personal data! Don't be fooled. Google is not protecting its users' privacy, it is protecting its own business. They want everyone's ads to be worse than Google's, so everyone is forced to use Google in the end. Hiding people's private data from everyone but themselves is part of the plan.

To be frank, we are not here to take on all aspects of such a huge discussion. That would be too much to chew and its the task of governments, really. We'd rather focus on one specific aspect of all this: Google's egregious attempt to disrupt time-honord HTTP mechanisms for the purpose of gaining an advantage over other players in the Ad Tech world. We are talking about Google’s attempt to “freeze” the User-Agent string, i.e. demolish a mechanism that has been in place for 30 years (and 3 major revisions of HTTP, -- HTTP 1.0, 1.1 and 2.0) and that is still used today by publishers and services around the planet. The final goal is always the same: make everyone competitor's life harder by creating new hurdles for everyone but Google itself.

Now that you have the context, we can talk about what we can do about this. This website is about making the world aware of the issue as much as orchestrating a reaction to the User-Agent freeze by publishers who are willing to react together to Google's arrogance.

The right to refuse service

Back in the 90’s, some privacy-conscious users would disable cookies through their browser settings, only to discover that many sites would become inaccessible in that case (users should enable their cookies and come back). Websites needed (and still need) cookies to support the login functionality. But we don’t need to go that far back in time to find examples of publishers that refuse service to some users for a good reason. In May 2018, GDPR, the European General Data Protection Regulation, became enforceable for all websites accessible by users based in EU (i.e. all of them). Some US-based websites (including a few major ones) decided to block EU-based users, as they felt that complying to the new regulation was far from simple and they needed more time to figure out how to avoid significant liability.

More recently, many websites are banning browsers that adopt ad-blocking techniques, and they are doing it for economic reasons this time. As many of us know, when ad-blockers are detected, many websites will politely ask users to disable them, as the livelihood of the websites depends on adverising. Other websites will straight out refuse service for the same reason.
We users don’t have a paid subscription to most websites we visit on a daily basis, and advertising is the only way websites can support themselves, i.e. pay the salaries of the people who work to offer content and services. The point here is that publishers are within their full rights to determine what the minimal browser requirements are in order to access their services. If publishers agree that freezing (effectively removing) the UA string is bad for the sustainability of their business, they should go ahead and let users know about it, so that users can tell device manufacturers to fix their browsers and re-establish a correct balance in the ecosystem.

The devil is in the details

The UA freeze is a big change, albeit its impact is hard to explain in simple terms to non-experts. Also, to muddy the water even further, Google is suggenting that Client-Hints are a replacement for all the use-cases in which the User-Agent in normally used. But this is not the case. First, there is no reason to freeze the User-Agent string in order to add Client-Hints. More importantly, there are plenty of cases where Client-Hints may not be quite up to the task of replacing the User-Agent. The user-agent string is needed to support a plethora of other use cases ranging from UX (user experience) optimization to image/video resampling to content security policy to fraud detection to Javascript polyfills to detecting webviews to bug workarounds to analytics and more. Google’s move will further restrict publishers’ ability to deliver meaningful service to their users. Among the examples:

  • Image/video resampling
  • Android OS / hardware capabilities detection for providing correct APK build for each user.
  • Content-Security-Policy: Dynamic on-the-fly adaptation so different browsers get different CSP headers.
  • Fraud detection (forged User-Agent detection).
  • Browser bug workarounds.
  • Content negotiation.
  • Detecting whether users are browsing within an Android/iOS app WebView.
  • Analytics

These are all situations in which companies have come to rely on the User-Agent string, and a frozen UA will adversely affect their ability to deliver their services. What we have now is Google trying to sneak in a big change (that is hard to explain to a lay person) as part of a browser update. Privacy concerns are an excuse. If we look at the top 150 User-Agent string in any given moment in any region, we will find that those are responsible for 50% of the traffic in that region, or, put another way, 1.5 billion people share the same UA string. The idea that the user-agent string is commonly used to identify users (Google's argument) is simply fake news. Google intends to kill the User-Agent string (essentially by freezing it to death) for the purpose of further reinforcing its dominant position in the internet ecosystem. Google hopes that this change is technical enough that publishers will fail to fully realize and nobody will complain. But the threat is real. If Google has its way now in this and in other areas, they will be emboldened and this will open up to more and more egregious abuse of its anti-competitive, monopolistic attitude.

Call to arms

Doing something about Google's behaviour is possible. We urge publishers to join in a collective reaction. This should happen in the form of detecting browsers with a frozen User-Agent string and denying service in that situation, or, at least, penalizing the browser by degrading the experience and making users aware of what is happening. Users should be invited to turn to their browser and device manufacturers, requesting that the previous configuration (i.e. a meaningful UA string) is restored. This will send a clear message from publishers and signal that bullying by the internet giants is not OK.

How to that in practice is relatively simple. We have included code samples here.

In this document, we tried our best to explain the issue at a level that provides enough information to technology-savy readers while keeping it at a sufficiently high level that everyone can see the big picture of what Google is doing and how the ecosystem can react. We have collected more specific information in the Frequently Asked Question page.

If you want to get in contact with the editor of this document, you are contact us on Twitter at @NoUAFreeze