How the score is made

Bye Spy reports a score out of 100 and a letter grade. Both summarise the same measurements the report lists check by check. This page explains how: how a single check produces a value, how the values are combined, why some checks weigh more than others, and what the score does not tell you.

From a measurement to a check's value

Each check reads raw measurements from the run and returns two things: a status, which the report shows as a coloured label, and a value between 0 and 1, where 1 means the browser fully withheld what the technique needs and 0 means it handed everything over. A value strictly between the two is partial protection. The rules for each check are written out on its area page.

Some outcomes are not scored at all. A check that could not run says so rather than guessing, and its absence is never counted as a pass or a fail.

Where a check's value is computed as a fraction (for example, the share of storage types kept separate), a result of at least 0.999 is reported as Safe, at most 0.001 as Leaks, and anything in between as Partly leaks with its exact value kept for the score.

Combining checks into a score

The checks fall into four scored areas. Within an area, the area's value is the weighted mean of the checks that produced a value:

area value   Sa = Σ (wi × si) / Σ wi      over the checks i in area a that were scored

score        S  = round( 100 × Σ (Wa × Sa) / Σ Wa )   over the areas with at least one scored check

Renormalising keeps the score a fair average of what was measured, at the cost of making scores from different runs not strictly comparable when different checks were left out. The report states the basis every time ("from 21 of 24 checks", for example), and it is worth reading alongside the number. Two cases are common:

• The real tracker servers are left out (the box is unticked). The tracker-blocking area then rests on the bait-file check alone, which carries the whole area's weight. Ticking the box gives a fuller picture of blocking.
• The tracker window is skipped or blocked. Every check that needs the tracker's own site drops out: third-party cookies, storage and cache partitioning, the cross-site fingerprint, click-ID stripping and window.name. The score then leans on the checks a single page can make.

A small worked example: an area with three checks weighted 4, 2 and 1, whose values are 0, 1 and 0.5, has the value (4 × 0 + 2 × 1 + 1 × 0.5) / 7 ≈ 0.36. If the first check could not run, the same area is worth (2 × 1 + 1 × 0.5) / 3 ≈ 0.83.

The weights

The table below is read from the test's own scoring when this page loads, so it always shows the weights the test is using. "Most points" is what a check can move the final score when every check in its area is scored.

The table of weights needs JavaScript to show here.

Why the areas weigh what they do

Third-party web tracking rests on one capability: recognising the same browser on unrelated sites. Mayer and Mitchell's survey divides the means into stateful ones, where the tracker stores an identifier in the browser, and stateless ones, where it derives an identifier from the browser's properties^[1]. Those are exactly the two heaviest areas:

• Cross-site tracking (stateful) and fingerprinting (stateless) share the largest weights equally. Each, on its own, is enough to build a cross-site profile, and each is what a browser's own privacy architecture is designed to stop: partitioned storage against the first, randomised or uniform values against the second.
• Tracker blocking weighs less. A request that never leaves the browser can neither store nor derive an identifier, which makes blocking a strong defence in depth. But it is list-based and always incomplete: a tracker missing from the list, or served from the site's own domain, gets through. It is scored as a layer, not as the foundation.
• Other leaks weighs the same as blocking. Each channel there is narrower (one reveals an address, another the previous page, another a click), but together they cover the ways identifiers and context travel beside cookies and fingerprints.

The context the report shows under What every site sees carries no weight: your IP address and the basic description your browser sends are visible to every server you talk to, and no browser setting changes that. Scoring them would reward or punish something the browser cannot control.

Why the checks weigh what they do

Within an area, a check weighs more when the technique it tests is (a) widely deployed, (b) able to identify a browser on its own rather than only narrowing it down, and (c) durable across time. Third-party cookies and the cross-site fingerprint comparison are the heaviest checks of their areas because they test the end result directly: whether one identifier was readable on two sites. Single fingerprint components weigh by how much they reveal. A canvas or WebGL rendering separates far more browsers than a screen size, which millions of people share. Each area page explains the weight of each of its checks.

The grade

The letter is a coarse reading of the score, so that a result can be told at a glance. The bands, as the test applies them:

The grade bands need JavaScript to show here.

The bands are deliberately uneven. An A takes a browser that withholds almost every identifier a tracker looks for, which today means a browser with tracking protection built in and switched on. A browser that only blocks third-party cookies, with no defence against fingerprinting, tends to land in the middle bands, because one of the two heaviest areas still leaks.

What the score does not say

• It is not a measure of anonymity. A browser can withhold every identifier Bye Spy looks for and still be recognised by the account you sign in with, the IP address you connect from, or data a site passes to its partners server to server.
• It describes one browser profile at one moment. Extensions, private windows, settings and browser updates all change the result. Run it again after a change.
• It is not uniqueness. The fingerprinting checks test whether a value is stable and readable across sites, not how many other people share it. A stable, common value is still an identifier when combined with others.
• The weights are a judgement. They follow the reasoning above, and other reasonable weightings exist. That is why the report always shows every check's outcome alongside the number.
• Some techniques cannot be tested from a web page at all. They are listed on What a web page cannot test.

References

1. J. R. Mayer and J. C. Mitchell. "Third-Party Web Tracking: Policy and Technology." IEEE Symposium on Security and Privacy, 2012. doi:10.1109/SP.2012.47