92%, 99%, 99.9%… The past few years have seen lots of flashy announcements touting accuracy statistics from various cross device solutions that are confidence inspiring at first glance. These results leave much to be desired, though, when we dig just one level deeper and ask exactly how datasets are matched and what constituent quantities are compared in order to produce those stats. For example, some have gone so far as to include cases where the cross device solution did *not* make a link in the accuracy computation. For example, if your algorithm make a link in 5% of possible cases these would be evaluated for veracity. However the 95% of the cases where a link was not made would also be included in the “correct” piece of the calculation if there was in fact no link between the ids. While this is in some theoretical sense appropriate, our position is that it doesn’t answer the basic question of interest as regards accuracy, which is: “When you declare to me that there is a link between cookie A and device B, what’s the probability that your declaration is true?”
Transparency when reporting our metric dictates exact detailing of how we arrive at the numbers we claim. This requires some preliminaries. Firstly, we have a probabilistic device graph that is the result of our internal processing. The final output of this is a list of cookie ids (Lotame ids), which are each matched to a number of mobile device ids (IDFAs on iOS and GAIDs on Android). On the other side, we have deterministic matches that comes from a number of providers and are indexed on the same ids as used by our graph. This results in a comparison link by link between the two graphs where there are four possible cases:
Once all these quantities are defined, accuracy and reach are consequently well defined and are computed in a straightforward fashion. Accuracy is the rate of true positive to all positive (true positive/(true positive + false positive)) and reach is the rate of true positive to all true links (true positive/(true positive + false negatives))
We suspect that our measure of accuracy is a lower bound on the true statistic as it is impossible to collect a deterministic graph which is in any sense complete or completely accurate. For example, if we use publisher or online service X, a user A may happen not to log into site X on their phone. Even worse, a friend user B may log in on his or her own phone and have logged in once on user A’s desktop browser. There is no way to ultimately verify truth. However, if both deterministic and probabilistic methodologies agree that two ids are linked, it stands to reason that it’s unlikely both are wrong.
Our default settings create a graph with above 90% accuracy and around 75% reach. We believe this to be sufficient for a wide variety of data propagation use cases including targeting, retargeting, and aggregate level analytics where the underlying data is unlikely to exhibit accuracy beyond this limit. We do not recommend expanding reach beyond our standard limits because the marginal accuracy for each extra link we add decreases. Our cutoff, while achieving a net 90%+ accuracy rate for the graph, yields a minimum marginal accuracy rate of around 65% for the lowest scoring links. We do not think it’s valuable for any customer to include links which a lower marginal accuracy. However, for more specific use cases where the accuracy is of utmost importance, the graph can be tuned up to almost 97% accuracy where reach is sacrificed down to around 20%. In between these two extremes, the tradeoff is that from almost any point on the curve, we give up around 50% of reach for every 2-3 points of accuracy.
This article was written by Lotame’s Chief Data Scientist, Omar Abdala, who was recently awarded the American Marketing Association’s 4 Under 40 Emerging Leaders Award!