You Kon’t Know Jacc(ard). When the Jaccard similarity index isn’t… | by Eleanor Hanna | Jul, 2024

I’ve been contemplating at the moment about one among my go-to info science devices, one factor we use pretty a bit at Aampe: the Jaccard index. It’s a similarity metric that you simply simply compute by taking the size of the intersection of two items and dividing it by the size of the union of two items. In essence, it’s a measure of overlap.

For my fellow seen learners:

Many (myself included) have sung the praises of the Jaccard index on account of it seems to be helpful for plenty of use cases the place you should decide the similarity between two groups of parts. Whether or not or not you’ve acquired a relatively concrete use case like cross-device id determination, or one factor further abstract, like characterize latent shopper curiosity lessons based mostly totally on historic shopper conduct — it’s really helpful to have a tool that quantifies what variety of components two points share.

Nevertheless Jaccard isn’t a silver bullet. Sometimes it’s further informative when it’s used along with totally different metrics than when it’s used alone. Sometimes it’s downright misleading.

Let’s take a greater take a look at plenty of cases when it’s not pretty acceptable, and what you might want to do in its place (or alongside).

The problem: The bigger one set is than the alternative (holding the size of the intersection equal), the additional it depresses the Jaccard index.

In some cases, you don’t care if two items are reciprocally comparable. Presumably you merely want to know if Set A largely intersects with Set B.

Let’s say you’re making an attempt to find out a taxonomy of shopper curiosity based mostly totally on looking out historic previous. You should have a log of the entire clients who visited http://www.luxurygoodsemporium.com and a log of the entire clients who visited http://superexpensiveyachts.com (neither of which can be keep hyperlinks at press time; fingers crossed no one creepy buys these domains in the end).

Say that out of 1,000 clients who browsed for great expensive yachts, 900 of them moreover appeared up some luxurious objects — nonetheless 50,000 clients visited the luxury objects web site. Intuitively, you might interpret these two domains as comparable. Virtually everyone who patronized the yacht space moreover went to the luxury objects space. Appears to be like like we could also be detecting a latent dimension of “high-end purchase conduct.”

Nevertheless on account of the number of clients who had been into yachts was quite a bit smaller than the number of clients who had been into luxurious objects, the Jaccard index would end up being very small (0.018) even though the overwhelming majority of the yacht-shoppers moreover browsed luxurious objects!

What to do in its place: Use the overlap coefficient.

The overlap coefficient is the size of the intersection of two items divided by the size of the smaller set. Formally:

Let’s visualize why this can be preferable to Jaccard in some cases, using in all probability essentially the most extreme mannequin of the difficulty: Set A is a subset of Set B.

When Set B is pretty shut in measurement to Set B, you’ve acquired a superb Jaccard similarity, on account of the size of the intersection (which is the size of Set A) is close to the size of the union. Nevertheless as you keep the size of Set A unbroken and improve the size of Set B, the size of the union will improve too, and…the Jaccard index plummets.

The overlap coefficient doesn’t. It stays yoked to the size of the smallest set. That means that while the size of Set B will improve, the size of the intersection (which on this case is your complete measurement of Set A) will always be divided by the size of Set A.

Let’s return to our shopper curiosity taxonomy occasion. The overlap coefficient is capturing what we’re involved about proper right here — the patron base for yacht-buying is expounded to the luxury objects shopper base. Presumably the online optimization for the yacht website online isn’t any good, and that’s why it’s not patronized as quite a bit as the luxury objects web site. With the overlap coefficient, you don’t have to stress about one factor like that obscuring the connection between these domains.

Skilled tip: if all you’ve got are the sizes of each set and the size of the intersection, you might discover the size of the union by summing the sizes of each set and subtracting the size of the intersection. Like this:

Further finding out: https://medium.com/rapids-ai/similarity-in-graphs-jaccard-versus-the-overlap-coefficient-610e083b877d

The problem: When set sizes are very small, your Jaccard index is lower-resolution, and usually that overemphasizes relationships between items.

Let’s say you’re employed at a start-up that produces cell video video games, and in addition you’re rising a recommender system that means new video video games to clients based mostly totally on their earlier having fun with habits. You’ve acquired two new video video games out: Mecha-Crusaders of the Cyber Void II: Prisoners of Vengeance, and Freecell.

A highlight group possibly wouldn’t peg these two as being very comparable, nonetheless your analysis displays a Jaccard similarity of .4. No good shakes, nonetheless it happens to be on the higher end of the alternative pairwise Jaccards you’re seeing — in any case, Bubble Crush and Bubble Exploder solely have a Jaccard similarity of .39. Does this suggest your cyberpunk RPG and Freecell are further fastidiously related (as far as your recommender is anxious) than Bubble Crush and Bubble Exploder?

Not basically. Because you took a greater take a look at your info, and solely 3 distinctive system IDs have been logged having fun with Mecha-Crusaders, solely 4 have been logged having fun with Freecell, and a pair of of them merely occurred to have carried out every. Whereas Bubble Crush and Bubble Exploder had been each visited by numerous of items. On account of your samples for the two new video video games are so small, a in all probability coincidental overlap makes the Jaccard similarity look quite a bit higher than the true inhabitants overlap would possibly be.

What to do in its place: Good info hygiene is always one factor to recollect proper right here — you probably can set a heuristic to wait until you’ve collected a certain sample measurement to ponder a set in your similarity matrix. Like all estimates of statistical power, there’s a element of judgment to this, based mostly totally on the on a regular basis measurement of the items you’re working with, nonetheless be mindful the general statistical best observe that greater samples are usually further marketing consultant of their populations.

Nevertheless another option you’ve got is to log-transform the size of the intersection and the size of the union. This output must solely be interpreted when evaluating two modified indices to at least one one other.

In case you occur to try this for the occasion above, you get a score pretty close to what you had sooner than for the two new video video games (0.431). Nevertheless since you’ve got so many further observations inside the Bubble type of video video games, the log-transformed intersection and log-transformed union are fairly a bit nearer collectively — which interprets to a quite a bit higher score.

Caveat: The trade-off proper right here is that you simply simply lose some determination when the union has plenty of parts in it. Together with 100 parts to the intersection of a union with 1000’s of parts could suggest the excellence between an on a regular basis Jaccard score of .94 and .99. Using the log rework technique could suggest that together with 100 parts to the intersection solely strikes the needle from a score of .998 to .999. It relies upon upon what’s important to your use case!

The problem: You’re evaluating two groups of parts, nonetheless collapsing the climate into items ends in an absence of signal.

That’s the reason using a Jaccard index to verify two objects of textual content material isn’t always a implausible thought. It might be tempting to try a pair of paperwork and want to get a measure of their similarity based mostly totally on what tokens are shared between them. Nevertheless the Jaccard index assumes that the climate inside the two groups to be in distinction are distinctive. Which flattens out phrase frequency. And in pure language analysis, token frequency is normally really important.

Take into consideration you’re evaluating a information about vegetable gardening, the Bible, and a dissertation in regards to the life cycle of the white-tailed deer. All three of these paperwork could embody the token “deer,” nonetheless the relative frequency of the “deer” token will differ dramatically between the paperwork. The quite a bit higher frequency of the phrase “deer” inside the dissertation possibly has a definite semantic affect than the scarce makes use of of the phrase “deer” inside the totally different paperwork. You wouldn’t want a similarity measure to solely neglect about that signal.

What to do in its place: Use cosine similarity. It’s not just for NLP anymore! (However moreover it’s for NLP.)

Briefly, cosine similarity is an answer to measure how comparable two vectors are in multidimensional space (whatever the magnitude of the vectors). The route a vector goes in multidimensional space relies upon upon the frequencies of the scale which can be utilized to stipulate the realm, so particulars about frequency is baked in.

To make it easy to visualise, let’s say there are solely two tokens we care about all through the three paperwork: “deer” and “bread.” Each textual content material makes use of those tokens a definite number of events. The frequency of these tokens grow to be the scale that we plot the three texts in, and the texts are represented as vectors on this two-dimensional airplane. As an illustration, the vegetable gardening information mentions deer 3 events and bread 5 events, so we plot a line from the origin to (3, 5).

Proper right here you need a take a look at the angles between the vectors. θ1 represents the similarity between the dissertation and the Bible; θ2, the similarity between the dissertation and the vegetable gardening information; and θ3, the similarity between the Bible and the vegetable gardening information.

The angles between the dissertation and each of the alternative texts is pretty big. We take that to suggest that the dissertation is semantically distant from the alternative two — as a minimum comparatively speaking. The angle between the Bible and the gardening information is small relative to each of their angles with the dissertation, so we’d take that to suggest there’s a lot much less semantic distance between the two of them than from the dissertation.

Nevertheless we’re talking proper right here about similarity, not distance. Cosine similarity is a change of the angle measurement of the two vectors into an index that goes from 0 to 1*, with the equivalent intuitive pattern as Jaccard — 0 would suggest two groups don’t have something in widespread, and nearer you get to 1 the additional comparable the two groups are.

* Technically, cosine similarity can go from -1 to 1, nonetheless we’re using it with frequencies proper right here, and there shall be no frequencies decrease than zero. So we’re restricted to the interval of 0 to 1.

Cosine similarity is famously utilized to textual content material analysis, like we’ve carried out above, nonetheless it might be generalized to totally different use cases the place frequency is important. Let’s return to the luxury objects and yachts use case. Suppose you don’t merely have a log of which distinctive clients went to each web site, you even have the counts of number of events the patron visited. Presumably you uncover that each of the 900 clients who went to every internet sites solely went to the luxury objects web site a couple of instances, whereas they went to their yacht website online dozens of events. If we take into account each shopper as a token, and subsequently as a definite dimension in multidimensional space, a cosine similarity technique could push the yacht-heads just a bit extra away from the luxury good patrons. (Bear in mind that you might run into scalability factors proper right here, counting on the number of clients you’re considering.)

Further finding out: https://medium.com/geekculture/cosine-similarity-and-cosine-distance-48eed889a5c4

I nonetheless love the Jaccard index. It’s simple to compute and usually pretty intuitive, and I end up using it frequently. So why write an entire weblog publish dunking on it?

On account of no one info science gadget can present you a complete picture of your info. Each of these completely totally different measures let you realize one factor barely completely totally different. You’ll get useful data out of seeing the place the outputs of these devices converge and the place they differ, as long as you understand what the devices are actually telling you.

Philosophically, we’re in direction of one-size-fits-all approaches at Aampe. After frequently we’ve spent what makes clients distinctive, we’ve realized the value of leaning into complexity. So we predict the broader the array of devices you need to make the most of, the upper — as long as you perceive tips on how to make use of them.