Does personalization always work?

Does personalization always work?

Does personalizing the experience always work?

What an easy question, heh? You might be thinking that the answer is Yes, because it makes sense and you may have read many posts about Personalization. But please, hang on in there and listen to what our data has to say.


I’ll use my favourite word in English to answer this. The answer is Yes(ish). Yes, I consider ish a word. And yes, personalizing the experience does not always work.

One of the features our search engine offers is Contextualise. Through Contextualise, we gather data about the user preferences based on:

  • Queries
  • Navigation
  • Product views

This data provides information to offer personalised results when a user

returns to a site and searches. To give a simple example, if you always search, navigate and view Red, next time you search for Dresses, red dresses score higher within the top results.

To measure how well Contextualise works we mainly use 2 metrics. One is Findability and the other is CTR. Findability, for those not familiar with it, shows the percentage of queries that had at least 1 click on the first page of results. If I search for Ankle boots and click on a result on the first page with results, Findability would be 100%. If instead of that, I don’t click on anything until I’m on the 2nd page with results, Findability would be 0% for that query. It is a key metric because the higher on the list the user finds a relevant result, the more likely it is that they will click on the result, go to the product page and start the purchase funnel process.

We do use Add to cart and CVR as metrics to measure performance too, but we do not focus on them in this case. Many factors affect Add to cart and CVR, factors such as the site’s checkout or the visitor’s intention (among others). It doesn’t just depend on how good the search results provided are.

Getting back to the initial question, does personalizing results always work? What does our data say about this?

As always it depends on the site, but overall Findability shows at least a 10% increase when comparing returning visitors that were Contextualised vs returning visitors that were not. The same when segmenting the data by device type. Findability increase for Contextualised users is higher on Desktop than on Mobile, which is most likely due to the number of results that are shown with no or little scroll on the results page (product list). That shows again how important making the relevant results visible at the top of the list for the user is.

Why do I say that the answer is Yes(ish) then?

We have done this analysis by breaking the data down by the number of words in the query. For queries with 1 or 2 words, which usually contain a product type, category or brand (for those with 1 word), or the same but with an attribute (for those with 2 words), Findability is again at least 10% higher for Contextualised users compared to non Contextualised users. This is great news because in industries like Fashion, around 70% of queries have 1 or 2 words. In other industries like Technology, the percentage is not as high, but it is still more than half of the searches.

What about for queries with more than 2 words? We still see positive results for queries with 3 words, however the increase is not as high as when the search is more generic. The reason behind this is that if we limit the number of results because they don’t match with the user affinities, we might end up showing no relevant results at all.

The more specific a query is, the less we need to personalise results. The query itself is doing the job of filtering what the user is really looking for. If the user searches for Midi party red dress, there is not much that can be personalised there. Well, yes, the size, but I’ll leave that for another post :-)

The conclusion is, yes, personalizing the experience does work most of the time. However, personalizing when there is no need to personalise can lead to a result in which the user has no experience at all.

This is why we don’t want to use the term “Personalise” since it is not about personalising but about being aware of the user Context, which sets a different path and understanding. Personalisation is adapting what you see to your profile while to us, the problem is not adapting to a user profile but to a momentary context.

If you want to learn more about how Contextualise works, please do get in touch.

Head of Performance & Analytics


Leave a reply

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.