In previous posts, we’ve explored how our Activity Ranking team moved from batch to real-time MLOps and migrated from XGBoost to deep learning transformers. Both are about ranking activities we already understand well. But what happens when we don’t yet understand them?
Let’s consider a brand new activity. How can you get a walking tour in Berlin noticed when it has zero bookings, zero reviews, and zero clicks? It took us three years, seven experiments, and one perspective shift to answer that properly. As it turns out, changing the question mattered more than any single experiment.
If you’re an engineer, systems designer or a complex problem-solver, read on to learn about our varied journey and the lessons we learned along the way.
{{divider}}
The “cold-start trap”
Our ranking system is, by design, a popularity engine. It learns from past behavior (clicks, bookings, and reviews) to decide which activities deserve the top of the page. That works well for activities with history. For activities without it, there is simply nothing to learn from yet.
A new activity joining the platform has none of those signals. By default, it ranks low. Low ranking means no exposure. No exposure means no clicks and no bookings. No bookings means no signals, and the activity stays at the bottom.

Reviews compound it. Even when a new activity surfaces, travelers are reluctant to book something without social proof. The first booking is the hardest one to get, and until it arrives, the system has little to go on.
We call this the cold start trap. And we call breaking out of it activation: the moment an activity earns enough bookings and reviews to sustain organic growth in the ranking on its own. Once activated, the flywheel spins on its own.
The status quo: A system built for throughput
Three years ago, our cold start solution had four parts, each one added to patch the last:
- Guaranteed exposure slots. New activities were placed in reserved positions on result pages so they could gather signals.
- A signal threshold. Once an activity collected enough clicks and bookings, it “graduated” out of assessment and into normal ranking.
- A queue. As supply grew faster than impressions in popular destinations, we couldn’t assess every new activity at once. A queue controlled the throughput into the assessment pipeline.
- A prioritization model. With a long queue, we needed to decide which activities to assess first.

It did what it was designed to do: collect signals. But suppliers could wait weeks after signing before their first booking, and even after assessment, a meaningful share of activities never activated. The pipeline was built to gather signals, not to activate activities, and all our optimization energy went into throughput.
We did not see this straight away. Experiments 1 and 2 below were both attempts to make the existing pipeline smarter, and both failed. What they taught us was that the pipeline was not the problem. The way we framed the question was the problem.
For years, we had framed cold start as a ranking problem, and therefore as a customer problem: here is a new activity, we have no data on it, how do we rank it correctly for the traveler? We couldn’t prove that ranking a new activity correctly had measurable long-term value. The framing made it impossible to show progress, and a problem where you cannot show progress is a problem you cannot justify investing in.
The change was to stop treating it as a customer problem and start treating it as a supplier problem. Rather than asking “How do we rank an activity we know nothing about?”, we asked “What is a first booking worth over the lifetime of an activity?”
Answering this question gave us three things we had never had:
- A metric in how many activities have started their flywheel
- A business case in the incremental lifetime value of a first booking
- A shared definition of what the team was for. We stopped building for throughput and started building for activation.
It also changed what we were willing to accept. Experiments four and six below both lowered overall conversion. Under the old framing, those were regressions, and we would have rolled them back. Under the new one, we could put a number to what the dip was buying, and it was positive. The reframe is what made those experiments shippable.
The experiments
1. Smarter prioritization with supplier features ❌
Hypothesis: A supplier’s track record (ratings, reviews, lifetime tours) is a strong signal for the quality of their new activities.
What we did: Added supplier features to the prioritization model so we’d assess the most promising activities first.
Result: Marginally better prioritization, not statistically significant. Learning: Improving the prioritization model wasn’t a strong lever for activation.
2. Filter out unpromising activities ❌
Hypothesis: If we stop wasting assessment capacity on activities that score very low, we can concentrate it on the most promising ones and lift total activations.
What we did: Filtered low-scoring activities out of the queue entirely.
Result: The activities we did assess converted better, but total bookings didn’t increase. We were concentrating on fewer bets rather than creating more winners. Learning: Being smarter within the existing framework had a low ceiling. To optimize for activation, we needed to rethink the framework itself.
3. Use the ranking model to pick assessment candidates ✅
Up to this point, once an activity was in the assessment pool, the boosted slot was filled essentially at random.
Hypothesis: Our ranking model, even with a limited signal, is still better than randomness at predicting which activity will convert.
This felt contradictory. Assessment existed precisely because the ranking model didn’t have enough signal on new activities, so using it to pick assessment candidates looked self-defeating.
What we did: Replaced the random selection of the boosted slot with selection based on the ranking model.
Result: A clear win. The conversion of activities in assessment increased, and so did the number of activities receiving a first booking. Even with sparse features, the ranking model picked better candidates than chance.
4. Remove the queue entirely ✅
Experiment 3 made us question the architecture itself. The queue and prioritization model were designed to manage scarce exposure slots. But if ranking could pick the best candidate in real time from any set of new activities, the queue was adding time to supplier onboarding without adding much in return.
What we did: Removed the queue and the prioritization model. All new activities became immediately eligible for the boosted slots. Ranking picked the best candidate.
Result: More nuanced. New activity conversion went up, but overall page conversion dipped: we were giving slots to new activities that were not yet competitive with established ones. A classic exploration/exploitation trade-off. We worked with our finance team to model the long-term value of adding more flywheels against that dip, and the results were positive. That calculation is the reframe doing its work. A year earlier, we had no way to value a first booking, and this would have read as a straightforward regression. The pipeline also got simpler, and supplier onboarding got considerably faster.
We rolled it out.
5. Add randomness back in ❌
With ranking selecting every boosted candidate, we worried we’d stop learning about activities the model was unsure about.
Hypothesis: Some controlled randomness would improve long-term learning and activation.
What we did: Made selection probabilistic — higher-scoring activities were more likely to be chosen, but lower-scoring ones still had a shot.
Result: Conversion trended negatively, and fewer activities received first bookings. The ranking model didn’t need our help diversifying. We closed the experiment and moved on.
6. Activate all unactivated activities ✅
Stepping back, we realized “new” wasn’t the only segment stuck in the cold start trap. Plenty of existing activities had passed through assessment without ever starting a flywheel. They were unactivated, too.
Hypothesis: The same boosting mechanism could help any unactivated activity, not just new ones.
What we did: Made the boosted slots eligible for all unactivated activities, with ranking choosing the best candidate from this larger pool.
Result: A small drop in overall conversion, but a meaningful lift in bookings per visitor for unactivated activities. The same trade-off as before, and the same calculation settled it: the incremental lifetime value of starting a flywheel outweighed the short-term cost. An intervention designed for a new supply ended up benefiting the entire marketplace.
7. From fixed slots to flexible boosting ✅✅
The entire system had been built around fixed boosted slots: guaranteed positions on the page. Guaranteed exposure has a cost: we promote the same activity to the same position regardless of context.
Hypothesis: Letting the system decide where an unactivated activity fits best, instead of forcing a fixed position, would be more efficient.
What we did: Replaced fixed slots with a blended score. The final score became a weighted combination of the standard ranking score and a boost term applied to unactivated activities. They still got help, but they competed for position with everything else.
Result: Two successive A/B tests, both strongly positive — including a meaningful lift in bookings per visitor for unactivated activities, without the conversion drag we’d seen with fixed slots.
Where we ended up
The system today looks remarkably different from where we started three years ago:
- No queue
- No prioritization model
- No fixed slots
A single ranking model does most of the work, with a lightweight boost factor giving unactivated activities a fair chance to compete. This is significantly more efficient and simpler than what it replaced.
.png)
Final takeaways
1. On a new problem, explore fast and wide
For a well-understood problem, such as iterating on our core ranking model, it pays to try many variations in the same direction. Cold start was not that. There was no obvious direction for improvement: each experiment touched a different part of the system and pointed in a new direction. The two early failures, supplier features and low-score filtering, failed quickly and cheaply. Each one narrowed down where the real lever was.
2. Treat handcrafted constraints as hypotheses
A pattern runs through all seven experiments. Every time we added a constraint, such as filtering low scorers or enforcing randomness, the result was flat or negative. Every time we removed one or widened the model’s freedom, by using ranking instead of random, removing the queue, widening eligibility, or dropping fixed slots, the result was positive. Handcrafted restrictions limit what an ML model can optimize. The lesson isn’t “never add constraints.” It’s “test them like everything else.”
3. The question is part of the problem
None of the wins above were available to us while we were asking how to rank an activity we knew nothing about. They became available the moment we asked, rather than how many activities had started their flywheel and what a first booking is worth over time.
The biggest thing we took away? Sometimes the most valuable thing you can do to find an answer is to change the question.
{{divider}}
Huge thanks to the Activity Ranking team behind this work, and their contributions to this post. If this kind of problem-solving sounds exciting, check out our open roles and learn more about the tech teams at GetYourGuide.














.png)





