Every time you open TikTok and find yourself watching video after video without consciously choosing any of them, or every time Shopee shows you a product you didn't search for but end up buying — you are experiencing a recommendation system at work. These invisible engines are among the most powerful and commercially significant applications of artificial intelligence in use today. Understanding how they operate is essential for any business operating in digital commerce, content, or marketing.
What Is a Recommendation System?
A recommendation system is an engine that predicts and surfaces content, products, or services that a specific user is likely to find relevant, based on that user's past behavior and the behavior of users who are similar to them. The goal differs depending on context: for TikTok, the primary goal is maximizing engagement (keeping users on the platform longer); for Shopee, the primary goal is maximizing conversion (turning browser behavior into purchases).
The fundamental distinction between a recommendation system and a search engine lies in user intent. When a user searches for "waterproof sunscreen SPF 50," they have a clear, explicit intent — they know what they want and are looking for it. A recommendation system operates in the absence of explicit intent. The user opens their Shopee homepage without a specific purchase in mind, and the system must infer from behavioral history what is most likely to be useful or desirable at that moment.
This makes the recommendation problem significantly harder than the search problem. There is no query to match against. The system must model the user's tastes, current context, and likely near-future interests from indirect signals alone.
The commercial significance is enormous. Netflix has reported that 75–80% of the content watched on their platform comes not from active search but from recommendations. For a platform with hundreds of millions of subscribers, this means that the recommendation engine is effectively Netflix's primary product discovery mechanism — far more important than its search bar. The pattern holds across platforms: Amazon attributes a significant portion of its revenue to "Customers who bought this also bought…" recommendations, and TikTok's entire content delivery model is built around algorithmic recommendation rather than subscription to specific creators.
In practical terms, a recommendation system takes as input: (1) a user's interaction history — what they watched, clicked, purchased, skipped, or rated; (2) item attributes — the category, price, description, tags, or content features of available products or videos; and (3) contextual signals — time of day, device type, location, current session behavior. It produces as output an ordered list of items predicted to be most relevant to that user at that moment.
The Three Main Types of Recommendation Systems
The field of recommendation systems has been developing since the 1990s, and three main architectural approaches have emerged, each with distinct strengths, weaknesses, and use cases.
Collaborative Filtering is the most widely known approach. It is based on the intuition that people who agreed in the past tend to agree in the future. If User A and User B have both purchased products X, Y, and Z, and User A has also purchased product W, then there is a reasonable chance that User B would also like product W — even if the system knows nothing about what product W actually is.
Collaborative Filtering operates on a user-item interaction matrix, where rows represent users, columns represent items, and values represent interactions (purchases, ratings, watch time, clicks). The system finds users or items with similar interaction patterns and uses those similarities to generate recommendations. There are two sub-variants: user-based collaborative filtering (find users similar to the target user, then recommend items those users liked) and item-based collaborative filtering (find items similar to items the target user has interacted with, then recommend those similar items). Item-based tends to be more scalable for large catalogs.
The major limitation of Collaborative Filtering is the cold start problem: it requires substantial interaction data to function well. A new user with no history gets poor recommendations. A new item with no interactions never gets recommended. Platforms address this with onboarding questionnaires, popularity-based fallbacks, or hybrid approaches.
Content-Based Filtering takes a different approach: instead of looking at what similar users liked, it looks at the attributes of items the user has interacted with and recommends items with similar attributes. If a user has been buying moisturizers with "hyaluronic acid" and "fragrance-free" in their descriptions, a content-based system will recommend other moisturizers with those attributes — even if no other user has made the same combination of purchases.
Content-Based Filtering works entirely from item metadata and a single user's own history. This makes it immune to the cold start problem for users: a new user who purchases one item immediately gets recommendations based on that item's attributes. However, it suffers from a different limitation: it tends to recommend more of the same, creating a narrow recommendation space. A user who bought one brand of sunscreen may only ever see that brand's products recommended.
Hybrid Systems combine both approaches, and they are what almost every major production recommendation system uses today. The combination can be implemented in several ways: an ensemble that blends the scores from both models; a weighted combination where Collaborative Filtering dominates for users with rich history and Content-Based takes over for new users; or a switching hybrid that detects which approach is appropriate and routes accordingly. Netflix, Spotify, Amazon, and all major e-commerce platforms use hybrid architectures, often augmented with deep learning layers on top.
How TikTok's For You Feed Works
TikTok's For You Page (FYP) is widely regarded as the most sophisticated content recommendation system deployed at consumer scale. Understanding its architecture reveals principles applicable across recommendation domains.
The system operates as a multi-stage ranking pipeline. The problem it solves is fundamentally one of scale: there are over 10 billion videos on TikTok, and the system needs to select approximately 20–30 videos to show a user in their next session. No single model can efficiently rank 10 billion items in real time. The pipeline progressively narrows the candidate pool.
Stage 1 — Candidate Generation. From the full corpus of 10 billion+ videos, the system retrieves approximately 500–1,000 candidates. This stage uses lightweight models focused on recall rather than precision. The retrieval is based on broad signals: user's followed accounts, videos similar to the user's historical interactions, trending content in the user's geographic region, content in categories the user has shown interest in, and content appropriate for the user's device and network conditions.
Stage 2 — Retrieval and Filtering. The ~1,000 candidates are filtered for quality, safety, and policy compliance. Duplicate content is removed. Content the user has already seen is filtered. The system also applies diversity constraints to ensure the candidate pool is not entirely from one category.
Stage 3 — Ranking. This is the core of the recommendation system. A deep learning model scores each candidate on multiple predicted outcomes simultaneously: predicted watch time (what percentage of this video will this user watch?), predicted like probability, predicted share probability, predicted comment probability, and predicted "not interested" probability. The final ranking score is a weighted combination of these predictions, with watch time typically weighted most heavily because it is the strongest signal of genuine engagement.
Stage 4 — Diversity Injection. TikTok deliberately injects 10–15% "exploration" content into the final feed — videos from categories or creators the user has not previously engaged with. This serves two purposes: it prevents the system from converging on an overly narrow filter bubble, and it allows the system to explore new content spaces that might become strong future preferences for that user.
The behavioral signals that drive TikTok's ranking are well-documented through reverse engineering and TikTok's own transparency reports. Watch time percentage is the most important positive signal — completing a 60-second video is a stronger signal than completing a 15-second video, and watching a video twice is a very strong positive signal. Shares are treated as a strong positive signal, stronger than likes, because sharing requires active decision to pass content to another person. Comments are positive. Likes are positive but somewhat weaker than shares. "Not interested" is a strong negative signal that suppresses similar content. Profile visits after watching a video are a very strong positive signal indicating high interest in the creator.
For content creators, TikTok's system means that new accounts can reach the For You Page relatively quickly if their content generates high watch time in its first 200–500 views. The system initially distributes new content to a small test audience; if engagement signals are strong, it distributes to a progressively larger audience. This creates TikTok's characteristic pattern where unknown creators can go viral within 48 hours if their content resonates.
How Shopee Suggests Products
Shopee's recommendation architecture differs from TikTok's in its primary objective — conversion rather than engagement — but the underlying technical approaches are similar. The major recommendation surfaces on Shopee include the homepage feed, "You Might Also Like" on product pages, "Frequently Bought Together" bundles, category recommendation pages, and search result personalization.
Item-to-item collaborative filtering is the foundation of Shopee's "Frequently Bought Together" feature. The system analyzes co-purchase patterns across millions of transactions: which products appear together in the same cart or within the same purchase session? A user buying a phone case is likely to also want a screen protector. A user buying a foundation is likely to also want a setting powder. These associations are computed from aggregate transaction data rather than individual user profiles, making them robust even for users with limited history.
Sequential recommendation models the user's purchase journey over time. Shopee tracks not just what a user has purchased but the sequence and timing of purchases. A user who bought a basic cleanser three months ago, a toner two months ago, and a serum last month is following a skincare routine build-up pattern. The system predicts that their next likely purchase is a moisturizer or sunscreen. This temporal modeling allows Shopee to surface the right product at the right point in a user's consumption journey.
Session-based real-time personalization adjusts recommendations based on what a user is browsing in their current session, not just their historical profile. If a user who typically buys clothing suddenly spends ten minutes browsing camping equipment, the session-based component will temporarily shift recommendations toward outdoor products — even if their historical profile suggests no interest in outdoor activities. This session context is especially important for seasonal or gift shopping scenarios.
Price sensitivity modeling is a Shopee-specific feature that learns each user's typical spending range per product category. A user who consistently purchases mid-range skincare products will rarely convert on luxury skincare recommendations, even if those products are popular. The system segments recommendations by inferred price sensitivity, ensuring that recommended products fall within the range where the user has historically been willing to transact.
The different recommendation surfaces serve different purposes. "You Might Also Like" on a product page uses collaborative filtering to find similar items — it targets users who are still in browse mode and haven't committed to a purchase. "Frequently Bought Together" uses co-purchase analysis and targets users who have decided to buy and may be receptive to add-ons. "Sponsored" recommendations are a paid overlay on the organic recommendation layer — advertisers can bid to appear in recommendation slots, with the system balancing commercial relevance and organic quality to avoid degrading user experience.
The commercial impact of Shopee's recommendation engine is significant: approximately 35% of Shopee's Gross Merchandise Value (GMV) is attributed to the recommendation engine rather than direct search-driven purchases.
Metrics for Measuring Recommendation System Effectiveness
Building a recommendation system is straightforward; building one that actually improves business outcomes requires rigorous measurement. The field has developed a set of standard metrics, though the appropriate metric depends heavily on the business objective.
Click-Through Rate (CTR) measures the percentage of users who click on at least one recommended item. It is the most commonly reported metric because it is easy to measure, but it is also the most gameable — a system optimized purely for CTR will recommend clickbait rather than genuinely useful items. CTR should always be paired with downstream conversion metrics.
Conversion Rate measures the percentage of clicked recommendations that result in a purchase (or other desired action). This is more aligned with business value than CTR, but it can create perverse incentives to recommend only the most universally popular products, which may not be personalized at all.
Revenue Per User measures the average revenue generated per user through recommendation-driven interactions. This captures both conversion rate and average order value, making it a more complete picture of commercial impact.
Novelty measures the degree to which a recommendation system surfaces items the user has not previously encountered. A system that only recommends items the user has already bought scores well on conversion but provides no discovery value. Novelty is important for user experience and for helping platforms surface new inventory.
Diversity measures the variety within a recommendation list. A list containing ten products from the same category and the same brand has zero diversity, even if each individual recommendation is relevant. Low diversity creates a poor user experience and misses cross-sell opportunities.
Serendipity is the hardest metric to optimize and arguably the highest-quality signal of recommendation excellence. A serendipitous recommendation is one that is surprising — the user would not have expected it — but turns out to be genuinely relevant. TikTok's "exploration" content injection is designed specifically to create serendipitous moments. Serendipity is typically measured through user surveys or by tracking engagement on content from categories the user rarely engages with.
In practice, production recommendation systems track all of these metrics and make tradeoffs between them. Optimizing purely for CTR creates clickbait. Optimizing purely for conversion creates boring, repetitive recommendations. The best systems find a balance that maximizes long-term user satisfaction and revenue, which often requires accepting lower short-term CTR in exchange for higher novelty and diversity.
The Filter Bubble Problem
The filter bubble is one of the most significant unintended consequences of recommendation systems, and it affects both content platforms and e-commerce platforms in different but important ways.
A filter bubble occurs when a recommendation system, optimized to surface content the user is most likely to engage with, progressively narrows the range of content a user is exposed to. Each interaction reinforces the system's model of the user's preferences, causing it to recommend more of the same, which causes the user to interact more with that type of content, further reinforcing the model. The result is a feedback loop that traps users in an information bubble — they see only content that confirms their existing interests and beliefs, and are never exposed to new perspectives, products, or ideas.
The most discussed examples involve political content. A user who watches one video expressing a particular political viewpoint is likely to be shown more videos with similar viewpoints. Over weeks or months, the system may have narrowed their media diet to a single ideological perspective, which can contribute to polarization at scale. TikTok, Facebook, and YouTube have all been studied in this context.
For e-commerce, the filter bubble problem has a different shape but is equally damaging commercially. If a customer's entire recommendation experience is dominated by one product category — say, skincare — they may never discover that the same brand has excellent haircare products. The recommendation system, optimized to maximize conversion probability, keeps surfacing the category where the user has the highest historical conversion rate, missing cross-category revenue opportunities.
The solutions that major platforms have implemented include several approaches. Exploration vs. exploitation balancing (sometimes called epsilon-greedy) deliberately injects a fraction of random or novel recommendations. With probability ε (typically 5–20%), the system recommends something outside the user's established preference zone; with probability 1-ε, it recommends the highest-predicted-relevance item. Diversity injection, as described in TikTok's system, is a structured version of this — ensuring the recommendation list contains a minimum percentage of content from categories outside the user's core interest areas. User control settings allow users to signal when they have seen too much of a particular type of content ("show more variety," "not interested in this topic"), giving the system direct negative feedback. Temporal decay applies decreasing weight to older interactions, allowing recent behavior to dominate over established long-term preferences — this means a user's interests can shift over time rather than being permanently anchored to early interactions.
AlgoData and Recommendation Systems
AlgoData applies recommendation system concepts to help Vietnamese brands understand and act on purchasing behavior patterns extracted from Shopee and TikTok Shop data.
The core application is market basket analysis — identifying which products are frequently purchased together by analyzing transaction data at scale. For a skincare brand selling on Shopee, AlgoData can identify that customers who buy a particular sunscreen also purchase Vitamin C serum within 14 days at a rate significantly above chance. This co-purchase signal is the foundation of "Frequently Bought Together" recommendations, and brands can use it to structure product bundles, optimize cross-promotion campaigns, and ensure their SKUs appear in each other's recommendation slots on the platform.
Cross-category opportunity identification extends this analysis beyond a single brand's product catalog. AlgoData examines whether customers who purchase heavily in one category show elevated purchase probability in adjacent categories. Customers who are building a skincare routine often expand into haircare products within 30–60 days. Identifying this transition point allows brands to time marketing campaigns to reach customers at the moment when they are most open to cross-category discovery.
Next-purchase timing prediction uses sequential pattern analysis to estimate when a specific customer is likely to make their next purchase in a given category. For consumable products like sunscreen or face wash that deplete at predictable rates, this prediction is relatively straightforward. For discretionary products like clothing or electronics, it requires modeling based on historical purchase intervals and seasonal patterns. Brands use this timing data to optimize campaign scheduling — reaching customers with relevant recommendations when they are in purchase mode rather than broadcasting uniformly across all customers at all times.
For brands asking "which of our products should appear in each other's Frequently Bought Together sections on Shopee?", AlgoData provides data-driven answers based on actual co-purchase behavior rather than intuition about product complementarity.
See also: What is AI Agent?
Building a Simple Recommendation System
For teams that want to understand the mechanics before committing to a production system, building a basic collaborative filtering engine from scratch is a useful exercise. The following walkthrough uses Python with pandas and scikit-learn.
Step 1: Build the user-item interaction matrix.
1import pandas as pd
2import numpy as np
3from sklearn.metrics.pairwise import cosine_similarity
4
5# Example transaction data
6transactions = pd.DataFrame({
7 'user_id': [1, 1, 1, 2, 2, 3, 3, 3, 4, 4],
8 'product_id': ['A', 'B', 'C', 'A', 'D', 'B', 'C', 'D', 'A', 'C'],
9 'purchase_count': [2, 1, 3, 1, 2, 1, 2, 1, 3, 1]
10})
11
12# Pivot to user-item matrix
13user_item_matrix = transactions.pivot_table(
14 index='user_id',
15 columns='product_id',
16 values='purchase_count',
17 fill_value=0
18)
Step 2: Compute user similarity.
1# Cosine similarity between users
2user_similarity = cosine_similarity(user_item_matrix)
3user_similarity_df = pd.DataFrame(
4 user_similarity,
5 index=user_item_matrix.index,
6 columns=user_item_matrix.index
7)
Step 3: Generate recommendations.
1def recommend(user_id, n=5):
2 # Find top similar users (excluding the user themselves)
3 similar_users = user_similarity_df[user_id].drop(user_id).nlargest(10)
4
5 # Get items purchased by similar users but not by the target user
6 target_purchases = set(
7 user_item_matrix.loc[user_id][user_item_matrix.loc[user_id] > 0].index
8 )
9
10 scores = {}
11 for similar_user, similarity in similar_users.items():
12 similar_purchases = user_item_matrix.loc[similar_user]
13 for product, count in similar_purchases.items():
14 if count > 0 and product not in target_purchases:
15 scores[product] = scores.get(product, 0) + similarity * count
16
17 # Return top N recommendations sorted by score
18 recommendations = sorted(scores.items(), key=lambda x: x[1], reverse=True)
19 return recommendations[:n]
20
21print(recommend(user_id=1, n=5))
This simple implementation illustrates the core mechanics: build the interaction matrix, measure user similarity using cosine distance, and recommend items that similar users have purchased but the target user has not yet bought.
Data requirements and limitations. This approach requires a minimum of around 1,000 users with at least 5 interactions each to avoid severe cold start degradation. Below that threshold, similarity scores are unreliable and recommendations are effectively random.
Production considerations. A production system cannot recompute cosine similarities across all user pairs in real time — at 10 million users, that matrix has 10^14 cells. Production systems use approximate nearest neighbor (ANN) algorithms (FAISS, HNSW, ScaNN) to find similar users in sub-linear time. User and item representations are pre-computed as dense embeddings (typically 64–256 dimensions) and updated on a schedule (hourly or daily) rather than in real time. The serving layer retrieves pre-computed recommendations from a low-latency store (Redis, DynamoDB) rather than computing them on demand.
See also: What is Vector Database?
Conclusion
Recommendation systems are the core engine driving modern e-commerce and content platforms. They are not optional features — for platforms like TikTok and Shopee, they are the primary mechanism through which users discover content and products. Understanding their architecture and mechanics is essential for any business competing in digital commerce or content.
The appropriate recommendation architecture depends on the maturity and scale of the business. Early-stage businesses with limited transaction data should start with simple market basket analysis and Content-Based Filtering — these approaches require less data and can deliver meaningful results even with a few thousand transactions. Growth-stage businesses should invest in hybrid collaborative filtering systems and begin modeling sequential purchase behavior. Enterprise-scale businesses with millions of transactions should explore deep learning recommendation architectures and multi-stage ranking pipelines.
For Vietnamese businesses selling on Shopee and TikTok Shop, even basic market basket analysis applied to existing transaction data can uncover cross-sell opportunities worth 15–20% additional revenue. The data exists in your transaction history; the question is whether you are analyzing it systematically. AlgoData provides this analysis, turning raw transaction data from Vietnamese e-commerce platforms into actionable recommendation signals for brands.
The recommendation systems powering TikTok and Shopee are not magic — they are systematic applications of well-understood statistical principles at enormous scale. The same principles, applied at a scale appropriate for a Vietnamese brand, can deliver the same category of commercial impact.
See also: What is RFM?
