Under The Hood
How tech-econ.com curates, ranks, and recommends content.
Ingestion
Before you can read, search, or stumble across something useful on this site, each piece of content has to go through a pipeline. From raw submission to ranked, enriched, and ready to discover. This tab walks through that journey.
What kinds of content?
Every entry on the site belongs to one of eight types. Each type lives in its own collection and shows up in different places (homepage rows, search results, dedicated browse pages).
| Type | What it covers | An example |
|---|---|---|
| Packages | Software libraries and tools | DoubleML, a library for causal machine learning |
| Datasets | Data collections and benchmarks | Nielsen retail scanner data for consumer research |
| Papers | Academic research organized by topic | "Mostly Harmless Econometrics" by Angrist and Pischke |
| Books | Long-form treatments worth reading cover to cover | "The Mixtape" by Scott Cunningham on causal inference |
| Talks | Videos, podcasts, lectures | Susan Athey on causal AI at Stanford |
| Resources | Blogs, tutorials, courses, anything that teaches | Andrew Ng's blog on applied AI |
| Career | Interview prep, hiring guides, industry overviews | Big Tech economist interview prep guides |
| Community | Conferences, meetups, labs, events | NBER summer institute and ASSA annual meeting |
How a single entry grows along the way
Each entry starts as a tiny record (a name, a link, a short description) and gets richer at every stage. Here is the same item at three points in the pipeline.
{
"name": "DoubleML",
"url": "https://docs.doubleml.org",
"description": "Double machine
learning for causal inference"
}{
"name": "DoubleML",
"url": "https://docs.doubleml.org",
"description": "Double machine
learning for causal inference",
"tags": ["causal inference",
"python", "machine learning"],
"best_for": "treatment effects
with high-dimensional controls",
"cluster": "Causal ML"
}{
"name": "DoubleML",
...
"tags": [...],
"cluster": "Causal ML",
"score": 0.87,
"rank": 4
}DoubleML arrived as a two-line stub: name, URL, and the phrase "double machine learning for causal inference." After the enrichment step it grew tags (causal inference, Python, treatment effects), a best-for note ("high-dimensional controls in observational studies"), a cluster label (Causal ML Methods), and seven synthetic questions that power semantic search. That richer record is what gets vectorized and ranked. The raw stub would have been nearly invisible in search.
Curation is a craft, not a pipeline. The pipeline handles tedium; judgment about what belongs is the part that can't be automated.
Where it comes from
Content arrives from three places. The primary channel is manual curation: reading the field, following researchers, spotting tools that actually get used in practice. A public submission form at /submit/ lets anyone suggest an addition. Behind the scenes, a small Cloudflare service receives the form data and routes it for review. There's also a modest archive channel, made up of older lists and reading stashes that get cleaned up and folded in over time.
Nothing lands on the site without a human deciding it belongs there. The pipeline automates the boring parts; curation stays manual.
Validation: the gate before anything enters
Once an item is ready to add, it has to pass a validation check. Before anything publishes, we run a quick schema check to make sure required fields are present (every entry must have at least a title, a URL, and a description). The check also scans for duplicate URLs to prevent the same resource appearing twice under different names, and it verifies that category values match the allowed list. If anything fails, the check reports exactly which entries are broken and why. Nothing merges until this passes.
Think of it as the spell-checker you run before publishing: it won't catch bad judgment, but it will catch bad data.
Enrichment: making the data richer
Raw entries are sparse. They have a title and a URL and maybe a short description, but they don't yet have tags, a "best for" use-case note, or a cluster label (the thematic bucket that groups similar content together). That's where the enrichment step comes in.
This step calls the Claude API (the same Claude that helped write this very page, which is a slightly recursive situation) and asks it to read each entry and generate structured metadata. The output gets written back into the content files. Enrichment runs for new entries and for entries that are missing metadata; it doesn't re-touch things that already look complete.
The generated tags and descriptions are what power search. When you type something into the search bar and get a result that feels semantically related to your query rather than just keyword-matched, that's because the enriched descriptions and tags were converted into vector embeddings (a way of representing meaning as a list of numbers) that the search engine can compare. More on that in the Processing tab.
Ranking: deciding what rises to the top
After an item is validated and enriched, it gets a score. That score determines where an item appears on any list page: higher score floats to the top.
The scoring is driven by a machine-learning model that pulls engagement signals from the site's analytics database (how often an item was clicked, how long people spent reading it, how far they scrolled, whether they found it through search). These signals feed into a model that is good at predicting how interesting an item is based on patterns in past behavior. Items with lots of strong signals get high scores; items that get lots of impressions but no clicks get penalized.
New items have no history, so they can't be scored by engagement. Instead, the ranker finds the items most similar to the new item (using the vector embeddings from the enrichment step) and borrows a fraction of their score. This is called cold-start propagation: a new item earns a provisional score based on its nearest neighbors until it accumulates its own engagement data.
Where it lives
The final resting place for all content is a set of structured files, one per content type. Hugo (the static site generator that powers tech-econ.com) reads these files at build time and converts them into the pages you see. The files are version-controlled in git, which means every addition, change, or removal is tracked. If something goes wrong, you can always roll back.
The build step also generates the search index and the vector embeddings that power semantic search, but that's the Processing tab's story.
Storage
Once content is ingested and ranked (see the Ingestion tab), it has to live somewhere. This tab explains where each kind of data sits: content files, ranking outputs, search indexes, analytics events, and the small cookie trail in your browser.
What lives where: the corpus at a glance
Eight content types, each stored separately, each with a different level of engagement relative to its size.
| Type | Entries | Approx. size | Notes |
|---|---|---|---|
| Packages | 551 | 2.3 MB | Largest collection; richest metadata per entry |
| Career | 639 | 2.4 MB | Most entries; interview guides, role overviews |
| Datasets | 443 | 1.7 MB | Includes benchmark and commercial datasets |
| Resources | 518 | 2.0 MB | Blogs, courses, tutorials; highest click density |
| Community | 452 | 1.8 MB | Conferences, labs, meetups |
| Talks | 265 | 1.1 MB | Videos, podcasts, interviews |
| Papers | 1,090 | 7.6 MB | Nested by topic; largest single file |
| Books | 102 | 379 KB | Smallest; highest per-item quality bar |
A single moment of attention, captured
Every interaction is compressed into a small event record before it travels from your browser to the edge database. The record is intentionally minimal: no names, no accounts, just an anonymous session and what happened.
{
"type": "click",
"session": "anon-8f3a", // random, not tied to an account
"page": "/packages/",
"ts": 1716547200000, // millisecond timestamp
"item": "DoubleML",
"section": "packages",
"exp": { "harness_aa_v2": "control_a" } // which experiment variant
}The server that receives your click is probably less than 20 milliseconds away. That's the point of running at the edge: the database lives close to the reader, not in a single central warehouse.
Content files: the source of truth
The primary home for all content is a set of structured files in the site's code repository, one per content type (packages, datasets, resources, and so on). Each file is a human-readable list of entries, with fields like title, URL, description, and tags. These files are version-controlled with git, which means every change is recorded and reversible. Before anything publishes, we run a quick schema check to make sure required fields are present and links aren't broken.
Humans can edit these files directly. The pipeline (enrichment, ranking, embedding generation) writes back into them automatically, adding or updating fields like the ranking score, tags, and cluster label.
Rankings and homepage layout
Two outputs represent the ranker's work. A ranking snapshot stores every item's current score after each ranking run. A separate pre-computed homepage layout describes a list of named rows ("Trending Now", "Top Packages", and so on), each containing the ordered items that should appear in that row. These files are the bridge between the machine-learning pipeline and the Hugo build step that assembles the actual pages.
Embeddings: numbers that represent meaning
Semantic search works by turning each item's text into a list of 1,024 numbers (an "embedding") that captures what the item is about in a way that can be compared mathematically. These embeddings live in a dedicated folder as two files. A compact binary file (roughly 16 MB) that the browser downloads only when someone uses the search bar for the first time. A smaller JSON file maps item IDs to their names and URLs so the browser can display results. A third file stores precomputed "items similar to this one" lists, used by the "Because You Viewed" recommendation row. This one loads quietly in the background when the page opens.
The analytics database
While the content files are static and human-readable, the analytics side is dynamic. Every time someone visits a page, clicks a card, searches, or scrolls, the site sends a small event to a Cloudflare Worker running at the edge (a server that lives close to the user, not in a central datacenter). That worker receives the event, validates it, and writes it into a SQL database. The database itself is replicated near every Cloudflare edge location for speed.
The database stores every raw event, along with aggregated counts of clicks per item and a record of what people searched for and how often. The ranking process reads from the analytics database to fetch engagement signals for each item before training the model.
Cookies on your browser
The site stores two small cookies: a random anonymous user ID (so the ranker can treat your session as a coherent visit, without knowing who you are) and a session counter that increments each visit. These cookies never contain personal information. They are not shared with third parties, and they are the only data the site stores on your device. Reading history (used for the "Because You Viewed" row) stays entirely in your browser's local memory and is never sent to any server.
Processing
Between "item added to a content file" and "item shows up ranked on the homepage" are several automated steps. This tab walks through each one, what it does, and how they depend on each other.
The rough shape of the ranking formula
Before the machine-learning model takes over, each item's raw engagement signals are combined using a weighted sum. The weights below are the actual values used in the current ranking configuration.
raw_score =
+ 5.0 * clicks
+ 3.0 * search_clicks // intentional: searched then clicked
+ 2.0 * scroll_90 // read the whole thing
+ 1.5 * deep_session // part of a high-engagement visit
+ 1.0 * dwell_minutes
+ 0.5 * impressions
+ 0.3 * coclick_count
+ 0.1 * coview_count
+ 0.15 * freshness_decay // half-life: 30 days
- 2.0 * rage_clicks // repeated frustrated clicks
- 1.0 * quick_bounces
- 1.0 * high_imp_no_click // seen often but never clicked
+ cold_start_boost // borrowed from 5 nearest neighbors
model_score = gradient_boosted_tree(raw_score, item_features)
// normalised to [0, 1]Stefan Wager's "Causal Inference: A Statistical Learning Approach" is the top-ranked item on the site as of the last ranking run. It collected 73 clicks against just 2 impressions, which is an unusually high click-to-impression ratio (most items with 2 impressions collect 0 or 1 clicks). The high ratio, combined with 487 co-views in the same sessions as other top items and 14 seconds of average viewable time, pushed it to the maximum normalized score of 1.00.
By contrast, "Causal Inference for the Brave and True" has 17 clicks and a score of 0.376. The gap is mostly in clicks, not in quality. The ranker rewards high intent signals, not just popularity.
Step 0: Validate (runs on every change)
The gatekeeper runs automatically whenever a content file changes and exits with an error if anything is wrong: missing required fields, duplicate URLs, or values that don't match the allowed list. Nothing proceeds until this passes. Think of it as a strict proofreader that catches structural problems before they become bugs on the live site.
Step 1: Enrich (runs for new or sparse entries)
The enrichment step calls the Claude API and asks it to read each item and generate structured metadata: descriptive tags, a short summary, a "best for" note explaining which audience would find it useful, and a cluster label (a thematic name like "Causal Inference" or "NLP Methods"). The output is written back into the same content files. Enrichment is selective: it only touches entries that are missing metadata or that have very sparse descriptions. Items with complete metadata are skipped to avoid unnecessary API calls.
The tags and enriched descriptions generated here are the raw material for the next step.
Step 2: Embed (runs after enrichment)
The embedding step turns each item's text (title, description, and tags combined) into a list of 1,024 numbers using an embedding model. These numbers are called an "embedding." Items about similar topics end up with similar-looking number lists, so the computer can find related items by comparing how close their numbers are, rather than by matching exact words. The output is saved as a compact binary file that the browser downloads when needed, and a smaller file for related-item recommendations.
Step 3: Cluster (runs after embedding)
The clustering step groups items within the Resources section into themed carousels for browsing. It starts from the cluster labels written during enrichment, then uses a standard grouping algorithm to handle orphans and merge clusters that are too small. The target is 5-10 items per carousel, and each carousel gets a human-readable name. These clusters become the browsable rows on the Resources page.
Step 4: Rank (runs on a schedule)
The ranking step is the most complex part of the pipeline. It fetches engagement data from the analytics database (clicks, time spent, scroll depth, search-result clicks), trains a small machine-learning model on those signals, and assigns each item a score between 0 and 1. That score is written back into the content files and determines where items appear on every list and row across the site.
For items with no engagement data yet (new additions), the ranker uses the embeddings from Step 2 to find the 5 most similar items that do have engagement, and borrows a fraction of their score. This is called "cold-start propagation." New items don't start at zero; they start from their nearest neighbors.
The feedback loop
The curved arrow in the diagram above shows the most important relationship in the whole pipeline: engagement data flows from the analytics database back into the ranking step. Every click, scroll, and search on the live site eventually becomes a training signal for the next ranking run. The pipeline is not a one-way conveyor belt; it is a loop that tightens over time as more engagement data accumulates.
Recommendation Surfaces
A recommendation system is only as useful as the places it shows up. This tab maps out the specific spots on the site where recommendations appear, what drives each one, and what you are supposed to get out of it.
The "Trending Now" row as of the last ranking run (May 2026) led with Stefan Wager's "Causal Inference: A Statistical Learning Approach" at score 1.00, followed by "Causal Inference for the Brave and True" by Matheus Facure (score 0.376), JD.com 2020 (MSOM-20) dataset (score 0.280), "Causal Econometrics Course" (score 0.257), and BestBuy dataset (score 0.244). Four of the five top items are causal inference resources. That's not a editorial choice; it's the community voting with its clicks.
How the homepage adapts to who you are
The server sends the same HTML to every visitor. Your browser is the one that rearranges things, based on reading history stored locally.
First visit
- Rows shown in default score order
- Hero row: top 5 items site-wide by ranking score
- No "Because You Viewed" row yet (nothing to base it on)
- Every item equally weighted for display
Returning reader (after a few clicks)
- Items similar to past clicks get a quiet boost in row order
- Items already seen are deprioritized (you won't get the same thing twice)
- "Because You Viewed" row appears with semantically related suggestions
- All adjustments happen in-browser; server sees nothing
Homepage rows: the main stage
The homepage is organized into named rows, each curated for a different purpose. The hero banner at the top rotates through the five highest-scoring items across the whole corpus. Below that are rows like "Trending Now" (items with a spike in recent clicks, blended with a freshness boost), "Top Packages," "Top Datasets," and so on. Each row is ranked independently by score within its category. The rows themselves are pre-computed by the ranking step and stored in a layout file, so the site build step is just reading a file, not running any algorithms.
Search: two kinds of matching blended together
When you type something in the search bar, two kinds of matching happen at the same time. Keyword matching finds items whose text contains the words you typed (or words close to them). Semantic matching finds items that mean something similar to your query, even if they share no words. Both produce a ranked list, and the two lists are blended together using a technique called Reciprocal Rank Fusion: an item gets a higher combined score if it appeared high in either list (or both). After blending, a diversity pass ensures the top results aren't all nearly identical to each other. For the algorithm behind this, see the "How Recs Work" tab.
Related items: "Because You Viewed"
Once you have clicked on a few items, a row called "Because You Viewed" appears on the homepage. It shows items that are similar (in the embedding sense, meaning similar in topic and style) to things you recently clicked. The similarity is pre-computed and stored in a file that loads quietly in the background when the page opens. Matching your reading history against that file happens entirely in your browser. Nothing about what you clicked is ever sent to a server.
Section pages: sorted lists
Every section page (Packages, Datasets, Resources, and so on) is a ranked list of all items in that category, sorted from highest to lowest score. There is no personalization here: the order is the same for every visitor. These pages are static HTML files generated at build time, which means they load quickly and work without JavaScript.
Personalization: a subtle re-rank on the homepage
The homepage rows respond to your reading history. If you have clicked on several machine-learning papers, the homepage will quietly surface more of them. If you have already seen an item, the homepage deprioritizes it so you see something new. This re-ranking happens in JavaScript on your device, after the page loads. It does not change the underlying score in the content files; it only adjusts the display order for you. No reading history is ever uploaded.
How Recommendations Work
This tab explains the actual algorithms. No jargon left unexplained. If you want to understand how a number gets assigned to every item on the site, or how the search bar decides what to show, this is the place.
Signal weights: what the ranker is actually measuring
The table below shows every signal the ranking model reads, its direction (positive or negative), and its weight. These are the actual values in the current configuration.
| Signal | Weight | What it means |
|---|---|---|
| Outbound clicks | +5.0 | Clicked the link and left to read; the strongest positive signal |
| Search-result clicks | +3.0 | Searched, saw it, chose it: high-intent |
| Scroll past 90% | +2.0 | Read the whole page |
| Deep session | +1.5 | Part of a high-engagement multi-item session |
| Dwell time | +1.0 /min | Minutes actually spent on the page |
| Impressions | +0.5 | Card was visible in the viewport |
| Co-clicks | +0.3 | Clicked alongside other engaged items in the same session |
| Co-views | +0.1 | Viewed in the same session as other items |
| Freshness | +0.15 max | Recency boost that decays with a 30-day half-life |
| Rage clicks | -2.0 | Rapid repeated clicks: frustration signal |
| Quick bounces | -1.0 | Left almost immediately |
| High impressions, zero clicks | -1.0 | Seen many times, never clicked: probably irrelevant |
Relevance is not binary. It is a distribution over users, queries, and contexts. A recommender system is really a probability estimator dressed up as a list.
How search blends two lists into one
When you search, two ranked lists are produced simultaneously and then fused. The fusion technique (Reciprocal Rank Fusion) rewards items that appear high in either list.
// step 1: get two ranked lists for query "diff in diff"
keyword_results = keyword_search("diff in diff")
// → ["DiD course", "mixtape chapter 9", "Callaway-Sant'Anna", ...]
semantic_results = semantic_search(embed("diff in diff"))
// → ["Callaway-Sant'Anna", "Event studies paper", "DiD course", ...]
// step 2: Reciprocal Rank Fusion score for each candidate
for item in all_candidates:
rrf_score = 1/(k + keyword_rank) + 1/(k + semantic_rank)
// k=60 is a smoothing constant; higher rank = lower position number
// step 3: sort by rrf_score, run MMR diversity pass on top-20
fused = sort_by(rrf_score, descending=True)
final = mmr_diverse(fused[:20], lambda=0.7)
// MMR trades some relevance for variety so the top 5 aren't near-identicalA search for "difference in differences" returns different results from keyword and semantic passes. The keyword pass finds items that literally contain those words in their titles or descriptions. The semantic pass finds items whose meaning is close to the query in vector space: it surfaces event-study papers, synthetic control resources, and "parallel trends" explainers even if they never use the phrase "difference in differences." After fusion, the blended top 5 includes results from both passes ranked by their combined position. The diversity step then ensures the top 5 aren't five near-identical DiD textbook chapters.
The ranker: learning from engagement
The core algorithm is a machine-learning model (a type of "gradient boosted tree," which you can think of as a very sophisticated weighted average that learns its own weights from data). The model takes a set of engagement signals for each item and outputs a single score between 0 and 1. Higher is better.
The signals and their rough weights are: clicks (5x), search-result clicks (3x, because clicking a search result is a very intentional act), scroll depth past 90% (2x, which means you read the whole thing), time spent reading (1x per minute), co-occurrence with other clicked items in the same session (0.3x), and freshness (recent items get a small decay-based boost). On the penalty side: rage clicks (rapid repeated clicks, a sign of frustration, -2x) and items with many impressions but zero clicks (-1x).
The model learns to combine these signals from historical data. It does not use rules written by hand; it figures out the combination that best predicts future engagement.
Cold start: what happens for new items
A brand-new item has no engagement at all. Clicks are zero, dwell time is zero, scroll depth is zero. A model trained on engagement signals has nothing to work with. This is the "cold start" problem.
The solution is nearest-neighbor lookup. Before scoring new items, the pipeline computes embeddings for every item (the 1,024-number representation from the Processing tab). It then finds the 5 existing items most similar to each new item in that embedding space, and assigns the new item a fraction of their average score. This fraction is weighted by how close the neighbors are: a very similar neighbor gives a stronger signal than a loosely related one. The new item then accumulates its own engagement data over time, and future ranking runs replace the borrowed score with a real one.
Search: two lists blended into one
When you search, two separate systems respond at the same time. The keyword system finds items where the text literally contains your query words (or words with the same stem). The semantic system converts your query into an embedding and finds items whose embeddings are close. Each system produces a ranked list of results.
The two lists are then merged using Reciprocal Rank Fusion. This is a straightforward idea: each item gets a score based on where it appeared in each list. Position 1 in either list is worth 1/(1+1). Position 5 is worth 1/(5+1). The positions are summed across both lists, and items are re-sorted by the total. An item that appears in the top 5 of both lists will beat an item that only appears in one, even if the one-list item was ranked first in its own list.
After blending, a diversity pass runs over the top candidates. It prefers results that are meaningfully different from each other, rather than showing ten near-identical items. It does this by computing the similarity between each candidate and the items already selected, and penalizing items that are too similar to what is already in the list.
Personalization: what changes for you
When you have clicked several items, the homepage performs a local re-rank in your browser. It retrieves the pre-computed similar items for each thing in your reading history, boosts the scores of items in that set, and reduces the scores of items you have already seen. This adjustment happens entirely in JavaScript on your device. The server sends the same page to everyone; your browser is the one that changes the order. No reading history is ever sent to a server.
How A/B Testing Works
Before rolling out a change to everyone, we run an experiment: show the change to half the users, keep the old version for the other half, and measure what happens. This tab explains how that process works from start to finish.
Declaring an experiment
Every experiment lives in a registry file, committed to the codebase. The declaration says who gets what, and what fraction of traffic goes to each variant. Nothing runs in secret.
{
"id": "homepage_row_mmr_vs_baseline", // unique, stable name
"status": "active", // active | paused | draft | ended
"started_at": "2026-06-01",
"primary_metric": "ctr", // what we're measuring
"variants": [
{ "id": "control", "traffic": 50 }, // current behavior
{ "id": "treatment", "traffic": 50 } // MMR diversity re-rank
]
}
// bucketing: hash(user_id + experiment_id) % 100 → variant
// same user → same variant, every visit, for this experiment's lifetimeThe first experiment, harness_aa_v1, ran from May 4 to May 23, 2026. Both variants were supposed to see the same thing; an A/A test should show identical click-through rates. Instead, the analysis showed control_a at 4.0% CTR (8,071 impressions, 322 clicks) vs control_b at 2.3% CTR (10,205 impressions, 237 clicks). A z-score of -6.5 and p near zero. That's not noise. A difference that large in a test where nothing differs means the measurement is broken.
The root cause: the anonymous user ID cookie was being written only after a first interaction (click or scroll), not on page load. The bucketing function read that cookie on the first impression, before the interaction, found it empty, and minted a temporary ID. Same user ended up in both variants. The bug touched 57 users across 19 days of data. It was fixed, the experiment was reset as harness_aa_v2, and the contaminated data was preserved for the record but flagged as invalid.
The most dangerous phrase in data science is "the numbers look about right." About right is how bad measurement hides behind intuition until it's too late to notice.
Switchback design (planned for next phase)
Some changes can't be tested by splitting users, because the two groups would interact with each other. The "Trending Now" row is one example: what one group clicks affects what appears as trending for another. The fix is a switchback: alternate the change on and off over time, then compare the periods.
Why A/B test at all
Opinions about which design or algorithm is "better" are cheap. Everyone on the team (or in this case, me) has intuitions about what will work. But intuitions are often wrong, and the site is too small for that to be obvious without measurement. A/B testing is the discipline of turning a question like "does sorting by diversity help more than sorting by score alone?" into a measurement: run both versions at the same time on real users, count what actually happens, and let the data decide.
How bucketing works
When you visit the site, your anonymous user ID (stored in a cookie) is combined with the experiment ID using a hash function called FNV-1a. A hash function takes any text as input and produces a number. The number is taken modulo 100 to get a value between 0 and 99. Experiments declare their traffic split as percentages: a 50/50 split means users in buckets 0-49 see control, users in buckets 50-99 see treatment. Because the hash function is deterministic (the same inputs always give the same output), you see the same variant every visit, for the duration of the experiment. Different experiments use different experiment IDs, so your bucket in one experiment is independent of your bucket in another.
What gets tracked
Every event your browser sends to the analytics service already includes an experiment field: a map of which experiments you are in and which variant you received. For example, something like "harness_aa_v2: control_a." The analytics service stores this alongside the event itself. An analysis script then reads those events, groups them by variant, and computes per-variant click-through rates (CTR, the fraction of users who clicked something) with confidence intervals. It then runs a standard statistical test (a two-proportion z-test) to judge whether any difference between variants is likely real or just noise.
The current experiment: an A/A sanity check
Before running a real treatment experiment (where the two groups see something different), we first run an A/A test: both groups see exactly the same thing. If the harness works correctly, the two groups should produce nearly identical click-through rates. A big difference in an A/A test is a sign that something is broken in the measurement pipeline, not that one version is better. The current experiment is this A/A check. It replaced an earlier version that turned out to have a bug: the cookie that identifies users was being written too late, so some users appeared in both groups at once. That bug was fixed before restarting the experiment.
Switchback experiments (planned)
Some changes cannot be A/B tested by splitting users. The "Trending Now" row, for example, shows the same content to every visitor. If you show different trending rows to different user groups, the groups are no longer independent: what one group clicks affects what appears as trending for the other. The fix for this is a switchback experiment: instead of splitting by user, you alternate the change on and off over time, then compare the periods. One hour you run version A, the next you run version B, and so on. The analysis then compares the A-periods to the B-periods. This design is more complex to analyze, but it avoids the contamination problem. Switchback experiments are planned as the next step after the current A/A check validates the harness.
How Well Does the Ranker Work?
Every week or so, the ranking system gets re-evaluated. This tab shows the latest results. The numbers here are real, pulled from the same evaluation runs that govern whether any change gets promoted to the live site.
What these numbers are measuring
The ranker is the part of the site that decides which items float to the top of every list. Measuring how good that ordering is requires checking, after the fact, whether the items placed highest were actually the ones people clicked. We do this by holding out a chunk of recent sessions (60 days of data by default) and asking: for each session, given only the items the ranker scored, did the thing the person clicked appear early in the list? NDCG, Hit-Rate, and MAP all measure this in slightly different ways.
NDCG (short for Normalized Discounted Cumulative Gain) is the primary metric. It scores between 0 and 1. A ranking that puts every clicked item at position 1 scores 1.0. A ranking no better than random scores near 0. For a content site at this scale, 0.4 to 0.5 is a reasonable starting point for a model that has only a modest amount of engagement data to learn from. We expect this number to grow over months as the corpus accumulates more clicks and the model can learn from richer patterns.
Hit-Rate@10 answers a simpler question: was at least one of the items in the top 10 something the visitor eventually clicked? This is easier to achieve than NDCG because it doesn't care about position within the top 10. MAP (Mean Average Precision) splits the difference: it rewards high positions, like NDCG, but uses a simpler average-of-precisions formula that some practitioners find more interpretable.
How evaluation actually runs
The evaluation is entirely offline: it does not run experiments on live users. Instead, it replays historical sessions against the current ranking and checks how well the ranking aligns with what people actually clicked during those sessions. This is a common technique in recommender systems research, sometimes called "leave-one-out" or "holdout evaluation." Sessions without any clicks are excluded from the count, because there is nothing to measure against.
The evaluation gate also governs when a new ranking approach is allowed to go live. The rule is: no default flip until we have at least three evaluation runs at the same holdout setting showing a consistent improvement. One measurement proving something looks better is not enough; it could be noise. Three measurements in the same direction is a pattern worth acting on.
Replay evaluation
Alongside the main evaluation, we run a replay: a head-to-head comparison of two scoring approaches on the same set of historical sessions. The most recent replay (2026-05-24) compared two configurations over 39 sessions. Both approaches produced a NDCG@10 of 0.2275. When the delta is zero, it means we tested the same configuration against itself as a sanity check, confirming the evaluation framework is deterministic.
What "good" looks like over time
We are not chasing a single target number. The goal is a steady upward trend over months as the ranker gets more engagement data, better cold-start propagation, and more carefully tuned features. The chart above shows the NDCG@10 trajectory. With one data point today, it is a single dot. Future evaluation runs will fill in the line. We track every run in a log that is the source of truth for the chart, so nothing is smoothed or cherry-picked.
All evaluation runs
| Date | NDCG@10 | Hit-Rate@10 | MAP | Sessions | Notes |
|---|---|---|---|---|---|
| 2026-05-03 | 0.4191 | 0.8000 | 0.4149 | 15 | rerank source=api |
What Experiments Have We Run?
Every change to the ranking, layout, or recommendation logic that could affect what you see gets tested in a controlled experiment before it goes live for everyone. This tab is the public registry of every experiment ever started, including the ones that went wrong.
Running now: harness_aa_v2 live
A/A sanity re-run after the cookie-timing fix. Both variants see the same experience. If CTR is balanced, the harness is validated and real treatment experiments can begin. Results are not available yet. Once we have a few days of data, the per-variant numbers will appear in the detail section below.
| ID | Started | Status | Kind | Verdict |
|---|---|---|---|---|
harness_aa_v1 | 2026-05-04 | paused | A/A sanity | broken -- cookie-timing bug caused 57 users to appear in both variants; 19 days of data were contaminated. Bug fixed; harness_aa_v2 started. |
harness_aa_v2 | 2026-05-24 | live | A/A sanity | collecting data |
Experiment details
A/A sanity test to validate the end-to-end experiment harness. Discovered a bucketing contamination bug: the anonymous user ID cookie was written too late, so the first-impression bucketing used a different ID than later events. Fixed before restarting as v2.
A/A sanity re-run after the cookie-timing fix. Both variants see the same experience. If CTR is balanced, the harness is validated and real treatment experiments can begin.
How we run experiments here
The experiment registry is a file committed into the site's code, so every experiment that has ever existed (including the ones that were paused, broke, or were never launched) has a permanent public record. The status field tells you what happened: "active" means it is running right now, "paused" means it was stopped before a conclusion, and "ended" means it ran to completion with a verdict.
Bucketing (deciding which variant each visitor sees) happens in JavaScript when you first load any page. Your anonymous user ID (a random number stored in a cookie) is combined with the experiment ID using a hash function to produce a number between 0 and 99. That number maps to a variant. The same user ID always produces the same bucket for a given experiment. If an experiment is paused, the bucket mapping is preserved but no new experiment data is collected.
We do not run A/B tests in secret or post-hoc select which results to publish. The table above shows every experiment, including the one that broke. The broken one is especially valuable: finding that the harness was contaminated before running any real treatment experiment saved us from drawing wrong conclusions about whether changes actually helped.