Skipper is available under the Apache 2.0 license. The code is on GitHub, alongside documentation and runnable examples. Our earlier engineering deep dive explains how we built it. Here, we’ll focus on the architecture of Skipper and where it fits in the ecosystem of options for development around workflows.

Why we developed Skipper

A process that spans multiple services, waits for human input, or involves a mix of synchronous and asynchronous steps, needs persistent storage for recording its progress. When a server restarts, it must know what has already happened, what (if anything) should be retried, and what is still awaiting processing. Implementing this separately for each process would mean repeatedly solving the same problems around partial failure, timers, and recovery, with associated testing and debugging; using a proven library is likely to improve quality and reduce toil.

Existing workflow engines meet these challenges. A common approach relies on a central orchestration service, whether self-hosted or fully managed, to handle durable execution. This setup provides a unified platform for multiple teams, streamlining central operations and monitoring.

For Airbnb, we wanted a new approach, one in which each service owned its workflows within its existing deployment and database. For critical services, that meant avoiding an additional shared orchestration dependency. It also let workflow execution stay in-process, without a separate coordinator between steps.

That requirement shaped Skipper’s architecture, as shown in the figure.

With Skipper, the service’s replicas also run the workflow runtime and scheduler. In a typical production deployment, they share a database that holds workflow state and scheduled work. The scheduler polls for runnable tasks and uses leases to coordinate ownership. If a replica fails, another can recover the work; takeover timing depends on scheduling and lease settings.

This reduces the number of separately deployed systems. It also puts responsibility for workflow execution, database capacity, and recovery with the service’s team. The duty of assuming those responsibilities is part of the tradeoff.

We believe that Skipper will appeal to teams that have a robust operational environment and need a workflow engine that adapts to their existing ways of working.

A workflow is a class

Skipper has two main abstractions. A workflow describes the orchestration: decisions, ordering, and waits. Actions perform work with potential side effects, such as calling an API or updating a database. Skipper records action results so they can be reused during recovery.

Consider a hypothetical listing-publication process. Some listings require a human review before publication; afterward, the host receives a notification:

class PublishListing : Workflow() {
  private val listing = actions<ListingActions>()

  @StateField var reviewApproved: Boolean? = null

  @WorkflowMethod
  suspend fun publish(request: PublicationRequest): Boolean {
    if (request.requiresReview) {
      val reviewed = waitUntil(
        { reviewApproved != null },
        Duration.ofDays(1)
      )
      if (!reviewed || reviewApproved != true) return false
    }

    listing.publish(request.listingId, request.requestId)
    listing.notifyHost(request.listingId, request.requestId)
    return true
  }

  @SignalMethod
  fun recordReview(approved: Boolean) {
    reviewApproved = approved
  }
}

waitUntil hibernates the workflow without holding a thread. The recordReview signal lets an API handler or queue consumer supply the review decision. A redeploy during the wait does not discard the persisted workflow. If approval never arrives, the timer allows the workflow to complete and deliver a negative result.

The action implementations and request type are omitted here. Each action must make its own side effects idempotent, for example, by deduplicating publication and notification separately using the stable request ID. Skipper does not make calls to downstream systems transactionally.

Java workflows use the same model, with methods that can return CompletableFuture. The examples include Java, Kotlin, Spring Boot, and Dropwizard projects. For tests, WorkflowTest provides an isolated in-memory runtime with controllable time, so you can exercise waits, signals, retries, and completion while supplying fakes for downstream collaborators.

Recovery through checkpoints and replay

When Skipper resumes an interrupted workflow, it re-executes the workflow method from the top. Actions with durably persisted checkpoints return their saved results. Execution continues through the remaining work.

This requires deterministic workflow code: orchestration must remain compatible with replay. Clock reads, randomness, and I/O belong in actions or explicit checkpoints rather than directly in the workflow body.

By default, Skipper batches action checkpoints until execution completes or pauses for a wait or retry. This reduces database writes, but a crash before that batch is persisted can require several previously completed actions to execute again. Actions can request immediate checkpointing to narrow that window. Even then, a downstream effect and its checkpoint are not one atomic operation, so side effects remain at least once.

On the normal execution path, a workflow can run directly in the service process, backed by persisted state and scheduled recovery work. It does not need a round-trip to an external orchestration service between actions. Database access and downstream calls still contribute to latency; the benefit is removing the separate orchestration tier from that path.

Durability does not mean every business operation succeeds. Developers define which errors are retryable and choose retry policies. Permanent failures and exhausted retries are surfaced for handling; scheduler tasks that exhaust recovery attempts can require manual intervention. Skipper also supports Saga-style compensation when a process needs explicit recovery actions for previously completed steps.

Where Skipper fits

Skipper’s production use already spans multiple teams at Airbnb. It is a good fit for a Java or Kotlin service that owns a multi-step process, and wants durable execution within its existing deployment. It is especially useful when the alternative is maintaining custom retry jobs, state tables, and polling loops for each new process.

What’s in the release, and what’s next

The open-source release includes the core engine, MySQL and SQLite adapters, test utilities, and the admin UI. Signals, queries, retries, compensation, and workflow evolution primitives are part of the programming model. It also includes Skipper State Machine, a Kotlin DSL for processes that are easier to describe as states, events, and transitions. The documentation covers these capabilities and JVM compatibility.

Next, we plan to explore adding more storage plugins, such as Postgres, DynamoDB, etc., and improve tooling for workflow evolution and replay debugging. Open source project contributions around automated compatibility checks, migration assistance, and clearer replay visualization could also help developers work through problems common to replay-based engines.

If this type of work interests you, check out some of our open positions!

Acknowledgements

A huge thank you to everyone who contributed code, adopted Skipper, shared feedback, and helped get this release over the finish line. This work reflects contributions from teams across Airbnb.

Special thanks to:

Our engineering leadership sponsors Navjot Sidhu, Musaab At-Taras, Mini Atwal. Jon Parise for guiding the team through the open source process, and Gary Leung and Harshit Gupta for helping us shape Skipper with its first use-case.

Read “How Airbnb evaluates coding models” on airbnb.tech

Frontier models, with hundreds of billions of parameters and internet-scale pretraining, are often good at these tasks. But for interactive products and high-volume workloads, accuracy is only part of the requirement. We also need low latency, manageable serving costs, and control over the tradeoff between false positives and false negatives.

Smaller open-weight models are faster and cheaper to run at scale. By extracting logprobs, we’re able to produce a score for thresholding. But these small models are trained to handle a vast range of conversations and our tasks occupy just a tiny corner of that training distribution. Out of the box, they often miss subtle distinctions that matter immensely for our product.

Better prompting has limits. (I’m often reminded of an assignment from elementary school: write instructions for how to make a PB&J. “Put the knife in the peanut butter,” you’d write, only to have the teacher stab through the jar or insert the handle instead of the blade. There seem to be no instructions detailed enough to avoid every possible lapse in common sense.)

GLOWC, an internal framework used by engineers at Airbnb, makes it easy to fine-tune open-weight language (and vision-language) models into fast, accurate classifiers. In this blog post, we put one of these classifiers head-to-head with open-weight and proprietary models and find that it matches frontier-level quality in under 100ms at roughly 1% of the cost.

Case study: benchmarking a production classifier

Public benchmarks rarely capture a real product’s constraints. Instead, we test a production classifier that screens content for violations of Airbnb’s policy on pricing and payment transparency. The task requires recognizing subtle policy distinctions with high precision, since false positives prevent legitimate content from being published. The model is called thousands of times per minute, and because it gates publication in real time, a score must return within 200ms.

We compare a fine-tuned classifier with 2 billion effective parameters (E2B) against open-weight and proprietary models on a held-out evaluation dataset of 10,000 weighted examples. ¹ The fine-tuned model is trained on de-identified examples from users who have not opted out of AI training. Open-weight models are self-hosted with vLLM 0.28.0 on a single NVIDIA RTX Pro 6000 GPU. Models up to 12B parameters run in bfloat16. Larger models use publicly available FP8 or NVFP4 checkpoints. Proprietary models are called through their vendor APIs. Unless noted, every model receives the same concise ~400-word prompt.

Quality vs. latency

Any model we deploy must be accurate enough to trust with a policy decision and fast enough to avoid disrupting the user experience. These two criteria often trade off, forming a quality-latency frontier that we plot below.

We report max F1 as the measure of classification quality. While we would normally prefer PR-AUC for imbalanced classification, most proprietary models return only a True or False verdict, so there is no curve to integrate. Max F1 puts both kinds of models on one axis: for open-weight models and decision models, we sweep over thresholds and report the best F1. For other proprietary models, we report the F1 of their verdict. The conclusions do not depend on the choice: PR-AUC, where available, and accuracy rank the models nearly identically to max F1. Latency is server-side handling time as recorded by our serving gateway.

Open-weight models are fast but sacrifice quality. Many open-weight models have latencies below our budget. Classification quality and latency tend to climb with model size. ² Off the shelf, their performance plateaus meaningfully lower than the best proprietary models or our fine-tuned classifier.

Proprietary generative models are accurate but slow and inflexible. Their classification quality does not meaningfully improve with more thinking, so we report the low-thinking results. Even in low-thinking mode, GPT-6.1 Sol and Claude Sonnet 5.5 have a median latency of about 900ms and a p99 of multiple seconds, many times our latency budget. GPT-6 Luna is the fastest proprietary generative model we tested, yet is both slower and less accurate than Gemma4-31B running on a single GPU. And because each model returns a verdict rather than a score, it gives a single, fixed precision-recall point. With the concise prompt, GPT-6.1 Sol lands at 0.82 precision and 0.96 recall, favoring recall in a task where false positives are costly. Moving that point means rewriting the prompt.

Decision models have strong performance for their speed. TypeSafe’s Jev returns a probability instead of generating a verdict, so it isn’t held back by token generation and can be thresholded like our own classifier. It is the most accurate off-the-shelf model we tested under 100ms.

Fine-tuning moves the quality-latency frontier instead of trading along it. Fine-tuning Gemma4-E2B on hundreds of thousands of domain-specific examples lifts its F1 from 0.563 to 0.902, on par with the best proprietary models, while median latency stays at 35ms. No other model we tested is close on both axes.

A detailed prompt improves classification quality but doesn’t close the gap. We also test every model with a detailed 2,700-word prompt that spells out each behavior type with worked examples and common false positives. Proprietary generative models gain one to three points and roughly match the fine-tuned classifier, albeit at much higher latency. For models under our latency budget, the gains are modest and a large gap remains. The detailed prompt also increases latency and cost. Together, these indicate that for models fast enough to deploy, prompt tuning does not necessarily close the quality gap.

Cost at scale

A GPU costs the same whether it’s busy or idle, therefore self-hosting is only cheap once you have sufficient traffic. The fine-tuned classifier can support thousands of RPM on a single GPU with low latency, therefore its serving cost is constant until another worker is needed. Proprietary models, called via vendor APIs, charge based on token usage. This forms a linear relationship with volume: the more traffic you send, the more expensive it becomes.

Costs for proprietary models are calculated from the input and output tokens observed on our evaluation runs, priced at each vendor’s publicly available list rates as of October 2026. They do not reflect any negotiated pricing, discounts, or other non-public commercial terms. For models that offer discounted pricing on cached input tokens, we assume a 71% cache hit rate based on the average shared prompt prefix. This is higher than the hit rate we observed in practice, so it favors the hosted models. GPU costs are based on publicly available cloud rental rates for an NVIDIA RTX Pro 6000, and all costs are shown as multiples of that single-GPU cost.

Self-hosting is cheaper at high volume. At thousands of RPM, the frontier models using the concise prompt cost two orders of magnitude more than a single self-hosted GPU, and the detailed prompt raises their cost by a further 20 to 40%. Lower-cost proprietary options still run nearly five times the cost of the GPU, with lower accuracy than the fine-tuned model. GPT-6 Luna and TypeSafe’s Jev cost about the same on this task, so we report Jev, the better-performing of the two. With the detailed prompt, Jev’s cost quadruples, as its API offers no discount for cached input.

Proprietary models only win at low traffic: below roughly 15 to 30 RPM, Claude Sonnet 5.5 and GPT-6.1 Sol are cheaper than a dedicated GPU.

When should you fine-tune?

Fine-tuning on in-domain data delivers quality, latency, and cost at the same time. But it isn’t always the right tool. For low-volume tasks, or tasks that change week-to-week, a frontier model behind a good prompt is often the right call. In our benchmark, fine-tuning paid off when:

1. Volume is high. Above the roughly 30 RPM on our task, one self-hosted GPU costs less than frontier APIs. At our volume, it costs orders of magnitude less.

2. Latency is tight. Our fine-tuned E2B classifier returned a score in about 85ms p99, while frontier APIs took around 900ms at the median and seconds at p99 to achieve similar quality. Faster tiers exist, but vendors charge a premium for them. ³

3. You need a threshold, not a verdict. Scores let you choose your tradeoff between recall and precision without rewriting the prompt.

4. You have labeled data. Examples teach distinctions and edge cases that instructions struggle to spell out. Hundreds of thousands of in-domain examples did what a 2,700 word prompt couldn’t.

This classifier is just one example, but the same tradeoffs have held across other models we’ve fine-tuned. At Airbnb, GLOWC provides shared infrastructure for fine-tuning, batch inference, and serving, so teams can focus on their task while improvements carry over to every model. If you’re interested in pushing the tradeoff between model quality, latency, and cost, check out our open positions!

Appendix

1. Example weights correct for how the evaluation set was sampled from production traffic. We validated a subset of models on a larger set of 184k examples; their max F1 was within 0.01 of the results reported here. ↩︎
2. Latency doesn’t track parameter count alone. Attention implementation matters too. vLLM’s current kernels are less optimized for hybrid linear-attention layers than for standard or sliding-window attention, so models with similar parameter counts can have quite different latencies. ↩︎
3. See OpenAI’s or Anthropic’s fast mode for more details. ↩︎

Acknowledgments

We would like to thank John Romano and Chuiwen Ma for their help fine-tuning the classifier and curating evaluation datasets, Anirudh Agarwal and Ben Mendler for the serving infrastructure and latency measurement behind the models in this benchmark, and Mukund Narasimhan, Nathan Brixius, and Ahmad Al-Dahle for their thoughtful reviews.

Natural language search lets guests express intent in their own words, such as “romantic cabin near skiing” or “pet-friendly beach house that sleeps eight,” rather than navigating filters and categories. We needed to introduce this kind of search for Airbnb, and to have it work well from the start. However, this required training the model before we had actual user queries to train the model on, and a reasonable set of relevance labels (inferred from the queries) to map queries to matching listings. Without prior traffic, both must be synthesized from scratch. This is the cold-start problem for natural-language search.

Other organizations have faced similar challenges and have created methods to address them. However, prior approaches address the need for a body of queries and the need for relevance labels separately. Passage-based methods such as InPars [1] prompt a large language model (LLM) to generate queries conditioned on a single document, then rely on additional LLM judgments to assign relevance labels, introducing a second source of error. Promptagator [2] improves query diversity using few-shot examples, but is still limited by the gap between its generation distribution and real user behavior.

In this post, we describe Contrastive And Seed-guided Training for natural Language sEarch (CASTLE), our framework for jointly generating queries and producing relevance labels from Airbnb’s own de-identified booking data. Our paper describing CASTLE was accepted as a full research paper at the 35th ACM International Conference on Information and Knowledge Management, more widely known as CIKM 2026 [3].

Challenges

Generating useful synthetic training data for search requires more than producing grammatically plausible queries. The distribution of synthetic queries must match how real guests actually search; their preferred phrasing, query length, and attributes they tend to mention. If queries fail to match real-world search patterns, the model trained on them will generalize poorly.

We measured this mismatch using Kullback–Leibler (KL) divergence (a measure of how much one probability distribution differs from a reference distribution) between synthetic and real query distributions — both over surface-level query statistics and over attribute types (property type, amenity, location, and so on). Prior approaches scored between 9.33 and 26.2 on the KL divergence metric: far from the real-user distribution. Beyond distributional alignment, we also needed relevance labels. Generating a query is only half the problem; knowing which listings are relevant to it is the other half.

Solution

CASTLE addresses both challenges through a single mechanism: contrastive listing pairs derived from de-identified booking sessions, selected in accordance with user privacy permissions. When a guest books a specific listing after viewing several possible alternatives, that event implicitly encodes preference: the booked listing was selected over the others. We use these (positive, negative) pairs of listings — where the negative is randomly sampled from the other viewed-but-not-booked alternatives — as the structured input for query generation. Listing features include amenities, property details, and a one sentence summary of recent public guest reviews, with personal identifying information removed. Because we instruct the LLM to generate a query that is satisfied by the positive listing but not the negative, the relevance label is correct by construction; no post-hoc judgment is required. Crucially, what is “positive” and “negative” is determined by actual, previous user booking behavior, not by the LLM itself, so there is no circularity in the subsequent generation of relevance labels.

Query generation. We combine three prompt variants to maximize coverage and realism. Template-based prompts ensure structural diversity across common query patterns. Few-shot prompts use approximately 500 seed queries collected from targeted user research as linguistic exemplars, grounding generated queries in authentic guest language. Attribute-grounded prompts focus the query on specific distinguishing attributes between the two listings, such as a fireplace or proximity to a ski resort. An explicit variety mechanism encourages the model to vary query length and phrasing across samples, preventing “query collapse,” where an unconditioned LLM converges on the same short phrase repeatedly.

Together, these variants produce a word-count distribution with standard deviation σ = 2.48, closely matching real users (σ = 2.49), compared to σ = 0.99 from baselines (InPars and Promptagator) without the seed-guided mechanism. It is worth noting that real Airbnb guests tend to search with concise, intent-focused phrases, typically four to eight words, rather than long natural-language sentences. CASTLE is designed to match this real-world distribution, not to generate artificially verbose queries.

Flexible input design. For listing pairs where contrastive grounding is not available, such as newly listed properties with no booking history, we supplement with a Virtual Judge (VJ), an LLM evaluator that assigns relevance labels to additional query–listing pairs. This extends coverage to long-tail intents without sacrificing the quality guarantees of the contrastive construction for the core training set. We also introduce a real-query seed variant, CASTLE-R, designed for production settings where some real traffic has begun to accumulate.

Results

We evaluate CASTLE against two previous approaches, InPars-style generation and Promptagator, on two axes: query distributional alignment and label quality.

On KL divergence against real user queries, CASTLE achieves 1.01 — a 9.2× improvement over the best baseline (Promptagator at 9.33). The improvement holds at the attribute-type level as well: CASTLE’s attribute-type KL of 0.08 is lower than even our hand-curated survey seed queries (0.09), meaning the generated queries capture the diversity of real guest intents more faithfully than the seeds themselves.

To validate label quality, we conducted a human evaluation with three independent subject matter experts reviewing 200 sampled triplets. Annotators agreed with CASTLE’s relevance assignments at 91–93% across all three raters, confirming that the contrastive construction produces high-quality relevance labels without any human annotation in the loop.

Beyond cold-start, CASTLE’s synthetic data continues to provide value once real traffic accumulates. Because generation is conditioned on specific listing characteristics, it naturally oversamples long-tail intents that appear rarely in organic traffic, such as property types or amenity combinations that guests only search for occasionally, but that have high importance when they do appear.

Conclusion

The key insight in CASTLE is that authentic booking behavior and LLM generation are complementary, not competing. Contrastive listing pairs from booking sessions ground query generation in real platform behavior and provide relevance labels by construction, while LLM prompting provides the query diversity and linguistic variety that behavioral data alone cannot supply. Together, they close the gap between synthetic and real user distributions in a way that neither approach achieves independently.

Looking ahead, we are exploring multi-turn conversational search, where a guest might begin with “cabin near skiing” and refine to “one that allows dogs” — a setting in which cold-start data generation is an even harder problem, and where the contrastive grounding approach may generalize further. For the full technical treatment, see our CIKM 2026 paper at arxiv.org/abs/2605.21812 (DOI: 10.1145/3799682.3841071).

Interested in tackling these kinds of challenges? Browse our previous publications or explore our open roles.

Acknowledgments

We would like to thank our collaborators across the AI Platform — Relevance & Personalization team for their support throughout this project.

References

[1] Jeronymo, Vitor, et al. “InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval.” arXiv:2301.01820, 2023.

[2] Dai, Zhuyun, et al. “Promptagator: Few-shot Dense Retrieval From 8 Examples.” ICLR, 2023.

[3] Wei, Wendy Ran, et al. “CASTLE: Contrastive and Seed-Guided Training for Cold-Start Natural Language Search.” ACM International Conference on Information and Knowledge Management (CIKM), 2026. DOI: 10.1145/3799682.3841071.

At Airbnb, MySQL-compatible databases are a critical backbone of our online database infrastructure: a fleet of hundreds of clusters supporting thousands of use cases at millions of queries per second (QPS). Operating databases at scale brings hard problems, including sizing clusters for future growth, keeping behavior consistent across version upgrades and migrations, and reproducing production incidents well enough to debug them. This post describes the database traffic capture and replay system we built to help us tackle them.

Challenges and motivation

A complex database operation such as Airbnb’s requires a number of supporting systems, and one of them is a way to capture database queries and replay them. This is needed for error recovery, governance, quality assurance testing and product improvement efforts.

Previously, Airbnb’s system for database query capture and replay was fragmented. Each language binding used its own logging framework to emit query traces to Kafka, which were then processed per use case and replayed by a custom replayer.

This system was borrowed from our observability pipeline. It worked as a fast initial solution, but over time, we found that it fell short of what we needed. Capturing queries client-side and per language carried a heavy maintenance burden whose ownership was ambiguous, and it scaled poorly across our service-oriented architecture (SOA). Most importantly, it lacked enough information to convey a complete picture of each transaction. As a result, it couldn’t accurately replay transactions, which made it impossible to verify that the same queries return the same results on the new database.

To close these gaps, we set out to build a system for query capture and replay, with three goals:

• Load testing with real production traffic. Synthetic benchmarks such as sysbench don’t capture the full spectrum of production queries. By replaying captured traffic at production rates, or at higher multiples to model forecast growth, we can right-size clusters: upscaling those with insufficient headroom for growth and downscaling over-provisioned ones.
• Query compatibility across upgrades and migrations. Subtle behavior differences (such as differences between MySQL 5.7 and 8.0, or between “official” MySQL and a MySQL-compatible engine) can break applications. By replaying the same traffic against two targets and comparing results, we surface incompatibilities before production migrations.
• Performance debugging. MySQL provides performance_schema digests for diagnostics, but these are abstracts of actual queries and don’t contain enough information to reproduce a problem. Capturing complete SQL statements with their transactional context lets us replay problematic workloads offline, both to investigate incidents and to validate fixes before they ship.

To meet these goals, we needed a single, client-agnostic capture point that needs no application changes. We built it on top of ProxySQL, an open-source proxy that understands the MySQL wire protocol and already sits between our applications and databases.

System architecture overview

All database traffic already flows through a ProxySQL deployment, which makes it a natural place to capture queries. The query capture and replay system has three components, all built in-house: a Log Mover that collects the query logs, a Log Processor that turns them into replayable, per-cluster datasets, and a Log Replayer that runs captured traffic against target databases.

Because captured queries can contain personal data, the pipeline treats them exactly as we treat production data. Logs are encrypted in transit and at rest, access follows the principle of least privilege and is limited to the database infrastructure team, and query logging is enabled for a given cluster only for the window a test requires. Replays run only in production-equivalent environments held to the same standards. Captured traffic is never replayed against lower environments or copied into a separate development tier.

Log Mover

ProxySQL has a built-in query logging feature controlled by its query rules, so we can turn logging on for one cluster’s traffic without touching anything else. When enabled, it writes query logs to the local disk, with little impact on query performance.

We complement ProxySQL with a Log Mover sidecar that monitors these log files and transfers them to cloud object storage, thus keeping local disk usage in check.

Log Processor

Our ProxySQL deployment runs on Kubernetes as a pool of stateless pods, each serving clients for several backend clusters. Therefore, a single log file can contain queries from many backend database clusters. An offline post-processing pipeline partitions these original query logs into a replayable dataset for each cluster.

The Log Processor job performs a few transformations:

Parsing, grouping, and ordering

The original query logs are in a binary format and intermingled with queries from multiple backend database clusters, so we decode them and write the queries for each cluster into its own file. Because the logs also interleave statements from concurrent connections, we reassemble each transaction’s statements in their original order so that replay reproduces the original behavior.

Timestamp-based bucketing

Each processed file is bucketed by query timestamp, holding a five-minute window of queries from one database cluster and ProxySQL pod. This allows the replayer to control the replay pace, throughput, and overall load at a fine granularity.

Metadata

Metadata, such as timestamp, username, database cluster and schema name, is stored alongside each processed log file, so users can locate specific query logs by timestamp and database cluster.

Query rewriting

The Log Processor also rewrites queries to keep auto-increment behavior consistent across databases. While MySQL’s default auto-increment values are monotonically increasing, MySQL 8.0 allows for customization to enhance performance, and certain MySQL-compatible databases may assign auto-increment values differently. As a result, an INSERT statement can produce a different last_insert_id than it did in production. A later read that depends on that value, such as a lookup of the related rows, would then fail or return nothing, skewing the test results.

To avoid this, the Log Processor rewrites each INSERT statement to explicitly include the last_insert_id captured in the logs. For example, an original MySQL statement INSERT INTO users (name) VALUES (‘bob’) that generates last_insert_id=1 will be rewritten to INSERT INTO users (id, name) VALUES (1, ‘bob’). This rewriting ensures that subsequent queries depending on last_insert_id values behave consistently across different database systems during replay tests, reducing false positives in compatibility testing. We accept the tradeoff here: by pinning last_insert_id, we no longer exercise the target’s native auto-increment generation during replay.

Log Replayer

The Log Replayer takes processed query logs and replays them against target databases. Users create replay jobs through a web UI with the following parameters:

• Source database cluster: the name of the database cluster whose traffic to replay
• Time range: the temporal scope of captured query logs to replay
• Target database endpoints: the destination databases for replay traffic
• Replay mode: which of the two modes to use (described below).

There are three components in the log replayer: an API Server (the control plane), a Replay Task Scheduler, and a fleet of Replay Task Workers.

API Server

The API Server is the control plane of the log replayer. Through a web UI, users can submit and control replay jobs, monitor job status, and view metrics and replay results. Job and task states are kept in a persistent database.

Users can choose one of two replay modes:

• Replay only (load testing): Queries are replayed to a single target database at a configurable speed, set either as a factor of the original traffic (1x, 2x, 3x) or as a target QPS. While query results are ignored, we track query errors, latency histograms per query digest, and resource utilization. This supports our load testing and capacity planning, and lays the groundwork for automated regression detection.
• Replay and compare (compatibility testing): The same queries are run against two target databases at once, and their results are compared with any discrepancies logged. This is used to validate database upgrades and migrations.

Replay Task Scheduler

The task scheduler breaks a job into batches of tasks, one task per log file. For each task, the scheduler pushes a message onto a managed message queue for replay task workers to consume.

To preserve the original timing, the task scheduler assigns every task in a batch the same “expected start time”, which is a future moment when that batch should begin. Because workers pick up tasks from the queue at different times, this shared start signal keeps them in step: each worker waits for it, then they all begin replaying their five-minute files together. This reproduces the concurrency and pacing of the original workload instead of replaying each file in isolation.

The task scheduler also monitors progress: if tasks start failing, it stops scheduling new ones to avoid cascading issues, and otherwise advances to the next batch.

Replay Task Worker

Workers scale horizontally, up to thousands of instances. Each polls the message queue for tasks, reads and parses its assigned query log file from cloud object storage, waits for the task’s “expected start time,” and then executes the logged queries against target databases in order. By spacing consecutive queries according to their original timestamps and the chosen speed factor, workers reproduce (or accelerate) the original traffic pattern while preserving relative timing.

Impact

This framework enables multiple business-critical capabilities for our online workloads and the teams operating them.

Upgrade and migration testing

For a database upgrade or migration, ensuring the new database has a compatible spec isn’t enough. We also have to preserve the performance baseline and backwards-compatible behavior. This became particularly apparent during our MySQL 5.7 to 8.0 upgrade, and replay testing was critical in identifying issues early and avoiding surprises in production.

Performance. MySQL 8.0’s performance profile differs from 5.7’s. While most changes were positive, some workloads regressed, for example more conservative metadata locking.

Pinning down the cause sometimes took more than one replay run. MySQL 8.0 removes the query cache entirely, so we first replayed against 5.7 with and without the query cache to isolate that effect, then compared 5.7-without-cache against 8.0. This bisection surfaced a pattern of duplicate, cacheable queries the application was sending, which the cache had quietly absorbed.

In the sharpest case, the latency for one query pattern went from 0.03 to 2.6 seconds, reading 273 MB per join instead of 5 MB, which we traced to an upstream change in how MySQL 8.0.20+ reads rows while sorting.

Correctness. Some default behaviors in MySQL 8.0 also changed in ways that can break applications, such as the default innodb_autoinc_lock_mode, which hands out interleaved, non-sequential auto-increment IDs that some applications assumed were sequential. To verify behavior stayed consistent, we ran “Replay and compare” against 5.7 and 8.0 restored from the same snapshot and diff’ed the results. This surfaced a common pattern: queries whose row order was never fully determined, either with no ORDER BY or an ORDER BY without a unique tiebreaker, which quietly return different rows on a new engine once a LIMIT is applied. By catching this proactively, we were able to work with the owning teams to add explicit ordering where it was implicitly expected by the application.

Capacity planning

To plan for growth, usually ahead of peak travel season, we replay a production cluster’s traffic in “Replay only” mode at higher speeds (for example, 2x) to model future load. This shows whether the current setup can absorb a seasonal peak or launch, or whether we need to scale up or out. On one large cluster, replaying 80% more write traffic pushed average commit latency up by almost 500% from about 6 ms to 34 ms, locating its ceiling well before real traffic did.

Teams at Airbnb can now request these traffic replays through a self-serve tool to help them prepare for expected growth, or identify current headroom on clusters which may be opportunities for more efficient bin-packing and cost optimization.

Conclusion

Synthetic benchmarks tell you how a database handles the workload you imagined. Replaying real traffic tells you how it handles the workload you actually have. That difference carried our MySQL fleet from 5.7 to 8.0 without a major production incident: we used replay to clear the highest-risk clusters first, catching latency regressions and non-deterministic queries offline instead of in production.

If this type of work interests you, check out some of our related positions!

Acknowledgments

Thanks to Ping Wang, Eugene Yedvabny, Ari Ekmekji, and Sam Lightstone for their support and feedback throughout this project.

We also want to thank Zheng Liu for their support in authoring this post during their time at Airbnb.

A significant share of Airbnb users arrive without a login, without a recent search history, or without any prior booking — especially those landing from paid advertising or organic search. For these users, the ML models that power search ranking, destination recommendations, and marketing landing pages face the cold-start problem: there is little to no user signal available.

In this blog post, we describe Proximity Features — a new class of ML features that addresses both the cold-start problem and evolving privacy constraints simultaneously. The core hypothesis: aggregate activity patterns within a proximity bucket can provide useful signals for cold-start personalization without relying on individual user identity. By grouping nearby users into buckets and aggregating their collective signals, we can construct rich, location-aware features available immediately for any user whose IP can be geo-located — no login, no prior history, and no persistent user identifier required.

The challenge: serving users without individual signals

Personalization at Airbnb has historically relied heavily on a user’s identifier to link together past searches, bookings, and browsing patterns. This works well for returning, logged-in users. But on marketing and acquisition surfaces, many users are logged out or visiting for the first time, and standard user identifiers are unavailable. In other cases, users are logged in, and standard user identifiers are available, but the features derived from them are stale or too sparse to carry meaningful signals.

Privacy regulations compound the challenge. Under GDPR, non-essential tracking for marketing personalization requires an appropriate legal basis and, in many contexts, user consent. Separately, browser restrictions and the deprecation of third-party cookies reduce the availability of cross-site identifiers. Any feature system targeting this traffic must work without relying on persistent user identifiers for model inference.

The result: users on these surfaces were receiving little to no personalization.

The solution: proximity features

The proximity key

The key insight behind Proximity Features is geographic correlation. Users browsing Airbnb from the same city or neighborhood tend to search for similar destinations, prefer similar price ranges, and often share travel patterns and local context. A user in Seoul is far more likely to book Jeju Island, the largest island that’s part of South Korea, or Osaka, Japan, than is a user in New York. This pattern holds however much Airbnb does or doesn’t know about the individual user.

We operationalize this insight through a proximity key: a compact group key representing a local geographic cluster of approximately 1,000 users. The key encodes a quantized latitude/longitude tile and, for densely populated areas, an IP hash bucket index within that tile — fine-grained for a city center, coarser for a rural location. In either case, the key identifies a neighborhood, not an individual.

This proximity key functions as an aggregation key analogous to user_id: any ML model that currently conditions on a user identifier can instead condition on a proximity key to serve cold-start users (e.g., logged-out, anonymous, or first-time). Even logged-in users with meaningful historical information available can benefit from the additional information that use of the proximity key enables. However, unlike logged-in user information, the features associated with a key reflect the collective behavior of the local group, not the actions of any single user. No persistent user identifier is required at inference time.

So what are the actual features? Once users are grouped into a bucket, features are computed daily across three categories: short-term engagement (e.g., top destinations, room types, and median prices from recent searches); long-term booking patterns (e.g., booked destinations and travel party signals); and aggregate bucket metadata (e.g., bucket size and geographic density). A user who has never searched on Airbnb gets a feature vector derived from approximately 1,000 nearby users, effectively borrowing signals from the crowd.

Adaptive clustering

Computing proximity keys requires grouping all global users into stable buckets of approximately 1,000 each. The challenge is that the geographic density of Airbnb users varies by orders of magnitude: a single coordinate near a major airport can represent vastly more daily users than a rural location with only a handful. A fixed geographic grid (like a standard geohash at one resolution) fails at both extremes — too coarse for cities, too fine for rural areas.

We solve this with a two-phase adaptive clustering algorithm. For any coordinate dense enough to fill a bucket on its own, we subdivide further using IP hash buckets, which preserves fine granularity in city centers. For the remaining coordinates, we apply multi-pass coarsening, progressively widening the geographic tile until enough users accumulate. The result is a zoom-adaptive map: fine-grained tiles over major cities, coarser groupings in rural areas. This runs efficiently at global scale on geo-IP coordinate data.

Proximity keys are also stable over time: the key partition bootstrapped in 2023 has remained valid in production without re-clustering, with a daily refresh handling new IPs and coordinate shifts. At serving time, a user’s IP is resolved to a proximity key in real time and features are fetched from a distributed key-value store. The lookup is a soft dependency. If it times out, the model proceeds without it, so personalization never blocks the core request path.

Privacy design

Privacy is built in at every layer. Every feature reflects a group of ~1,000 users, not an individual. The clustering input excludes users who have not provided consent. Geo-IP coordinates are coarse and group-level, aggregated around population centers rather than specific addresses. The pipeline is also integrated with consent management and data-governance deletion controls throughout.

The benefits: what proximity features unlock

Proximity Features have now launched across multiple Airbnb surfaces, with measurable results from production A/B experiments.

Marketing landing pages. Pages where users arrive from paid advertising and organic search have especially high cold-start rates. Before this work, the listing recommender had severely sparse feature vectors for much of this traffic and served static listing cards. Adding proximity features gave the model access to booking and browsing patterns from nearby users, enabling meaningful personalization for cold-start traffic.

Homepage AutoSuggest. For users without history, the AutoSuggest feature on the Homepage previously fell back to a static global list (e.g., Paris, Barcelona, London, Rome), regardless of where the user was browsing from. Proximity features replace this generic fallback with location-aware signals: a new user browsing from Beijing now sees Hong Kong and Tokyo instead.

The experiment showed directional gains concentrated among never-booked and dormant users on Airbnb, exactly the population proximity features are designed to serve. The diversity shift corroborates this: destinations like Kuala Lumpur, Dubai, and Jeju Island, which in the past had rarely been displayed to users in relevant regions, appeared consistently in the treatment arm.

Engagement emails. The same serving layer is being extended to personalized email campaigns using recent coarse location signals. Experiment validation is underway.

Conclusion

Proximity Features show that the cold-start problem does not require a persistent user identity to solve. By grouping users geographically and aggregating their collective signals, we can construct features immediately available for any user whose IP can be geo-located, without relying on persistent user identifiers. The design is intentionally reusable; any model conditioned on a user identifier can be extended to cold-start users by conditioning on a proximity key instead.

For the full technical details, see our paper accepted to the TSMO Workshop at KDD 2026. The paper includes the system architecture, clustering algorithm, privacy design, and complete experiment results.

If this type of work interests you, check out some of our related positions!

Acknowledgments

The authors thank the Relevance & Personalization, Marketing Technology, and Infrastructure teams at Airbnb for their collaboration, feedback, and support in bringing Proximity Features to production.

We also want to thank Hui Gao for his support in authoring this post during his time at Airbnb.

That design worked well, but it left a gap. Activity from earlier the same day wouldn’t show up in the embedding until the following day’s run, on top of the pipeline’s own processing lag — in practice, up to nearly two days of staleness. For a guest actively planning a trip, that meant the ranking model was often working from a slightly outdated picture of what they wanted, and the recent activities are often highly relevant to current search needs. This is a limitation that our original JourneyFormer research had already flagged as needing new serving infrastructure to solve.

In this post, we describe how we closed that gap by adding two new capabilities to Chronon, Airbnb’s feature platform: Near-real-time Model Transform and Push Mode. Chronon is an open source project, and these capabilities have been contributed back to our public repo.

Background

To keep a multi-layer Transformer off the critical serving path, our original design split inference into two stages. Offline, the sequence encoder would run as a scheduled batch job: each night, it would process the previous day’s guest activity and write a fresh embedding to a low-latency store for any guest who had something new to show for it. Online, retrieval was already real-time: the moment a guest ran a search, the ranking model read that guest’s embedding straight out of the store and combined it with the live query to score candidate listings.

This split kept serving latency low while still letting every ranking decision draw on years of guest history — but it meant an embedding was only ever as fresh as the last completed batch run.

Challenges

Moving from daily batch updates to near-real-time updates introduced three potential challenges that we needed to address:

• First, staleness had two separate sources: the batch schedule itself, which only ran once a day, and processing lag within that job, which pushed effective staleness closer to two days. Fixing only one of these wouldn’t have been enough to fully close the gap; we needed a pipeline that could react to a guest’s activity as it happened, not simply run more often.
• Second, our sequence encoder was built to run as a scheduled inference job over a full day’s snapshot of guest events, not as a service reacting to individual activity events one at a time. Wiring model inference into a real-time pipeline meant rethinking where and how the encoder got called, without duplicating the offline logic used for training.
• Third, guest activity signals — page views, searches, and other interactions — arrive as independent event streams. Reacting to any one of them in isolation risked missing the fact that these events need to be merged with a guest’s longer-term profile and routed through the encoder consistently, so the resulting embedding stays comparable to the one produced by the batch pipeline it replaces.

Solutions

We addressed these challenges by building two general-purpose capabilities into Chronon, rather than by building a modified pipeline specific to our use case.

Push Mode in Chronon

The first is Push Mode. Chronon already runs streaming jobs that keep feature values up-to-date as new events arrive. Push Mode extends this by having a streaming job publish a lightweight notification as soon as it commits a new feature value, instead of waiting for something downstream to poll for it. This turns a reactive step into an event-driven trigger: the moment a guest’s activity feature updates, downstream consumers are notified and can act immediately. For our use case, this meant a guest’s newest search or listing view could kick off embedding generation right away, instead of waiting for a scheduled job to notice it.

Near-real-time model transform in Chronon

The second is Near-real-time (NRT) Model Transform, which lets model inference run as part of the same streaming pipeline. Historically, running a model meant either embedding its logic directly into application code or waiting for an offline batch job. NRT Model Transform lets Chronon call an already-deployed model — in our case, the same Transformer sequence encoder from the batch pipeline — directly from within the streaming flow, and write the result back as a feature value that is immediately available for serving.

Integration

Combining the two, our new pipeline works like this: a guest’s activity event, such as a page view or a new search, is captured by a streaming job that merges it with the guest’s existing long-term and short-term sequence data. Push Mode then signals that new sequence data is ready, and NRT Model Transform runs the sequence encoder over it, producing a fresh embedding without waiting for the next scheduled run. The embedding is written to the same low-latency store the batch job uses, so the ranking model retrieves it exactly as it always has — the only difference is how quickly the embedding it retrieves has been updated.

Because both capabilities live in the platform rather than in a use-case-specific pipeline, other teams working on different near-real-time use cases can build on the same foundation.

Results

The impact showed up on two relevant dimensions of our search results: freshness and quality.

On freshness, we replaced a pipeline where embeddings could lag actual guest behavior by roughly two days with one where the typical delay is well under a minute. In practice, updates often land within 10 to 30 seconds of the underlying activity. A guest who views a handful of new listings can now have that context reflected in their very next search.

On quality, offline evaluation showed a +1.67% improvement in Normalized Discounted Cumulative Gain (NDCG) over the daily-batch baseline — a large jump for a ranking system that has already been refined for over a decade, where even gains of a fraction of one percent are considered meaningful. Online A/B tests confirmed an approximately one-third of a percent increase in uncancelled bookings. This confirms something we suspected, but hadn’t measured directly: freshness itself is a meaningful source of ranking quality. The same guest representation becomes more valuable to the ranking model, and to the guest themselves, simply by being more current.

Rather than a one-off integration, Push Mode and NRT Model Transform represent general platform capabilities that we expect to serve as a foundation for other near-real-time modeling efforts across Airbnb. Since Chronon is open source, and since both features are already available in the public repository, their impact extends far beyond Airbnb. With these additions, any team running Chronon to build a near-real-time model is able to build on the foundation we’ve created.

Conclusion

Our previous post described how encoding a guest’s full history — their bookings, reviews, and recent browsing — allows our ranking system to better prioritize relevant listings based on current search intent. This post closes the remaining gap: making sure that understanding reflects a guest’s most recent activity, not just what they did as of last night’s batch run.

By combining Push Mode’s event-driven triggering with NRT Model Transform’s in-pipeline model inference, we turned a daily batch process into a near-real-time one, cutting effective staleness from roughly two days to less than a minute, and improving offline ranking quality by +1.67% NDCG in the process. More broadly, the pattern we used here: react to an event, merge it with existing state, run inference immediately, and serve the result; is one we expect to generalize to other guest-facing models that depend on freshness.

You can learn more about our team’s work on personalization and search ranking by checking out our previous post on sequence modeling the guest journey and other engineering blog posts from Airbnb. To learn more about Chronon, check out our previous blog posts on the project, and browse the code at our repo: https://github.com/airbnb/chronon.

Interested in learning more about our technical journey? Browse our previous publications to see how our systems have evolved. If tackling these kinds of challenges excites you, explore our open roles.

Acknowledgments

We would like to especially thank the following people for their great collaboration (listed alphabetically): Ashish Jain, Ben Mendler, Bin Xu, Casey Getz, Gil Forsher, Han Zhao, Hao Li, Jiawei Yao, Jun Shi, Kedar Bellare, Linyun He, Liwei He, Michael Kinoti, Michael Sestito, Mingyang Xu, Pallavi Adusumilli, Ruirong Yang, Shashank Dabriwal, Sid Reddy, Sophie Wang, Tanya Piplani, Tracy Yu, Vijay Velagapudi, Xiaowei Liu, Yangbo Zhu, Yan Zhang, Yi Li, Yiwei Wang, Zach Barahal, and Zhiwei Wang.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

LLMs certainly make for confident scientists, but we need them to be responsible ones. Smarter models help, but intelligence has never been the whole of science, in people or in machines. The method is as much the product as the answer. That is the idea behind the agent harness we built for data science: the methodology itself, built as infrastructure around the model. It governs how an AI agent operates, from framing a question to selecting evidence to recording decisions, so results can be reproduced, audited, and challenged, and the method shared, inspected, and built on.

The challenge of unstructured data exploration

In 2025, Airbnb was preparing to launch an AI customer service assistant. Before it could ship, we needed to understand exactly what kinds of situations it would face in the real world. That included rare events that could be risky for AI to interact with, and involved examining their taxonomy and prevalence to create the datasets that would help us build a more responsible product.

The investigative work to do this was rigorous, but the process was deeply artisanal. Months of high-touch iteration went into each investigation, from finding the right data, reviewing samples with experts, and generating representative datasets, and the method was manually curated across notebooks, tables, docs, and individual judgment.

This was fine for one investigation — but as we carried the same investigation into new languages, new geographies, and new LLM-based products at a near-weekly cadence, the workload outgrew the process. To bring the same rigor, thought, and quality at this new pace, and involve more people, we needed to make each investigation less bespoke. In short, we needed a way to replicate the methodology itself.

Insight Miner: Agent harness for unstructured text understanding

Insight Miner starts with the engineering (the queries, the scaled labeling and embedding, the clustering and tracking), so that no one has to learn new infrastructure to run analysis at scale. The core method is well established: extract, embed, cluster. Around that core we assembled the methods we had come to rely on, drawn from internal investigations and industry research: prompt tuning, hard-example mining, contrastive labeling, and how to start with unsupervised exploration then mature into classification. These pieces used to live in separate notebooks, manually iterated and shared by copy and paste. Now they are in one shared package, which gets updated whenever an individual investigation teaches us a better technique.

An investigation begins with a research question in a chat session, with an agent that is at once research partner, executor, and expert in the methods the harness holds. Insight Miner isn’t tied to a single dataset, domain, or question: it runs over any unstructured text source, through whatever lens the question needs, with the same rigor behind every investigation.

As we expanded our community support AI assistant to new languages and countries, we used Insight Miner to carry out investigations that used to take months in just a matter of days. But its bigger impact was ensuring that rigor and speed both increased, instead of trading off.

With this harness, data scientists were able to shift their focus from executing analyses to improving the techniques used for each step of the process. Because the mechanical parts of investigations can scale and parallelize, we are able to spend more time on the careful parts of an investigation: directly inspecting the most ambiguous or strategic data that helps us deeply understand the product, testing hypotheses and groupings, and building a robust qualitative and quantitative understanding of our data and products. Rather than automating analysis, we’re increasing the amount of human judgment in the most strategic parts of it.

Going beyond technical teams

Insight Miner was initially designed as a tool for data science teams. However, it quickly evolved beyond that: a year in, dozens of teams are using it for hundreds of types of investigations. In fact, it has more users outside of technical roles than within them, with a particularly heavy representation among operations and product-insights teams. A UI to make data exploration and agent conversations more accessible further increases expert participation.

Subject matter experts who have never written a line of code have been able to directly conduct scaled analyses rather than waiting on scarce eng or DS resourcing. Projects that were previously unresourced or were informed by the manual review of hundreds of examples can instead use our shared best practices and work across several orders of magnitude more data. Use cases span coding open ended survey answers, evaluating model performance, understanding fraud patterns, and many more.

A new category of infrastructure

Harnesses like Insight Miner are full-stack systems, a new type of infrastructure that must be developed, maintained, evaluated, and continually improved. This type of work is a natural fit for other agentic systems. For Insight Miner, separate agentic systems help us update instructions for new model releases, fold in new best practices, review live use to identify pain points, and watch for, reproduce, and propose fixes for new bugs. These systems form a larger agentic environment reshaping our day to day work in domains well beyond just data science.

A harness can be useful wherever experts carry a methodology worth encoding: legal review, policy analysis, any field where the method is as much the product as the answer. Such systems are critical to adopting AI-first knowledge work. Our CTO has written that as models commoditize, what endures is proprietary data, deep workflow integration, and above all taste. A harness is where all three accumulate. Expert taste becomes the method every team runs, the workflows deepen with every adopter, and the feedback loops are built in: every run can leave the harness improved for everyone.

If this type of work interests you, check out some of our related positions!

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

The Forecasting Data Science team at Airbnb produces many of the forecasts the rest of the company plans around: demand, bookings, cancellations, and a range of finer cuts by market and segment, refreshed continuously across thousands of markets. The targets differ, and the models differ, but they have one thing in common: Other teams build on top of them.

This means a forecast that is casually wrong is not a clean miss, as it might be in an academic setting. That’s because a small bias does not stay small once a lot of decisions are riding on it. So when one of those forecasts starts to drift, what to do about it is not really a methods question. It is a risk question, and an easy one to get wrong, which we have from time to time.

One of these forecasts had been missing, compared to what actually happened after the forecast was released, in the same direction for a couple of quarters. This bias persisted after several routine refreshes. The usual solution would be to fully retrain the model: pull in the recent data, refit the model again, and ship. But we wanted to understand the source of the bias, rather than simply hoping an update would eliminate it.

If you’re interested in other posts on this topic, you can learn more about how COVID impacted Airbnb’s financial models or how we dealt with disruption to our models during the pandemic. This post is about the discipline that came out of both: how we now decide whether a struggling forecast needs new data, a new model, or no changes at all.

One word, three decisions

The easy mistake is treating the choice to “retrain” a model as a single action. It is three separate actions — refitting, respecifying, or holding — and none of them is particularly similar to the others.

Refitting is the cheaper option. Same model, same structure, same features, updated with newer data. This is what most people mean by “retrain.” For the ordinary drift that builds up in a model as the world moves on, it’s usually the right choice.

Saying that this is the cheaper option is not the same as saying that it’s cheap. A refit still has to be validated and shipped. The refitting process can degrade a model that was actually fine if the recent data happens to be unusual. And, given the importance of the production forecast, every refresh is a small risk you are choosing to take. It is the least expensive of the three options, but not a free one. None of this is an argument against refitting. On most cycles it is the right move, and if we had to run one of the three blind, it would be this one. The claim is narrower. A refit is a decision with a price, and pricing it is what lets you notice the cycles where a different option was worth more.

Respecifying is a different animal. You change the model itself: add a feature, drop one, change the structure, the priors, the likelihood. On the production forecast, which so many of our teams plan around, this is a real commitment; you are replacing something you understand, and have watched the behavior of for years, with something partly new that you have not. It is also where almost all the actual improvement lives. Refitting keeps a good model current. Respecifying is how a model that is in some way wrong in its operations is made right.

Holding is the one nobody likes. You look at the miss, decide it does not warrant action, and leave the model alone. This takes the most nerve, because to anyone watching the forecast it looks like you are ignoring a problem. On a forecast that people are relying on, “We decided to do nothing” is a hard sentence to say out loud. It is also, more often than you would think, the correct call.

What we have learned from years of running these models is that the trouble comes from filing all three under one verb. “We should retrain” gets said when any of the three is the actual answer, and the word drags everyone toward the one that feels cheapest, because it is the one with a name.

Most of the time, though, nobody says it at all. Retraining is rarely a decision someone makes because a model is drifting. Retraining runs on a schedule, monthly or quarterly, by convention, because a standing cadence is one less thing to think about. Which means the choice among the three is hardly ever made on purpose. The cadence makes the choice in advance, every cycle, and the cadence always picks refit.

Three ways to hold on too long

Each of the options contained in the term “retrain” — refit, respecify, or hold — has a matching failure, and underneath each sits the same mistake: a model holding onto a shock, or a surprise, after the cause of the shock or surprise is no longer relevant to forecasts. That mistake can grab a blip and treat it as the new normal; keep a crisis assumption after the crisis is over; or build the shock so far into a new model that the model can never move past the shock.

Chasing noise. A forecast misses for a quarter, the residuals look alarming, and someone asks whether we should update the model now rather than wait for the next scheduled run. That impulse is the failure, and it is a specific one. What triggers an off-cycle refit is always a surprise, which means the window you are rushing to absorb is the window you understand least.

A couple of years ago one of our markets ran hot for a quarter against the published forecast, because a large event landed on the calendar and pulled a wave of bookings forward. We refit early. The model took the unusual quarter as the new level, the next two forecasts came in high, and it settled only once the event aged out of the training window. Waiting for the cadence would have meant fitting that spike alongside the quarters that came after it, rather than as the last thing the model saw, which is a much weaker pull.

Carrying ghosts. The quieter error, and the opposite mistake. A structural assumption that was true during a shock stays switched on long after the shock has passed, hidden behind a run of refits that all look like routine maintenance. We have had one of these hide in plain sight. During the worst of the COVID disruption, cancellation timing shifted in a way it never previously had, and the model learned from it. Market behavior came back to normal well before the assumption did, and since every refit looked clean, the stale piece sat there until the forecast had leaned the same way for long enough that someone went looking, found the lean baked into the model, and respecified the model to wring out the now-faulty assumption.

Respec-as-panic. The failure of overcorrecting. Something genuinely moves — a foreign exchange rate (FX) swing from a major source market, a regional disruption that reroutes demand — and the reflex is to rebuild. A fresh model, new structure, stood up under deadline.

We have caught ourselves reaching for this solution. An FX move on a major source market threw one of our forecasts off for a couple of months, and the first instinct was to stand up a new model around the new regime. We widened the priors instead, the existing model rode it out, and we avoided a move to a more fragile model, tuned in response to a shock that was already on its way out. (Figure 3)

How we decide now

The rule we now hold to is easy to state and hard to follow. On a standing cadence, the question of whether to retrain mostly answers itself. So the question that matters is which of the three options a given miss calls for: refit, respecify, or hold? And the right answers turns on what changed.

If the process generating the data is still the one the model assumes, and the parameters have just drifted, refit. The structure is fine, it just needs current numbers.

If the process has changed in a way the model cannot represent no matter how you estimate it, respecify, because fresh data cannot help a model being asked to describe a world it has no language for. The tell is direction: a misspecified model misses the same way over and over.

A respecify can cut both ways. It can add structure the model was missing, the way borrowing across geography did after 2020, or it can take structure out, retiring an assumption the world has outgrown.

The second kind is the forgetting, and it is the one teams skip, because adding feels like progress and removing feels like giving something up. But it’s an important tool to remember, and to use, whenever needed.

And if the miss sits inside the range the model already calls normal, hold. In practice this usually means declining to refit off-cycle, on the theory that a surprise you cannot yet explain is the worst possible reason to move a model early. The cadence will get to it. Skipping a scheduled refit is the rarer and more expensive version, since stale parameters have a price of their own, and it is worth it only when you can name the thing in the window that you do not want the model to absorb.

There is a more formal version of this that we have written up separately [1][2]. The one-line version is that updating is not free: moving a model toward new data costs something, staying put costs something, and the call is a comparison of the two, at the level of parameters and again at the level of structure.

In production, running real forecasts, the math is not what we reach for. We reach for the three questions above, in order.

The time we got it right

The clearest case we have of making this call correctly is the one from our second post on the Covid era, seen through this lens.

Before 2020, our destination-market forecasts leaned on a hierarchy. Markets with long, stable histories anchored the estimates, and other, “thinner” markets borrowed strength from them, on the assumption that a destination behaves like comparable destinations elsewhere. The borrowing was the structure, and for years it was true enough to be both invisible and unquestioned.

COVID broke the assumption, not the parameters. The shock was larger than anything we had previously experienced. Recovery did not arrive everywhere at once, or in the same shape. Some markets came back fast, while others stayed flat for quarters. Markets s that used to move together were suddenly on different paths. As a result, the “borrowing” of one market’s stability, by other markets that had previously tracked it, which had previously stabilized the forecasts, was now contaminating them. Markets were being pulled toward a kind of average that no longer described any of them.

The instinct was to refit our models with new data, and we tried that. It failed in a way that turned out to be the whole lesson: the estimates did not just shift, they went unstable, swinging quarter to quarter as the model tried to reconcile markets that no longer belonged in the same pool. Over the recovery window, the refit-only error ran about three times the pre-shock baseline, and it would not settle as new data came in.

That instability was the signal. A parameter problem gets noisier at the edges, but holds its shape; this was the shape itself coming apart, which is what a structural problem looks like.

The fix was a respecification, which is the subject of the previous post, so in one line: we changed what the model borrows across. Instead of pooling by a fixed hierarchy, the prior borrows along geographic adjacency and shared recovery dynamics, so a market draws strength from places actually behaving like it today, rather than places it used to resemble before the crisis. On a held-out recovery window, the respecification cut error by a little over half against refitting, and brought it back to within a couple of points of the pre-shock baseline.

The point for this post is that our new framework, which gives a prominent role to holding, would have told us not to refit. That would have prevented the failed refit that generated so many errors.

The miss was directional, not noisy, so it was not a hold. Fresh data made it worse, not better, which is the signature of a problem in structure and not in parameters, which indicated that the remedy was not a refit. What was left was to respecify.

We got there by trial, and errors, that time. The reason to write the decision down is to get there without the trial the next time, on a forecast where the trial costs a quarter of accuracy that other teams feel downstream.

Why this is hard on forecasts that carry weight

None of this is hard to understand. It is hard to do, and it is hardest precisely on the forecasts that matter most, for three reasons that we keep running into.

Holding looks like negligence. When a forecast a lot of people watch is missing, “We are choosing not to act” is an unpopular thing to say, and the pressure is always toward visible activity. Instead, when we hold, we use the modern, Buddhist-adjacent maxim: “Don’t just do something; sit there.” Refitting, on the other hand, is visible activity. It photographs well even when it is wrong.

Respecifying is expensive, and a little frightening. Replacing the structure of a model that has produced reliable numbers for years means giving up something you understand for something you do not understand yet, and the cost of being wrong is not abstract. That fear is healthy, and it is also why teams under-respecify and let ghosts accumulate.

And the last few years trained the wrong reflex. Forecasting through COVID meant changing models under real pressure, often correctly. Coming out the other side of the pandemic, a lot of teams kept reaching for the rebuild when a refit would do, while leaving pandemic-era assumptions switched on because nobody wanted to touch a model that had survived.

The same period produced both the panic and the ghosts. The discipline now is not how to react fast. It is how to tell the three options apart again, with time to think, on models where being repeatedly wrong is expensive.

Learning to forget

Refitting keeps the model current. It does not keep it honest. What goes stale is not the numbers but the assumptions underneath them, and those are exactly what a refit leaves alone.

A shock makes you add structure to cope: an elevated risk, a wider band, a special case for a world that has stopped behaving. Taking that structure back down once the shock has passed, before it quietly biases everything downstream, is harder than any retrain. It rarely has a deadline. Nobody schedules it.

A good forecasting team doesn’t just learn from shocks. It learns to forget them, when needed, on purpose.

If this type of work interests you, check out some of our related positions.

Acknowledgments

Thanks to Liz Medina, Jess Needleman, Linhan Liu, and Flavio Stanchi for building and improving the forecasting systems described here, and to Peter Coles, Yuanyuan Cui, and Adam Liss for their support of this work and its publication.

Harrison Katz leads Finance Data Science & Strategy at Airbnb. His research focuses on Bayesian methods for compositional and hierarchical time series, Bayesian decision theory, & forecast governance.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Our authentication system had grown organically over a decade, adding new login methods like Social Login, Email OTP, and Phone over time. We recognized that our users have diverse needs: many, such as hosts, use Airbnb daily, while others visit less frequently, when they are planning trips or traveling. To provide a seamless experience, we needed to ensure we were doing a great job supporting all of these usage patterns, helping users pick up exactly where they left off, regardless of how much time had passed or which device they were using.

In this blog post, we’ll walk through how we rebuilt Airbnb’s authentication flows using a new paradigm we call Flexible Authentication, and how each architectural decision was guided by insights into the user’s experience.

Identify first then Challenge

The ‘Identify first then Challenge’ model was the product insight that reframed the project. Our old system treated authentication as a single question: can this person prove who they are? The real question is more nuanced: given the multiple aspects of the user session, what’s the easiest way for them to verify it?

A traveler in Brazil who registered with a phone number is better served by a WhatsApp one-time password (OTP) than by SMS, since WhatsApp penetration far exceeds SMS in this market. A returning host in South Korea is better served by their Naver login, the leading South Korean identity provider, than by their Google ID. The right challenge depends on the person and context.

Based on this insight, we adopted an Identify first then Challenge model that splits the flow into two distinct stages. First, the person tells us who they are, using whatever method they want. This may be an email address, phone number, or social login. Once we know which account they are trying to log in to, a configurable policy engine picks the challenge that is most likely to succeed. We lead with that challenge and offer all other options as fallbacks.

The policy engine can use all available aspects of the user session and the account they are logging into, including historical information, to pick the challenge most likely to succeed. We have only started to scratch the surface of optimizing this policy engine.

The key architectural decision is that the client never decides which challenge to present. The server makes that call, and the client renders whatever screen it receives. This separation lets us tune authentication strategy per-region without shipping new client code across the many devices Airbnb guests and hosts use. This has made experimentation much easier.

Try another way

Our old system had a frustrating failure mode: if you couldn’t complete the challenge presented to you, you were stuck. Forgot your password? You could reset it, but the reset flow was its own journey with its own drop-off points. Couldn’t receive an SMS? Tough luck, go back and enter your email so you can try password login instead.

The product team articulated a principle that became a hard engineering requirement: every authentication screen must offer an escape. No dead ends. If someone can’t complete the primary challenge, they should always see a “try another way” option with all other options they can use to log in.

We built this as the Challenge Picker, a server-driven component that accompanies every challenge screen. The server returns not just the primary challenge but a ranked list of alternatives, ordered by predicted success rate for that person’s context. The ranking considers what the person has successfully used before, what methods are registered on their account, and what’s available on their platform. When someone taps “Try another way,” the flow doesn’t restart. It adapts.

Moving faster than our client release cycles

This insight came from a pattern we kept seeing: we’d identify a potential authentication improvement, build it, then wait weeks for app review and rollout before we’re able to even start an experiment. For a flow as critical as login, that feedback cycle was unacceptable.

The solution was to make the authentication experience fully server-driven, with the screen as the fundamental unit of abstraction. Each step (identifier input, challenge, account picker, or error recovery) is a screen defined entirely by server response. The client is only required to serve as a thin renderer that knows how to display a set of screen types and send actions back, with no opinion about sequence, copy, or flow logic.

Because our screens are defined by a server-side schema, our clients (Web, iOS, and Android) can use automatically generated type definitions. This setup ensures engineers can catch mismatches between client code and server requirements immediately during development.

Results

Our former approach had much of the authentication logic on the client. Airbnb supports nearly every client type that there is, so this approach presented many challenges, including anticipating various challenges and reproducing bugs reported across many different kinds of clients. So keeping authentication logic on the server was a big change for us, and it had many beneficial effects.

As we implemented this change, the first thing we noticed was how much code disappeared. By moving screen sequencing and flow logic to the server, our client state management became much simpler. Our transition to a server-driven model allowed us to not only remove client-side logic but also streamline design systems and server-side localization. Together, these initiatives drove a 60% reduction in code and a 100KB reduction in the web client bundle. A smaller bundle is its own win for people on slow connections and constrained devices, but the leaner client mattered most for what it let us do next.

That leanness bought velocity. When the rules that govern a flow live behind a server response rather than inside a shipped binary, changing them no longer requires on-app review and staggered rollout. In the three months since launch we’ve run 20+ experiments against the new flows, and for most that did not require client changes, the turnaround between an idea and a measured result collapsed from weeks to days. We achieved considerable success with the initial launch, and we have been able to iterate quickly on new ideas and changes since them.

The numbers we care about most here are the ones people feel. Time to login has dropped significantly as the flow now leads with the challenge most likely to succeed, instead of one we’d hard-coded in advance. With fewer dead ends, and with a fallback always within reach, the share of visitors who successfully authenticated has risen by 2.6%, from an already high base, which is an improvement that affects millions of sessions. Outages that prevent one challenge from working no longer block users, because they can easily find alternatives.

Because more returning people find their way back into their existing account, the rate of duplicate accounts has fallen by 27%, which keeps trip history and reservations intact across sessions. And as more of those journeys are resolved through challenges a person can actually complete, we send fewer one-time SMS codes, cutting OTP costs by roughly 11%. None of these changes happened in isolation; a faster, more forgiving flow is the same flow that reaches more people, fragments fewer accounts, and spends less in doing so.

Conclusion

There’s a constant flow of new devices and potential new ways to authenticate, and we are always looking for ways to make authentication easier. So Flexible Authentication is a framework for continuous product improvement, not a finished product.

The broader lesson for teams building authentication at global scale is that product insights and engineering architecture aren’t separate concerns. Identify-then-Challenge, the Challenge Picker, and server-driven screens are each direct expressions of our beliefs about how people behave when they’re trying to log in. The architecture’s flexibility, namely its ability to adapt per-region, per-experiment, and eventually per-person, comes from giving the server authority over decisions the client used to make. That pattern generalizes: for any flow where context determines the right experience, moving the decision boundary off the client unlocks the iteration speed needed to act on what you learn.

Interested in working on systems that serve hundreds of millions of people worldwide? Check out our open roles.

Acknowledgments

Flexible Authentication was a cross-functional effort spanning engineering, product, design, content, and data science. We’d like to thank the Identity and Authentication team for building and shipping the system, the product and content design teams for the principles that shaped the architecture, and the data science team for building the measurement infrastructure that lets us keep improving.

This project was a collective effort, shaped by the ideas, care, and hard work of several people. Thank you to everyone who helped bring it to life: Susan Stevens, Husayn Abdul Hakeem, Katie Ta, Nathanael Ji, Miyuan Zhao, Jordan Smith, Antonio Fuentes, Gloria Yang, Tan Tho Le, Kush Baldha, Evan Krasts, Jacob Parsons, Xiao Li, Tatu Sacheri, Jack Bao, James Zhan, Brandon McQuarie, Renal Khabibulin, Alice Lew, Alan Yao, Jing Liu, Chloe Fan, Beth Soucy, Vicki Siolos, Emily Kellner, Gabiella Lalli Martins, Christian De La Paz, Jill Peloquin, Sai Vinay, June Woo Suk, Sindhu Ravichandran, Richa Khandelwal, Arjun Kumar, and Pat Moynahan.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Generative AI breaks a lot of the assumptions that used to hold true for software testing. Unlike traditional software, LLM outputs are non-deterministic, and “correct” is subjective. Because so much judgment is involved, you often need an AI to evaluate an AI, which introduces its own potential failure modes. Making matters more complicated, a single interaction with an LLM can chain retrieval, reasoning, tool calls, and generation, each of which can fail independently.

At Airbnb, we build LLM-powered features across our product, with recent launches including review highlights, AI customer support, smart communication features for guests and hosts, and more. Behind the scenes, we also use AI to help us spot trends and understand what’s working, guiding where we improve the product next.

Each product team may have its own evaluation criteria, process, workflows, etc. However, these are built on top of some common foundations and principles. An infrastructure team provides tooling and best practices, incorporating learnings across domains so that they are shared with everyone building products at Airbnb.

In this article, we wanted to share some of these best practices and learnings with the broader engineering community. Please note that the recommendations here are not intended to be prescriptive; there is no one-size-fits all approach when it comes to running evals.

1. Foundation

Evaluating LLM-based systems is challenging work, and this should be planned for at the outset. Without a deliberate strategy, three things tend to happen:

• False confidence: A generic “helpfulness” metric scores well, you ship, but it didn’t capture the failure mode people actually hit.
• Undetected regressions: A prompt change subtly degrades a dimension you weren’t measuring.
• Wasted effort: You build a scaled eval pipeline for metrics that don’t correlate with outcomes.

Expect to spend a meaningful share of your total project effort on evaluation. This is not unnecessary overhead, it’s how you build products that actually work.

1.1 The one rule

When in doubt, look at your data. Manually reviewing your data and building an intuition for what counts as success is always the starting point we recommend to teams. Build your prototype, and run it through 100 examples (synthetic is fine). Then read the outputs. Read the traces and find the model’s mistakes. Categorize them and build an eval.

This single habit will do more for your product quality than any framework, tool, or methodology in this document.

1.2 Eval-driven development

Formalized, that habit becomes eval-driven development (EDD), the GenAI analogue of test-driven development. Rather than predicting every failure upfront, EDD builds the infrastructure and habits to discover, encode, and continuously test for failure modes as they appear. It also forces stakeholders to externalize what “good” means, which shapes the product roadmap.

Five principles anchor EDD:

1. Define goals and gates upfront. What are you optimizing for? What must be true before you ship? These answers may not be clear right away; you might discover them as part of your data exploration.
2. Let real errors guide your metrics. Co-develop them with cross-functional partners based on observed failures. Don’t invent them in a vacuum.
3. Keep your evaluator set small and sharp. 3–5 well-calibrated LLM-as-judge evaluators beat 20–30 noisy ones. Each should target one specific correctness dimension.
4. Appoint a decision-maker. While what constitutes correctness should be a team discussion, people will sometimes disagree. Include a final (human) decision-maker who makes the ultimate call on what constitutes good vs. bad system behavior.
5. Collaborate continuously. Have your product partner regularly answer: “Is X better or worse than Y?” and “What’s actually wrong with this output?”

2. The three evaluation methods

Every evaluation you run will use one or a combination of these three methods.

Layer 1: Programmatic checks (fast, low resource — catches obvious failures)

                         ↓

Layer 2: LLM-as-a-Judge (nuanced - catches quality issues)

                         ↓

Layer 3: Human evaluation (high resource - validates edge cases, 
 calibrates the stack)

2.1 Programmatic & heuristic metrics

Deterministic, code-based checks that don’t require an LLM call should be your first filter, catching obvious failures before you send anything to a judge or human labeler.

✅ Do: Use structured outputs (JSON schemas) to ensure strict typing.

❌ Don’t: Rely on prompt instructions alone to format data. This breaks downstream data pipelines.

2.2 LLM-as-judge (Virtual judges)

Use a stronger LLM to evaluate another LLM’s output against a carefully designed rubric. This is how you assess nuanced qualities e.g. tone, coherence, faithfulness, relevance, at a fraction of the resources needed for human evaluation.

Rubric design matters. Ambiguity is the enemy. Something like “Is the provided explanation readable and up to our standards?” isn’t likely to be effective — if a human can’t apply the rubric consistently, an LLM certainly can’t.

Here is a simplified example of a single virtual judge’s rubric:

Score the readability of listing explanations. A good explanation sounds
like a friendly travel agent: warm but professional, 
simple, natural, grammatically complete.

Score 1 if it reads cleanly. 
Score 0 if it has ANY of these problems:

- Tone: too formal/jargony, too casual
("awesome vibes"), too salesy ("amazing!"), or robotic.

- Internal terms: never use internal terminology.

- Formatting: no quotation marks, no bullets, no fragments. End every
explanation with a period - never "!" or "?".

- Grammar: use articles/determiners/prepositions for natural flow
("this home has a pool", "close to downtown"). In a series, use the
article once then drop it: "a backyard, grill, and kitchen" - not
repeated, not omitted entirely.

- Complexity: plain words over jargon ("pool" not "aquatic recreation
area"; "near" not "proximate").

Examples:
- "Host mentions a pool and hot tub available near downtown." → 1
- "The listing mentions a pool!" → 0 (internal term "listing"; ends in "!")
- "This domicile encompasses aquatic amenities." → 0 (complex words; jargon)

Return ONLY:
{
"reason": "<list of [error_type, explanation] tuples as a string, or []>",
"score": <1 or 0>
}

2.2.1 Calibration: Making your virtual judge trustworthy

A virtual judge that hasn’t been calibrated is worse than no judge at all, because it gives you false confidence. Here are the calibration steps we recommend:

1. Create a golden dataset of 50–100 examples. This MUST include bad examples (not just good ones).
2. Run your virtual judge against the golden set.
3. Measure agreement. Target percentages in the high 80s-90s. Possible options to measure disagreement are Cohen’s kappa or Krippendorff’s alpha. (Perfect agreement isn’t achievable — even humans disagree.)
4. Analyze disagreements. Refine the prompt and update your few-shot examples. Then re-run the loop until you hit the target agreement.
5. Recalibrate periodically as failure modes evolve.

2.3 Human evaluation

Human judgment remains the gold standard for ground truth, high-stakes domains, and resolving disagreements between automated evaluators.

2.4 Evaluation scenarios and recommended methods

Overall, the rule of thumb is to start with 20–100 rows labeled by subject-matter experts. Move to a scaled annotation workforce only when the rubric is rock-solid and volume is the bottleneck.

And if your experts disagree on a label, stop. Solve human disagreement before automating anything.

3. Evaluating agentic systems

Agentic systems involve multi-step reasoning, tool calling, branching logic and intermediate state transitions. Evaluating only the final output is insufficient: a correct final answer can mask a broken reasoning path, wrong tool parameters, or an inefficient trajectory.

Therefore, you will need to evaluate across three layers:

To achieve this, you can take advantage of the fact that an agent generally pushes traces and spans under an application root. This contains information about the type of agent, the sub agent if invoked, input/output of the agent, tools invoked if any, and more. These traces can be written out to an observability platform or persistent storage.

Then, you can use DFS or another type of tree traversal to reconstruct the trace in memory. This lets you ensure certain subagents were invoked at the right time, the agent called the right tools, etc. And you can scope your evaluation to specific agents/subagents.

4. A practical walkthrough

Here’s what the full process looks like end-to-end, using a fictionalized and simplified version of a real use case.

Scenario: You’re building an AI assistant that answers questions about a travel platform’s support policies.

Step 1: Explore & discover. Run 100 inputs through your prototype and read every output. You find:15 responses generated policy details not in the source documents (faithfulness issue); 8 correct but too verbose (conciseness); 5 refused valid questions (over-refusal); 3 had broken JSON (format).

Step 2: Build evals. Add programmatic checks for JSON validity and length bounds. Write a virtual judge for faithfulness (separate prompt, different model, chain-of-thought) and another for conciseness. Have your PM or subject matter expert label 60 examples, including failures, as a golden set.

Step 3: Calibrate & iterate. Your faithfulness virtual judge agrees with the PM 78% of the time. Not good enough. Analysis reveals the judge is penalizing accurate paraphrases as “unfaithful.” Update the rubric and add few-shot examples. Agreement jumps to 88%. Improve the retrieval step; faithfulness failures drop significantly.

NOTE: Here, we find that when iterating on models and prompts, it’s best to fix one variable at a time. First fix the model and vary the prompt, then fix the prompt and vary the model, then fix both and vary the serving configuration. At each stage, virtual judge results narrow the candidate pool. Then, you can improve the virtual judge(s) using samples from the top candidates. The evaluators and the candidates sharpen each other until both stabilize.

Step 4: Scale & monitor. Scale evaluation across 5,000 examples. Set up production monitoring: sample 5% of live de-identified traffic daily, run programmatic checks + virtual judges, and surface flagged outputs for human review. A weekly PM review closes the loop, with new failure modes introducing new evals and subsequent system improvements.

NOTE: We sample live traffic continuously using privacy-preserving techniques. All data undergoes robust de-identification prior to human review, and usage is strictly purpose-limited to safety and quality assurance, aligning with Airbnb Privacy Principles.

Key takeaways

1. Look at your data. Read outputs and traces before building anything else.
2. Avoid generic metrics. Build evaluators for your product’s real failure modes.
3. Start with 50–100 rows. Fail fast, iterate cheaply.
4. One evaluator per dimension. No “God evaluators.”
5. Calibrate to high 80s-90s% agreement before trusting your Virtual Judge at scale.
6. Use all three methods. Programmatic, Virtual Judge, and human as layered defenses.
7. Include bad examples in your Gold Set. You can’t test discernment without them.
8. Evaluate the system, not just the model. Test retrieval, tool calls, the full pipeline. For agents, evaluate the trajectory, not only the final answer.
9. Mirror evals in production. Pre-production metrics are not one-and-done.
10. Evaluation is a team sport. Evaluation is about shaping what product success looks like, and that takes contributions from many people. The teams that succeed with AI aren’t the ones with the best models, they’re the ones with the best communication and clearest product vision.

If this type of work interests you, check out some of our related positions!

Acknowledgments

We would like to thank Tania Myronivska, Haozhen Ding, and Sebastian Wickenburg for their thoughtful feedback and contributions to this guide, Jisheng Liang and John Hewson for their guidance and insights, and Min Yi and Yi Li for their constant support.

We would also like to thank Evelyn Xu for their support in authoring this post during their time at Airbnb.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Planning a trip on Airbnb rarely happens in a single session. A guest searching for a place to stay in San Francisco might browse dozens of listings over several days, leaving behind a trail of views. Typically, over a period of years, that same guest will have accumulated many previous bookings, reviews, and the occasional cancellation. Taken together, these events reveal a great deal about what that guest values in a stay. For years, Airbnb’s search ranking captured this through hand-crafted features: aggregated statistics such as total past bookings or average listing price. These worked well, but as the feature count grew into the hundreds, the approach became harder to scale and increasingly limited in expressiveness. In this blog post, we describe how we built a sequence modeling system that encodes the full guest journey using a Transformer, learning richer representations of guest preferences to deliver more personalized search results.

Challenges

Event sequences per guest present three core challenges. First, they are dominated by listing views, which account for the vast majority of all events — some guests accumulate hundreds of thousands of them — making raw sequences computationally intractable to model directly.

Second, unlike social media platforms, which optimize for engagement, Airbnb optimizes for booking conversion. Bookings are rare, compared to events, and deliberate, whereas a listing view could reflect genuine intent or simply idle browsing. Building a model that generalizes from these sparse, noisy signals to surface the listings most relevant to a guest is fundamentally different from social media engagement modeling. Third, training a sequence model on hundreds of millions of search-label pairs is expensive; targeted optimizations are needed to reduce costs and improve the throughput of model training.

Solution

Designing the guest sequence

How to tackle this challenge, which is also a rare opportunity to use Transformer technology to improve guest experience?

After research into similar use cases, we came up with a plan of action. Rather than modeling all events uniformly, we split the guest sequence into two parts:

• The long-term sequence captures infrequent but informative events from the past seven years: bookings, reviews, cancellations, and other past interactions. We cap this sequence at 80 events.
• The short-term sequence captures listing views from the past 21 days. We cap the short-term sequence at 200 events.

The 80-event and 200-event capping thresholds are set so that only the longest 2% of sequences are truncated. Together, the two sequences give the model both depth from a guest’s booking history and the immediacy of their most recent browsing activity. Events are encoded using a shared feature pool with a unified embedding table [1] for high-cardinality IDs such as listing and host identifiers, along with hierarchical geographic IDs.

Making training & serving efficient

We’ve implemented three strategies that, together, deliver roughly a 4x improvement in training throughput. The most impactful is batching of searches. Consider a guest who performs three searches — at T=4, T=7, and T=10 — interspersed with events at other timestamps. Because our encoder uses a causal mask, it processes the sequence cumulatively. The embedding produced at position T=3 captures everything the guest has done up to that point, making it the correct sequence representation for the search at T=4. Similarly, no events are ignored between searches: the embedding at T=6 captures the entire sequence from T=1 through T=6 to serve the search at T=7, and the embedding at T=9 captures the full history up to T=9 to serve the search at T=10. Rather than running the encoder three separate times from scratch, we run it once over the full event sequence and route each search to its corresponding intermediate embedding, allowing three searches to share a single encoder forward pass.

The other two strategies reduced padding waste. (Data is processed in fixed chunks, and empty space in actual data is filled with zeros, referred to as “padding.”) To minimize padding within batches, we bucketize sequences by length. And to eliminate padded searches from the ranking model, we use sparse calculation of searches.

The serving design reflects the same efficiency principle. Running a multi-layer Transformer encoder over a full guest sequence — spanning up to seven years of events and 21 days of views — at query time would be expensive. Instead, we decouple the two stages of inference. The sequence encoder runs as a daily batch job, processing event history and storing the resulting embeddings for guests who have generated new events. When a guest performs a search, the ranking model retrieves their stored embedding in real time and combines it with the live search query, then uses the information to produce scores for the retrieved listing candidates. This split keeps latency low at serving time, while still incorporating the full depth of a guest’s history into every ranking decision.

Three milestones

Before rolling out any changes, we rigorously evaluate them using A/B testing, where guests are randomly but consistently assigned to either our current model or a new model. These tests typically run for three weeks, during which we closely monitor key business metrics and “guardrail” metrics to ensure there are no unintended negative side effects. A new model is only launched if there is strong, statistically proven evidence that it improves the guest experience without failing any safety checks.

Following this standard, we rolled out the system in three stages.

• First, we tested the system with the long-term sequence alone, establishing that sequence-learned guest representations could meaningfully improve our existing ranking system.
• Second, we added the short-term view sequence, which introduced data scaling challenges that are addressed by the efficiency strategies above.
• Third, we introduced a setwise ranker [2, 3], co-trained with the sequence encoder. Unlike a pointwise ranker that scores each listing independently, the setwise ranker considers a set of candidates together, allowing it to reason about relative differences across listings. Combined with the rich guest representations from the sequence encoder, it can make more personalized ranking decisions by jointly understanding the guest’s travel history and how the retrieved listings compare to each other.

Results

Offline, the long-term sequence alone improved Normalized Discounted Cumulative Gain (NDCG) of booking labels by +0.44% over our existing system. Adding the short-term view sequence brought the combined improvement to +1.48%. The setwise ranker added a further +2.3% improvement in offline ranking quality. In total, the improvement added up to 3.78%. At Airbnb, where the ranking model has been refined over more than a decade, a 0.3% improvement is considered significant. This substantial improvement means our model can rank the listings a guest likes higher, which may potentially help guests find what they want more easily.

Online A/B tests confirmed the gains. With the long-term sequence alone: +0.31% uncanceled bookers (guests who have booked), +0.38% views. Adding the short-term sequence: +0.55% uncanceled bookers, +0.82% uncanceled nights, +0.90% views — all statistically significant. The rise in views suggests that the model is effectively surfacing listings that capture guests’ interests, which directly translates into the higher booking conversions we observed. The setwise ranker delivered a further +0.28% uncanceled bookings and +0.32% booking requesters.

We also applied the system to promotional email listing ranking. Using the same sequence embeddings, without any architecture changes, produced +0.16% uncanceled bookers, +0.23% uncanceled nights, and +5.04% email clicks — demonstrating that the guest representations generalize beyond search.

Conclusion

What we built learns directly from the full arc of a guest’s relationship with Airbnb; their past trips, recent browsing activity, and everything in between. Moving from hand-crafted feature aggregation to learned sequence representations gives our ranking system access to a kind of guest understanding that was previously out of reach: not just what a guest has done across their history at Airbnb, but what those actions suggest about what they are looking for right now. The results across search ranking and promotional emails, which are two independently iterated surfaces, validate that this signal is both powerful and general.

The work is far from finished. We are exploring near-real-time sequence embedding updates, richer event types such as wishlists and map interactions, target-aware ranking, generative recommender, and continued improvements to setwise ranking. Each of these directions builds on the same foundation: the more faithfully we can represent the guest journey, the better we can connect guests with homes they will love. For a deeper technical treatment, see our KDD ’26 paper JourneyFormer: Encoding Airbnb guest journey with sequence modeling [4].

Interested in learning more about our technical journey? Browse our previous publications to see how our systems have evolved. If tackling these kinds of challenges excites you, explore our open roles.

Acknowledgments

We would like to especially thank the following people for their great collaboration and for continuing to advance sequence modeling at Airbnb (listed alphabetically): Ashish Jain, Gil Forsher, Hao Li, Haozhen Ding, Jiawei Yao, Kedar Bellare, Linyun He, Mingyang Xu, Pallavi Adusumilli, Pengyu Hou, Shashank Dabriwal, Sid Reddy, Sophie Wang, Tanya Piplani, Vijay Velagapudi, Yangbo Zhu, Yan Zhang, Yi Li, Yiwei Wang, Yuli Han, and Zhiwei Wang.

References

[1] Coleman, Benjamin, et al. “Unified embedding: Battle-tested feature representations for web-scale ML systems.” Advances in Neural Information Processing Systems, 2023.

[2] Tang, Jie, et al. “Learning to Comparison-Shop.” ACM International Conference on Information and Knowledge Management. 2025.

[3] Haldar, Malay, et al. “Beyond Pairwise Learning-To-Rank At Airbnb.” ACM International Conference on Information and Knowledge Management. 2025.

[4] Zha, Daochen, et al. “JourneyFormer: Encoding Airbnb Guest Journey with Sequence Modeling.” ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2026.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Shipping a production LLM system means iterating fast on improvements to something that is, by construction, non-deterministic. Models drift, judges disagree with themselves, references regenerate as different strings, and bugs may persist until the next release, because retraining takes weeks. Most of this friction comes from infrastructure challenges, not model quality, and the fixes come from classical software engineering techniques.

At Airbnb, we built reliable LLM infrastructure by addressing four layers. Three correspond to engineering enhancements we’ve made; the fourth is the integration layer that ties the rest together — the one that is easiest to overlook, because each individual component looks fine in isolation. The approach rests on two observations: the seams are where things break, and finding those breaks requires exercising the full path, not just validating each component in isolation.

Figure 1. The four layers of the production LLM stack. Bounded model mutation requires trustworthy measurement, and end-to-end validation requires the eval foundation to be fast enough to run on the combined path.

Layer 1: Name it before trying to remove it

Layer 1 is diagnostic framing of evaluation noise. This layer addresses two different sources of indeterminacy: data and judging uncertainty.

Classical ML metrics are deterministic: F1, BLEU, and accuracy return the same number on the same input. With LLMs in the evaluation loop, that assumption dies. Judges score identical inputs differently across runs, and LLM-generated references regenerate as different strings.

A two percent score movement can mean the model improved, the judge drifted, the references shifted, or some combination. We cannot tell which without naming which kind of noise we are looking at.

The question is not only, “Which average is higher?,” but “Where do the systems differ, and are those differences meaningful, stable, and product-relevant?” Meaningful, as we use it, means surviving perturbation: the conclusion holds when we rotate judges, version metrics, and re-stratify samples. Formal significance testing is a complement, not the whole answer; a difference that passes a t-test but flips under judge swap isn’t a difference worth shipping on.

We use the frame of dual indeterminacy: noise in LLM evaluation has two potential sources, each requiring separate diagnoses & fixes. Abbasi Yadkori et al formalizes this separation, showing that epistemic uncertainty (model/judge limits) and aleatoric uncertainty (task ambiguity) require different detection strategies, and that conflating them produces wrong conclusions. Specifically, methods that fail to separate them misclassify high-entropy responses as hallucinations. Ling et al. (2024) extends this to the in-context learning, demonstrating that epistemic uncertainty is the more actionable signal, while aleatoric uncertainty reflects properties no judge improvement resolves.

In practice, the two kinds of uncertainty can appear together. A generated answer may depend on missing user preferences (aleatoric) while the judge lacks domain knowledge to verify it (epistemic). Some judge disagreements are fixable; others reflect properties of the task.

In our setting, roughly three-quarters of LLM-generated references differ across labeling runs on identical inputs, and the same judge drifts about one percent across runs on the same dataset. When the real signal is one to three percent, much of what we observe is noise — and not the kind more samples will resolve.

Naming the sources isn’t enough; we need to tell them apart in a given experiment. That requires judge outputs stable across runs on identical inputs. Layer 2 makes that possible.

Layer 2: A deterministic evaluation foundation

The instinctive responses to a noisy judge are probabilistic: sample and majority-vote, or model the noise Bayesian-style. Both fall short. Majority voting converges toward the judge’s central tendency, not toward accuracy. Bayesian methods need a centralized store for priors and per-sample posteriors across runs; effectively the same infrastructure as a per-sample cache, but without the reproducibility you get for free by caching.

Repeated sampling is effective at identifying examples where judges’ scores vary significantly (judge-sensitive examples). For instance, a stable case shows low variation (e.g., [0.78, 0.80, 0.79, 0.81, 0.78]), while an unstable case shows high variation (e.g., [0.45, 0.83, 0.52, 0.88, 0.60]). However, applying this method broadly is resource-prohibitive because the evaluation resource burden scales quickly with the number of examples, systems, and judges.

The simpler move is to stabilize the inputs to the judge rather than model around its instability. When we instrumented the workload, more than half of model outputs across candidates were identical strings. So most experimental changes only affect a subset of inputs, and candidates sharing a base model diverge only on contested examples. Most reference outputs would also have matched had we not regenerated them. The noise we experienced was largely generated by the testing process itself.

We built a per-sample cache on both axes:

• References: Keyed by sample identifier (eg text or more) and reference-generation configuration.
• Judge scores: Keyed by sample, model output, judge configuration, and metric.

Identical inputs return cached results. Evaluation becomes deterministic, more efficient, and comparable across runs, which is exactly what Layer 1 needs to diagnose whether a disagreement is epistemic or aleatoric.

Keying at the experiment level (not the infrastructure level) also makes partial progress durable: a job failing at example 8,000 will resume from the cache. Each new candidate and metric runs against existing cached outputs for free. Reproducibility at this stage is not a convenience; it is a precondition for the next layer.

Layer 3: bounded, scoped model mutation

Once evaluation is deterministic and fast, the bottleneck shifts from “Can we measure whether this change is an improvement?” to “Can we make a small change quickly?” Full adapter retraining is the wrong tool because it is neither quick nor safe. A LoRA adapter is a small set of extra weights trained on top of a frozen base model; its rank sets how much capacity it adds, with higher ranks learning more but costing more to train. On an 8-billion-parameter base model, training a high-rank adapter (a few hundred in LoRA terms) takes days, and every weight change risks regressing inputs that were already working.

The research gives us both the justification and the warning. Meng et al. (NeurIPS 2022) shows factual behavior in transformers is partially localized; targeted weight updates to specific MLP modules can succeed without disturbing unrelated behavior. Cohen et al. (TACL 2024) shows the limits: even precise edits produce ripple effects on logically related knowledge. Pletenev et al. (2025) puts a practical boundary on it; LoRA adapters can absorb targeted corrections reliably up to a few hundred examples, beyond which reasoning degrades and the model becomes overconfident in hard-to-detect ways.

Our solution is the micro adapter: a small LoRA patch — rank less than 50, well below the few-hundred range used for full retraining — layered on top of an existing shared adapter without modifying its weights. The patched adapter learns only the minimal correction for a specific bug, trains in under an hour on one GPU, and ships as a software hotfix would: scoped to one issue, validated behind two gates (no regression on expert-reviewed domains; high-uncertainty outputs flagged for human review), canary-deployed with automatic rollback. The loop runs under a same-day turnaround.

Each micro adapter is self-contained in isolation, but stacked patches interact: a direct case of the rule in Sculley et al. ( 2015) rule, captured in the acronym CACE (changing anything changes everything). Three lifecycle rules keep the stack from drifting:

• Fuse co-triggering patches. Patches firing on overlapping inputs share relational neighborhoods; learnable fusion resolves the subspace interference that naive stacking creates (Gao et al., 2024).
• Retrain on accumulation. When the number of patches against one category approaches the empirical ceiling given in Pletenev et al., fold the patches into a clean retrain.
• Unload unused patches. Every loaded patch needs revalidation when upstream changes. Patches not triggered in a defined window are unloaded automatically.

This layer only works because Layer 2 exists. Without deterministic evaluation, the same-day turnaround collapses; there is no time to validate behind two gates if validation itself takes a day.

Layer 4: end-to-end validation at the seams

This final layer is the layer that is easiest to overlook, because each of the individual components upstream is likely to look fine. We had validated language detection, preprocessing, the modeling layer, and the serving path. Yet the combined production behavior was still capable of surprising us, because none of the component-level checks exercised the full path under realistic conditions.

Component-level confidence creates false assurance at the seams. This is a known failure mode of ML-enabled systems. Kästner et al. (2021) argues that ML components resist the compositional reasoning that makes traditional software testable. They carry no formal specifications, so their interactions at the seams can only be observed empirically. Sculley et al. (NeurIPS 2015) identifies the same structural problem through the CACE principle: in ML pipelines, components are entangled in ways that make isolated validation insufficient. Validating each part does not remove the need to validate the whole.

The fix is unexotic: a small set of representative inputs are run through the entire production path, with quality and tail-latency measured on the combined configuration. This is done using the same eval framework that powers Layers 2 and 3, so deterministic measurement applies end-to-end. Selecting truly “representative” inputs is what matters, and this is best performed by combining traffic-weighted sampling with deliberate over-representation of the tail — the locales, input modalities, and historically incident-prone patterns that individual component tests most often miss.

We stratify across the highest-volume traffic segments, include the long tail of locales and input modalities where component coverage is weakest, and explicitly seed the set with regression cases from prior incidents. The set is small enough to run efficiently on every release candidate, and broad enough that bugs which occur at the seams surface before deployment. These bugs include language detection misclassifying a code-mixed input, preprocessing truncating a field the model relied on, and latency spikes from cache-warmth interactions.

When the foundation is strong enough, an end-to-end pass is not a separate project; it is one more invocation of infrastructure that you already trust.

Conclusion

The four layers are a dependency stack, not a checklist. Deterministic evaluation makes same-day fixes possible. Same-day fixes make micro-adapter discipline tractable. End-to-end validation only means something when the measurement underneath it is trustworthy. Remove any one layer and the others degrade.

The deeper lesson is about where the leverage actually is. The field’s instinct (and our own, early on) is to reach for model improvements when systems feel unreliable. But the debt accumulates at the seams, not in the components.

LLM pipelines are not categorically different from the ML pipelines described a decade ago in Sculley et al. They are pipelines with a nondeterministic component in the middle, which makes the seams harder to reason about, but all the more important to test.

The novelty in the field is real. The leverage is still in the boring, well-understood patterns of systems engineering, applied with judgment to where the new failure modes actually live.

If this type of work interests you, check out some of our open roles.

References

Abbasi Yadkori, Y., Kuzborskij, I., György, A., & Szepesvári, C. (2024). To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty. NeurIPS 2024. https://openreview.net/pdf?id=k6iyUfwdI9

Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685

Ling, C., Zhao, X., Zhang, X., Cheng, W., Liu, Y., Sun, Y., … & Chen, H. (2024). Uncertainty Quantification for In-Context Learning of Large Language Models. https://arxiv.org/pdf/2402.10189

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J., & Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf

Liu, Y., et al. (2024). FashionGPT: LLM instruction fine-tuning with multiple LoRA-adapter fusion. Knowledge-Based Systems, Vol 299. https://doi.org/10.1016/j.knosys.2024.112043

Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022. https://arxiv.org/abs/2202.05262

Cohen, R., Biran, E., Yoran, O., Globerson, A., & Geva, M. (2024). Evaluating the Ripple Effects of Knowledge Editing in Language Models. TACL 2024. https://arxiv.org/abs/2307.12976

Kästner, C., Kang, E., & Apel, S. (2021). Feature Interactions on Steroids: On the Composition of ML Models. arXiv:2105.06449. https://arxiv.org/pdf/2105.06449

Pletenev, S., Marina, M., Moskovskiy, D., Konovalov, V., Braslavski, P., Panchenko, A., & Salnikov, M. (2025). How much knowledge can you pack into a LoRA adapter without harming LLM. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-naacl.243

Acknowledgments

I am grateful to the colleagues and reviewers who shaped this work through technical discussions, infrastructure support, and feedback.

Special thanks to Yashar Mehdad, Nikolaj Nielsen, Haotian Li, Richa Khandelwal, Dan Miller, Jisheng Liang, Julie Xiang, and Atul Kale for their guidance and support at different stages. I’m also grateful to Yi Li for the support and review that helped bring this work forward. I also appreciate the broader ML and infrastructure teams for the discussions that helped refine the ideas in this article.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

But getting there wasn’t straightforward. This fundamental shift surfaced a critical question for our data organization: How do you evolve your offline data architecture to support new product lines without introducing disorder in vital analytics services?

We knew the approach we took would have long-lasting implications. A fragmented strategy risked creating data silos, inconsistent analytics, and a tangled web of technical debt that would likely slow down future innovation. In this post, we’ll take you behind the scenes to share key decisions that we made, the framework that emerged, and the lessons that helped reshape our offline data warehouse for the future. Note that we focus specifically on our offline data warehouse (the analytics-oriented data infrastructure owned by our data engineers and analytics engineers) rather than the online data systems that serve the app directly, as the two domains have fundamentally different requirements, constraints, and design philosophies that warrant separate treatment.

The core dilemma: Separate vs. monolithic

The first and most critical question was how to structure offline data for the new, three-product world, with Homes, a refreshed Experiences product, and the new Services offering. This involved a trade-off between two main approaches:

1. Separate data models: This approach creates distinct sets of tables for each product line, keeping the data for each business clean and highly tailored, but incurring a higher incidence of duplicated logic across models.
2. Monolithic model: This approach combines all product lines into a single, unified set of tables to maximize code reusability and ensure consistency, but risks becoming unwieldy and less well-suited to the unique attributes of each product.

It became clear that neither approach was universally superior. The optimal choice depended heavily on the specific business domain. A model that was perfect for guest data, for example, would be suboptimal for payments data.

We chose a path that balanced consistency with flexibility. We established a framework that combined firm, centralized principles with decentralized modeling guidelines, empowering each data team to make the right choice for its domain.

Our three foundational principles

To ensure a baseline of consistency across all teams, and to keep the door open for any new product categories that emerge in the future, we established three foundational principles. These principles ensured that no matter which modeling path a team chose, the results would be consistent, scalable, and easy for all data consumers to understand:

• Principle 1: No hybrid data models. A domain’s data model had to be either completely separate by product type or completely monolithic. We viewed this choice between two distinct paths as key for future scalability. It was important to avoid confusing, inconsistent situations down the road where some products might use combined data tables while others do not.
• Principle 2: Consistent identifier naming. To ensure reliable table joins and prevent confusion, we established a strict convention where the structure of primary identifiers was directly dependent on the modeling choice. Teams using separate models were required to use product-specific IDs (e.g., id_experience, id_service). In contrast, teams using a monolithic model had to use a generic product descriptor ID (e.g., id_product_listing) and include a product type column (e.g., dim_product_type) to differentiate between Homes, Experiences, and Services.
• Principle 3: Clear namespace organization. We used namespaces to define clear placement for every table. Core, product-specific tables were placed in dedicated product namespaces while monolithic, cross-cutting tables lived in a global namespace. This structure was supplemented by team-specific namespaces, giving individual teams the flexibility to manage their own assets and intermediate tables.

These principles set firm boundaries for our modeling efforts and ensured a consistent foundation across the company.

The modeling guidelines

With the foundational principles in place, we gave each team a set of guidelines to help them decide how to model their data. This empowered them to pick the right model for their specific domain, using a common set of considerations.

• Shared vs. unique product attributes: Do the different product lines (Homes, Experiences, and Services) share a common set of data attributes? Or are there many unique attributes for each product line that would make a unified model impractical?
• Future evolution: Will this model scale if we add a fourth product line? A fifth?
• Upstream alignment: How does the online production database structure its data?
• Downstream consumers: How will analysts and data scientists query this offline data?
• Code maintainability: Does one path lead to cleaner, more modular code?
• Data volume & performance: Can the model handle the scale of the new data that’s being added efficiently?
• Compatibility: How can we maximize backward and forward compatibility with the existing data and ensure future changes for Experiences and Services do not impact resiliency of the models powering the Homes product line?
• Business Continuity: How can we ensure accurate reporting for existing key metrics after introducing new data into the models, especially considering the relatively low volume of data that is processed initially and the need to handle unexpected values?

These guidelines provided a framework for every domain team to use in analyzing their specific situation.

Putting the framework into action

When teams applied this framework to their specific data, a clear pattern emerged. While every guideline played a role, one question in particular proved to be the most decisive factor: Do the product lines share mostly common data attributes, or do they have significant unique attributes?

Separate data models: For product-specific logic

Teams working on features closest to the user experience found that the product attributes were too distinct to combine. They overwhelmingly chose to build separate models to capture the unique nature of each product, particularly as we introduced several new concepts with our Services product. Some examples include:

• Listings: The biggest conceptual change came from the introduction of Service offerings. These are distinct activities or variants nested under a single parent Service listing. For example, a fitness host might offer yoga, pilates, and strength training, or a chef might offer several different menus. This many-to-one relationship between offerings and a parent listing was a new core concept that had no parallel in our Homes or Experiences products, making a separate data model necessary.
• Availability: For Service providers, we introduced the concept of business hours. Previously, hosts set availability with specific calendar dates (for Homes) or specific start times (for Experiences). Business hours allowed a guest to book at any time within a set timeframe (e.g., Mon-Fri, 9am-5pm). This required us to change our data models to translate these flexible hours into discrete, bookable time slots for guests.
• Location: Prior to this launch, listings for Homes and Experiences were generally at a single, fixed location specified by the host. With Services, a host could have guests travel to them, or they could choose to travel to their guests. This introduced the need for a Service area, allowing hosts to designate the geographic areas they were willing to travel to. This flexible, radius-based location concept required its own data model.
• Guests: The redesign of the app created entirely new user journeys. The way a guest interacted with the app, from the homepage and search all the way through the checkout steps, differed across product types. By its nature, interaction data has very high volume, which risks performance degradation. To accurately capture these distinct funnels and user behaviors, and ensure scalability and performance, separate data models were necessary.

Monolithic data model: for cross-cutting concepts

While product-facing domains chose separation, other teams managing more foundational, cross-cutting concepts found that a monolithic model was a much better fit.

• Messaging: On Airbnb, a message thread is the common unit of analysis, and it can occur between guests and hosts, guests and customer support, or even between guests. Messaging can span multiple product types within a single thread. Separating the data models by product type would fragment these conversations, create operational challenges, and make it impossible to get a holistic view, so a monolithic approach was the choice.
• Payments: Our payments data models are largely product-agnostic by design. They are built to handle transactions, refunds, and payouts, regardless of what product is being purchased. New product types like Services could be onboarded within the existing monolithic data models with minimal changes.
• Customer support: Much like messaging, customer support requests can span multiple product types. A guest might contact support with an issue that involves both a Home they stayed in and a Service they booked. A monolithic model was the only way to maintain a single, unified view of a user’s support history.

This clear delineation, choosing the separate models approach for product-specific logic and monolithic models for cross-cutting concepts, allowed us to successfully model our new, complex business landscape. The framework gave us the flexibility we needed within a consistent and scalable structure.

Navigating the challenges of standardization

Designing a framework on paper and implementing it across a large, fast-moving organization are two very different things. To make this initiative successful, we would need to navigate two significant, real-world challenges.

Creating the foundation of actionable insights

Our offline data warehouse doesn’t exist in a vacuum. Upstream online data models and their corresponding event logging were rightly optimized for the immediate needs of running the app, such as transactional speed and stability, rather than the structural clarity that is ideal for offline analytics.

As a result, the raw data flowing into our warehouse was often structured in ways that were not ideal for analytics. This reality underscores a core tenet of our data strategy: the offline data warehouse must act as a crucial translation layer. Taking the raw production data and transforming it into a standardized, reliable source of truth is a key function performed by our data engineers and analytics engineers, enabling our downstream consumers to surface insights quickly and accurately.

Managing data debt while making progress

The new standards gave us a clear vision, but they also highlighted legacy tables and dashboards, especially from the older version of Experiences, that no longer met the criteria. Migrating and deprecating these assets is always a massive undertaking. These tables often have hundreds of downstream consumers, so the process requires extreme care, involving extensive communication, dual pipeline runs for validation, and a painstakingly slow deprecation cycle to avoid breaking the business processes that depend on each asset.

These challenges are the reality for any data organization supporting rapid product innovation. Months after our initial launch, we continue to refine our translation layers and to carefully migrate legacy assets.

The takeaway

The journey to a multi-product data architecture was as much about people and process as it was about technology. By establishing clear principles while empowering teams with a flexible set of modeling guidelines, we successfully navigated the data complexity of launching the new Services product line and overhauling the existing Experiences product line, while ensuring that functionality within the core Homes product line was not placed at risk.

This journey reinforced a key principle of data modeling at scale. The best answer is rarely “one size fits all.” It’s about creating a system that balances central consistency with domain-specific flexibility, and having the discipline to not only build for the future but also to thoughtfully address the past. It’s a continuous effort, one that has already paid dividends in scalability, clarity, and our ability to deliver insights quickly and accurately.

If this type of work interests you, check out some of our related positions!

Acknowledgments

This was one of Airbnb’s biggest product releases ever and the data work behind it was a true team effort. Thanks to all who contributed big and small.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

In our previous post, we explored Airbnb’s dynamic configuration system, Sitar, with a focus on service architecture and configuration change safety. Now for the harder question: once a config change is committed, which happens several times each minute, how does it actually reach the thousands of Airbnb’s service instances reliably, quickly, and without redeploying the services?

This post describes sitar agent: a lightweight Kubernetes sidecar that runs alongside every subscribed service pod, continuously synchronizing the latest configurations from the service backend and making them available on the local filesystem for reads. In this post, we will first go through the configuration delivery life cycle, and then discuss some key design choices for the sitar-agent sidecar.

Config delivery life cycle

The diagram below illustrates the end-to-end journey of a configuration change, from the developer-facing layer to the production service fleet.

Step 1 — Config creation/update

Developers create or update configuration values through either Git flow or the web UI. These changes are committed to the Sitar Service, where they are stored with full versioning, change logs, and ACL enforcement.

Step 2 — Hourly snapshot upload

The Snapshot Service periodically packages the full state of all config groups and uploads compressed snapshots to AWS S3.

Step 3.1 — Preload snapshot from S3 (on pod startup)

When a production service pod starts, the sitar-agent sidecar runs first. It downloads the latest snapshot for each subscribed tenant’s configs from S3 to the mounted disk (shared between sitar-agent and the main container). This allows the agent to bootstrap from a known-good state without fetching every config from the Sitar Service from scratch on every restart. Preloading the snapshots from S3 enables faster restarts, makes the service resilient to transient Sitar Service unavailability, and avoids load spikes during deployments.

Step 3.2 — Preload latest config from Sitar Service (on pod startup)

After loading the S3 snapshot, the agent performs an initial sync with the Sitar Service to catch up on any changes published since the last snapshot. Once this step succeeds, the agent signals readiness, unblocking the application main container from starting.

Step 4 — Periodic update

After startup, the agent enters a continuous polling loop (order of seconds with jitter). On each cycle, the sitar agent queries the Sitar Service for changes across all subscribed groups.

Step 5 — Read config

The application main container reads configurations from the mounted disk through the Sitar client library, which maintains an in-memory cache. The client detects file changes and refreshes its cache transparently.

With the delivery lifecycle in mind, the following sections walk through the major architectural choices that shaped the sidecar’s design.

Key design decisions

In 2024, the sitar-agent underwent a full rewrite from Ruby to Java, Airbnb’s mainstream JVM language, giving the team an opportunity to modernize the architecture alongside the language migration. The snapshot-based S3 preload introduced in the previous section is one outcome of this effort: it dramatically reduces cold start time for the pod and decouples startup reliability from Sitar Service availability. The rewrite also led to several other deliberate design decisions around reliability, performance, and operational safety. The sections below walk through each of these choices.

Requirements for the Sitar System

Before diving into specific design choices, it helps to understand the constraints that shaped every decision. At Airbnb, dynamic configuration delivery isn’t just a convenience: it controls critical features across thousands of services. That means configs must always be available, even when the Sitar Service itself is down; a slightly stale value is tolerable, but an unreadable config is not. At the same time, when an engineer pushes a change, it needs to reach every subscribed service within tens of seconds, not minutes. Making that work at scale is non-trivial: with tens of thousands of pods fetching updates simultaneously, the system has to absorb that load without degrading. And since Airbnb’s service fleet spans Java, Python, Go, Typescript, and Ruby, the solution needs to serve all of them, ideally minimizing the effort of maintaining separate per-language implementations.

The above requirements for reliability, performance, scalability, and multi-language support aren’t independent. As you’ll see, most of our design decisions, described below, come back to balancing one against another.

Main container vs sidecar

The question of whether sitar-agent should run as a sidecar container or a process in the main container surfaced as a key architectural decision during the Java rewrite. We evaluated the pros and cons of each option as follows:

Pros of moving to the main container:

• Cost reduction. This is the main driver for moving to the main container: running the agent as a library eliminates the per-pod JVM overhead, allowing memory and CPU to be shared with the main container.
• Reduced operational surface. One fewer container means one fewer component for service owners to configure and tune. However, this advantage weakens when considering Airbnb’s multi-language service fleet.

Cons of moving to the main container:

• Multi-language complexity. Airbnb service languages span Java, Python, Go, Typescript, and Ruby. A library approach would require the existing sidecar logic to be implemented in all languages, significantly increasing development and maintenance effort.
• No isolation. Bugs or resource spikes in sitar logic can crash or starve the main container, and vice versa. This coupling increases incident blast radius and complicates resource attribution during debugging.
• Operational noise. Having the logs for Sitar and its cpu/memory usage mixed with the main process logs and its metrics makes it harder to debug both sitar and main process issues.
• Optimizability. Having a separate container allows the container to be optimized for its purpose, and eases testing and debugging.

Decision:

Despite the cost savings and reduced operational surface which would result from moving the sitar-agent logic to the main container, the projected savings were insufficient to justify the tradeoffs in reliability and operational overhead, and the development overhead of supporting the sidecar logic in multiple languages. We therefore decided to maintain the sitar-agent as an isolated sidecar container.

The pull model and server-side optimization

Sitar-agent fetches configuration updates by polling the Sitar service every 10 seconds. This is a pull model: the agent drives the update cycle by periodically asking the server for changes. This pull-based architecture, while being simple and easy to maintain, generates unnecessary load on the server when there is no update needed.

A push-based architecture change can greatly reduce the server-side load and change propagation time, at the expense of a more complicated architecture. In order to keep the current simple architecture while reducing the service-side load, the sitar system implements the following optimizations:

1. Since the sitar config is mostly changed manually, which takes longer than several seconds, a slight delay in config update delivery is acceptable. Therefore, a server-side cache with a short TTL (10s) is a great way to reduce sitar server-side processing. Most of the sitar-agent calls to services hit the cache layer without triggering heavy server-side compute or database access, thus greatly reducing the resource usage of handling requests.
2. When there is a cache miss and the request actually triggers database access, it passes along a token (last scanned db row), which tells the service to skip scanning for changes before the last fetch, thus greatly reducing server-side processing and database access time during each periodic pull.

Given the above optimizations, the sitar-service can scale and perform quite well in handling the pull request from all service pods at Airbnb, and we can preserve the simple, stateless server-with-pull architecture.

Decision:

For sitar’s use case, polling latency on the order of seconds is acceptable; dynamic config is not a real-time signaling mechanism, and most config changes are manual, making a few seconds of propagation delay inconsequential. The pull model’s stateless simplicity is a strong operational advantage at Airbnb’s scale. The team elected to keep the pull model and invest instead in reducing per-poll cost.

Local datastore selection

Sitar-agent maintains a local on-disk key-value store that the main container reads from. The legacy datastore is a Sparkey-backed internal implementation, with a thin layer around the Sparkey datastore for concurrent coordination. As the usage of Sitar continues to grow and evolve, the mismatch of the Sparkey-backed datastore and sitar’s needs have become evident:

• Sparkey is purpose-built for write-once, read-many workloads with no support for multi-thread read-write coordination. This requires a wrapper around the Sparkey datastore for concurrent coordination to support sitar’s frequent write to the datastore, adding to complexity and potentially becoming a source of latent bugs.
• Sparkey doesn’t include native concurrency support by design, and we needed an external locking mechanism that locks the entire datastore file on write. As update frequency increased across the datastore, this lock contention began to limit concurrent read/write performance.
• Since Sparkey’s design requires re-indexing of the entire datastore on each write, writing frequently to the Sparkey backed datastore became increasingly expensive. However, as Sitar has become widely used across almost all Airbnb services, the write to the datastore is very frequent; we see updates in configs in almost every pull cycle (every ~10 seconds)
• Sparkey has limited multi-language support: it does not have implementations in all languages Airbnb services require, and supporting all languages in Airbnb would require complex interop.

The team evaluated and benchmarked two candidates to replace the legacy Sparkey-based datastore: SQLite and RocksDB. A matrix of experiments were run across varying dataset sizes, read QPS, and memory allocations, fixing two of the three dimensions and varying the third in each run. We also researched community support, open source activity, supported languages, and adoption breadth of both. The following summarizes our findings:

SQLite:

Pros:

• Mature, widely-adopted library with officially maintained bindings for Java, TypeScript/Node.js, Python, Go and Ruby: all languages used by sitar’s service consumers.
• Built-in write-ahead logging (WAL) mode supports concurrent reads during writes, eliminating the need for a custom concurrency wrapper.
• Simple operational model: a single file, no background compaction or tuning required.
• Read and write performance is dramatically better than the Sparkey-backed datastore, and sufficient for sitar’s workload.

Cons:

• Read latency is 2–3x slower than RocksDB, and increases linearly with data size.
• Write latency also increases with larger data sizes.

RocksDB:

Pros:

• Best raw read/write performance across all test dimensions.
• Consistent read high-QPS performance; tested up to 1500 ops/sec with minimal degradation.

Cons:

• A more complex operational model; requires tuning of compaction, block cache, column families, and memory settings.
• The multi-language library ecosystem is less mature and less uniformly maintained than SQLite’s.
• Higher operational burden for a team without deep RocksDB expertise.

Decision:

In our tests, both RocksDB and SQLite significantly outperform Sparkey-backed datastores for our workload across all three test dimensions: data size, memory allocation, and read QPS. While RocksDB delivers better raw performance, sitar-agent’s workload operates comfortably within SQLite’s envelope. SQLite’s first-class multi-language library support, native WAL-based concurrent access model, and simpler operational footprint made it the better overall fit for a team supporting multiple language runtimes. The team selected SQLite as the replacement for the Sparkey-backed datastore.

Safe migration from Sparkey to SQLite

Operational safety was a top priority. Beyond extensive testing, we also we relied on two mechanisms to keep the rollout safe:

1. Shadow reads: Before migrating each service, we ran a shadow read-and-compare phase; services continued reading from Sparkey while SQLite results were fetched in parallel for validation.
2. Feature flag-gated gradual rollout: We migrated incrementally, starting from the least critical services and progressing toward the most critical. Some critical Tier 0 services were onboarded last, with dedicated coordination at each step.

Conclusions

Sitar-agent sits at the core of Airbnb’s dynamic configuration delivery system. This post walked through how it works and the key tradeoffs we navigated during the Java rewrite: between cost and isolation, simplicity and push-based efficiency, and raw performance and operational practicality. Every decision came back to the same constraints: configs must always be available, changes must propagate quickly across a fleet of tens of thousands of pods, and the solution must work across Airbnb’s polyglot service stack without compounding the maintenance burden.

If this type of work interests you, check out some of our related positions!

Acknowledgments

Our progress with Sitar would not have been possible without the support and contributions of many people. We’d like to thank Craig Sosin, Nikolaj Nielsen, Daniel Fagnan, Alex Edwards, Nick Morgan, Carolina Calderon, Hanfei Lin, Yunong Liu, Lucas Rosa Galego, Yann Ramin, Denis Sheahan, Richa Khandelwal, Swetha Vaidy, Adam Kocoloski, Adam Miskiewicz, and all the other engineers and teams at Airbnb who joined design reviews and offered valuable feedback, as this work would not have been possible without them.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Almost every forecasting system is built on the same implicit assumption: the future will resemble the past. You train on historical data, you validate on holdout periods, and you trust that past patterns will at least roughly indicate future performance. When this assumption breaks, the model does not gracefully degrade; it fails confidently. It produces precise, well-calibrated intervals around the wrong answer.

The acute phase of COVID, from early to late 2020, was a clear illustration of this, and we wrote about it in a previous post. But the more interesting forecasting problem was not the shutdown. It was everything that came after.

The period from late 2020 through 2022 was not a single coherent regime. It was a sequence of overlapping, asynchronous changes: vaccine rollouts that reached some markets months before others, border reopenings that followed their own country-level timelines, reclosures triggered by new variants that hit different corridors (a pairing of the traveler’s origin city and destination city) at different moments.

Demand was not recovering uniformly. It was rebounding unevenly across every corner of the world, in ways that had no historical precedent and no single governing pattern.

The standard response to a shock is to wait for each affected market to accumulate its own post-shock data and retrain locally. But Covid was among the biggest shocks the travel industry has faced in decades. With markets worldwide reopening and reclosing on staggered schedules, waiting for markets to settle meant forecasting blind for months at a time, across all markets, just when timely projections were most needed, in the circumstances.

So we started building something different. When we could not simply look backward in time for relevant examples, we looked sideways across geographies instead.

The insight: geography as a time machine

The key observation was that the recovery was not happening everywhere at once. It was unfolding sequentially, and messily, often punctuated by further reclosings and reopenings. Vaccines reached some markets in early 2021 and others months, a few quarters, or many quarters later. Some borders reopened in spring and reclosed by autumn. Demand in one corridor could be surging while an adjacent corridor was still effectively shut.

This sequential, asynchronous structure was operationally painful. But it contained more information than might have been obvious on initial consideration.

One of the clearest signals we track is the mean lead time for bookings: how far in advance guests book relative to their travel dates, measured as a ratio against the same period in a baseline set in 2019, the last fully pre-pandemic year. When there is a disruption, and the pandemic as a whole was the largest disruption we’ve ever seen, lead times compress sharply as travelers shorten planning horizons for the trips they do take, then lengthen again as conditions stabilize.

The figure below shows this signal for Europe and North America across the major phases of the pandemic. The key observation is not the shape of either curve in isolation. It is the lag between them.

Europe’s first wave of booking lead time compression hit in February 2020. North America’s came roughly four to six weeks later, but following the same trajectory. The reopening recovery was partial in both regions, because the travelers who returned first were booking short-lead-time trips rather than resuming normal planning horizons. And when vaccine rollout arrived, the direction reversed: North America turned the corner in December 2020, while Europe was still in its second wave trough, and did not begin its recovery until February and March 2021.

Once we could see how demand responded to reopening in one of the two markets, we had a genuine signal about how demand was likely to respond when the other market reopened later. It was not a perfect signal. The markets were distinct, the timing varied, and the traveler mix was somewhat different. But the underlying dynamics were related.

Travelers responded to reopened borders, to restored flight routes, and to lifted entry requirements, in ways that were not completely idiosyncratic to each corridor. Corridors in the earlier-reopening market were ahead of the later-reopening market in time, but they were observing the same underlying phenomena.

Doing the math for demand increases with reopening

In Bayesian terms, the structure is as follows. A brief glossary: c is a corridor; θ_c denotes the demand parameters for corridor c; φ denotes hyperparameters shared across all corridors; and w(c, c’) denotes the similarity weight between corridors. Each corridor has demand dynamics governed by corridor-level parameters θ_c, drawn from a shared population distribution:

where φ are hyperparameters estimated jointly across all corridors and y_{c,t} is the observed demand signal in corridor c at time t. This is a standard hierarchical setup.

The innovation is what happens when a change hits corridor c at time τ_c, and a similar corridor c’ experiences the same change later at τ_{c’} > τ_c. Rather than waiting for local data in c’ to accumulate, the posterior from the early-affected corridor (the updated belief about its parameters after observing local data) becomes an informative prior for the late-affected one:

You are not extrapolating from the past. You are propagating observable evidence from one part of the world to another, in real time.

How it worked in practice

We did not design this system in advance. We built it as events unfolded

By mid-2020 it was becoming clear that the recovery was not going to be a single global event. Areas were going to open and close on their own timelines, and our models, which had been designed for a world with a stable shared regime, were not equipped for a world where each corridor was effectively in a different phase of the same phenomenon at any given moment.

The first version wasn’t perfect, but it gave us something concrete to build on. We started by identifying corridors where meaningful demand data had returned, where borders had reopened enough to observe actual traveler behavior, and using those as reference points for similar corridors that were still closed or just beginning to reopen. The process was manual, and heavily dependent on human judgment, at the start.

Over time we formalized the process. Airbnb operates across a wide range of origin-destination corridors globally, and that breadth is what made the approach tractable at scale.

The core idea was simple: not all corridors are equally informative about one another. A market with similar traveler composition, similar reliance on international versus domestic demand, and similar accommodation mix should receive a stronger prior from an early-recovering corridor than one that differs substantially on those dimensions. We weighted the information transfer accordingly, so the signal flowed most strongly between corridors that were genuinely structurally similar.

The system earned its keep across all of it: the initial reopenings, the reclosures triggered by new variants, and the uneven rollout of vaccines across regions. In each phase, some corridors were ahead of others, and the ones ahead had something useful to say about the ones that were not yet there. The information was never perfect, but it was available immediately, rather than weeks or months later; that timeliness was often the entire point.

The result was that we could generate informative forecasts across the corridor network throughout the recovery period, including in markets where local data was thin, at precisely the moments when Finance needed the most reliable read on where demand was heading.

Why Airbnb’s data structure made this possible

This approach is not universally available. It requires a specific data structure that Airbnb happens to have.

First, you need enough geographic breadth and granularity for the information sharing to be meaningful. The approach works because some markets are ahead of others in time, and because structurally similar markets exist to borrow from. The more corridors you observe, and the more resolution you have within each one, the richer the signal you can propagate. Airbnb’s global footprint across a wide range of origin-destination pairs gave us enough diversity to find genuinely informative analogues for almost any market we needed to forecast.

Second, you need consistent data across those corridors. The measurements you take in one region need to be directly comparable to the priors you set in another. Airbnb’s booking data is collected in a consistent format globally, which means a demand response measured in Europe translates cleanly into a prior for Asia-Pacific, without a translation layer.

Third, you need a modeling framework that can incorporate informative priors at the corridor level and update them as local data arrives. A standard time series model estimated on a single corridor’s history cannot do this. A hierarchical Bayesian framework that treats corridor-level parameters as draws from a shared distribution can: the prior propagation across geographies is a natural extension of the hierarchical structure.

The framework generalizes beyond COVID

We built this during the pandemic recovery, but the underlying logic applies to any change that rolls out sequentially across geographies or market segments. The change does not have to be a crisis.

Consider how a platform introduces new product features. A new payment plan option, a change to cancellation policy, or a new booking flow does not launch everywhere simultaneously. It typically rolls out to a subset of markets or corridors first.

The early markets are a source of information about what to expect when the feature reaches the next wave. If the first corridors to receive a new payment plan show a measurable shift in booking lead times or cancellation rates, that signal can inform the priors for corridors where the feature has not yet launched. You are not waiting to observe the effect everywhere before you can say anything; you are propagating what you already know into relevant forecasting.

The same logic applies to regulatory changes, which rarely hit all regions simultaneously. Or to commodity price shocks: an energy price spike hits origin markets with high fuel-cost sensitivity before it ripples through to destinations, and the corridors that feel it first carry information about how demand will shift when it arrives elsewhere. Or to any macroeconomic or geopolitical development that affects different origin-destination corridors at different moments, which in travel is most of them.

In each of these cases, the question is the same: is your modeling infrastructure set up to learn from the early-signal markets in real time, and to propagate that learning before the change arrives everywhere else?

At Airbnb, this approach has become a standing part of how we think about forecasting when demand conditions are shifting unevenly across markets. It does not apply to every problem. But when the environment is evolving sequentially, and structurally similar corridors exist to borrow from, waiting for local data is no longer the only option.

What we learned

Three things stand out from building and operating this system.

Geographic structure is underutilized information. Most forecasting teams treat each market or region as an independent problem. The shared dynamics across geographically and economically similar corridors are a source of signal that is almost entirely ignored in standard approaches, but which are likely to provide actionable information. This matters during any sequentially-rolling change, not just crises.

Sequential rollouts are an underused source of signal. The instinct, when a change hits some markets before others, is to treat the unaffected markets as a separate problem until local data accumulates. The more useful instinct is to identify which markets were affected first and treat them as leading indicators. This reframe shifts you from “we have no data yet” to “we have early evidence from analogous markets.”

Bayesian hierarchical models are a powerful tool for this problem. The prior-propagation mechanism is not a hack or a workaround. It is exactly what hierarchical Bayesian models are designed to do: share information across related units while allowing local data to update the shared prior as it arrives. As observations accumulate in a later-affected corridor, the update follows the standard form:

The balance between the propagated prior and the local likelihood shifts automatically as data accumulates. Early on, when c’ has little local data, the prior from similar corridors dominates. As observations arrive, the local likelihood takes over and the corridor estimate converges toward its own experience; no manual tuning required. The information sharing is heaviest when it matters most, and gracefully recedes as it is no longer needed.

In addition to long-lasting disturbances due to Covid, the world has not been short of disruptions since 2020. Many of them have arrived with their own geography, their own timeline, and their own reasons why the historical record was not quite the right guide. When disruptions do unfold sequentially across a heterogeneous corridor network, that is exactly the structure this framework is designed to exploit most effectively. It will not be the right tool every time. But it has been the right tool often enough that we have stopped treating it as a crisis response and started treating it as standing infrastructure.

We did not plan to build this system. We built it because the alternative, waiting for each market to tell its own story on its own schedule, could not keep pace with rapidly evolving conditions. That turns out to be a reasonable description of how most useful forecasting infrastructure gets built.

If this type of work interests you, check out some of our related positions.

Acknowledgments

Thanks to Liz Medina and Jess Needleman for building and improving the forecasting systems described here, and to Carolina Barcenas, Yuanyuan Cui, and Adam Liss for their support of this work and its publication.

Harrison Katz leads Finance Data Science & Strategy at Airbnb. His research focuses on Bayesian methods for compositional and hierarchical time series, Bayesian decision theory, & Forecast governance.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Airbnb’s identity graph captures relationships between users in a graph database. The identity graph serves aggregated insights that enable user identity resolution and relationship understanding. These capabilities support a wide range of Trust and Safety use cases, from detecting suspicious activities to identifying linked accounts. Over time, the identity graph has grown into one of the largest and most complex graph data products at Airbnb, both in terms of scale and the complexity of queries it supports.

In 2024, Airbnb began investing in a new, internally managed, paved-path graph data platform to build a unified knowledge graph infrastructure. Airbnb’s identity graph became one of the first systems to adopt this platform. In this post, we’ll walk through the foundations and challenges of the identity graph, introduce the architecture behind the graph infrastructure, and highlight several key optimizations that emerged during the onboarding process.

Airbnb’s identity graph

Airbnb’s identity graph is a critical foundation layer, playing an important role in Trust and Safety applications. It contains two major components:

• Graph data storage: a storage layer composed of a graph database and a key–value (KV) caching layer. It models users and relationships as vertices and edges. Most data is ingested in near real-time through asynchronous events and served through low-latency, real-time service calls.
• Graph service: this service provides a unified interface for accessing graph data. It retrieves data from underlying sources, including the graph database, applies aggregation logic or models as needed, and serves the results to downstream customer services.

Evolution of the identity graph architecture

The identity graph architecture progressed through three major iterations. It started with a relational database for user and entity data, paired with a KV store holding JSON‑encoded edge lists, a pattern that became difficult and expensive to scale as graph density increased. A third‑party SaaS graph database replaced the KV store in 2021, improving horizontal scalability but introducing long‑tail latency, operational instability, and limited ability to tune performance or enforce fine‑grained access controls. As part of ongoing system enhancements, the system was migrated to a graph infrastructure: an internally managed, high‑performance graph platform built to support low‑latency, large‑scale graph workloads.

A number of challenges persisted during the evolution of the identity graph:

• Scalability: The identity graph consists of 7 billion nodes and 11 billion edges. The data is fast-growing, at a speed of roughly 5 million new edges per day, which imposes a huge scalability challenge on the write side.
• Query complexity: The graph read queries typically contain 4–8 hops for the majority of identity graph use cases, posing a huge challenge in meeting the latency requirements of critical data flows.
• Long-tail latency: Naturally, the density of the graph data structure varies within each subgraph, and hitting high fanout nodes during the graph traversal can significantly increase the amount of data accessed, which would result in huge performance degradation. The P95 and P99 of graph queries can grow disproportionally long compared to P50.
• Stability: Slow queries can take up a large amount of resources during their executions, which can result in stability issues in graph DB performance.

Airbnb’s graph infrastructure

Before the centrally managed graph infrastructure, graph adoption at Airbnb was fragmented. Teams typically fell into one of four anti-patterns, each with significant operational overhead:

• Relational “graphs”: Modeling nodes and edges in SQL tables, resulting in expensive joins during traversal.
• Offline graphs: Building graphs in the data warehouse, which limited data freshness to daily snapshots.
• DIY open source: Self-managing community versions of graph DBs, leading to high operational toil.
• Managed PaaS: Using third-party vendors, which introduced vendor lock-in and performance bottlenecks.

To solve this, we built an internal graph infrastructure, a paved-path, multi-tenant platform designed to bring our use cases together under a single, supported infrastructure.

The tech stack: JanusGraph + DynamoDB

Graph databases vary widely in storage design, schema models, and query languages. We evaluated options based on four requirements:

1. Scalability for online queries
2. Expressive schema and query capabilities
3. Fit with Airbnb’s infrastructure and operational model
4. A visible, extensible codebase

We chose JanusGraph (a distributed, open-source graph database built on Apache TinkerPop), with DynamoDB as the storage backend and OpenSearch for indexing. JanusGraph’s labeled property graph model provides strong schema support, and Gremlin enables expressive traversal queries.

This combination offers a unique advantage: Storage separation. Because JanusGraph supports pluggable storage backends, we were able to leverage the scalability and reliability of AWS DynamoDB for the persistence layer while maintaining full control over the graph logic layer. This allowed us to iterate quickly on graph features without reinventing the wheel on distributed storage operations. Moreover, we have the ability to evolve the storage layer over time as our internal persistence platform matures.

Architecture & optimizations

Airbnb’s knowledge graph infrastructure provides a managed experience where each tenant (such as the identity graph) operates in an isolated namespace. We built a management service on top of JanusGraph to handle schema enforcement, index management, and schematized Thrift APIs.

To meet Airbnb’s latency requirements, we made several key optimizations to the core JanusGraph engine:

• Optimized transactions: JanusGraph’s default locking can be heavy. We implemented a custom transaction strategy leveraging DynamoDB’s conditional writes and transaction APIs to ensure data integrity with lower overhead.
• Parallel query execution: We improved the getMultiSlices interface to fetch data in parallel, significantly reducing latency for high-fanout queries.
• Observability: We integrated Airbnb’s distributed tracing into our internal fork, closing the observability gap present in the open source version.

This architecture now supports critical use cases across the company, including fraud detection, inventory knowledge graphs, and data lineage.

The migration

We moved from a vendor-provided solution to an internally-built solution based on the open source Apache Tinkerpop graph computing framework, which includes the Gremlin graph query language. The change has delivered considerable improvements in performance and reliability.

Airbnb’s knowledge graph infrastructure architecture

The identity graph service consists of four applications: two for event-based data ingestion and bulk loading, another two for data serving and pre-computation of complex graph queries. For both the internal solution and the previous third-party vendor solution, read and write traffic are isolated in the graph computation engine layer.

Both graph engines also support Gremlin, enabling us to benchmark Gremlin queries side-by-side when serving shadow traffic. After benchmarking, internal graph infrastructure was used to start serving production traffic before the vendor solution was deprecated.

Client-side query optimization

Even though both graph engines support the Gremlin query language, they applied very different optimizations over TinkerPop query steps during the query planning phase. Thus, during the migration, identical Gremlin queries produced significantly different performance between Airbnb’s graph infrastructure and the third party vendor. To address this and enhance the performance of graph queries, optimizations were made on both the JanusGraph side and the client side.

Client-side optimization included a series of query rewriting improvements, including:

• Removal of Path steps: Path steps in Gremlin, such as Path or SimplePath, are not optimized as batched queries within JanusGraph, and many of the queries would fall back to slow, non-batched backend queries, which can occupy large connections in the backend storage thread pool. Thus, the path steps are removed wherever possible, and replaced with a series of conditional queries to ensure the returned results are acyclic.
• Side-effect step optimization: Aggregation within side-effect steps may not be fully optimized within JanusGraph query planning strategy, resulting in non-batched substeps. Thus, side-effect steps were modified to minimize the amount of computation.

Gains from the migration

Integrating the identity graph with the internal infrastructure has delivered meaningful improvements in performance, stability, and scalability:

• Performance: The new, internal solution outperforms the previous third-party vendor in all graph query patterns. This led to huge improvement in the latency of the end-to-end read API, which typically involves multiple graph queries. Significant P99 latency reduction also demonstrates our success in reducing long-tail latency in complex graph queries.
• System stability: The previous vendor’s solution required periodic manual instance reboots to maintain optimal performance.These reboots are no longer needed with the internally hosted solution. Plus, our ability to manage the new solution internally yields shorter response times for incidents and more transparent incident investigations.
• Scalability: The new, internal service supports auto-scaling. During load tests, the write queries per second (QPS) was successfully scaled to ten times the previous solution’s write QPS.

Conclusion and future plans

Airbnb’s knowledge graph has demonstrated significant improvement in both query performance and system stability compared to the third-party vendor identity graph we had been relying on. Enabling multi-step queries in a broader range of queries in JanusGraph has improved overall query performance, while client-side query optimizations helped greatly in shrinking long-tail query latency. Our ability to manage the new system internally enables faster and more transparent incident investigations.

If this type of work interests you, check out some of our open roles.

Acknowledgements

Special thanks to Pawan Rathi, Zach Fein, Cong Zhao, Peter Li, Haiyang Han, Jisheng Liang, Abhishek Ravi, Yi Li, Adam Kocoloski, Kaushik Srinivasan, and everyone else who supported this project. The infrastructure upgrade would not be such a huge success without contributions from these people.

We also want to thank Primus Lam and Rajan Jon for their support in authoring this post during their time at Airbnb.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

We’re excited to announce the 1.0 release of the Viaduct. This release marks a shift from Viaduct being an Airbnb-internal tool that happens to be open source to a true community-driven project with a stable public API. The 1.0 release includes substantial new features and enhancements which we describe in the Viaduct blog.

Viaduct is for platform engineers building a company-wide data API, service owners who want to contribute to a shared graph without spinning up their own server, or engineering organizations that have outgrown a single GraphQL service.

What is Viaduct?

Viaduct is Airbnb’s data-oriented service mesh, a GraphQL-based system that provides a single interface for accessing and interacting with any data source. For years it has supported Airbnb’s data infrastructure, allowing product engineers to access data efficiently and safely, while enabling service owners to decouple implementation details from the API surface.

What is a data-oriented service mesh?

A Viaduct service mesh is defined in terms of a GraphQL schema consisting of:

• Types (and interfaces) describing data managed within your service mesh
• Queries (and subscriptions) providing means to access that data, abstracted from the service entry points that provide the data
• Mutations providing ways to update data, again abstracted from service entry points

Why Viaduct?

Viaduct was built to solve a specific problem faced by most organizations that adopt GraphQL strategically: decentralized development of a central schema.

Why a central schema?

A central schema provides a single, consistent interface to the full range of an organization’s data and capabilities. Instead of every client needing to know which backend service to call, they interact with one unified graph that connects all of an organization’s domains. This makes APIs easier to discover, enables richer cross-domain queries, and provides a consistent place to enforce policies, observability, and schema governance.

Why decentralized development?

A central schema only works if it can evolve quickly. The domain experts who understand each part of the business must be able to design and implement the parts of the schema they know best. A central team cannot own everything, and shouldn’t try to. The challenge is giving teams autonomy over their own domain contributions while preserving the coherence and stability of the shared schema.

Viaduct solves this through multi-tenancy. A shared multi-tenant runtime hosts independently developed and tested tenant modules, each owning a portion of the schema. A team wanting to contribute simply creates a directory for their module, defines their schema definition language (SDL) and resolvers, and they are ready to serve. There is no need to set up or operate a separate GraphQL service, manage router composition, or become experts in GraphQL infrastructure. Teams focus on domain logic; the platform handles execution, scaling, and integration.

Viaduct and GraphQL Federation

We’re frequently asked how Viaduct compares with GraphQL Federation. Both address the same problem — decentralized development of a central schema — but they take different approaches.

Federation distributes development through services. Each team owns and operates its own GraphQL subgraph server; those subgraphs are composed by a federation router into a single unified graph. Viaduct distributes development through modules. A shared multi-tenant runtime hosts tenant modules that define and implement portions of the schema.

Federation distributes development by distributing servers. Viaduct distributes development by distributing modules.

We don’t see Viaduct as an alternative to federation, but as a complement to it. Viaduct can participate as a subgraph within a federated architecture. In a large organization where hundreds of teams contribute to the overall graph, a federated approach requires running hundreds of independent subgraph servers. With Viaduct, organizations can instead run a smaller number of Viaduct instances, each hosting many closely related tenant modules. Federation can then compose those instances into a larger enterprise graph.

Community

The “1.0” designation for Viaduct is a commitment to stability. Until now, Viaduct has evolved rapidly to meet internal needs, often with breaking changes managed through our internal monorepo tooling. Public release required a different approach. We have applied @StableApi, @ExperimentalApi, and @InternalApi annotations across all public surfaces, and we run Kotlin’s binary compatibility validator in CI to catch breaking changes before they ship. Viaduct is now published to Maven Central with automated releases and Dokka-generated API documentation.

We are committed to developing Viaduct in the open. Our intent is to involve the community in major architectural decisions before code is written, not after. Our first public discussion is the Connections RFC on GitHub, and we plan to continue in that direction. Our goal going forward is to be a genuine community project, not simply an internal project that happens to be open source.

Whether you are looking to unify your data layer, contribute to the core engine, or build on top of the graph, now is the time to get involved. Begin with Getting Started →

If this type of work interests you, check out some of our open roles!

GraphQLConf 2026

Heading to GraphQLConf next week? Check out four Viaduct-powered conversations by Airbnb engineers on May 20th.

Speaker: James Bellenger

Time: 3:50–4:15 PM PST

This talk explains how probabilistic testing exposes hidden bugs in complex GraphQL systems — demonstrated by Airbnb’s launch of a new GraphQL engine — and shows how you can use the same approach to harden your own systems.

Speaker: Vickey Yeh

Time: 1:55–2:20 PM PM PST

A look at how Airbnb’s Viaduct system lets each team easily monitor and debug its own code — using built-in ownership tags, automatic alerts/dashboards, and cost-aware tracing — so everyone can treat their part of the shared service as if it were their own.

Speakers: Linquan Zhang and Cetlin Sahin

Time: 2:30–2:55 PM PST

We will cover how we architected our sharding solution and how it improved our operational abilities. You will gain a clear understanding of how our implementation tradeoffs have fared over time, key production insights gathered since rollout, and strategies to evolve a GraphQL gateway towards greater isolation without fragmenting the API surface.

Speaker: Michael Rebello

Time: 3:05–3:30 PM PST

Producing valid and realistic mock data for prototyping and testing has been an unsolved challenge for years. Mock data is tedious to write and maintain, but attempts to improve the process such as random value generation and field stubbing fall short as they lack essential domain context to make test data realistic and meaningful. In this talk, I’ll share how we’ve reimagined GraphQL mocking at Airbnb by combining existing GraphQL infrastructure, rich product and schema context, and LLMs to generate convincing, type-safe mock data simply by adding a directive (@generateMock) to a field or operation.

Whether you’re fine-tuning a single service or running a multi-tenant gateway, these sessions will equip you with practical strategies to build robust, observable, and developer-friendly GraphQL systems. See you on May 20th!

Acknowledgments

Thanks to the entire Viaduct team, and especially Aileen Chen and Raymie Stata, for the tireless work on Viaduct Modern.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

When an incident hits, teams lean on observability to answer the only questions that matter: what’s broken, and why? Monitoring systems are designed to help you answer these questions, and they usually do.

But what happens when your observability stack is dependent on the same systems that are failing? In that moment, the dashboards go dark, alerts stop firing, and the tools meant to guide recovery become part of the outage.

This is an increasingly common challenge as organizations consolidate onto shared platforms like Kubernetes, service meshes, and other common infrastructure components. At Airbnb, where thousands of services rely on shared infrastructure to deliver a reliable experience for guests and hosts, we traced a key reliability risk back to a circular dependency: our metrics pipeline was built on the same systems it was meant to observe.

Reliability is foundational to how we build trust and deliver exceptional experiences for our global community. Airbnb’s core values include being a responsible, reliable host to millions of guests and hosts around the world. Breaking that dependency chain became essential to operating responsibly at scale and sustaining trust.

In this blog post, we’ll walk through how we identified and eliminated dependencies in Airbnb’s metrics platform. We’ll start with why circular dependencies pose such a risk to observability, then dive into the specific design patterns we used to break them across compute, networking, and meta-monitoring. The takeaway is a set of practical approaches you can apply to make your observability stack more reliable than the systems it observes.

The hidden risk: Circular dependencies in observability

Reliable observability isn’t just about collecting data — it’s about ensuring that the observability stack itself is more reliable than the systems it monitors.

At Airbnb, platform teams provide shared infrastructure that lets developers deploy, operate, and run services with minimal friction. Our team relied heavily on those foundations — and that’s where we hit a subtle snag: our observability stack depended on the same systems it was supposed to monitor. In other words, we had circular dependencies.

Fixing this wasn’t optional. Left unsolved, it undercuts a core purpose of observability: enabling engineers to diagnose root causes when something goes wrong. If the system that detects outages relies on the very infrastructure that is failing, visibility disappears exactly when it’s needed most.

Our solution was simple in principle: give every internal customer a redundant, highly available path for collecting metrics. Just as important, we drew a clear line around our responsibilities: we didn’t try to guarantee the availability of metrics for systems that weren’t our internal customers, and we didn’t design for redundancy across those external fault domains.

Isolating compute: Dedicated clusters without the overhead

Among the many stakeholders of our metrics platform, the Airbnb Cloud team was one of the most critical. Airbnb operates Kubernetes clusters for everything from personal dev environments to the production workloads powering Airbnb.com, and our system had to meet the Cloud team’s expectations for metrics availability.

When considering where to run our observability components, we effectively faced two extremes:

1. Run Observability on shared production clusters.
   This minimized our operational overhead, but tightly coupled us to the very applications we needed to monitor.
2. Operate our own Kubernetes clusters.
   This provided full isolation but required deep operational expertise and ongoing maintenance — work the small but mighty Observability team wasn’t eager to take on.

Neither extreme worked. The first introduced circular dependencies, jeopardizing our availability targets; the second imposed undue operational burdens.

Our “just right” solution was to isolate our workloads onto dedicated Kubernetes clusters. These clusters aren’t shared with product or infrastructure applications, but they’re still administered and maintained by the Cloud team. This preserved Kubernetes as a managed foundation while reducing shared failure domains.

To keep this setup reliable, we coordinate changes with the Cloud team so that only one major change lands at a time, and so that changes are validated on lower-priority clusters before reaching operational clusters. This struck the balance we needed: high availability for our internal customers with minimal operational overhead for our team.

Rethinking networking: Breaking free from the service mesh

Observability data is uniquely high-volume. At Airbnb’s scale, we send orders of magnitude more observability traffic than business traffic, which makes networking a key foundation of our observability stack.

Airbnb uses Istio as its service mesh, and while Istio is excellent for many infrastructure benefits, it wasn’t the right fit for our observability workloads. The immediate challenge was obvious: we couldn’t rely on the same data plane for monitoring product and infrastructure applications as for business traffic. That would create a circular dependency — metrics for the data plane would depend on that same data plane to be delivered.

The second issue was more subtle: observability traffic behaves very differently from typical business traffic. It’s far larger and (arguably) demands higher availability and priority. Our service mesh was originally designed around business workloads, not a world where every service continuously pushes telemetry to a central store. Sharing the same transport channel meant that as usage grew, congestion could make metrics unavailable, eroding critical debuggability for both platform engineers and product developers. Worse, telemetry spikes could also consume shared capacity and degrade or disrupt application traffic, directly impacting Airbnb.com availability.

The first major step was to rethink the network path altogether. To break free from the service mesh, we built a custom Layer 7 network ingress layer based on Envoy that load-balances traffic and routes read and write requests to the right backends. Running this proxy independent of the shared compute layer added fault tolerance and shielded our ingest path from service-mesh failures.

You might wonder why we chose to manage our own networking layer but not our own compute layer. For compute, Kubernetes was already a mature, managed foundation operated by the Cloud team, and adding dedicated clusters for observability was a relatively small increment to their existing footprint. The networking layer was different: our service mesh couldn’t cleanly isolate and prioritize observability traffic from business traffic at our scale, and the features we needed — strict prioritization, isolation, and custom routing for telemetry — sat squarely within our team’s domain. Owning this layer gave us the control we wanted and, compared to running Kubernetes ourselves, it was a much more straightforward surface to operate.

Decoupling from the service mesh also unlocked important new capabilities. Airbnb runs over 1,000 services, each mapped to its own tenant in a single, global user space. Our custom load-balancing tier makes this practical: we map each service name to a specific cluster backend, and every request must include a tenant header that informs routing. This header-based routing was a key motivation, relieving clients of complex configuration and ensuring even, predictable load distribution.

Finally, managing our own network layer gives us the flexibility to implement powerful custom features. For example, we can mirror metrics to alternate destinations for testing or enforce fine-grained access controls, which is critical when working with external vendors or specialized use cases. This extensibility continues to pay dividends as the platform evolves.

Monitoring the monitors

Once we’d built resilience into the core layers of our metrics infrastructure, a natural question followed: how do we know when the metrics engine itself is having issues? To answer that, we added another layer whose purpose is to monitor the monitors — a concept often called meta-monitoring.

At Airbnb, we run a separate set of Prometheus instances dedicated to monitoring our observability stack. These Prometheus servers alert us when a component misbehaves. To avoid correlated failures, they run on Kubernetes nodes isolated from the observability stack and in different availability zones. Each Prometheus instance is part of a high‑availability set, as are the corresponding Alertmanagers, and we ensure no Prometheus–Alertmanager pair can land on the same shared infrastructure, further reducing shared fault domains.

This naturally raises the next question: how do we know if the meta-monitoring layer is down? Spinning up yet another monitoring stack would just lead to an infinite regress.

Instead, we use a Dead Man’s Switch — a mechanism that sends a steady signal. The recipient of the signal can assume something is wrong when the signal disappears. In our setup, we maintain an alerting rule that always fires as long as Prometheus is scraping correctly. Alertmanager continuously sends these alerts to an external AWS SNS topic, and a CloudWatch alarm monitors the rate of incoming messages. If they stop — because Prometheus is down, scraping has stalled, Alertmanager can’t send, or something else has degraded — the CloudWatch alarm triggers and on-call is paged.

Together, these components form a robust signal chain that surfaces issues in the meta-monitoring layer itself, helping ensure that our “monitoring of the monitors” is protected against silent failures.

Conclusion

Reliable monitoring requires designing for uncomfortable moments such as partial outages, degraded networks, and failing dependencies — exactly the conditions where traditional observability architectures often go blind. In this post, we covered how we strengthened reliability by eliminating circular dependencies: running observability workloads on dedicated (but managed) Kubernetes clusters, separating telemetry transport from the service mesh with a purpose-built proxy tier, and adding meta-monitoring backed by a dead man’s switch to avoid silent failure.

These approaches generalize well beyond Airbnb. Any organization can improve monitoring reliability by mapping critical dependencies, intentionally isolating failure domains, and ensuring there is always an independent path for the signals that drive paging and incident response. The specific technologies will vary, but the principle holds: treat monitoring as a production system whose availability must exceed that of what it observes. Doing so preserves visibility during incidents, speeds recovery, and helps teams operate services with the confidence their users — and their business — depend on.

If this type of work interests you, check out some of our related positions.

Acknowledgments

Thank you to the Observability team — Callum Jones, Eugene Ma, Natasha Aleksandrova, Rishabh Kumar, Rong Hu, Wei Song, and Yann Ramin — and our partners across the company who helped make this a reality.

We also want to thank Suman Karumuri and Xuan Lu for their support in authoring this post during their time at Airbnb.

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Picture this hypothetical flow: A host submits an insurance claim about their listing to Airbnb. The system needs to validate the claim, run trust and safety checks, assess estimates, process the payout, and send notifications. Halfway through — after the validation passes, but before the payout — the server crashes.

What happens next?

In a traditional architecture, the answer is often “it depends.” Maybe the operation times out and the guest retries, triggering duplicate processing. Maybe partial state corrupts what comes next. And for workflows spanning minutes, hours, or even days, such as our insurance claim example, interruptions are all too likely.

The industry has developed solutions for this, such as dedicated orchestration clusters and cloud-managed workflow services. However, these solutions come with their own costs: operational complexity, infrastructure dependencies, and architectural constraints.

We needed durable execution without the overhead. This article describes how we built Skipper, an embedded workflow engine now powering critical workflows across insurance, payments, media processing, and infrastructure automation at Airbnb.

Why existing solutions fall short

Before building Skipper, we evaluated existing workflow solutions extensively. Each had merits, but none fit our specific constraints.

The external orchestration problem

External orchestration engines are the industry gold standard for durable workflow execution, providing exactly-once semantics and battle-tested reliability. However, they require dedicated infrastructure — a cluster of servers and a persistence layer, along with operational expertise — to maintain. For our highest-criticality, “Tier 0” services (services which directly impact user-facing transactions), adding a new critical dependency was problematic. An orchestration cluster outage would mean every dependent service would lose the ability to start or advance workflows.

Cloud-managed workflow services eliminate operational overhead but introduce vendor lock-in, regulatory and data-handling requirements beyond our own infrastructure, limits on execution, and the same fundamental concern: a critical external dependency.

Meanwhile, homegrown, queue-based systems avoid external dependencies but trade them for bespoke complexity: each team implementing and maintaining its own retry logic, state management, and compensation flows.

The domain logic problem

Beyond the infrastructure problems, we noticed something subtler; and this isn’t unique to Airbnb, it’s an industry-wide pattern. When teams wire up multi-step processes using queues or ad-hoc async plumbing, the domain logic ends up fragmented. A single business workflow, such as processing an insurance claim, gets scattered across queue consumers, scheduled jobs, callback endpoints, and reconciliation scripts. There’s no single place in the code where you can read what the business process actually does. And to make things worse, each of these fragments tangles domain rules together with infrastructure concerns: retry backoff, deduplication checks, timeout handling, async coordination.

Why we built Skipper

Across Airbnb, teams were running into the same problem independently when trying to solve durable execution. In each case, engineers built something bespoke. These solutions worked, but they shared the same issues: they were expensive to build, difficult to test, and each one re-discovered the same edge cases around idempotency, partial failure, and crash recovery. We were paying the cost of solving durable execution over and over again, and every new implementation carried its own set of subtle bugs.

We stepped back and looked at the pattern as a whole. Instead of having every service re-invent this wheel, we decided to build a shared library. Skipper is a workflow engine that can be embedded directly into any service, with succinct ergonomics, no external runtime dependency, and allowing developers to focus on writing domain logic instead of plumbing code.

Skipper doesn’t enforce architectural purity, but its programming model promotes it. By exposing workflows and actions as plain Java/Kotlin classes, with a minimal, annotation-based contract, Skipper enables developers to write business logic that looks like business logic, not framework boilerplate.

Primitives such as conditional waits, durable retries, and signals are available, but they surface through the same straightforward class structure, rather than requiring developers to learn a separate execution model or wire up infrastructure abstractions. The contract stays out of the way: a workflow method reads like the process it represents, and an action method looks like the service call it wraps. And, because domain rules are cohesive in a single class rather than scattered across infrastructure components, it becomes straightforward to test workflows end-to-end.

Here’s what that looks like in practice: a durable, multi-step business process expressed as a single workflow class, with side effects isolated behind actions and one annotation:

// 1) Invoke it like a normal typed call (no codegen client)
val out = workflow<ChargeAndAccept>("reservation:${req.id}").execute(req)

// 2) Define workflow logic as normal-looking Kotlin
class ChargeAndAccept : Workflow() {
  private val billing = actions<BillingActions>()
  private val reservations = actions<ReservationActions>()
  @StateParam var paymentCaptured = false

  @WorkflowMethod
  suspend fun execute(r0: Reservation): Reservation {
    val r1 = billing.charge(r0)                // durable side-effect boundary 
    waitUntil { paymentCaptured }              // durable wait (resumes after restart)
    return reservations.markAccepted(r1)
  }
}

// 3) Side effects live in Actions; one annotation makes it checkpointable
class BillingActions : Actions() {
  @Execute(checkpoint = true)
  suspend fun charge(r: Reservation): Reservation =
    billingApi.chargeAsync(r.id, r.amount).await()
}

Skipper isn’t the first system to offer “write normal code, get durable execution” — other workflow engines do as well — but Skipper’s focus is on removing the adoption friction: fewer required constructs and less setup, so teams that use Java/Kotlin can get to a first durable workflow with minimal ceremony.

Our requirements

We crystallized our requirements through these evaluations:

1. No new critical dependencies: Tier 0 services cannot add single points of failure
2. Leverage existing infrastructure: Use the database the service already depends on
3. Self-service integration: Enable teams to adopt Skipper without dedicated support
4. Simple programming model: Support rapid development and easy maintenance
5. Performance neutrality: The workflow engine shouldn’t constrain host service scalability

These requirements pointed toward an embedded architecture: a library that runs within each service, rather than a central orchestration system.

Design philosophy

Five principles guided Skipper’s design.

Succinct ergonomics. Workflow code should read like the business logic it represents. A workflow that “waits for approval, then processes payment, then sends confirmation” should look just like that in code.

No single point of failure. Skipper runs embedded within each host service. If one service’s workflow processing fails, other services continue independently. There’s no central coordinator that can bring everything down.

Leverage existing dependencies. Skipper stores its state in the same database the host service already uses: either MySQL or Airbnb’s internal Unified Data Store. There’s no separate persistence layer to manage.

Self-service ready. Skipper is a library dependency: add it to your build, provide some configuration, and start defining workflows. No complex setup, no dependencies on a central team.

Performance-neutral. Skipper uses separate thread pools, configurable concurrency limits, and efficient hibernation patterns to coexist peacefully with latency-sensitive request handling.

How Skipper works

Skipper’s programming model centers on two abstractions: Workflows and Actions. Workflows define the orchestration logic: what happens in what order, and under what conditions. Actions encapsulate individual operations such as API calls, database updates, notifications. Each action is automatically checkpointed, so the result of an action survives crashes and restarts.

A workflow in practice

On the Airbnb platform, hosts submit photos of their property listings that go through a review process before the listing can go live. In this fictional example, a host submits their photos to be approved for quality and accuracy, then the listing is activated; then, if all looks good, the host is notified. This simplified fictional workflow demonstrates the core concepts of Skipper in action:

class ListingPublicationWorkflow : Workflow() {
    private val actions = actions<ListingActions>();

    @StateField val photosApproved: Boolean? = false;

    @WorkflowMethod
    suspend fun publishListing(submission: ListingSubmission): PublicationResult {
        // Submit photos for review
        val reviewId = actions.submitPhotosForReview(submission.getListingId());
        // Wait for photo review completion (manual or automated)
        val reviewTimedOut = waitUntil(() -> photosApproved != null,
Duration.ofHours(24));
        if (reviewTimedOut || !photosApproved) {
            actions.notifyHost(submission.getHostId(), "Photos require updates");
            return PublicationResult.rejected("Photo review failed");
        }
        // Publish the listing
        actions.activateListing(submission.getListingId());
        actions.notifyHost(submission.getHostId(), "Your listing is now live!");
        return PublicationResult.success(submission.getListingId());
    }

    @SignalMethod
    fun completePhotoReview(approved: Boolean) {
        photosApproved = approved;
    }
}

This code reads naturally: submit photos, wait for photo review, publish. There’s no retry logic, queue management, or async coordination visible in the workflow itself. Signals (@SignalMethod) let external events push data into a running workflow, updating @StateField fields that the workflow’s waitUntil conditions evaluate against.

The following diagram shows how these components interact at runtime:

Because domain logic is cohesive in a single class, testing is equally straightforward; no queues to set up, no infrastructure to mock:

class ListingPublicationTest : SkipperTest() {
    @Test
    fun testListingPublication() {
        val workflow = workflowBuilder(ListingPublicationWorkflow::class.java).build();
        workflow.publishListing(ListingSubmission("listing-123", "host-456", true));
        helper.expectWorkflowToWait();       // workflow waits for photo review
        workflow.completePhotoReview(true);  // photos approved
        helper.waitForWorkflowToComplete();
        val result = workflow.getResult();
        assertEquals(PublicationResult.Status.SUCCESS, result.getStatus());
    }
}

Replay: How durability works

Skipper achieves durability through its replay mechanism with checkpointed actions. When a workflow starts, Skipper executes the workflow method and checkpoints each action’s result to the database. If the workflow needs to wait (via waitUntil), Skipper persists the current state and the workflow hibernates, consuming no compute resources.

When conditions change — a signal arrives, a timer expires, or the service restarts — Skipper replays the workflow method from the beginning. Previously executed actions don’t re-execute; they return their checkpointed results instantly. The workflow picks up from where it left off.

Unlike event-sourced orchestration systems that reconstruct state by replaying an entire event history, Skipper persists state fields directly. There’s no event log to replay, just current state and checkpointed action results. This makes execution leaner, especially for workflows with many signals or long histories, though it trades some auditability for that efficiency. The next section explains how this also translates to minimal runtime overhead in the most common case.

The happy path: Getting out of the way

Most workflow engines impose overhead on every execution, even when nothing goes wrong. External orchestration engines require network round-trips to a central cluster for every activity invocation — the worker executes the activity, then calls back to the cluster to persist the result before the workflow can advance. This is fundamental to their architecture; the cluster is the coordinator.

Skipper takes a different approach. When a workflow starts, two things happen at the database level: the workflow instance is created, and a delayed timeout task is scheduled as a durability guarantee. Then the workflow executes entirely in-process. Actions run as normal method calls on an in-memory execution queue on a dedicated thread pool, checkpoints are batched, and the workflow can run to completion without any further coordination.

The delayed task acts as a safety net: if the process crashes mid-execution, the persistent scheduler picks up the workflow after a lease period expires and replays it. If the workflow completes normally, the timeout task fires harmlessly and is discarded.

The result is that, in the happy case: the workflow runs and all actions succeed, with no crashes; Skipper adds very little overhead (just a few database writes). The workflow executes almost as if there were no workflow engine at all. The engine is only called into action when something goes wrong: a crash triggers a replay, a waitUntil hibernates the workflow, or an error invokes compensation.

This is what makes Skipper viable for latency-sensitive, high-throughput services — durability is guaranteed, but you only pay for it when you need it.

The determinism requirement

Replay imposes one key constraint: workflow methods must be deterministic. Given the same inputs, checkpointed action results, and state fields, the workflow must make the same decisions and call actions in the same order. All side effects, such as API calls, time-dependent logic, and randomness, belong in actions, never in the workflow method directly.

Error handling and compensation

Skipper distinguishes between retryable errors (temporary failures such as network timeouts, which are retried automatically, with configurable backoff) and non-retryable errors (permanent failures such as a declined card, which halt the workflow’s normal flow).

When a workflow fails partway through, you’re left in an awkward state: some actions completed successfully, but the workflow as a whole didn’t. For example, a listing might pass content validation and quality checks, but then fail during photo review submission.

The content validation result is now dangling; the work was done in preparation for a publication that now isn’t going to happen. In traditional architectures, teams handle this with ad-hoc cleanup logic: a scheduled job that scans for orphaned records, a reconciliation script that runs nightly, or manual intervention. These approaches are fragile, often delayed, and easy to forget as the workflow evolves.

We made compensation a first-class primitive to prevent workflow code from getting cluttered with error-handling plumbing that obscures the business logic. The @Compensate annotation lets developers pair each action with a method that undoes its effect. If an action fails after prior actions have succeeded, Skipper automatically executes compensation methods in reverse order (releasing held inventory, refunding charges, reverting state changes), walking the system back to a consistent state. Developers express what “undo” means for each action; Skipper handles the orchestration of when and in what order the undos run. The result is eventual consistency without distributed transactions, and workflow code that stays focused on the business process rather than cleanup choreography.

Key tradeoffs

What we gained

No infrastructure to manage: Skipper runs inside your service. No separate cluster to deploy, monitor, or page on.

Uses existing dependencies: If your service depends on MySQL, Skipper uses that MySQL instance. If it uses Airbnb’s Unified Data Store, Skipper uses that. No new data stores or failure modes.

Simple programming model: Workflows are Java/Kotlin classes. Actions are method calls. Developers use familiar tools and debugging workflows.

Independent scaling: Each service manages its own workflow processing. High load on one service’s workflows doesn’t affect others.

What we traded off

Determinism requirement: The replay model requires deterministic workflow methods, which can be unintuitive for developers new to the pattern.

At-least-once execution: Actions may execute more than once in edge cases (crash after execution but before checkpoint). Actions should be idempotent.

Evolution complexity: Changing a workflow’s structure can break in-flight workflows. Teams need versioning strategies for workflow evolution.

These tradeoffs are inherent to the embedded model; teams needing cross-language support or cross-service orchestration may find a dedicated orchestration system more appropriate.

Production impact

Skipper has been running in production for more than a year, powering 15+ use cases across insurance, payments, media, infrastructure, incentives, and wallet teams. Use cases include multi-step claim processing, policy lifecycle management, resilient transaction orchestration, and scheduled financial operations that can span days or weeks. The Media Foundation team uses Skipper to coordinate video processing pipelines — validation, transcoding, thumbnail generation — surviving pod restarts across multi-hour jobs. Infrastructure teams rely on it for durable Flink job lifecycle management and reliable data pipeline CRUD operations. Across all domains, Skipper guarantees that every workflow reaches a terminal state, even through infrastructure failures, deployments, and infrastructure disruptions. At peak, Skipper has scaled to 10,000 workflows per second on Amazon DynamoDB, enabled by its lean execution model.

Lessons learned

What worked well. The embedded model reduces operational burden dramatically. Teams adopt Skipper without new infrastructure, deployment procedures, or on-call rotations. Supporting multiple storage backends (MySQL and our internal UDS) means that no team is blocked by database incompatibility. And the simple API (“actions are checkpointed, workflows must be deterministic”) accelerated learning, with workflow code reading like straightforward business logic.

What we’d reconsider. Workflow evolution remains the biggest friction point. While we have versioning patterns (create new method versions, migrate traffic, deprecate old versions), better tooling — automated compatibility checking, migration assistants, runtime versioning support — would smooth the experience. Debugging replayed workflows also requires mental model adjustment: engineers must understand that log timestamps and call sequences reflect replays, not original execution. Better observability tooling, particularly replay visualization, would help.

Conclusion

Durable workflow execution is a fundamental capability for reliable distributed systems. Skipper represents a specific point in the design space: an embedded engine that trades centralized orchestration for operational simplicity, running inside services rather than alongside them, using existing databases, and providing a straightforward Java/Kotlin programming model.

This approach won’t fit every situation. But for services seeking durable execution without infrastructure overhead, particularly those where minimizing dependencies is paramount, the embedded model offers compelling advantages. The core insight — that replay-based execution with checkpointed actions can provide durability without coordination services — generalizes beyond Airbnb’s implementation to anywhere you’re building long-running, failure-prone workflows.

If this type of work interests you, check out some of our open roles.

Acknowledgements

Skipper wouldn’t have been possible without the support and contributions of many people across different teams. Special thanks to Navjot Sidhu, Musaab At-Taras, Mini Atwal, Gary Leung, Harshit Gupta, Alex Zhang, Gerum Haile.

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement

Airbnb is no exception. As our products and infrastructure have evolved, each new feature and each new incident has added another layer of instrumentation. Unsurprisingly, we were generating 1.3 billion active time series on average, at 50 million samples every second.

When we made the decision to move from a hosted metrics provider to an internally operated solution, the sheer scale introduced a set of engineering challenges. Our initial mandate was straightforward in principle but challenging in practice: persisting and serving this data performantly.

This blog post walks through several of the key challenges we encountered while building a system capable of operating at this speed and this scale.

Navigating tenancy

Isolating read and writes

There were several possible approaches for organizing tenants. We considered mapping tenants by team, but this was discarded because team ownership of applications changes frequently. With roughly 1,000 services at Airbnb, assigning a tenant to each service or process offered a more logical and stable grouping. This approach enables precise attribution of metric growth to individual applications and also lays the groundwork for future chargeback mechanisms. Key write and read guardrails were established per application, e.g. the amount/rate of metrics ingested, total number of rules, evaluation interval/offset etc.

Shuffle sharding

Shuffle sharding is a technique that isolates different tenants’ workloads and gives each tenant a single-tenant experience in a shared cluster. This improves fault tolerance and isolates failures for both read and writes in scenarios such as node outages. When incoming metrics data is received, shuffle sharding ensures that each tenant writes only to a subset of storage nodes. Similarly, when a query comes in via Grafana, shuffle sharding randomly selects a subset of query workers (the shuffled set). This ensures that no single tenant can overwhelm the entire query layer or fleet of ingesters. The diagram below illustrates that if there is a DDoS or other attack from application A, the first and second shards may go down, impacting B & C, but the data of other applications (D & E) are still preserved in the third and fourth shard.

Strategizing operational aspects

Shifting to a multi-tenant architecture forced us to handle multiple configurations per tenant, leading to significant operational complexity for the team. Key challenges which were identified:

• Tenant onboarding involved numerous manual steps across multiple components and required a series of code changes and deployments, often consuming a lot of time
• It was unclear which configuration parameters needed adjustment when a tenant approached or exceeded limits.

These challenges were solved with a consolidated control plane. New tenants were automatically onboarded by monitoring new services’ creation, and updates were automatic upon configuration changes, allowing a single deployment for any applicable changes. The second challenge was addressed by exposing only the necessary limits and deriving other limits; e.g., series limits were exposed, but ingestion rate/ingestion burst size was derived based on the series limit.

Observability at scale

Our initial effort to set out and research requirements identified challenges that our effort would have to meet. As a first step in meeting them, we took steps to ensure that we could run single clusters with a high degree of reliability.

Observability requirements and challenges

Our assessment of the existing vendor-based observability backend provided us with a clear understanding of the active time series volume and ingestion rates. As we set out to build a reliable and scalable metrics backend, some key requirements identified were:

• The system should be capable of handling over 50 million samples per second and 1.3 billion active timeseries.
• It should support up to 10,000 dashboards and 500,000 alerts, while maintaining a p99 query execution time under 30 seconds.

To validate these requirements, we deployed shadow clusters using production traffic. This surfaced a number of challenges:

• Reliability issues in both write and read components, primarily due to unpredictable demand (such as spikes in active timeseries) and the absence of robust query guardrails.
• Compaction, which optimizes long-term storage by merging small and fragmented time series data, incurred significant delays for large tenants. This adversely impacted query performance and resulted in excessive resource consumption when accessing historical data.
• Slow query performance when requests involve a large payload (e.g, more than 5,000 series or when 500MB+ data is being processed). The absolute number may vary depending on the querying system.
• Degraded observability across all tenants if the cluster experiences issues, resulting in significant risk of “flying blind” (reduced observability) for these tenants.

Addressing these challenges required a strategic shift: our first focus was to improve the reliability of a single cluster. Building on that, we are moving towards a multi-cluster architecture to establish multiple failure domains and minimize the blast radius of issues, thereby enhancing overall system resiliency.

Running a single cluster reliably

We started with stabilizing writes, followed by reads and compaction.

Writes: We started with benchmarking write components to figure out resource usage based on metrics ingested per second. This helped in capacity planning and component sizing and setting per-replica limits based on ingestion rate and inflight requests.

Write guardrails were set, starting with the maximum number of time series emitted for a tenant. We began with a one-month lookback period, which was continuously adjusted over time.

Reads and compaction: As compared to writes, benchmarking read components presented unique challenges due to query payload variability. We used query sharding to normalize the reads on query workers. Read guardrails, such as limits on the number of fetched series/chunks per query, were set on a per-tenant basis to ensure that a few bad queries would not cause outages. The evaluation query path was isolated from the ad hoc query (dashboards, continuous integration jobs) path due to varied criticality. Autoscaling was enabled for read components, allowing dynamic fleet scaling in response to variable throughput patterns throughout the day. For very large tenants, compaction workloads were sharded, with each worker processing up to eight million series, ensuring data being read is always compacted.

Both writes and reads had stateful components which were made zone-aware and were deployed in three zones to be more fault tolerant during a zonal outage, and less vulnerable to outages in events such as node rotation/deployments.

The outcome

Per-replica limits provided actionable scaling signals for fleet management. Tenant-level controls shielded the system from potentially disruptive behavior. Multi-zone deployments enhanced both fault tolerance and deployment agility. With reliable single-cluster deployment, the strategy was to move to a multi-cluster architecture to achieve a reduced blast radius and to gain flexibility to launch in different regions, which is covered in the next section.

Multi-cluster environment and federation

Due to blast radius and flexibility concerns, we adopted a multi-cluster architecture approach in which we distributed tenants across multiple clusters. Clusterization strategy involved creating dedicated clusters for specialized workloads (such as compute and mesh infrastructure) and multiple application clusters, ensuring that failures in one area did not affect others. Once multiple clusters were created and their respective workloads onboarded, we needed to establish a rollout strategy that followed a progressive approach: starting with test and internal clusters, advancing through application clusters, and finally deploying infrastructure clusters to achieve over 99.9% availability and high reliability. This ensured that workloads are sequenced based on criticality, as shown below:

However, there were multiple concerns which came up as part of adopting a multi-cluster architecture:

• Complex metrics discovery and querying: Metrics discovery became increasingly challenging as we introduced the additional complexity of routing reads to the correct cluster, on top of supporting multiple tenants. Many use cases required joining application metrics with host or client metrics, which were sometimes attributed to various clusters, making querying technically demanding.
• Operational overhead: Managing numerous clusters meant higher operational complexity, with more resources needed for configuration, maintenance, and monitoring.

We addressed these concerns by building tooling to manage tenant cluster mapping, which was a source of truth for all the components that needed cluster-level awareness. Deployment of stateful apps was automated using Grafana Kubernetes OSS roll-out operators, enabling coordinated, multi-AZ rollouts across StatefulSets within a namespace. This replaced a manual, sequential process that took days, and also reduced configuration drift across clusters. Seamless deployment also helped in reducing configuration drift across clusters.

We leveraged the Promxy OSS project, which is a proxy over Prometheus, and added some custom functionality such as native histogram support and. query fanout optimization. These enhancements enable cross-cluster querying and alerting, tailored to Airbnb’s needs.

Key learnings

Moving to a multi-cluster architecture, along with seamless deployment strategies, have been instrumental in addressing blast radius concerns and ensuring platform resilience. Below are selected key learnings, some of which we discovered as we started adopting a multi-cluster approach, and some which we discovered later through varied query patterns from customers:

• Cross-cluster querying cost: Federated queries are significantly more resource-intensive, typically 5–10x costlier than queries within a single cluster. We encountered multiple scenarios in which a few sets of expensive queries were enough to cause read reliability issues across multiple clusters. This experience led us to adjust aspects of tenant consolidation, particularly in relation to hot read patterns.
• Deployment consistency: When using a single-cluster approach, stateful apps were deployed manually, which was operationally intensive as we steered toward multi-cluster deployments. We spent a good deal of time in making OSS Kubernetes rollout operators compatible with Airbnb cloud infra, which involves respecting strict pod disruption budget requirements while rollouts and other node operations are happening. We use automation and standardized deployments, which help to prevent configuration drift and maintain reliability.
• Cluster management philosophy: Our single-cluster scaling and stabilization work enabled clusters to self-tune. This approach allowed us to add or replace clusters with minimal additional operational overhead, effectively allowing us to treat clusters as cattle, not pets.

Conclusion

Building a reliable, large-scale observability platform at Airbnb has been as much about architecture and operations as it has been about culture and expectations. We began with a straightforward mandate: persist and serve billions of time series at high throughput. We quickly discovered that true reliability required us to rethink tenancy, failure domains, and dependencies end-to-end.

The work is ongoing. We’re actively exploring ways to further reduce metric volume, optimize cross-cluster querying, and simplify the developer experience, while continuing to scale with Airbnb’s growth. But the principles we’ve leaned on — isolation, automation, guardrails, and independent paths for critical signals — will continue to guide how we evolve the system.

Ultimately, observability at scale is not just about having the most advanced tools or sophisticated dashboards. It’s about designing scalable systems that can clearly communicate what they are doing, why they are doing it, and when something goes wrong. It establishes a continuous feedback loop between systems and teams, empowering rapid learning and ongoing improvement, enabling us to uphold our responsibility to guests and Hosts around the world.

Does this type of work interest you? Check out our open roles.

Acknowledgments

Thank you to the Observability team — Abdurrahman Allawala, Callum Jones, Eugene Ma, Natasha Aleksandrova, Rong Hu, Wei Song, and Yann Ramin— who helped in building this storage system. We would also like to thank the Cloud Infrastructure, Cost, and other partner teams for their invaluable collaboration throughout this project.

We also want to thank Suman Karumuri and Xuan Lu for their support in authoring this post during their time at Airbnb.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

At Airbnb, our hosts and guests form the heart of our community. As shared by CEO Brian Chesky, we’re evolving into a more social ecosystem. Airbnb Experiences now highlight the people involved as much as the activity. Guests can see Who’s going on an Experience, message co-guests directly, and view people they’ve met through the Connections section in their Airbnb profile. Guests are able to choose to share their profile for each new Experience. Guests who choose not to share their profile will not have their photo or profile info shared with others outside of their travel group before or during the experience. Our goal is to foster meaningful connections while giving guests control over their privacy.

In this post, we’ll share how we built new social features with privacy by design at their core. You’ll learn about our approach to user privacy, the technical decisions we made, and how we’re empowering guests to control their visibility every step of the way.

👩‍💼 Users vs. Profiles: Why we separate them

At Airbnb, building trust while protecting privacy is fundamental. To achieve this, we’ve made a clear distinction between the concepts of User and Profile.

User represents the complete, internal record we hold about an Airbnb user, including names, email addresses, phone numbers, and account details. This is the information we collect and use when providing our services. Whereas a Profile includes a subset of information about a User and is their public-facing representation. The information displayed on a Profile varies based on whether the user is a host, guest, or both. Users can choose to add as much or little to their profile as they see fit.

For example, as a guest, you might need your host’s phone number for check-in, but other guests reading your review after the stay shouldn’t see any of your contact info.

One user can have multiple profiles. For example, we have Host Profiles for prospective guests to learn more about the host they’ll be staying with.

We also have Guest Profiles for hosts to learn more about the guests they will be hosting.

We now also create Experience-specific Guest Profiles, which manages how a guest’s profile information is shown to other guests on that experience outside their travel group. If a guest chooses not to share their profile, their profile will not be viewable by others looking to book the Experience. Only their first name will be visible to other guests on the Experience outside of their travel group.

🆔 How we enable this: Separate identifiers

To deliver this context-aware experience, we’ve introduced two distinct types of identifiers:

• A User ID that represents the internal user entity.
• A Profile ID that identifies how that user appears in a specific context.

Each user will only have one User ID, but could have multiple Profile IDs that are used in different contexts. By decoupling these, we enable:

• Context-awareness: Each profile is visible only where it’s relevant.
• Flexible representation: Guests can choose how they appear to other guests on each Experience.
• Privacy controls: By decoupling the User ID and Profile ID, we put mitigations in place to make it more challenging to link profiles across contexts.

Ultimately, this empowers guests to control when their profile data is shared with other guests and hosts, and keeps identity management simple and intuitive.

For example, let’s say Marie chooses to attend an Airbnb Experience called “Pasta Making with Nonna” and decides to remain private. We will create Profile A for Marie, associated with “Pasta Making with Nonna,” which will only surface her first name and will not include her profile photo. If Marie also attends an Airbnb Experience called “Goat Yoga”, we will create a separate Profile B for her associated with “Goat Yoga.” If Marie opts-in to social features on the “Goat Yoga” Airbnb Experience, other co-guests will be able to see her profile photo, guest stats, and other “About me” information, which she can curate from her Profile edit page.

If Alex also attends the Airbnb Experience “Goat Yoga,” he will see Marie’s full Profile B. If Alex happens to browse “Pasta Making with Nonna”, however, he won’t see Marie’s profile photo in the “Who’s going” list. As a result, Alex will not know that Marie from the “Goat Yoga” Airbnb Experience will be attending the “Pasta Making with Nonna” Airbnb Experience as well.

As another example, say Alex is a host of a mountain cabin in the Swiss Alps. When he attends the “Goat Yoga” Experience as a guest, his host profile remains separate from his guest profile for that Experience. This means that fellow guests like Marie won’t be able to tell that Alex also hosts a cabin in the Alps, because Alex’s host profile and guest profile are not linked. If Marie later searches for places to stay in the Swiss Alps, she might come across Alex’s cabin listing. However, if Alex has chosen to remain private on the “Goat Yoga” Airbnb Experience, Marie will only see his first name, and won’t be able to know that his guest profile on “Goat Yoga” and his host profile for the mountain cabin represent the same person.

🔐 Permissions

Airbnb users interact with a range of people: fellow travelers, hosts, Airbnb support personnel, and more. Each interaction requires the right privacy boundaries. We use least-privileged access to ensure everyone sees only the data they need.

To manage these permissions, we use Himeji, our in-house authorization system. Himeji enforces access controls at the data layer, ensuring privacy and security beyond just the user interface. One of Himeji’s key optimizations is its ability to perform configurable relation denormalization at write time, when profile information or permissions change. This makes permission checks at read time extremely fast and scalable, enabling users to have a seamless experience even as privacy needs grow more complex.

🔄 Implementation

In order to launch, work was needed to ensure that Airbnb’s platform utilized the right identifier in the right context.

1. Automated auditing

We developed Python scripts to audit the codebase, searching for known patterns associated with user data access. This gave us a comprehensive list of candidate locations.

2. Determining team ownership

Our scripts mapped each finding to the owning team via the directory structure. This let us assign migration work directly and efficiently.

3. Manual review for context

Code owners manually reviewed findings, determining whether each instance was internal-only or externally used. This hands-on review layer was critical for accuracy and confidence.

4. AI-powered refactoring

We leveraged AI-powered refactoring tools to suggest code changes based on our audit findings. However, engineers always remained in the loop by reviewing, refining, and applying code updates, which ensured correctness and protected nuanced business logic.

5. Company-wide collaboration

Perhaps the most important ingredient was company-wide alignment. Teams across Airbnb (engineering, product, privacy, legal, and beyond) came together with a shared mission. This collective commitment ensured prioritization, smooth coordination, and ultimately, a successful migration.

🧰 Type safety and migration quality

Strong typing and automated tests were our safety net. We made sure profile IDs and user IDs couldn’t be mixed up accidentally. Code reviews, linters, and type checks enforced boundaries. Progress was tracked in a shared hub, keeping everyone aligned and accountable.

🎯 Our privacy principles

As Airbnb becomes more social, guest privacy stays at the heart of everything we build. Our new context-aware profile IDs lay the groundwork for future features without compromising trust and reflects our commitment to privacy in our Privacy Principles.

If this type of work interests you, check out some of our related positions.

Acknowledgments

It takes a village to build a robust privacy-oriented infrastructure. Special thanks to:

• Usman Abbasi
• Ansuman Acharya
• Matt Byington
• Simon Cao
• Ananth Dandibhotla
• Matt Gadda
• David Haggerty
• Weiwei Huo
• Jasmine Lee
• Houkun Li
• Hugh McCauley
• Margot Nack
• Ezlie Nguyen
• Ashley Quitoriano
• Omar Cruz Salgado
• Evelyn Shen
• Jordan Smith
• Andrew Sorensen
• Yupeng Yan
• Ximin Zhang

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

But for our project, this approach wasn’t straightforward: most of our metrics were instrumented with StatsD libraries, OpenTelemetry was gaining traction, and our new storage system was based on Prometheus. We were left with a lot of open questions. Where do we fork the metrics? Should we adopt OpenTelemetry? Do our metrics work well with Prometheus? The task of collecting metrics required us to answer these questions and rethink our metrics infrastructure.

Instrumentation and collection

Our services originally used the StatsD protocol, with a Veneur sidecar collecting and forwarding metrics to our vendor.

For the migration away from this architecture, we defined a clear set of supported and preferred instrumentation protocols. OpenTelemetry Protocol (OTLP) is now recommended for internal services, while Prometheus is preferred for OSS workloads. StatsD (DogStatsD format) remains as a fallback for legacy paths or applications that are difficult to update / instrument.

This led us to adopt the vendor-neutral OpenTelemetry Collector (OTel Collector).

Dual-write migration approach

Roughly 40% of our services use a shared platform-maintained metrics library, which was the logical starting point for OTLP adoption and enabled broad rollout of OTLP emission across those services. By enabling this library to dual-emit metrics, StatsD for the existing pipeline and OTLP for the new OTel Collector, we made broad migration progress with low friction.

The collection pipeline during migration:

Benefits realized by moving to OTLP

Moving to OTLP offered significant benefits over remaining on StatsD:

• JVM profiling data shows that CPU time spent on metrics processing dropped from 10% to under 1% of total CPU samples in production after migrating from StatsD to OTLP.
• OTLP provides greater reliability than StatsD, which relies on UDP and is more vulnerable to packet loss under high throughput.
• Emitting OTLP natively removed the need for an intermediate StatsD-to-OTLP translation step inside the OTel Collector.
• Because our backend is Prometheus-based, remaining on StatsD would have restricted users to classic histograms. OTLP enabled us to adopt Prometheus-native capabilities such as exponential histograms with full fidelity.
• OTLP is a CNCF-sponsored, vendor-neutral protocol and the de facto industry standard for telemetry. Moving to OTLP positions us to support future observability use cases without protocol barriers.

A challenge encountered with OTLP

While most services handled the dual-write OTLP + StatsD mode without issues, our highest-volume metric emitters (many of which are also among our most critical services) experienced significant regressions in performance after enabling OTLP. In particular, these services began to suffer from memory pressure, increased garbage collection activity, and heap growth.

Through investigation, we connected the performance issues to the fact that those services emit very high-cardinality metrics (on the order of 10K+ samples per second per instance). To mitigate this regression, we updated our common metrics library configuration to use delta temporality (AggregationTemporalitySelector.deltaPreferred()) for a select set of services. That change reduces the in-process memory burden, because delta temporality avoids retaining full state for all metric-label combinations between exports.

Based on our observations, we found that switching to delta temporality was only necessary for services emitting extremely high metric volume, so we left the rest of the services using default cumulative mode. That said, with delta metrics, unexpected failures result in visible gaps in the data, a trade-off we accepted in exchange for reduced memory usage and improved stability. With cumulative metrics, those same failures typically appear as a larger jump between data points rather than a gap.

The final collection picture

With OTLP now serving as the primary data flow and the fallback StatsD path maintained only where necessary, we arrived at a consistent and future-ready collection architecture. The OTel Collector anchors the pipeline, feeding directly into our Prometheus-based storage backend, with the legacy path available for compatibility.

This groundwork was essential before tackling more complex challenges like aggregation.

The diagram illustrates this final design and the complete flow of metrics through the system.

Streaming aggregation

To keep costs under control, our previous pipeline used an internal fork of Veneur to aggregate away instance-level labels (e.g. pod, hostname) instead of submitting the raw metrics directly to our vendor. In the Prometheus world, we needed a similar aggregation step.

Rejected options

There were several options that we considered using, but discarded:

• Keep using Veneur: Continuing with Veneur required a significant rewrite to support Prometheus’ protocol and data model, and the fork was an ongoing maintenance burden.
• Recording rules: While recording rules are great for improving query performance, they require storing the raw data in the TSDB first, which undermines our cost saving goal.
• m3aggregator: Although battle-tested at other companies, the architecture seemed more complex than we wanted.
• OpenTelemetry Collector: Lack of support for metric aggregation, despite proposals.
• Vector: no built-in scaling capabilities. Written in Rust, which is not widely adopted at Airbnb.

Choosing vmagent

We ended up looking into vmagent from VictoriaMetrics.

Pros:

• Supports streaming aggregation for Prometheus metrics
• Supports sharding, providing a way to scale horizontally
• Documentation is extremely user-friendly and easy to set up
• It has a small codebase (~10K LOC), so it’s easy to understand and modify if needed.

Our architecture has two layers of vmagents: routers and aggregators. Routers are stateless and shard metrics by consistently hashing all labels except the ones you want to aggregate away (e.g. “pod”, “host”). Aggregators do the aggregation (i.e. summing over counters) and maintain the in-memory state of outputs (i.e. the current running total). On the command line, each router is passed a static list of aggregator hostnames (Kubernetes gives StatefulSet pods stable network identity). This simple approach avoids additional service discovery dependencies and keeps the metrics sharding consistent between pods.

Once we figured out how to scale vmagent, we made a few internal customizations to support our metrics environment (including native histogram support and Mimir-style multitenancy). More generic changes were contributed back upstream (5931, 5938, 5990).

The outcome

We achieved our goal of aggregating metrics: our single production cluster scaled out to hundreds of aggregators, ingesting over 100 million samples per second.

In addition to reducing the overall cost by an order of magnitude, our new aggregator pipeline provides another benefit. Because it’s centralized and processes ALL our metrics, it’s a convenient and powerful tool for other types of transformations. When users deploy a bad instrumentation change, it’s handy to drop problematic metrics. When users temporarily need raw metrics, we use it to dual-emit raw metrics on the fly.

Sparse counters

After migrating to our new Prometheus-based metrics pipeline, we noticed our PromQL queries over certain counters were consistently undercounting compared to our previous vendor.

Problems with rate()

The root cause lies in how Prometheus computes rates of change from cumulative counters. In a StatsD monitoring system, a data point measures the delta observed during a flush window (typically 10 seconds). In Prometheus, a data point measures the cumulative count starting from zero. In practice, you’ll use something like rate() or increase() to derive deltas, which are more meaningful to graph.

However, there is an edge case: when a counter is created, it may also be incremented at the same time.

Consider a counter that increments once and resets (due to pod restart) before it can increment again:

1 1 1 1 1 - - - 1 
          ↑ reset

Each time the counter resets, the increment is lost before PromQL’s rate() function has a chance to derive a delta.

Initially, we anticipated that the edge case would have minimal impact, given Prometheus’s widespread adoption and proven reliability in diverse environments. However, as we migrated more users, we started seeing this issue more frequently, and it stalled migration. Airbnb systems have a tendency to generate many low-rate counters with high dimensionality. A counter tracking requests per currency per user per region might see any given data series increment only a handful of times a day, and these updates can be lost due to the issue with rate(). Yet, these were often critical metrics that product teams needed to monitor accurately.

Rejected solutions

Before arriving at our final approach, we considered several workarounds — each with significant drawbacks.

• Pre-initialize all counters to zero: Not practical given the number of callsites, in addition to not being able to predict all label combinations ahead of time.
• Ask teams to use logs for exact counts: Logs and metrics are separate systems and serve different use cases. We needed real-time dashboards and alerting on metrics. Telling teams to restructure their observability around our limitations wasn’t acceptable.
• Emit gauges instead of counters: Internally, Prometheus doesn’t distinguish between gauges and counters. However, by doing this, we’d be swimming upstream against Prometheus conventions.
• Add hacky PromQL adjustments: We could have asked users to pad their queries with various hacks. But this would push complexity onto every team writing dashboards and alerts.

Solution: zero injection

Clearly, the solution had to be transparent and systematic. The migration was already demanding a lot from our users, and it wasn’t reasonable to expect our team to modify every callsite ourselves.

The idea was simple: we needed to seamlessly inject zeroes in our metric stream. Our aggregation tier was already processing all our metrics in a centralized location, so it was the perfect spot to implement the change.

We implemented an Aggregator tweak: when we flush an aggregated count for the first time, flush a zero instead of the actual running total.

Consider a counter that increments by one every flush. Here’s what this looks like in practice:

By injecting the synthetic zero, all counters are implicitly initialized to zero to match Prometheus counter semantics. The delayed flush is also important, it ensures the zero’s timestamp doesn’t conflict with any previous samples. There is one downside: the initial increment will lag by one flush interval, which isn’t a big deal in practice.

This approach required minimal changes to vmagent, kept the fix invisible to end users, and solved the problem at the source rather than papering over it downstream.

Conclusion

In this blog post, we talked about how we collected all existing StatsD metrics into a new, Prometheus-based metric pipeline.

We implemented a dual-write instrumentation strategy, adopting the OpenTelemetry Protocol (OTLP) while maintaining support for StatsD. This switch yielded performance gains and unlocked the use of features such as native histograms.

To manage costs and scale, we introduced a centralized, two-layer streaming aggregation pipeline using a sharded vmagent setup. Finally, we solved the issue of consistently undercounted sparse counters by implementing a transparent “zero injection” technique within our aggregation tier.

Our work demonstrates a practical, production-tested blueprint for migrating a massive, high-volume metrics pipeline to modern, open-source standards such as OpenTelemetry and Prometheus, all while maintaining data accuracy and cost efficiency.

If this type of work interests you, check out our open roles.

Acknowledgements

Thank you to the Observability team — Abdurrahman Allawala, Rong Hu, Callum Jones, Rishabh Kumar, Wei Song, and Yann Ramin — and our partners across the company who helped make this a reality.

We especially want to thank Helen Lee and Vlad Kostyukov for their support on OpenTelemetry instrumentation.

We also want to thank Suman Karumuri and Xuan Lu for their support in authoring this post during their time at Airbnb.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Joining the big leagues

Like many kids, I wanted to grow up to become a professional athlete. Playing professional football was my dream, but over the years, it faded into the background because it seemed unrealistic — until it wasn’t.

I was born in California, moved with my family to Maryland and Kentucky, and settled in Tennessee, where I attended middle school and high school. I started playing football in seventh grade, but I didn’t know then how fateful a decision that was. It wasn’t until ninth grade that I really fell in love with the game, and only after my junior year of college that I realized I might be able to play at the professional level.

After being drafted in 2016, I tore my Achilles tendon and missed the entire first season. It felt like my career — my dream — might have ended before it began. Getting that close to my dream was tough. To this day, I have a note I wrote in elementary school about my dreams of playing professionally.

Healing was my priority, but as I recovered, I used my spare time to complete my MBA. Looking back, that decision was almost as pivotal as the one to play football. Little did I know that I would need to practice adapting to new circumstances and making the most of them many more times.

From training camp to boot camp

My injury was far from the end for me: From 2016 to 2022, I played for the Jaguars, Falcons, Dolphins, and Bills as a defensive end. But I knew, as many athletes do, that a career in sports is most often just one leg of life’s journey.

So, in 2021, while playing in the Canadian Football League, I started exploring my career options — what I wanted, and what I knew I didn’t want.

My biggest priority was finding a career path that wasn’t just “good enough.” I wanted a career that truly suited me.

That’s when I discovered coding. I taught myself to code during the season, and I couldn’t get enough of it. When I wasn’t practicing, playing, or studying plays, I used my time to dive deeper into coding.

When I retired from football, I knew software was my first choice for the next step in my career. I applied to the Nashville Software School bootcamp, where I took a full-stack web development course. My experience there convinced me I would never get tired of learning more about technology.

There, I learned programming fundamentals, saw the benefits of structured learning over self-study, and transitioned from a mostly active work environment to a more sedentary one. That took some getting used to!

Throughout, I found myself flexing muscles I didn’t know were there. Everywhere I turned, I found new skills and interests I didn’t know I had. Every opportunity to exercise them felt like a chance to grow as a person.

Going long to advance my career

Looking back, it feels like fate, or the best luck of my career: I was scrolling through LinkedIn one day and stumbled on a post about Airbnb’s Connect Engineering Apprenticeship program. I thought, “Why not?”, took a chance, and applied.

On the one hand, I really did think I was a fit; on the other hand, I wondered how stiff the competition was. When the acceptance came through, it felt surreal. I was a fan of Airbnb as a user, and the chance to see how it all worked on the inside, and potentially join the company itself, was beyond exciting.

The program accepted 10 apprentices, including me, and offered a 6-month program split into two phases.

In the learning phase, we heard about the Airbnb tech stack, learned the company’s working methods, and we built and presented projects.

In the team placement phase, each apprentice was matched to an Airbnb team. We worked on projects of gradually increasing complexity until the program culminated in a group presentation and the possibility of conversion to a full-time role.

The apprenticeship program reminded me of how valuable it was to play on a team — and learn from a team — that really has your back.

There were weekly technical mentor meetings where I could ask all sorts of questions, and I had plenty. Some were about corporate operations — norms and expectations that someone who’d worked in corporate for years might take for granted. I had questions about different career paths in tech and how different decisions might play out. And because I had a bootcamp background, I often had questions about computer science fundamentals that my education hadn’t yet covered. My alumni mentor, Jeremiah Boyle, made a particularly big impact. Our conversations helped open my mind to backend engineering, which I hadn’t considered before.

When we started the team placement phase, we each got a team buddy who served as the primary day-to-day point of contact and helped guide the technical work. Kyler Mejia, who’s a Senior Software Engineer and has been at Airbnb for almost 9 years now, was my team buddy and had a tremendous impact on my growth as an engineer.

As we went from phase to phase, the program also provided an alumni host system, which offered peer guidance from people who’d been through the program before me. I had support from every angle, and I flourished, eventually getting the offer to join the company full-time.

Running containment with higher stakes

Today, I work on the secure development engineering team, which covers cloud and infrastructure protection, vulnerability management, product security, and identity and access management, with my focus on the latter two.

While working on Airbnb’s largest permissions platform, I found the challenge of balancing security and safety with LLM integration, an exciting new dimension of security software engineering. This challenge was exciting to pursue in part because of the engineering culture here. My team is highly collaborative, and everyone at the company prizes transparency and cross-team collaboration.

I remember the day I got accepted into the apprenticeship program, and the day I was offered the chance to work full-time; each time, the reality exceeded my expectations. Airbnb has such a reputation for being creative and collaborative, and the product is a part of so many people’s best memories, including mine. It felt really special, and, as it turned out, it was as great on the inside as it appeared on the outside.

Three years in, I’ve found numerous, sometimes surprising, parallels between football and security engineering.

In football and in security, you will inevitably face intense, high-pressure situations, often with little warning.

In football, that might be the final play that determines the outcome of the game, and in security, that might be a critical incident that threatens security if it’s not addressed. Now that I’m an engineer, I find myself drawing on that high-intensity experience, switching from one pace of work to another to make quick decisions and think on the fly.

Another parallel I’ve found is that a strong team, supported by a leader who knows how to bring out the best in each person while navigating adversity together, can make all the difference. I’ve been fortunate enough to experience that both on the football field and here on the Secure Development Engineering team, led by Kadia Mashal.

But I’ve also had to adapt: Football is a game with a strict set of rules you operate within, whereas the software and security spaces require much more flexibility because there are many more variables to defend against. Football gave me experience to build on, but software has offered me directions for growth.

Now, I can look backward and close the loop on my apprenticeship experience by supporting new apprentices as an alumni host, and look to the future by learning about the evolving ways AI is changing security.

As a defensive end, I frequently ran containment, which prevented other teams from making big gains. Now, I’m part of a team running containment with much higher stakes. I might be behind a keyboard instead of on a defensive line, but every day is still just as exciting.

If you’re from an underrepresented background or are taking an unconventional path into tech, like me, then I highly recommend applying to the Connect Engineering Apprenticeship program. There’s no better way, in my opinion, to make a career transition. And if you’re interested in Airbnb’s approach to security and the opportunity to support a vital function at the company, then check out our openings.

In March 2020, the forecasting models that had served us well in stable times faced a new challenge: predicting outcomes in a world that had suddenly changed.

At Airbnb, many of the financial metrics we forecast depend on two separate events: when guests book, and when they actually travel. A booking made today might correspond to a trip three days from now or three months from now. The distribution of that gap, what we call the lead-time composition, drives how we translate today’s bookings into future revenue (see Figure 1).

The pandemic reshaped what was being booked and where. Hosts shifted their inventory toward longer stays and remote-friendly properties, while guests pivoted from urban centers to rural and suburban destinations. These supply and demand shifts interacted with each other over time, as hosts adapted listings to match evolving guest preferences and new inventory attracted new traveler segments.

Each of these dynamics introduces its own forecasting challenges, but they fall outside the scope of this article. Here, we focus on a more fundamental structural problem: the disruption of the lead time relationship between booking and travel.

Our models had learned stable patterns from years of data: seasonal rhythms, day-of-week effects, gradual trends. Then, as COVID took hold, lockdowns hit. A reopening would trigger a surge of demand, which would then reverse a week later. Rolling closures across different regions created patterns no historical data could have anticipated.

This is not a story unique to Airbnb. Every forecasting team at every company that deals with time-shifted metrics, such as insurance claims, subscription renewals, and supply chain orders, faced a version of the same problem. The specific challenge we want to highlight here is not just that the numbers changed. It’s that the structure of the data changed, and our models were not designed to detect or adapt to structural shifts.

This post describes what we observed, the architectural decision it forced us into, and the second surprise we discovered once the dust settled.

The architectural insight: separate the what from the when

Before COVID, we forecasted trip-date metrics more or less directly. The models learned the relationship between bookings and eventual trips as an integrated process. This worked when the relationship was stable. During the pandemic, it stopped working because two things were moving at once: the volume of bookings, and the temporal distribution of when those bookings would be realized as trips. A third variable shifted as well: whether bookings would be realized at all. Cancellation rates spiked and became far less predictable, adding another layer of instability to the booking-to-trip relationship. That problem deserves its own treatment and is not the focus here, but it compounds everything that follows.

The lockdown-reopening-relockdown cycle was the forcing function. Different regions would go into lockdown, reopen at different times, then often, and unpredictably, go back into lockdown again.

Gross booking volumes were swinging wildly as restrictions changed, but so was the lead-time composition. When a region reopened, people would book trips for the subsequent week. When a new variant hit, last-minute bookings dried up, but far-out bookings for “someday, when this is over” persisted.

The two signals were tangled together, and models that tried to learn them jointly could not disentangle which changes were driven by surges in volume and which were structural shifts in booking behavior. This led us to a fundamental architectural decision: decompose the forecast into two separate components:

1. Gross metrics on the booking date axis. How many bookings (nights, fees) are we recording each day? This is a standard univariate (or multivariate) time series problem.
2. Lead-time composition. Of the bookings recorded today, what fraction correspond to trips happening in each future time window? This is a compositional time series, a vector of proportions which must sum to one. The final trip-date forecast is the product of the two: gross bookings, multiplied by the lead-time composition vector, so the booking-date axis generates the trip-date axis.

This decomposition might sound obvious in hindsight, but it required a modeling framework that could handle compositional data natively. Standard time series methods do not respect the simplex constraint: that is, proportions must be non-negative and sum to one. Naive approaches such as forecasting each lead-time bucket independently produce incoherent results: the shares do not add up, and individual components can go negative.

We developed a family of models called B-DARMA (Bayesian Dirichlet Auto-Regressive Moving Average) specifically for this class of problem. The name unpacks the approach: the Dirichlet distribution enforces valid compositions, the auto-regressive and moving average components capture temporal dynamics in how those compositions evolve, and the Bayesian framework provides full predictive distributions with calibrated uncertainty while allowing us to encode domain knowledge through priors. The core model treats the lead-time composition as Dirichlet-distributed, with the mean vector evolving through a vector autoregressive process in log-ratio space. It learns temporal dependencies between the proportions and produces honest uncertainty bands, not just point forecasts.

We published the original methodology in theInternational Journal of Forecasting (B-DARMA; Katz, Brusch, & Weiss, 2024), and we have since extended the framework with variants for time-varying volatility (B-DARCH; Katz & Weiss, International Journal of Forecasting 2026), shrinkage priors for high-dimensional compositions, and the structural break intervention mechanism described below.

The two-part architecture (Figure 2) gave us something critical: the ability to diagnose which component was responsible when forecasts went wrong. During the lockdown cycle, gross volumes were volatile, but at least moved in interpretable ways (correlated with restriction announcements, case counts, and vaccination rates). The lead-time composition was the harder problem, and separating it out let us focus our modeling effort where the instability actually lived.

The second surprise: the distributions did not snap back

Here is where the story gets more interesting. By late 2021, gross booking volumes had largely recovered. You could look at top-line booking counts and conclude that things were back to normal. But when we looked at the lead-time compositions, the picture was different. The distributions had shifted during the pandemic and stayed shifted (Figure 3).

To quantify this, we started measuring the full distributional divergence between the current lead-time composition and a pre-pandemic baseline using a normalized L1 (Manhattan) distance. Instead of reducing the distribution to a mean or median, this captures the entire shape change: which parts of the distribution shifted, by how much, and whether they recovered. When we applied this approach to booking data across four major U.S. cities from 2018 to 2022, we found a clear two-phase pattern: an abrupt disruption at the pandemic’s onset, followed by a partial recovery that plateaued well short of pre-pandemic norms. The distribution did not return to its old shape; it settled into a new one (Figure 4). The divergence was especially pronounced for international guests, whose booking horizons compressed more sharply and recovered more slowly than domestic travelers. This is consistent with the layered effect of travel restrictions, visa uncertainty, and longer planning cycles that characterize cross-border travel. Even by late 2022, international lead-time distributions remained further from their pre-pandemic baseline than domestic ones.

We published this analysis in Annals of Tourism Research: Empirical Insights (Katz, Savage, & Coles, 2025). We found similar persistence in stay-length distributions: the share of stays lasting 28 nights or longer nearly doubled during the pandemic, and remains significantly above its 2019 baseline, many years later.

This kind of monitoring turned out to be useful well beyond detecting exogenous shocks. The same distributional divergence metric that flagged the pandemic shift also became a diagnostic tool for model health. If the predicted lead-time composition is consistently diverging from the realized composition, that’s a signal that the model is misspecified, not just that the world changed. It lets us separate two failure modes that are easy to conflate: “the model is right but the inputs are unusual” versus “the model’s assumptions no longer hold,” even if the inputs were to revert toward previous norms.

Monitoring also gives us a measure of forecast stability over time. A model whose predicted distributions of lead times, cancellation rates, or regional booking shares are drifting gradually, even if point forecast accuracy looks fine, is a model heading toward a cliff. Catching that drift early, before it shows up in top-line error metrics, is where the real operational value is.

This was the second insight: even after the acute crisis passed, the compositional structure of our data had undergone a durable change. A model that assumed that pre-pandemic dynamics would eventually reassert themselves was going to produce systematically biased forecasts indefinitely.

Why distributional shifts hit the bottom line

These compositional changes were not just a statistical curiosity. At Airbnb, lead-time distributions feed directly into revenue forecasting, cash flow planning, and capacity decisions. The time gap between when a booking is made and when the guest travels determines when revenue is recognized, when cancellation risk resolves, and how far ahead operations teams can see incoming demand. When the composition shifts, every one of those downstream processes inherits the error. Getting the distributions right was a prerequisite for accurate financial operations.

Building models that learn structural shifts

The discovery of persistent distributional shifts, even as gross metrics stabilized, meant we needed B-DARMA to handle structural breaks natively rather than treating them as temporary anomalies to be dummied out. (“Dummying out” means adding a binary indicator variable, often called a dummy variable, that flags an unusual period so the model can ignore it rather than learn from it.)

We tried the obvious fix first: adding dummy variables for the COVID period. This worked in-sample (the model fit its training data well) but failed out-of-sample (it could not forecast new data accurately), and the reason is instructive. The pandemic was not a single fixed effect. It was a rolling sequence of distinct shocks: initial lockdowns, partial reopenings, the Delta variant, and the Omicron variant, each with its own directional impact on the lead-time composition. Fitting a separate dummy for each disruption period means estimating a new parameter every time the world changes, with no ability to generalize across shocks or project the trajectory forward. You end up with a model that perfectly explains the past and has nothing to say about the future.

What we needed was a model that could learn the direction and speed of a structural shift, not just flag that one has occurred. We extended B-DARMA with a directional-shift intervention mechanism that decomposes a structural break into three interpretable components:

1. A direction vector specifying which components of the composition gain share and which lose share during the transition.
2. An amplitude parameter controlling how large the redistribution is.
3. A logistic gate function governing the timing and speed of the transition, allowing for shifts over time rather than instantaneous jumps.

The logistic gate is the key. Instead of a binary before/after indicator, it learns a smooth S-shaped curve that ramps from 0 to 1 over a window of time. This means the model can represent both sudden shocks (steep curve) and gradual structural transitions (gentle slope), and crucially, it can project the transition forward into periods it has not yet observed (Figure 5).

We tested this against the baseline B-DARMA and the dummy-variable approach using a rolling one-step-ahead forecast evaluation during peak COVID disruption (July 2020 through March 2021). The evaluation window begins in July because the intervention model requires several months of post-shock observations to estimate the new regime’s dynamics. In practice, this also reflected the reality of the moment: the early months of the pandemic were spent understanding the disruption before we could begin modeling it formally.

• The intervention model reduced Aitchison distance (the natural error metric for compositional data) by 31% compared to baseline.
• Its 80% prediction intervals achieved close to 80% coverage, while the baseline model’s intervals covered only about 54% of outcomes. The dummy-variable model sat in between at roughly 71%.
• The log score (the proper scoring rule for density forecasts) favored the intervention model by a wide margin, indicating better calibrated uncertainty, not just better point forecasts.

The point forecast improvements mattered, but the calibration improvements mattered more. During a crisis, decision-makers need honest uncertainty bounds, not confident predictions that turn out to be wrong. The intervention model gave us that.

The full methodology and simulation study are available as a working paper: Directional-Shift Dirichlet ARMA Models for Compositional Time Series with Structural Break Intervention.

What we learned

Three lessons that generalize beyond our specific use case:

1. Consider decomposing volume from timing, especially for delay-aware forecasting. Separating gross metrics from lead-time compositions was the single highest-value architectural decision we made during the pandemic. It isolated the source of instability, made diagnostics manageable, and let us direct modeling effort where the actual complexity lived. This is not always the right move for every forecasting problem, but it pays off when you are dealing with time-shifted metrics where booking time and event time diverge. In those settings, the lead-time distribution is not just a nuisance parameter on the way to a trip-date forecast. It is a vector of genuine interest in its own right. Pricing changes, cancellation policies, product launches, and marketing campaigns all directly affect how far in advance guests book, which means the lead-time composition responds to levers the business can actually pull. Modeling it separately makes those effects visible and forecastable rather than burying them inside an aggregate.

2. Monitor distributions, not just point statistics. The normalized L1 divergence we use for lead times has become a standard monitoring tool for us. It alerts us to distributional shifts long before they show up in averages.

3. Build break detection into the model rather than treating disruptions as one-off anomalies. The acute disruption was dramatic, but ultimately temporary. The persistent structural shift in distributions, visible only after gross metrics recovered, was the harder modeling problem and the one with long-term consequences. Building break detection into the model rather than treating breaks as one-off anomalies is what made our forecasts resilient to the slow, ongoing changes that followed the shock.

Looking ahead

The pandemic was an extreme shock, but there are other shocks that have a similar structure. Currency markets, regulatory changes, competitive dynamics, and macroeconomic shifts all induce structural breaks in the compositional data we forecast. The intervention framework we built for COVID is domain-agnostic. We have since applied variations of it to currency share forecasting and energy mix data, and the same principles hold.

The next shock will not look like the previous one. The question is whether your models can learn the difference.

If this type of work interests you, check out some of our open positions!

Acknowledgments

Thanks to Liz Medina and Jess Needleman for building and improving the forecasting systems described here, and to Carolina Barcenas, Peter Coles, and Adam Liss for their support of this work and its publication.

Harrison Katz leads Finance Data Science & Strategy at Airbnb. His research focuses on Bayesian methods for compositional and hierarchical time series & Bayesian decision theory.

Like many of its peers, Airbnb started out by outsourcing its observability needs to vendors. But, as the company matured, our needs diverged from the typical vendor’s incentives. Vendors charge by the amount of data ingested, so Airbnb’s costs were rising, but more data does not automatically lead to faster insights or reduced mean time to detect (MTTD) or repair (MTTR). Also, by being out of the feedback loop of how observability data is consumed, our ability to enhance the monitoring workflows of our customers or pursue cost optimizations on observability spend was severely hampered.

To meet our objectives, Airbnb embarked on a complex migration, overhauling every part of our metrics infrastructure. This challenging journey involved replacing our instrumentation, collection, storage, and visualization systems as we transitioned from a third-party, vendor-managed observability platform to a custom, in-house solution built on open-source technology based on Prometheus.

This transition enabled us to gain end-to-end ownership over metrics collection, storage, and querying. Through this project, we successfully utilized our automation tooling to migrate a massive volume of data across 1,000 services: 300 million timeseries, 3,100 dashboards, and more than 300,000 alerts. While one does not undertake a project of this scope lightly, the restrictions around what we could do with our own data left us with little alternative.

Though there’s no one-size-fits-all migration guide, we have some learnings from this specific migration that we believe will be helpful to others. We will first describe a simplistic, but ultimately flawed, approach to migration, and then detail the successful strategy we ultimately employed. Our plans for this migration, begun five years ago, evolved significantly as we proceeded. This journey provided us with valuable, hard-won lessons, offering clear insights into the best practices and pitfalls for executing a large-scale migration.

Migration v1

Climbing Everest on day one

When approaching a large-scale migration, the general strategy is to aim for the largest, most impactful service first. Tackling the biggest challenge upfront allows the project team to demonstrate immediate, significant progress, build confidence in the new infrastructure, and prove the migration strategy’s viability to the wider team and stakeholders. This high-visibility win then sets the tone for the rest of the rollout.

With this approach in mind, it (initially) made sense to take a complex service — for example, something that has sparse data, or emits metrics in a way not standard to the new system — and make this the first service to migrate onto our new system. That would expose the differences between the two systems and allow us to design solutions that mimicked the behavior of the old system, ensuring that the dashboards and alerts would look the same.

When following this strategy, a team will spend a lot of time getting the first service to work, and is likely to have false alarms, dashboards that don’t align between the systems, and a lot of education to get the first customer on board; all to prove to the wider company that the overall migration is worthwhile.

Same baggage, thinner air

When migrating systems, a smooth transition for your customers is paramount, as enough is already changing for them. Even if the legacy system has led to flawed or inaccurate queries that could be fixed, it’s important to approach fixes strategically. For example, avoid trying to resolve issues with existing dashboarding and alerting patterns all at once.

Instead, focus on a complete migration to the new system first. Once everyone is brought over and centralized, you can address and solve these dashboard and alerting improvements in a subsequent, second set of changes. Customers want the initial migration completed first. If moving to the new system already presents friction, we shouldn’t exacerbate the difficulties involved by also altering the functionality of their queries.

The documentation explains everything, for those who stop to study it.

To facilitate customer adoption of the new system, which likely includes extensive changes, it’s crucial to provide comprehensive documentation on the new query language and interfaces. This should be paired with extensive training to ensure user proficiency.

Even for infrequent users, familiarity with the observability query language is important, similar to any programming language. This knowledge, supported by the documentation and training materials, will be particularly valuable for quickly diagnosing problems during incidents.

Migration v2

Start with an achievable target

Choosing an initial service to migrate that aligns closely with your destination system can eliminate much of the distraction and noise that typically accompanies a complex migration. When your initial goal is to prove (to your team and to the broader company) that the migration is worthwhile (in terms of cost reduction, capabilities, etc.), it’s crucial to focus on a tractable, high‑leverage service.

This allows your team to:

• Validate that the storage engine can handle the required scale and observe how it behaves during incidents
• Deliver translator tooling to migrate existing dashboards and alerts to the new system
• Invest in customer‑facing documentation, education, and training
• Introduce the new visualization tool to a small set of users, giving them time to adjust to the new experience while your team gathers feedback on UX gaps before rolling the service out at scale

Successfully migrating this less complex service achieves two things: it demonstrates to your team that the migration is technically and operationally feasible; and, by narrowing the scope and collaborating closely with the customer, it helps you validate the migration strategy to the entire company.

Switching vendors will often necessitate a new visualization tool, so this is also a good opportunity to surface any UX differences early. The feedback your team collects from this initial cohort will prove invaluable when rolling the new tooling out to the rest of the organization.

Migrate the intent of query

With any long‑running system, you eventually accumulate a wide variance in how latency, error rates, and other signals are measured across dashboards. In a democratized observability environment, this is almost guaranteed: some metrics will be computed incorrectly — for example, averaging a result that should be a p95, or summing total latency. Customers can easily construct flawed queries and, without guardrails, the results of those queries get treated as ground truth.

A migration puts your team back in the loop. It’s an opportunity to correct these patterns — not by erasing or resetting them, but by understanding the original intent and mapping it to a preferred, standardized query.

With this in mind, we adapted our translation system to focus on intent rather than one‑to‑one query translation. For example, if a query included a p95 calculation, we would ignore any additional aggregations layered on top and instead return a canonical histogram query for that metric.

In practice, this required introducing a dedicated metadata engine. In Prometheus, metric types are often inferred from naming conventions (for example, a _total suffix indicating a counter). But, because we chose to preserve existing metric names to avoid divergence between code and observed data, naming alone was no longer reliable.

To address this, we built a metadata engine directly into the translation layer. It periodically scans all known metrics and uses an internal label (otel_metric_type) to construct and maintain a reliable mapping from metric to type.

With this change in place, we’re now significantly more confident that teams have accurate, consistent visibility into the performance of their applications.

A new query language can be daunting

Customers are, understandably, going to need some time to reset the muscle memory they have developed for the old system. In some cases, your new system may not provide the same polish for easily crafting queries.

During our migration, our users’ lack of familiarity with the new query language was solved by using our translation tooling, which could map a given query to the new system to help bootstrap a self-service migration. But, post-migration, we didn’t want to continue providing a shim between the old language and the new, as that would slow down or inhibit the needed reset of muscle memory.

By adopting PromQL as the query language for our new system, we benefit from a mature ecosystem and a well-documented query language that is widely understood by both humans and modern LLMs.

We further reduced the barrier to entry by pairing this with in-house skills for our AI tooling, which supplies rich semantic metadata about each metric — for example, whether it is a counter or histogram, or what units it is reported in. Together, these capabilities allow agents to generate correct PromQL with far less manual effort, enabling developers to bootstrap their understanding more quickly and explore metrics more effectively. This has become critical for incident debugging.

Consequently, customers can now complete many common tasks, such as diagnosing incidents or creating dashboards, in a matter of minutes, a significant reduction from the hours these tasks previously required.

Now is the time to correct outdated patterns

Over time, old systems may be neglected, and their (lack of) functionality may become a drag on productivity. Thus, the benefit of any large migration is that you get to take a step back, reflect on how your product is being used, and identify additional areas for improvement.

While initially we were afraid of introducing new concepts, as our initial set of customers began using the new system, it became clear to us that the existing alert framework was in need of a replacement. We made the choice to pull forward the investment our sister team, Reliability Experience, had been making in the alert authoring space, and promote their work as the preferred alert-authoring experience.

This new alert experience replaced the outdated system with a new alert-authoring experience that treated each alert as a development workflow, rather than as a sparsely documented config file. Alerts are authored as code, with autocomplete and builder-style assistance for writing queries, built-in backtesting to understand when an alert would have fired historically, and diffing to clearly reason about the impact of changes, all before deployment.

This was a change from our original strategy of maintaining dashboards and alerting unchanged by the initial, major migration. But we found that the migration’s success was significantly amplified by the improved alerting framework. This change helped customers recognize the inherent value of the migration, beyond just control and cost objectives, shifting the perception of the project. Furthermore, the centralized nature of the new alerts streamlined the process by reducing the overall migration effort. This centralization lowered the amount of work customers needed to spend on migration tasks.

Conclusion

This migration ultimately became much more than a change in which storage engine to use and where it was hosted. What began as an effort to “just port the system” evolved into a reset of how we think about observability, ownership, and the developer experience.

By owning the full lifecycle of our metrics, from emission through querying, we materially improved our reliability posture. We moved from fragile, hand-crafted queries toward more intentional patterns, better tooling, and higher-confidence alerts and dashboards. The investment in metadata, AI tooling, and modern authoring workflows paid dividends not only in correctness, but in how quickly engineers can explore possibilities and respond during incidents.

Equally important, the migration reshaped our relationship with our internal customers. Instead of outsourcing core observability concerns, we were forced to engage deeply with how teams actually use metrics under real operational pressure. That feedback loop has been invaluable, and it continues to guide where we invest next.

Automation played a key role in making this migration feasible, but we learned that automation alone is not enough. Blindly translating existing patterns (especially flawed patterns) simply transfers technical debt into a new system. The most impactful moments came when we deliberately chose to break compatibility in favor of better defaults, even when that caused short-term friction.

Finally, this effort produced a repeatable blueprint. For future migrations, we are clear about what we want to own: the experience and interfaces for how engineers write, read, and reason about their data. Storage engines and backends may change, but the responsibility for making observability usable, trustworthy, and evolvable should remain firmly in our hands.

Even for teams not facing an imminent migration, there are steps worth taking now to reduce future friction. The most impactful is to own the interaction layer: the frontend, authoring tools, and workflows engineers use to explore and act on observability data. During our migration, we not only had to change the backend system, but also had to move every team to a new visualization tool and to our new alerting framework, significantly increasing the switching cost. Had we already controlled those touchpoints, the backend transition would have been far simpler and more incremental.

While every organization’s constraints are different, we hope these lessons help others view large-scale migrations not as a necessary burden, but as a rare opportunity to meaningfully improve systems and, as a result, your observability posture.

If this type of work interests you, check out some of our open roles.

Acknowledgements

We would like to express our gratitude to the following teams for their essential contributions to this migration:

Observability Team

Abdurrahman Allawala, Rishabh Kumar, Eugene Ma, Natasha Aleksandrova, Suman Karumuri, Wei Song, Yann Ramin, and Xuan Lu. This team was instrumental in migrating Airbnb to the new observability system.

Reliability XP Team

Doug Smith, Kevin Goodier, Harry Shoff, Rich Unger, and Vlad Vassiliouk. This team developed our new Alerting framework.

Finally, a thank you to everyone across Airbnb who helped successfully migrate onto our new observability system.

In this blog post, we describe how we help users in the exploration stage by recommending travel destinations. There are multiple unique challenges in modeling destination intent: for example, how to effectively integrate diverse signals (users’ long term interests vs. short term interests), how to balance dormant user behavior vs. active user behavior, and how to encode rich geolocation knowledge.

To address these challenges, we developed a framework that predicts users’ destination intent based on their actions on the Airbnb platform. While the framework is inspired by language modeling, we introduce several key adaptations in training data creation, model architecture, and loss function to tailor it to the destination recommendation problem in the travel domain. Lastly, we present two applications, autosuggest and abandoned search email notifications, that help users explore destination possibilities and facilitate booking decisions.

Model architecture

Travel destination is one of the primary aspects users explore during trip planning, as it largely determines subsequent decisions such as travel timing, budget, and accommodation preferences. User travel destination preferences are driven by a combination of historical behavior, contextual signals, and temporal factors, etc. For instance, users who previously booked listings in Hawaii may exhibit a preference for beach or tropical destinations, while seasonal context (e.g., summer) may shift their intent toward cooler locations.

In our model, we generalize the destination prediction based on historical user preference data. (Users are able to opt out of this personalization.) As shown in Figure 1, we treat each user action as a token, inspired by language modeling. We use transformers to model sequences of user actions as recorded in various sources: booking history, view history, and search history. Each action is represented by the sum of embeddings of city / region / days to today. We also use contextual information, such as the current time, to capture seasonality. This setup enables the model to summarize user’s short-term interests (views, searches), and long-term interests (bookings), and make a holistic prediction of destination intent.

Figure 1: model architecture.

Balancing active users and dormant users

At Airbnb, we need to make predictions not only for “active users,” but also for “dormant users”. They exhibit different behaviors, for example:

• Active users: User A recently issued a search in the California Bay Area last week. She is currently looking for more affordable listings in the Bay Area.
• Dormant users: User B made several bookings in 2025, and hasn’t returned to Airbnb since then. He is currently exploring ideas for a summer vacation in 2026.

Motivated by these two different types of goals, we design the training data shown in Figure 2. For each booking, we create 14 training examples in total. There are two parts:

1. Seven training examples for active users, from 1, 2, 3…7 days before the booking date. For these 7 examples, we use the up-to-date booking/view/search data. This is to mimic the late booking stage when users have a rough idea where to go.
2. Seven training examples for dormant users, randomly sampled from 8 to 365 days before the booking. For these 7 examples, we only use booking data, to mimic the early planning stage when users don’t have a concrete idea and haven’t come to Airbnb.

Figure 2: T is the date for the latest booking. The arrows at the bottom show the training examples used for the planning stage; the arrows in the upper-right corner illustrate the training examples used for the booking stage.

Improving location understanding

At Airbnb, we have rich geolocation information about cities and their relationships. For example, the California Bay Area contains many closely related cities; a user interested in staying in San Francisco may also consider nearby cities such as San Jose. For the purposes of destination recommendation, the Bay Area can be viewed as a broader “region” that encompasses multiple cities.

To incorporate this information into our framework, we use multi-task learning. Specifically, we add multiple prediction heads at the final layer of the model, each corresponding to a different prediction task. As shown in Figure 1, the model is trained to predict both the region-level and the city-level destination. By jointly learning these tasks and encouraging consistency between region and city predictions, the model learns richer geolocation representations of cities.

Applications

We deployed the resulting model in two features of the Airbnb platform. The first is autosuggest. When users click on the search bar, multiple city recommendations are presented. Online A/B testing shows significant booking gains in regions where English is not the primary language; further analysis indicates that these recommendations benefit not only users who have not yet decided on a destination, but also users who are open to booking more affordable listings in neighboring cities.

The second application is abandoned search email notifications. When a user abandons a search on Airbnb, we send follow-up emails featuring listings from areas predicted by the destination recommendation model. This helps drive bookings by encouraging users to explore alternative listings within the recommended destinations and re-engaging them to complete a booking on Airbnb.

Conclusion

In this post, we described a destination recommendation framework designed to support users in the exploration stage of trip planning, when intent is often ambiguous and preferences are still forming. Our framework includes several key innovations: modeling multiple sequences of user actions to balance short-term and long-term interests, designing training data to accommodate both active and dormant user behaviors, and using multi-task learning to incorporate rich geolocation information. Deployed in autosuggest and abandoned search email notifications, the model helps users discover relevant destination alternatives and drives measurable booking gains. Looking ahead, this framework provides a solid foundation for modeling other preferences, such as travel times and price preferences, enabling broader and deeper personalization across the travel planning journey.

If this type of work interests you, check out some of our open roles.

Acknowledgments

We would like to especially thank the following people for their great collaboration throughout this project: Kidai Kwon, Phanindra Ganti, Kedar Bellare, Malay Haldar, Soumyadip Banerjee, Michael Kinoti, Yi Li, Amisha Patel, Rachel Zhao, Zhentao Sun, Wei Jiang, Jackie Liu, Ying Xiao, Hongzhao Huang, Chen Qian, Haiyang Han, Pengyu Hou, Haichun Chen, Sherry Chen, Pavan Tapadia, Stephen Simburg, Clarence Quah, Chris Tarello, Eric Kostenbauder, Linda Yu, Gary Chang.

Yet there’s a critical gap in most OaC workflows. While we bring rigor to alert definitions through code review and version control, the actual behavior of those alerts often can’t be validated until they’re live. Production becomes the proving ground. Problems surface either as noise that erodes trust or silence that hides real incidents.

This tolerance of high alert noise might appear to be a culture problem, but we realized it was actually a gap in the developer workflow. We solved it by building accessible, fast feedback loops to preview, validate, and surface actionable insights on alert behavior before PR submission. With these changes, development cycles collapsed from weeks to minutes, and we successfully migrated 300,000 alerts from a vendor to Prometheus, a feat that wouldn’t have been possible otherwise.

Airbnb’s OaC North Star

Our Observability as Code North Star is for product teams to receive out-of-the-box, best-practice monitoring from platform teams. When a product engineer adopts Kubernetes, a service framework, or a database, they should inherit battle-tested alerts, dashboards, and SLOs. The best monitoring gives product engineers the benefit of all of Airbnb’s infra and platform domain expertise immediately. We call this “zero touch.”

We began our OaC journey more than 10 years ago when we built Interferon, starting with 1,000 alerts. Today, we manage 300,000. By any measure, Interferon was a success, scaling our monitoring practices towards our North Star.

However, this success introduced new operational challenges. With so many alerts in production, validating any OaC changes became costly, and it became harder to iterate. Engineers faced a difficult tradeoff: tuning an alert template might reduce noise, but it also risked losing an important signal. Without a way to preview alert behavior, the safer choice was often to leave things as-is.

The problem: Traditional code review can’t validate alert behavior

The problem wasn’t Interferon itself, but rather a development workflow gap. Our North Star requires platform teams to define and maintain monitoring patterns at scale, but we lacked the necessary tooling to effectively validate those patterns against reality.

Traditional code review can validate syntax and logic, and unit tests can verify outputs. But neither can answer the questions that matter most: “How will these alerts behave in production? What noise might they generate? Will they needlessly wake up on-call engineers at 3 AM?”

That’s why you have to validate alerts against real-world data. If your assumptions about the data are wrong, your alert is wrong.

However, off-the-shelf query visualization tools are insufficient for the job. They don’t account for alert-specific parameters, and most notably, they show downsampled data that masks actual alert behavior — step sizes don’t match evaluation intervals. Also, the further you move from static configs, the harder validation becomes. With templating, reviewers must manually copy-paste queries and fill in variables. For a single alert, this is tedious and error-prone; for changes affecting hundreds of services, it’s impossible.

So in practice, developers often had to resort to a weeks-long process of deploying a new alert side-by-side with the existing one, waiting for real-world data to come in, validating, and then iterating.

The solution: Making alert behavior visible

What if an infrastructure engineer could validate a Kubernetes alert template, one that fans out to thousands of services, in 30 seconds instead of 30 days?

That question among others prompted us to rethink and rebuild our OaC platform from the ground up. Building on top of Prometheus’ open source foundations, we could develop the exact UX our engineers needed, particularly local diffs and pre-deployment validation.

The core of this workflow is local-first development: The same code and inputs run in production must run identically on a developer’s laptop and in CI. In addition, we built Change Reports that show how alerts will be modified and bulk backtesting that simulates alerts against historical data.

We rolled out this platform incrementally, with each milestone providing compounding value.

Phase 1: Text-based diffs everywhere

We first generated markdown alert diffs with field-level granularity and query links — the “terraform plan” of OaC. We met developers where they work, in terminal via CLI and in PRs via CI. This solved the basic visibility problem: engineers could finally review the OaC generated alerts without error-prone copy-pasting.

Phase 2: Dedicated Change Report UI

We then built a Change Report UI showing side-by-side alert diffs exactly as they will appear in production, removing the guesswork and mental mapping between config and UI. However, the user was still responsible for mentally simulating alert behavior, which is challenging even for Prometheus experts to get right.

Phase 3: Historical simulation via bulk backtesting

Finally, we built a backtesting system that runs proposed alerts against historical data, hooking directly into Prometheus’s rule manager. Backtesting allows users to understand which alerts would have fired, when, and why, as if they had existed the entire time. Displaying this simulated state inline in the Change Report UI answered the question that matters most: “How would this alert behave in production?”

We backtest in bulk for the entire diff — hundreds or thousands of related alerts — and surface quality signals to help reviewers focus their attention. We compute a “noisiness” metric and show alert firing timelines in the table view, letting users sort by potential problems and focus their effort.

Putting it all together

On the new platform, when a user makes an OaC change, they will now generate a Change Report via CLI or CI. We post a Change Report on all PRs.

The user reviews their changes via the UI. In this example, a one week backtest was conducted, and the changes are sorted by noisiness, seen in the “Tuning” column, to help direct users’ attention.

The user can dive into individual alerts to learn more. This one looks to be problematic, firing once per day.

The user can set overrides. Given the graph, 1.14 looks like a better trigger threshold.

They can then see the impact of their changes. No more alert firings. This looks good and is ready to ship.

Our learnings

Compatibility over novelty

A key architectural decision was compatibility over novelty. Rule groups are Prometheus’ standardized format. By taking that as input, we hooked directly into Prometheus’s rule evaluation engine rather than reimplementing it. We wrote results as Prometheus time series blocks, exposing the data via the standard query API. This meant building analysis tools once. This standardization made our system portable, allowing us to reach all developers in their existing workflows.

Guardrails aren’t optional

Simulating thousands of alerts over 30 days quickly and without service degradations required careful design. Each backtest runs in its own Kubernetes pod with autoscaling to prevent resource contention. Concurrency limits, error thresholds, and multiple circuit breakers prevent cascading failures. A backtesting system that can destabilize production is worse than no system at all.

Perfect is the enemy of shipped

Our simulator doesn’t account for recording rule dependencies. We could have built a more sophisticated dependency resolver, but users can separate this into two distinct tasks — modify the recording rule first, then backtest alerts that depend on it — assuming they know they should. The Change Report UI helps, because when modified dependencies are detected, it highlights them and prompts resolution. This turns a technical limitation into a guided workflow. We shipped the 80% solution that immediately delivers value, leveraging the UI to close gaps.

Own the full surface area

Monitoring is often an afterthought — engineers have limited time under tight deadlines. Our job is to make that time as effective as possible. Prometheus is powerful but exposes a low-level API that requires expertise most engineers don’t have. To achieve our North Star, we introduced abstractions like anomaly detection, burn rate alerts, and change detection. But abstractions only simplify things when you own all the touchpoints: the input language engineers write, the generation process, the UI that displays results, and the validation tools that provide feedback. Partial ownership creates leaky abstractions. Full ownership lets us ruthlessly optimize for developer experience.

The impact

Successful migration from a vendor to Prometheus

We migrated 300,000 alerts from a vendor to Prometheus. Rewriting every alert would have been impossible with our old workflow, but we achieved it thanks to our Change Report UI, bulk backtesting, and an additional vendor-specific integration. By codifying our domain knowledge in the UX, what originally promised to be a multi-year slog of manual effort became a structured, confident migration.

Collapsed development cycles

The typical developer workaround for making alert changes — deploy side-by-side, wait, then iterate — became obsolete. Engineers now make and validate alert changes all within a single PR. Platform teams confidently deploy template changes affecting thousands of services. What once took a month of iteration now takes an afternoon.

Culture transformation

Even though we realized we had a workflow problem, not a culture problem, solving this problem still ended up transforming our culture. We reduced companywide alert noise by 90%, and engineers stopped tolerating noisy alerts and started competing to improve them. Platform teams resumed iterating on shared patterns. Alert hygiene became a point of pride, not a chore to avoid.

Turnkey alert testing is the biggest positive improvement to alerts management in Airbnb’s history.

I’ve seen people spend hours debating the merits of changing an alert only to do nothing because of fear, uncertainty, and doubt. This new alert testing capability completely evaporates the stop energy and allows us to monitor with confidence.

– Gregory Szorc, Senior Staff Software Engineer

Conclusion

Local-first development, Change Reports, and bulk backtesting give us the necessary tools to incrementally reach our North Star. Platform teams can now confidently iterate on monitoring for their domains. Zero touch is becoming how we operate, one cycle at a time.

Now that we’ve introduced pre-deployment visibility and validation to the alert lifecycle, our next step is to introduce that same rigor to on-call analysis.

If this type of work interests you, check out some of our open roles.

Acknowledgements

Thank you to the Reliability Experience team — Kevin Goodier, Harry Shoff, Rich Unger, and Vlad Vassiliouk — and our partners across the company who helped make this a reality.

But some of the basic assumptions underlying ranking, built for a world where search results are presented as lists, simply break down for maps.

What Is Different About Maps?

The central concept that drives ranking for list-results is that user attention decays starting from the top of the list, going down towards the bottom. A plot of rank vs click-through rates in Figure 1 illustrates this concept. X-axis represents the rank of listings in search results. Y-axis represents the click-through rate (CTR) for listings at the particular rank.

To maximize the connections between guests and Hosts, the ranking algorithm sorts listings by their booking probabilities based on a number of factors and sequentially assigns their position in the list-results. This often means that the larger a listing’s booking probability, the more attention it receives from searchers.

But in map-results, listings are scattered as pins over an area (see Figure 2). There is no ranked list, and there is no decay of user attention by ranking position. Therefore, for listings that are shown on the map, the strategy of sorting by booking probabilities is no longer applicable.

Uniform User Attention

To adapt ranking to the map interface, we look at new ways of modeling user attention flow across a map. We start with the most straightforward assumption that user attention is spread equally across the map pins. User attention is a very precious commodity and most searchers only click through a few map pins (see Figure 3). A large number of pins on the map means those limited clicks may miss discovering the best options available. Conversely, limiting the number of pins to the topmost choices increases the probability of the searcher finding something suitable, but runs the risk of removing their preferred choice.

We test this hypothesis, controlled by a parameter . The parameter serves as an upper bound on the ratio of the highest booking probability vs the lowest booking probability when selecting the map pins. The bounds set by the parameter controls the booking probability of the listings behind the map pins. The more restricted the bounds, the higher the average booking probability of the listings presented as map pins. Figure 4 summarizes the results from A/B testing a range of parameters.

The reduction in the average impressions to discovery metric in Figure 4 denotes the fewer number of map pins a searcher has to process before clicking the listing that they eventually book. Similarly, the reduction in average clicks to discovery shows the fewer number of map pins a searcher has to click through to find the listing they booked.

Launching the restricted version resulted in one of the largest bookings improvement in Airbnb ranking history. More importantly, the gains were not only for bookings, but for quality bookings. This could be seen by the increase in trips that resulted in 5-star rating after the stay from the treatment group, in comparison to trips from the control group.

Tiered User Attention

In our next iteration of modeling user attention, we separate the map pins into two tiers. The listings with the highest booking probabilities are displayed as regular oval pins with price. Listings with comparatively lower booking probabilities are displayed as smaller ovals without price, referred to as mini-pins (Figure 5). By design, mini-pins draw less user attention, with click-through rates about 8x less than regular pins.

This comes in handy particularly for searches on desktop where 18 results are shown in a grid on the left, each of them requiring a map pin on the right (Figure 6).

The number of map pins is fixed in this case, and limiting them, as we did in the previous section, is not an option. Creating the two tiers prioritizes user attention towards the map pins with the highest probabilities of getting booked. Figure 7 shows the results of testing the idea through an online A/B experiment.

Discounted User Attention

In our final iteration, we refine our understanding of how user attention is distributed over the map by plotting the click-through rate of map pins located at different coordinates on the map. Figure 8 shows these plots for the mobile (top) and the desktop apps (bottom).

To maximize the chances that a searcher will discover the listings with the highest booking probabilities, we design an algorithm that re-centers the map such that the listings with the highest booking probabilities appear closer to the center. The steps of this algorithm are illustrated in Figure 9, where a range of potential coordinates are evaluated and the one which is closer to the listings with the highest booking probabilities is chosen as the new center.

When tested in an online A/B experiment, the algorithm improved uncancelled bookings by 0.27%. We also observed a reduction of 1.5% in map moves, indicating less effort from the searchers to use the map.

Conclusion

Users interact with maps in a way that’s fundamentally different from interacting with items in a list. By modeling the user interaction with maps in a progressively sophisticated manner, we were able to improve the user experience for guests in the real world. However, the current approach has a challenge that remains unsolved: how can we represent the full range of available listings on the map? This is part of our future work. A more in-depth discussion of the topics covered here, along with technical details, is presented in our research paper that was published at the KDD ’24 conference. We welcome all feedback and suggestions.

If this type of work interests you, we encourage you to apply for an open position today.

Since first writing about Mussel in a 2022 blog post, we have completely deprecated the storage backend of the original system (what we now call Mussel v1) and have replaced it with a NewSQL backend which we are referring to as Mussel v2. Mussel v2 has been running successfully in production for a year, and we wanted to share why we undertook this rearchitecture, what the challenges were, and what benefits we got from it.

Why rearchitect

Mussel v1 reliably supported Airbnb for years, but new requirements — real-time fraud checks, instant personalization, dynamic pricing, and massive data — demand a platform that combines real-time streaming with bulk ingestion, all while being easy to manage.

Key Challenges with v1

Mussel v2 solves a number of issues with v1, delivering a scalable, cloud-native key-value store with predictable performance and minimal operational overhead.

• Operational complexity: Scaling or replacing nodes required multi-step Chef scripts on EC2; v2 uses Kubernetes manifests and automated rollouts, reducing hours of manual work to minutes.
• Capacity & hotspots: Static hash partitioning sometimes overloaded nodes, leading to latency spikes. V2’s dynamic range sharding and presplitting keep reads fast (p99 < 25ms), even for 100TB+ tables.
• Consistency flexibility: v1 offered limited consistency control. v2 lets teams choose between immediate or eventual consistency based on their SLA needs.
• Cost & Transparency: Resource usage in v1 was opaque. v2 adds namespace tenancy, quota enforcement, and dashboards, providing cost visibility and control.

New architecture

Mussel v2 is a complete re-architecture addressing v1’s operational and scalability challenges. It’s designed to be automated, maintainable, and scalable, while ensuring feature parity and an easy migration for 100+ existing user cases.

Dispatcher

In Mussel v2, the Dispatcher is a stateless, horizontally-scalable Kubernetes service that replaces the tightly coupled, protocol-specific design of v1. It translates client API calls into backend queries/mutations, supports dual-write and shadow-read modes for migration, manages retries and rate limits, and integrates with Airbnb’s service mesh for security and service discovery.

Reads are simplified: Each dataname maps to a logical table, enabling optimized point lookups, range/prefix queries, and stale reads from local replicas to reduce latency. Dynamic throttling and prioritization maintain performance under changing traffic.

Writes are persisted in Kafka for durability first, with the Replayer and Write Dispatcher applying them in order to the backend. This event-driven model absorbs bursts, ensures consistency, and removes v1’s operational overhead. Kafka also underpins upgrades, bootstrapping, and migrations until CDC and snapshotting mature.

The architecture suits derived data and replay-heavy use cases today, with a long-term goal of shifting ingestion and replication fully to the distributed backend database to bring down latency and simplify operations.

Bulk load
Bulk load remains essential for moving large datasets from offline warehouses into Mussel for low-latency queries. v2 preserves v1 semantics, supporting both “merge” (add to existing tables) and “replace” (swap datasets) semantics.

To maintain a familiar interface, v2 keeps the existing Airflow-based onboarding and transforms warehouse data into a standardized format, uploading to S3 for ingestion. Airflow is an open-source platform for authoring, scheduling, and monitoring data pipelines. Created at Airbnb, it lets users define workflows in code as directed acyclic graphs (DAGs), enabling quick iteration and easy orchestration of tasks for data engineers and scientists worldwide.

A stateless controller orchestrates jobs, while a distributed, stateful worker fleet (Kubernetes StatefulSets) performs parallel ingestion, loading records from S3 into tables. Optimizations — like deduplication for replace jobs, delta merges, and insert-on-duplicate-key-ignore — ensure high throughput and efficient writes at Airbnb scale.

TTL

Automated data expiration (TTL) can help support data governance goals and storage efficiency. In v1, expiration relied on the storage engine’s compaction cycle, which struggled at scale.

Mussel v2 introduces a topology-aware expiration service that shards data namespaces into range-based subtasks processed concurrently by multiple workers. Expired records are scanned and deleted in parallel, minimizing sweep time for large datasets. Subtasks are scheduled to limit impact on live queries, and write-heavy tables use max-version enforcement with targeted deletes to maintain performance and data hygiene.

These enhancements provide the same retention functionality as v1 but with far greater efficiency, transparency, and scalability, meeting Airbnb’s modern data platform demands and enabling future use cases.

The migration process

Challenge

Mussel stores vast amounts of data and serves thousands of tables across a wide array of Airbnb services, sustaining mission-critical read and write traffic at high scale. Given the criticality of Mussel to Airbnb’s online traffic, our migration goal was straightforward but challenging: Move all data and traffic from Mussel v1 to v2 with zero data loss and no impact on availability to our customers.

Process

We adopted a blue/green migration strategy, but with notable complexities. Mussel v1 didn’t provide table-level snapshots or CDC streams, which are standard in many datastores. To bridge this gap, we developed a custom migration pipeline capable of bootstrapping tables to v2, selected by usage patterns and risk profiles. Once bootstrapped, dual writes were enabled on a per-table basis to keep v2 in sync as the migration progressed.

The migration itself followed several distinct stages:

• Blue Zone: All traffic initially flowed to v1 (“Blue”). This provided a stable baseline as we migrated data behind the scenes.
• Shadowing (Green): Once tables were bootstrapped, v2 (“Green”) began shadowing v1 — handling reads/writes in parallel, but only v1 responded. This allowed us to check v2’s correctness and performance without risk.
• Reverse: After building confidence, v2 took over active traffic while v1 remained on standby. We built automatic circuit breakers and fallback logic: If v2 showed elevated error rates or lagged behind v1, we could instantly return traffic to v1 or revert to shadowing.
• Cutover: When v2 passed all checks, we completed the cutover on a dataname-by-dataname basis, with Kafka serving as a robust intermediary for write reliability throughout.

To further de-risk the process, migration was performed one table at a time. Every step was reversible and could be fine-tuned per table or group of tables based on their risk profile. This granular, staged approach allowed for rapid iteration, safe rollbacks, and continuous progress without impacting the business.

Migration pipeline

As described in our previous blog post, the v1 architecture uses Kafka as a replication log — data is first written to Kafka, then consumed by the v1 backend. During the data migration to v2, we leveraged the same Kafka stream to maintain eventual consistency between v1 and v2.

To migrate any given table from v1 to v2, we built a custom pipeline consisting of the following steps:

1. Source data sampling: We download backup data from v1, extract the relevant tables, and sample the data to understand its distribution.
2. Create pre-split table on v2: Based on the sampling results, we create a corresponding v2 table with a pre-defined shard layout to minimize data reshuffling during migration.
3. Bootstrap: This is the most time-consuming step, taking hours or even days depending on table size. To bootstrap efficiently, we use Kubernetes StatefulSets to persist local state and periodically checkpoint progress.
4. Checksum verification: We verify that all data from the v1 backup has been correctly ingested into v2.
5. Catch-up: We apply any lagging messages that accumulated in Kafka during the bootstrap phase.
6. Dual writes: At this stage, both v1 and v2 consume from the same Kafka topic. We ensure eventual consistency between the two, with replication lag typically within tens of milliseconds.

Once data migration is complete and we enter dual write mode, we can begin the read traffic migration phase. During this phase, our dispatcher can be dynamically configured to serve read requests for specific tables from v1, while sending shadow requests to v2 for consistency checks. We then gradually shift to serving reads from v2, accompanied by reverse shadow requests to v1 for consistency checks, which also enables quick fallback to v1 responses if v2 becomes unstable. Eventually, we fully transition to serving all read traffic from v2.

Lessons learned

Several key insights emerged from this migration:

• Consistency complexity: Migrating from an eventually consistent (v1) to a strongly consistent (v2) backend introduced new challenges, particularly around write conflicts. Addressing these required features like write deduplication, hotkey blocking, and lazy write repair — sometimes trading off storage cost or read performance.
• Presplitting is critical: As we shifted from hash-based (v1) to range-based partitioning (v2), inserting large consecutive data could cause hotspots and disrupt our v2 backend. To prevent this, we needed to accurately sample the v1 data and presplit it into multiple shards based on v2’s topology, ensuring balanced ingestion traffic across backend nodes during data migration.
• Query model adjustments: v2 doesn’t push down range filters as effectively, requiring us to implement client-side pagination for prefix and range queries.
• Freshness vs. cost: Different use cases required different tradeoffs. Some prioritized data freshness and used primary replicas for the latest reads, while others leveraged secondary replicas to balance staleness with cost and performance.
• Kafka’s role: Kafka’s proven stable p99 millisecond latency made it an invaluable part of our migration process.
• Building in flexibility: Customer retries and routine bulk jobs provided a safety net for the rare inconsistencies, and our migration design allowed for per-table stage assignments and instant reversibility — key for managing risk at scale.

As a result, we migrated more than a petabyte of data across thousands of tables with zero downtime or data loss, thanks to a blue/green rollout, dual-write pipeline, and automated fallbacks — so the product teams could keep shipping features while the engine under them evolved.

Conclusion and next steps

What sets Mussel v2 apart is the way it fuses capabilities that are usually confined to separate, specialized systems. In our deployment of Mussel V2, we observe that this system can simultaneously

1. ingest tens of terabytes in bulk data upload,
2. sustain 100 k+ streaming writes per second in the same cluster, and
3. keep p99 reads under 25 ms

— all while giving callers a simple dial to toggle stale reads on a per-namespace basis. By pairing a NewSQL backend with a Kubernetes-native control plane, Mussel v2 delivers the elasticity of object storage, the responsiveness of a low-latency cache, and the operability of modern service meshes — rolled into one platform. Engineers no longer need to stitch together a cache, a queue, and a datastore to hit their SLAs; Mussel provides those guarantees out of the box, letting teams focus on product innovation instead of data plumbing.

Looking ahead, we’ll be sharing deeper insights into how we’re evolving quality of service (QoS) management within Mussel, now orchestrated cleanly from the Dispatcher layer. We’ll also describe our journey in optimizing bulk loading at scale — unlocking new performance and reliability wins for complex data pipelines. If you’re passionate about building large-scale distributed systems and want to help shape the future of data infrastructure at Airbnb, take a look at our Careers page — we’re always looking for talented engineers to join us on this mission.

References

1. https://medium.com/airbnb-engineering/mussel-airbnbs-key-value-store-for-derived-data-406b9fa1b296

Every request lookup on Airbnb, from stays, experiences, and services search to customer support inquiries ultimately hits Mussel, our multi-tenant key-value store for derived data. Mussel operates as a proxy service, deployed as a fleet of stateless dispatchers — each a Kubernetes pod. On a typical day, this fleet handles millions of predictable point and range reads. During peak events, however, it must absorb several-fold higher volume, terabyte-scale bulk uploads, and sudden bursts from automated bots or DDoS attacks. Its ability to reliably serve this volatile mix of traffic is therefore critical to both the Airbnb user experience and the stability of the many services that power our platform.

Given Mussel’s traffic volume and its role in core Airbnb flows, quality of service (QoS) is one of the product’s defining features. The first-generation QoS system was primarily an isolation tool. It relied on a Redis-backed counter, client quota based rate-limiter, that checked a caller’s requests per second (QPS) against a configurable fixed quota. The goal was to prevent a single misbehaving client from overwhelming the service and causing a complete outage. For this purpose, it was simple and effective.

However, as the service matured, our goal shifted from merely preventing meltdowns to maximizing goodput — that is, getting the most useful work done without degrading performance. A system of fixed, manually configured quotas can’t achieve this, as it can’t adapt in real time to shifting traffic patterns, new query shapes, or sudden threats like a DDoS attack. A truly effective QoS system needs to be adaptive, automatically exerting prioritized backpressure when it senses the system has reached its useful capacity.

To better match our QoS system to the realities of online traffic and maximize goodput, over time we evolved it to add several new layers.

• Resource-aware rate control (RARC): Charges each request in request units (RU) that reflect rows, bytes, and latency, not just counts.
• Load shedding with criticality tiers: Guarantees that high-priority traffic (e.g., customer support, trust and safety) stays responsive when capacity evaporates.
• Hot-key detection & DDoS mitigation: Detects skewed access patterns in real time and then shields the backend — whether the surge is legitimate or a DDoS burst — by caching or coalescing the duplicate requests before they reach the storage layer.

What follows is an engineer’s view of how these layers were designed, deployed, and battle-tested, and why the same ideas may apply to any multi-tenant system that has outgrown simple QPS limits.

Background: Life with Client Quota Rate Limiter

When Mussel launched, rate-limiting was entirely handled via simple QPS rate-limiting using a Redis-based distributed counter service. Each caller received a static, per-minute quota, and the dispatcher incremented a Redis key for every incoming request. If the key’s value exceeded the caller’s quota, the dispatcher returned an HTTP 429. The design was simple, predictable, and easy to operate.

Two architectural details made this feasible. First, Mussel and its storage engine were tightly coupled; backend effort correlated reasonably well with the number of calls at the front door. Second, the traffic mix was modest in size and variety, so a single global limit per caller rarely caused trouble.

As adoption grew, two limitations became clear.

1. Cost variance: A one-row lookup and a 100,000-row scan were treated equally, even though their load on the backend differed by orders of magnitude. The system couldn’t distinguish high-value cheap work from low-value expensive work.
2. Traffic skew: Per-caller rate limits provided isolation at the client level, but were blind to the data’s access pattern. When a single key became “hot” — for example, a popular listing accessed by thousands of different callers simultaneously — the aggregate traffic could overwhelm the underlying storage shard, even if each individual caller remained within its quota. This created a localized bottleneck that degraded performance for the entire cluster, impacting clients requesting completely unrelated data. Isolation by caller was insufficient to prevent this kind of resource contention.

Addressing these gaps meant shifting from a request-counting mindset to a resource-accounting mindset and designing controls that reflect the real cost of each operation.

Resource-aware rate control

A fair quota system must account for the real work a request imposes on the storage layer. Resource-aware rate control (RARC) meets this need by charging operations in request units (RU) rather than raw requests per second.

A request unit blends four observable factors: fixed per-call overhead, rows processed, payload bytes, and — crucially — latency. Latency captures effects that rows and bytes alone miss: two one-megabyte reads can differ greatly in cost if one hits cache and the other triggers disk. In practice, we use a linear model. For both reads and writes, the cost is:

RU_read = 1 + w_r × bytes_read + w_l × latency_ms
RU_write = 6 + w_b × bytes_written / 4096 bytes + w_l × latency_ms

Weight factors w_r, w_b, and w_l come from load-test calibration 
based on the compute, network and disk I/O. 
bytes_read, bytes_written and latency is measured per request

Although approximate, the formula separates operations whose surface metrics look similar yet load the backend very differently.

Each dispatcher continues to rely on rate-limiter for distributed counting, but the counter now represents request-unit tokens instead of raw QPS. At the start of every epoch, the dispatcher adds the caller’s static RU quota to a local token bucket and immediately debits that bucket by the RU cost of each incoming request. When the bucket is empty, the request is rejected with HTTP 419. Because all dispatchers follow the same procedure and epochs are short, their buckets remain closely aligned without additional coordination.

Adaptive protection is handled in the separate load-shedding layer; backend latency influences which traffic is dropped or delayed, not the size of the periodic RU refill. This keeps rate accounting straightforward — static quotas expressed in request units — while still reacting quickly when the storage layer shows signs of stress.

Load shedding: Staying healthy when capacity evaporates or develops hotspots

Rate limits based on request units excel at smoothing normal traffic, but they adjust on a scale of seconds. When the workload shifts faster — a bot floods a key, a shard stalls, or a batch job begins a full-table scan — those seconds are enough for queues to balloon and service-level objectives to slip. To bridge this reaction-time gap, Mussel uses a load-shedding safety net that combines three real-time signals: (1) traffic criticality, (2) a latency ratio, and (3) a CoDel-inspired queueing policy.

The latency ratio is a ratio that serves as a real-time indicator of stress on the system stress. Each dispatcher computes this ratio by dividing the long-term p95 latency by the short-term p95 latency. A stable system has a ratio near 1.0; a value dropping towards 0.3 indicates that latency is rising sharply. When that threshold is crossed, the dispatcher temporarily increases the RU cost applied to a designated client class so that its token bucket drains faster and the request rate naturally backs off. If the ratio keeps falling, the same penalty can be expanded to additional classes until latency returns to a safe range.

The estimate uses the constant-memory P² algorithm [1], requiring no raw sample storage or cross-node coordination.

The Control-Delay (CoDel) thread pool tackles the second hazard: queue buildup inside the dispatcher itself [2]. It monitors the time a request waits in the queue. If that sojourn time proves the system is already saturated, the request fails early, freeing up memory and threads for higher-priority work. An optional latency penalty can also be applied to RU accounting, charging more for queries from callers that persistently trigger the latency ratio.

Together, these layers — criticality, a real-time latency ratio, and adaptive queueing — form a shield that lets guest-facing traffic ride out backend hiccups. In practice, this system has cut recovery times by about half and keeps dispatchers stable without human intervention.

Hot-key detection and DDoS defence

Request-unit limits and load shedding keep client usage fair, but they cannot stop a stampede of identical reads aimed at one record. Imagine a listing that hits the front page of a major news outlet: tens of thousands of guests refresh their browser, all asking for the same key. A misconfigured crawler — or a deliberate botnet — can generate the same access pattern, only faster. The result is shard overload, a full dispatcher queue, and rising latency for unrelated work.

Mussel neutralises this amplification with a three-step hot-key defence layer: real-time detection, local caching, and request coalescing.

Real-time detection in constant space

Every dispatcher streams incoming keys into an in-memory top-k counter. The counter is a variant of the Space-Saving algorithm [2] popularized in Brian Hayes’s “Britney Spears Problem” essay [4]. In just a few megabytes, it tracks approximate hit counts, maintains a frequency-ordered heap, and surfaces the hottest keys in real time in each individual dispatcher.

Local caching and request coalescing

When a key crosses the hot threshold, the dispatcher serves it from a process-local LRU cache. Entries expire after roughly three seconds, so they vanish as soon as demand cools; no global cache is required. A cache miss can still arrive multiple times in the same millisecond, so the dispatcher tracks in-flight reads for hot keys. New arrivals attach to the pending future; the first backend response then fans out to all waiters. In most cases only one request per hot key per dispatcher pod ever reaches the storage layer.

Impact in production

In a controlled DDoS drill that targeted a small set of keys at ≈ million-QPS scale, the hot-key layer collapsed the burst to a trickle — each dispatcher forwarded only an occasional request, well below the capacity of any individual shard — so the backend never felt the surge.

Retrospective and key takeaways

The journey from a single QPS counter to a layered, cost-aware QoS stack has reshaped how Mussel handles traffic and, just as importantly, how engineers think about fairness and resilience. A few themes surface when we look back across the stages described above.

The first is the value of early, visible impact. The initial release of request-unit accounting went live well before load shedding or hot-key defence. Soon after deployment it automatically throttled a caller whose range scans had been quietly inflating cluster latency. That early win validated the concept and built momentum for the deeper changes that followed.

A second lesson is to prefer to keep control loops local. All the key signals — P² latency quantiles, the Space-Saving top-k counter, and CoDel queue delay — run entirely inside each dispatcher. Because no cross-node coordination is required, the system scales linearly and continues to protect capacity even if the control plane is itself under stress.

Third, effective protection works on two different time-scales. Per-call RU pricing catches micro-spikes; the latency ratio and CoDel queue thresholds respond to macro slow-downs. Neither mechanism alone would have kept latency flat during the last controlled DDoS drill, but in concert they absorbed the shock and recovered within seconds.

Finally, QoS is a living system. Traffic patterns evolve, back-end capabilities improve, and new workloads appear. Planned next steps include database-native resource groups and automatic quota tuning from thirty-day usage curves. The principles that guided this project — measure true cost, react locally and quickly, layer defences — form a durable template, but the implementation will continue to grow with the platform it protects.

Does this type of work interest you? We’re hiring, check out open roles here.

📚 References

1. Raj Jain and Imrich Chlamtac. 1985. The P² algorithm for dynamic calculation of quantiles and histograms without storing observations. Communications of the ACM, 28(10), 1076–1085. https://doi.org/10.1145/4372.4378
2. Erik D. Demaine, Alejandro López-Ortiz, and J. Ian Munro. 2002. Frequency estimation of internet packet streams with limited space. In Algorithms — ESA 2002: 10th Annual European Symposium, Rome, Italy, September 17–21, 2002. Rolf H. Möhring and Rajeev Raman (Eds.). Lecture Notes in Computer Science, Vol. 2461. Springer, 348–360.
3. Kathleen M. Nichols and Van Jacobson. 2012. Controlling queue delay. Communications of the ACM, 55(7), 42–50. https://doi.org/10.1145/2209249.2209264
4. Brian Hayes. 2008. Computing science: The Britney Spears problem. American Scientist, 96(4), 274–279. https://www.americanscientist.org/article/the-britney-spears-problem

Across the more than 220 global markets that Airbnb operates in, cards are the primary way that guests pay for stays, experiences, and services. However, to help make our platform accessible to more people, reduce friction at checkout, and drive more adoption, we introduced trusted, locally preferred payment methods — called local payment methods or LPMs. By offering and supporting these payment methods, Airbnb enables guests everywhere to choose what works best for them.

In this blog post, we’ll discuss the implementation details behind our Pay as a Local initiative, which allowed us to launch 20+ local payment methods across multiple markets in just over one year.

LPMs: What they are, why they matter, and our discovery and selection process

Local payment methods go beyond traditional cards and include:

• Country or region-specific digital wallets (such as M-Pesa or MTN, MoMo)
• Online bank transfers (such as Online Banking Czech, Online Banking Slovakia)
• Real-time or instant bank payments (such as PIX, UPI)
• Local payment schemes (such as EFTPOS, Cartes Bancaires)

By embracing LPMs, Airbnb helps make travel more inclusive and seamless for people around the world. LPMs help the platform to:

• Boost conversion and bookings by offering guests familiar, trusted payment options.
• Unlock new markets where credit card usage is low or non-existent.
• Build accessibility for guests without credit cards or traditional banking access.

Through our research on local payment methods (LPMs), we identified over 300 unique payment options worldwide. For the initial phase of the LPM initiative, we used a structured qualification framework to select which local payment methods we would support. We evaluated the top 75 travel markets and selected the top one to two payment methods per market — excluding those without a clear travel use case — and arrived at a shortlist of just over 20 LPMs best suited for integration into our payment platform.

Background on Airbnb’s payment platform

Airbnb’s payments platform is designed to decouple payment logic from the core business (i.e., stays, experiences, and services), allowing for greater flexibility and scalability. The platform efficiently coordinates both guest pay-ins and host payouts by working with regulated payment service providers and financial partners.

Beyond payment processing, the system also supports robust payment trust and compliance functions.

Modernization

As part of a multi-year replatforming initiative for our payments architecture called Payments LTA (long-term architecture), we shifted from a monolithic system to a capability-oriented services system structured by domains, using a domain-driven decomposition approach. This modernization approach reduced our time to market, increased reusability and extensibility, and empowered greater team autonomy.

The core payment domain delivers essential capabilities for pay-in, payout, and payment intermediation. It consists of multiple subdomains, including Pay-in, Payout, Transaction Fulfillment, Processing, Wallet & instruments, Ledger, Incentives & Stored Value, Issuing, and Settlement & Reconciliation.

Replatforming as an enabler for local payment method expansion

The processing subdomain enables integration with third-party payment service providers (PSPs) and supports API and file-based vendor integration, as well as switching and routing capabilities. As part of our replatforming initiative, we adopted a connector and plugin-based architecture for onboarding new third-party payment service providers. This strategy has significantly reduced the time required to integrate new PSPs in different markets.

During this replatforming effort, we also introduced Multi-Step Transactions (MST): a processor-agnostic framework that supports payment flows completed across multiple stages. MST defines a PSP-agnostic transaction language to describe the intermediate steps required in a payment, such as submitting supplemental data or handling dynamic interactions. These steps, called Actions, can include:

• Redirects
• Strong customer authentication (SCA) frictions (challenges, fingerprinting)
• Payment method — specific flows

When a PSP indicates that an additional user action is required, its vendor plugin normalizes the request into an ActionPayload and returns it with a transaction intent status of ACTION_REQUIRED. This architecture ensures consistent handling of complex, multi-step payment experiences across diverse PSPs and markets.

LPM integration architecture

While our modernized payment platform laid the foundation for enabling LPMs, these payment methods come with a unique set of challenges. Many local methods require users to complete transactions in third-party wallet apps. This introduces complexity in app switching, session hand-off, and synchronization between Airbnb and external digital wallets.

Each local payment vendor also exposes different APIs and behaviors across charge, refund, and settlement flows, making integration and standardization difficult.

Technical approach

We analyzed the end-to-end behavior of our 20+ LPMs, and identified three foundational payment flows that capture the full spectrum of user and system interactions. By distilling LPM behaviors into these standardized payment flow archetypes, we established a unified framework for integration:

1. Redirect flow: Guests are redirected to a third-party site or app to complete the payment, then return to Airbnb to finalize their booking (e.g., Naver Pay, GoPay, FPX).
2. Async flow: Guests complete payment externally after receiving a prompt (such as a QR code or push notification), and Airbnb receives payment confirmation asynchronously via webhooks (e.g., Pix, MB Way, Blik).
3. Direct flow: Guests enter their payment credentials directly within Airbnb’s interface, allowing real-time processing similar to traditional card payments (e.g., Carte Bancaires, Apple Pay).

This standardized approach has enabled significant reusability across integrations and substantially reduced the engineering effort required to support new payment methods.

Asynchronous payment orchestration

Since many guests complete payments through external providers, we redesigned our payment orchestration — building on top of MST — to support payment flows that require user actions outside Airbnb (redirect flows and async flows).

For redirect flows, where guests complete the payment on a third-party app or website:

• Airbnb’s payments platform sends a charge request to the local payment vendor, whose response includes a redirectUrl.
• Our platform redirects the user to the external app or website to complete the payment.
• Once the payment is successfully completed, the user is redirected back to Airbnb with a result token. Airbnb’s payments platform then uses this token to securely confirm and finalize the payment with the local processor.

For async flows (which typically involve scanning a QR code):

• Airbnb’s payments platform sends a charge request to the local payment vendor, whose response includes a qrCodeData.
• The checkout page displays the QR code for the user to scan and complete the payment in their wallet app.
• After the payment succeeds, the vendor sends a webhook notification to Airbnb’s payments platform, which updates the payment status to success and confirms the user’s order.

Naver Pay: Redirect To Naver Pay Website

Naver Pay is one of the fastest-growing digital payment methods in South Korea. As of early 2025, it has reached over 30.6 million active users, representing approximately 60% of the South Korean population. Enabling Naver Pay in the South Korean market not only helps deliver a more seamless and familiar payment experience for local guests, but also expands Airbnb’s reach to new users who prefer using Naver Pay as their primary payment method.

Pix: Scan A QR Code

Pix is an instant payment system developed by the Central Bank of Brazil, enabling 24/7 real-time money transfers through methods such as QR codes or Pix keys. Its adoption has been extraordinary — by late 2024, more than 76% of Brazil’s population was using Pix, making it the country’s most popular payment method, surpassing cash, credit, and debit cards. In 2024 alone, Pix processed over BRL 26.4 trillion (approximately USD 4.6 trillion) in transaction volume, underscoring its pivotal role in Brazil’s digital payment ecosystem.

Config-driven payment method integration

Airbnb embraced a config-driven approach, powered by a central YAML-based Payment Method Config that acts as a single source of truth for flows, eligibility, input fields, refund rules, and more. Instead of scattering payment method logic across the frontend, backend, and various services, we consolidate all relevant details in this config. Both core payment services and frontend experiences dynamically reference this single source of truth, ensuring consistency for eligibility checks, UI rendering, and business rules. This unified approach dramatically reduces duplication, manual updates, and errors across the stack, making integration and maintenance faster and more reliable.

These configs also drive automated code generation for backend services using code generation tools, producing Java classes, DTOs, enums, schema, and integration scaffolding. As a result, integrating or updating a payment method is largely declarative — just a config change. This streamlines launches from months to weeks and makes ongoing maintenance far simpler.

Payment widget

Our payment widget — the payment method UI embedded into the checkout page — includes the list of available payment methods and handles the user’s inputs. Local payment methods often require specialized input forms (such as CPF for Pix) and have unique country/currency eligibility.

Rather than hardcoding forms and rules into the client, we centralize both form-field specification and eligibility checks in the backend. Servers send configuration payloads to clients defining exactly which fields to collect, which validation rules to apply, and which payment options to render. This empowers the frontend to dynamically adapt UI and validation for each payment method, accelerating launches and keeping user experiences fresh without frequent client releases.

For example, Pix in Brazil requires the guest’s first name, last name, and CPF (tax ID), which we collect and transmit as required to complete the payment.

Below is a diagram illustrating how dynamic payment method configurations are delivered from the backend to the frontend, enabling tailored checkout presentations for each payment method.

Building confidence through better testability

Testing local payment methods can be difficult, because developers often don’t have access to local wallets. Yet with such a broad range of payment methods and complex flows, comprehensive testing is essential to prevent regressions and ensure seamless functionality.

To address this, we enhanced Airbnb’s in-house Payment Service Provider (PSP) Emulator, enabling realistic simulation of PSP interactions for both redirect and asynchronous payment methods. The Emulator allows developers to test end-to-end payment scenarios without relying on unstable (or nonexistent) PSP sandboxes. For redirect payments, the Emulator provides a simple UI mirroring PSP acquirer pages, allowing testers to explicitly approve or decline transactions for precise scenario control. For async methods, it returns QR code details and automatically schedules webhook emission tasks upon receiving a /payments request — delivering a complete, reliable testing environment across diverse LPMs.

Scaling observability for local payment methods

Maintaining high reliability and availability is critical for Airbnb’s global payment system. As we expand to support many new local payment methods, we face increasing complexity: greater dependencies on external PSPs and wide variations in payment behaviors. For example, a real-time card payment and a redirect flow like Naver Pay follow completely different technical paths. That diversity makes observability difficult — a single “payment success rate” may represent card health well, but say little about an asynchronous LPM. Without proper visibility, regressions can go unnoticed until they affect real users. As dozens of new LPMs go live, observability has become the foundation of reliability.

To address this, we built a centralized monitoring framework that unifies metrics across all layers, from client to PSP. When launching a new LPM, onboarding now requires a single config change; add the method name, and metrics begin streaming automatically:

• Client metrics — user-level flow health from clients
• Payment backend metrics — API-level metrics for payment flows
• PSP metrics — API-level visibility between Airbnb and the PSP
• Webhook metrics — async completion status for redirect methods or refunds

We have also standardized the alerting rules across our platform’s Client, Backend, PSP, and Webhook layers using composite alerts and anomaly detection. Each alert follows a consistent pattern (failure count, rate, time window), e.g., “Naver Pay resume failures > 5 and failure rate > 20% in 30 minutes.” This design minimizes false positives during low-traffic periods.

This framework scales effectively, providing end-to-end visibility from user click to PSP confirmation. It enables engineers to trace issues in minutes rather than hours, whether those issues were caused by internal changes or external outages. By turning observability into a shared, automated layer, we were able to strengthen the backbone of payment reliability while accelerating the rollout of new LPMs worldwide.

Impact

The Pay as a Local initiative delivered significant business and technical impact:

• Meaningful booking uplift: We observed meaningful uplift in bookings and new users in markets where we launched local payment methods
• Faster integrations: Reduced integration time significantly through reusable flows and config-driven automation.
• Stronger reliability: Improved observability for early outage detection, standardized testing to prevent regressions, and streamlined vendor escalation and on-call processes for global resilience.

Conclusion

Supporting local payment methods helps Airbnb to stay competitive and relevant in the global travel industry. These payment options help improve checkout conversion, drive adoption, and unlock new growth opportunities.

This post outlined how the Airbnb payment platform has evolved to support local payment methods at scale — through asynchronous payment orchestration, config-driven onboarding, centralized observability, and robust testability. Together, these capabilities enable faster integrations, lower maintenance overhead, and offer a more seamless, localized checkout experience for guests worldwide.

As Airbnb continues to expand globally, our payments platform will keep evolving with the same principles of extensibility, reliability, and scalability, ensuring that guests everywhere can pay confidently, using the methods they know and trust.

Acknowledgments

We had many people at Airbnb contributing to this big rearchitecture, but countless thanks to Mini Atwal, Ashish Singla, Musaab At-Taras,Linmin Yang, Yong Rhyu, Yohannes Tsegay, Livar Cunha, Praveena Subrahmanyam, Steve Ickes, Vijaykumar Borkar, Vibhu Ramani, Aashna Jain, Abhishek Ghosh, Abhishek Patel, Adithya Tammavarapu, Akai Hsieh, Akash Budhia, Amar Parkash, Amee Mewada, Ankita Balakrushan Tate, Bharath Kumar Chandramouli, Bo Shi, Bo Yuan. Callum Li. Carlos Townsend Pico, Chanakya Daparthy, Charles Tang, Cibi Pari, Cindy Jaimez, Cindy Shi, Dan Yo, Daniela Nobre, Danielle Zegelstein, David Cordoba, David Drinan, Dawei Wang, Dechuan Xu, Denise Francisco, Denny Liang, Dimi Matcovschi, Divya Verma, Feifeng Yang, Gabriel Siqueira, Sunny Wallia, Prashant Jamlakar, Daniel Kriske, Giovanni Iniguez, Haojie Zhang, Haokun Chen, Haoti Zhong, Harriet Russell, Harshit Gupta, Henrique Moreira Indio do Brasil, Ishan Ishan, Jenny Shen, Jerroid Marks, Jiafang Jiang, Joey Yin, Jon Chew, Karen Kuo, Katie Turley, Letian Zhang, Maneesh Lall, Manish Singhal, Maria Daneri, Mark Jang, Mengfei Ren, Michelle Desiderio, Mohit Dhawan, Nam Kim, Nerea Ruiz Alvarez, Nikita Kapoor, Oliver Zhang, Omer Faruk Gul, Pallavi Sharma, Prateek Sri, Rae Huang, Rohit Krishnan Dandayudham, Rory MacQueen, Ruize Liu, Sam Bitter, Sam Tang, Saran Singh. Sardana Sai Anil, Serdar Yildirim, Shwetha Saibanna, Silvia Crespo Sanchez, Simon Xia, Stella Dong, Stella Su, Stephanie Leung, Steve Cao, Sumit Ranjan, Tay Rauch, Thanigaivelan Manickavelu, Tiffany Selby, Toland Hon, Trish Burgess,Vishal Garg, Vivian Lue, Vyom Rastogi, William Betz, Xi Wen, Xing Xing, Xuanxuan Wu, Yangguang Li, Yanwei Bai, Yeung Song, Yixia Mao, Yujia Liu. Yun Cho, Zhenhui Zhu, Ziyun Ye

****************

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Across these conferences, Airbnb researchers engaged directly with academic and industry peers by publishing and presenting papers, learning about the latest innovations, launching new collaborations, and mentoring emerging researchers. In this blog post, we’ll recap the conferences and key papers we presented in 2025, organized by research themes.

Applied machine learning for search, ranking, and personalization

KDD (Knowledge and Data Mining)

KDD is a flagship conference in data science research. Hosted annually by a special interest group of the Association for Computing Machinery (ACM), it’s where researchers learn about some of the most groundbreaking developments in data mining, knowledge discovery, and large-scale data analytics, which are critical to Airbnb’s efforts to improve core products like search and recommendations.

Our participation

We’ve been presenting at KDD since 2018, and 2025 was another strong year for us. We received multiple contributions across the applied data science track and workshops, which were well-received by the broader community and even inspired us to consider open-sourcing some of our technology. We were also inspired by the related research in this area and are eager to explore these methods through new collaborations.

Research highlights

1. Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking: While A/B tests are crucial for developing ranking algorithms and recommender systems, they’re difficult to set up and can take extensive time to reach statistical significance (especially for products with long conversion cycles, like accommodation booking). In this paper, we shared techniques for rapid pre-A/B online assessments that help teams identify the most promising experiments, streamlining the overall process without sacrificing accuracy.
2. High Precision Audience Expansion via Extreme Classification in a Two-Sided Marketplace: Airbnb search balances diverse global inventory with varied guest preferences for location, amenities, style, and price. This process requires efficient location retrieval to find the listings guests might realistically book by determining which geographic areas to query. We introduce a new approach to location retrieval by using a set of relevant, high-precision categorical location cells.

Link to all papers

• Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking (Qing Zhang, Alex Deng, Michelle Du, Huiji Gao, Liwei He, Sanjeev Katariya)
• High Precision Audience Expansion via Extreme Classification in a Two-Sided Marketplace (Dillon Davis, Huiji Gao, Thomas Legrand, Juan Manuel Caicedo Carvajal, Malay Haldar, Kedar Bellare, Moutupsi Paul, Soumyadip Banerjee, Liwei He, Stephanie Moyerman, and Sanjeev Katariya)
• TSMO: Two-sided Marketplace Optimization

CIKM (Conference on Information and Knowledge Management)

CIKM is a premier forum for discussing and presenting research at the intersection of information and knowledge management, including topics like AI, data mining, database systems, and information retrieval. Many of these topics directly intersect with our core product challenges, such as search, ranking, and recommendations.

Our participation

At CIKM 2025, Airbnb’s Relevance and Personalization team had five peer-reviewed papers accepted for publication, building on our participation in 2023 and 2024. These papers focused on advanced AI/ML techniques for search and recommendations, and sharing real-world insights from using these technologies at Airbnb’s scale. Industry and academic researchers, especially those working on two-sided marketplaces, engaged with our work and provided valuable feedback.

Research highlights

1. Augmenting Guest Search Results with Recommendations at Airbnb: When guests use overly narrow criteria to search for accommodations, they often receive insufficient results, leading to a frustrating experience. This paper introduces a recommendation system that dynamically suggests alternatives — different dates, relaxed amenities, or adjusted price ranges — to help guests find suitable accommodations and improve the platform’s booking rate. Authors: Haowei Zhang, Philbert Lin, Dishant Ailawadi, Soumyadip Banerjee, Shashank Dabriwal, Hao Li, Kedar Bellare, Liwei He, Sanjeev Katariya
2. Maps Ranking Optimization in Airbnb: Maps play a crucial role in Airbnb search and bookings, accounting for roughly 80% of search interactions. Yet map ranking has traditionally reused feed-ranking assumptions, which break down when we examine the NDCG (Normalized Discounted Cumulative Gain) metric. This paper explains why list-based NDCG fails to model user attention on maps, introduces a map-specific NDCG, and reports experiments showing that optimizing it yields booking gains. Authors: Hongwei Zhang, Malay Haldar, Kedar Bellare, Sherry Chen, Soumyadip Banerjee, Xiaotang Wang, Mustafa Abdool, Huiji Gao, Pavan Tapadia, Liwei He, Sanjeev Katariya, Stephanie Moyerman
3. BListing: Modality Alignment for Listings: To improve search ranking, we introduce BiListing (Bimodal Listing) embeddings to use unstructured text and photo listing data as ranking signals. BiListing leverages large-language models and pretrained language-image models to create unified representations of diverse unstructured data into a single embedding vector per list and modality. Our experiment results show a 0.425% increase in NDCB (Normalized Discounted Cumulative Booking) gain and drove tens of millions in incremental revenue. Authors: Guillaume Guy, Mihajlo Grbovic, Chun How Tan, Han Zhao
4. Beyond Pairwise Learning-To-Rank At Airbnb: In this paper, we introduce a method to improve the accuracy of pairwise learning-to-rank algorithms, the bedrock of modern search stacks. This approach captures interactions between items during pairwise comparisons, thereby giving us a better sense of what searchers truly want. We also share ways to implement this algorithm performantly, and results from online and offline experiments. Authors: Malay Haldar, Daochen Zha, Huiji Gao, Liwei He, Sanjeev Katariya
5. Learning to Comparison-Shop: Traditional ranking models often evaluate items in isolation, disregarding the context in which users compare multiple items on a search results page. In this paper, we propose a novel ranking architecture, the Learning-to-Comparison-Shop (LTCS) System, that explicitly models and learns users’ comparison-shopping behaviors. Our experiments show statistically significant improvements of 1.7% in Normalized Discounted Cumulative Gain (NDCG) and 0.6% in booking conversion rate. Authors: Jie Tang, Daochen Zha, Xin Liu, Huiji Gao, Liwei He, Stephanie Moyerman, Sanjeev Katariya

NLP & building LLM systems in production

EMNLP (Empirical Methods in Natural Language Processing)

EMNLP is a top-tier NLP conference that brings together practitioners and researchers to discuss new architectures and training strategies for language models, safety and evaluation strategies for LLMs, and real-world NLP applications. These research areas directly intersect with many of Airbnb’s product surfaces, such as customer support, search & discovery, and trust & safety. Additionally, each EMNLP cycle includes the release of new datasets, evaluation suites, and open-source libraries to help teams benchmark their progress against community standards.

Our participation

In 2025, we sponsored EMNLP and presented two papers on humans-in-the-loop in AI systems and advanced summarization techniques. We also used EMNLP’s community datasets to benchmark our system, which showcased where we excel and where we can build upon our success with additional best practices. The conference deepened academic collaborations through discussions on LLM evaluation, safety, and agentic AI design, including mentoring students and early-career researchers.

Research highlights

1. Agent-in-the-Loop, A Data Flywheel for Continuous Improvement in LLM-based Customer Support: To improve our LLM-based customer support system, this paper introduces an Agent-in-the-Loop (AITL) framework that leverages new interaction data to continuously enhance model performance. This flywheel can help the system stay up to date with new product features, shifting user preferences, and updated support policies and procedures. We launched a pilot in the US, and the results demonstrate significant improvement in accuracy and helpfulness. Authors: Cen Mia Zhao, Tiantian Zhang, Hanchen Su, Yufeng Wayne Zhang, Shaowei Su, Mingzhi Xu, Yu Elaine Liu, Wei Han, Jeremy Werner, Claire Na Cheng, Yashar Mehdad
2. Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback: Customer service agents multitask during support interactions, identifying core issues, tracking prior actions, and producing accurate notes. To streamline this workflow, we introduced an incremental summarization system that intelligently determines when to generate concise bullet notes during conversations, reducing agents’ context-switching effort without sacrificing quality. To improve the system over time, we also introduced a learning framework that enables agents to make real-time edits, immediately refining online note generation. Authors: Yisha Wu, Cen Mia Zhao, Yuanpei Cao, Xiaoqing Su, Yashar Mehdad, Mindy Ji, Claire Na Cheng

COLING (International Conference on Computational Linguistics)

COLING is a top-tier NLP conference that covers both foundational research and industry applications of language models, including reasoning, evaluation, multilingual NLP, and real-world LLM systems. The work presented at this conference helps validate Airbnb’s technical direction and directly informs future investments.

Our participation

In 2025, Airbnb presented at COLING for the first time, sharing a paper titled “LLM-Friendly Knowledge Representation for Customer Support” by Hanchen Su, Wei Luo, Wei Han, Yu Elaine Liu, Yufeng Wayne Zhang, Cen Mia Zhao, Ying Joy Zhang, and Yashar Mehdad. The paper presents a new format, Intent, Context, and Action (ICA), for structuring business knowledge in LLM-based QA and customer support workflows. Initial experiments in production show promising results. We also discovered relevant research in knowledge retrieval, LLM evaluation, and hallucination detection that will inspire future projects.

Optimization, causal inference, and measurement science

MIT CODE (Conference on Digital Experimentation)

MIT CODE is one of the premier venues for researchers and practitioners to discuss topics in online digital experimentation, causal inference, and data-driven product innovation. The conference supports our commitment to data-driven decision-making and using experimentation to understand the long-term impacts on guests, hosts, and marketplace health.

Our participation

In 2025, we had another strong showing at CODE, with a cohort of 6 data scientists and 3 academic collaborators. We gave talks in two sessions and presented a poster, which led to meaningful discussions with peer companies and interest in collaborating with academic research groups.

Research highlights

1. Beyond the Experiment Window: Prospective Impacts Under Long-Term Ranking Dynamics: Product teams frequently leverage A/B tests to assess different rankers. While these experiments are typically conducted over shorter periods, we also recognize the value of understanding longer-term dynamics (such as seasonality and user evolution) to further support sustained business objectives, like marketplace health. To solve this problem, we developed a causal framework that allows us to estimate the long-term impacts of ranking changes with strategic goals (like marketplace health) using A/B testing data.
2. Trustworthy Bayesian Inference in Batch-Adaptive Experimentation: Adaptive experimentation, like multi-arm bandit methods, can improve experiment efficiency by reallocating traffic toward promising treatments. Continued advancements in these approaches are expanding our ability to maintain high standards of statistical validity. This paper introduces a practical Bayesian framework for inference in batch-adaptive experiments, specifically tailored to the operational realities of online platforms.

Link to all papers

• Beyond the Experiment Window: Prospective Impacts Under Long-Term Ranking Dynamics (Lo-Hua Yuan)
• Trustworthy Bayesian Inference in Batch-Adaptive Experimentation (Yicheng Li)
• Experimental Design for Product Launches with Collaborative User Networks (Monu Kala)

INFORMS (Institute for Operations Research and the Management Sciences)

INFORMS brings together academics and industry professionals to discuss and share research across data science, machine learning, economics, behavioral science, and analytics.

Our participation

In 2025, our data science team was invited to INFORMS to present two talks in a session about bridging the gap between statistical methods and industry applications.

Research highlights

1. Beyond Multi-Arm Bandits: Tackling Challenges in Adaptive Experiments at Airbnb. In this talk, we walked through the metrics and infrastructure challenges when using classic bandit algorithms, which make it difficult to operationalize adaptive experiments. We propose a hybrid approach that incorporates bandit algorithms into A/B experiments to enable adaptive testing. We also discussed how we onboard and validate adaptive experiments across individual product domains at Airbnb.

Link to all papers

• Beyond the Experiment Window: Prospective Impacts Under Long-Term Ranking Dynamics (Lo-Hua Yuan)
• Beyond Multi-Arm Bandits: Tackling Challenges in Adaptive Experiments at Airbnb (Yicheng Li)

LION (Learning and Intelligent Optimization)

The LION conference is a premier gathering of researchers exploring the intersection of machine learning, artificial intelligence, and mathematical optimization.

Our participation

While Airbnb has attended LION in the past, 2025 was the first time we presented at the conference. Nathan Brixius presented “Optimal Matched Block Design For Multi-Arm

Experiments,” which introduces a new optimization formula using mixed-integer programming (MIP) to group subjects in multi-armed experiments, leading to more balanced groups and, in turn, more accurate experimental results. We also connected with leading experts in metaheuristics and AI fairness to help shape our future roadmap and sponsored the awards for the best papers presented at the conference.

Data systems

VLDB (Very Large Data Bases)

The VLDB Conference is one of the top 2 flagship conferences in data management and large-scale data systems, with over 1,500 researchers and practitioners attending.

Our participation

“In 2025, we published our first paper at VLDB: ‘SQL:Trek Automated Index Design at Airbnb’ by Sam Lightstone and Ping Wang. The paper presents a novel approach for automated index design (code-named SQL:Trek). It uses query compiler cost models to identify effective indexes across many relational databases, including most MySQL and PostgreSQL derivatives. Additionally, the Airbnb team attended sessions on system efficiency, graph computing, and AI databases, and had the opportunity to meet other researchers.

Conclusion

Conferences remain a big part of our research program at Airbnb, helping us validate and refine our ideas through community feedback and providing a forum to share real-world insights that advance the field. In 2025, we doubled down on this vision by publishing papers for the first time at conferences in domains such as NLP, optimization, causal inference, and data systems, reflecting our ongoing commitment to using these technologies to create the best possible travel experiences.

As we look to 2026, we’re eager to expand our presence at these conferences and discover new ways to use AI, machine learning, and data science to build a best-in-class travel and living platform. If you’re interested in doing this type of work with us, consider joining us. Apply for one of our open positions.

Like any powerful tool, dynamic configuration is a double-edged sword. While it enables fast iteration and rapid incident response, a bad change can cause regressions or even outages. This is a common challenge across the industry: balancing developer flexibility with system reliability.

In this post, we will outline the expectations of a modern dynamic configuration platform, then walk through the high-level architecture of Airbnb’s dynamic config platform and how its core components work together to enable safe, flexible config changes.

Modern config platform essentials

As Airbnb’s business grows, our expectations for the dynamic config platform have evolved over time through our own learnings as well as industry best practices. These shape our view of what a good dynamic config platform should provide, including:

• A coherent experience for config management: The platform provides a streamlined, end-to-end experience for defining, reviewing, testing, and rolling out config changes. It covers the most common needs out of the box with rich built-in features, while still offering escape hatches for edge cases.
• Strong reliability, availability and safety guarantees: All config changes are validated, reviewed, and rolled out progressively, with clear ownership and well-defined access control. Treating config as code is a key focus: config changes are versioned, reviewed, and auditable like service code, but remain dynamic at runtime. The platform itself must be highly available so that services can reliably fetch and apply configs. Changes should be observable, with support for fast rollbacks when needed.
• Safe testing in isolated environments: Developers can validate config changes in isolated local or canary environments before they reach production.
• Flexible multi-tenant support: In a multi-tenant platform, different tenants have different risk profiles. The platform should allow config owners to customize how their configs behave per tenant, including deployment triggers, guardrails, and rollout strategy (for example, AWS zone or Kubernetes pod percentage-based rollouts).
• Fast and controlled incident response: During an incident, responders can ship emergency configs as needed with clear auditability. The platform also provides observability for config changes, so incident responders can tell what changed, who was affected, when the change was made, and who made the change. This enables them to effectively identify the culprit and take action.

High-level architecture

At Airbnb, Sitar is the internal name for our dynamic config platform. It provides a common way for teams to manage runtime behavior safely. At a high level, Sitar has four main parts: a developer-facing layer, a control plane, a data plane, and the clients and agents that run alongside application code.

The developer-facing layer is where config changes are created and reviewed. By default, configs are managed through a Git-based workflow, while a few exceptions are managed in the web interface (sitar-portal), which is also used for admin operations such as emergency deployments.

The control plane is responsible for orchestrating config changes. It enforces schema validation, ownership, and access control, and decides how each change should be rolled out: for example, which environments or AWS zone to target, what percentage of Kubernetes pods to start with, and how to progress the rollout over time. The control plane also specifies how to roll back the changes when needed, and supports routing in-flight configs to specific environments or slices of subscribers for fast testing.

The data plane provides scalable storage and efficient distribution of configs. It acts as the source of truth for config values and versions, and propagates updates to services reliably, consistently, and quickly.

On the product services side, an agent sidecar running alongside each service fetches the subscribed configs from the data plane and maintains a local cache. Client libraries inside the service then read from this cache and expose configs to application logic with fast, in-process access and optional fallbacks.

Putting these together, a typical change starts from a Git flow, proceeds through control-plane validation and rollout decisions, into the data plane for distribution, and finally to agents and client libraries that apply the config updates to application logic.

Key design choices

In this section, we highlight a few key design choices that shape how the platform looks and is operated.

Configs as code with a Git-based workflow

Config changes are by default managed by a Git-centric workflow. We use GitHub as the primary interface for managing configs, because we have an established and responsive internal team to manage GitHub Enterprise. GitHub integrates naturally with our existing CI/CD tooling, so we can reuse rich validation and deployment pipelines without re-inventing the wheel. This approach gives developers a consistent experience to make code changes: open a pull request, get reviews, merge, and deploy. GitHub also brings additional benefits such as mandatory reviewers, review and approval flows, and a change history. Configs under the same theme are grouped into tenants, with clear owners, customizable tests, and a dedicated CD pipeline.

While the Git-based flow is the default, we keep a UI portal for teams that prefer a portal-based experience and as a shortcut for specific operational needs, such as fast emergency config updates that can bypass the normal CI/CD pipeline.

Staged rollouts and fast rollbacks

When a change is proposed, schema validation (checking that the config matches the expected structure and types) and other automated checks run in CI. The change is always reviewed and approved before rollout.

Once merged in the main branch, the control plane performs a staged rollout where the change is first deployed to a limited scope, then gradually expanded to a larger scope if things look good. At each stage of this rollout, the change is evaluated, the author and the stakeholders are notified if regressions are detected, and a fast rollback can be triggered if needed. Staged rollouts can greatly reduce the blast radius of bad changes and improve the overall reliability of the platform.

Separated control and data planes

We separate the “decide” and “deliver” responsibilities. The control plane focuses on validation, authorization, and rollout decisions, while the data plane focuses on storing configs and distributing them reliably at scale. This separation allows us to evolve rollout strategies and policies without disrupting the underlying storage and delivery mechanisms, and vice versa.

Local caching and resilient clients

On the product services side, we introduced a local caching layer between the agent sidecar and the client library to improve resilience and availability. The agent sidecar runs alongside the main service container, regardless of which language the service is written in, and periodically fetches subscribed configs from the backend and persists them locally. The client libraries then read from this local cache. Even if the backend is temporarily unavailable or degraded, services can continue operating on the last known good configs from the local cache.

Impact on product teams

It is essential for the Sitar system to make life easier for product teams. In practice, its architecture changes how teams ship and operate in a few ways:

• Rollouts become safer and more predictable. New behaviors, such as refined authorization rules, can be introduced gradually, verified on a small slice of traffic or in a specific environment, and rolled back quickly if needed. Teams spend less time worrying about “big bang” releases and more time iterating on behavior.
• Teams get more flexibility in how their configs are managed and rolled out. Each team can tailor a config flow to its own risk profile and release schedules. For example, teams can choose between automatic, manual, or cron rollouts, select the rollout strategy, and add extra checks. This lets teams keep their existing ways of working while still benefiting from a common platform and shared guardrails.
• Incident mitigation becomes faster and more controlled. When something goes wrong in production, incident responders can use observability tools that integrate config events to quickly locate the culprit change, then take quick action using the portal’s emergency flow. These emergency updates are fully auditable for future review.

Besides these examples, the platform includes other improvements in usability, safety, and observability that we will not cover in detail here. Together, they contribute to a smoother day-to-day experience for teams that rely on dynamic configuration.

Conclusions and next steps

Dynamic configuration is a foundational capability of modern infrastructure. It enables fast iteration and rapid incident response, but only when it is equipped with strong safety features and provides a good developer experience. In this post, we shared how we think about a modern dynamic config platform at Airbnb, and how we developed Sitar’s architecture to meet those expectations.

The work is ongoing. As Airbnb’s business grows, we are continuing to refine rollout strategies, improve config testing, invest in observability and smart incident response tooling, and evolve other platform components.

In future posts, we plan to dive deeper into specific areas of the platform, such as how we optimize the Kubernetes sidecar that delivers config updates and how we design the developer experience around config management.

If this type of work interests you check out our open roles.

Acknowledgments

Our progress with Sitar would not have been possible without the support and contributions of many people. We’d like to thank Craig Sosin, Nikolaj Nielsen, Daniel Fagnan, Alex Edwards, Xian Gao, Nick Morgan, Carolina Calderon, Hanfei Lin, Joyce Li, Yunong Liu, Alex Berghage, Brian Wolfe, Yann Ramin, Denis Sheahan, Richa Khandelwal, Swetha Vaidy, Abhishek Parmar, Adam Kocoloski, Adam Miskiewicz, and all the other engineers and teams at Airbnb who joined design reviews and offered valuable feedback, as this work would not have been possible without them.

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Discovering a passion after the Soviet Union

I grew up in Eastern Ukraine, and the year I was graduating from high school, the Soviet Union collapsed. Despite the political turmoil, it was an interesting time to get into technology, and I have my brother to thank for that.

I was always a nerdy kid, at school and at home, and my older brother really stoked that curiosity. He was studying computer hardware in Moscow, and he’d bring home computer parts to play with. I still remember the first computer he’d assembled, which required a cassette player to load programs. Only after many minutes of buzzing and clicking would the computer finally whirr to life.

Thinking back, that was really my first inspiration to work in technology. Seeing the inner workings of this new thing, a computer, and watching how the parts came together to form a whole — that’s what made me realize I wanted to work with computers, too.

Of course, I didn’t know the Soviet Union would end, which made studying in Moscow impossible. But technology was still my future.

English: Harder than any programming language

I got my start learning programming in a local Ukrainian university, and after four years of studying, I immigrated to America.

When I arrived, I knew how to program, and I knew how to write and read English, but I couldn’t communicate well. I took ESL classes at a community college and, in parallel, enrolled in Berkeley Extension classes to advance my C++ knowledge and learn Java, which was still very new at the time.

Throughout my first couple of jobs, I was more likely to run into challenges with the English language rather than with programming languages.

My first job was in computer hardware diagnostics at a tiny company with only five engineers, where we communicated directly with hardware manufacturers. This was right before the dot-com bubble burst.

I almost didn’t get the job, though. The interview process for this job included a written portion that tested my knowledge of key computer science terms before getting to solve the coding problems. Given my prior education, I knew all the terms, but I ran out of time because the language gap slowed me down. Luckily, my interviewer happened to be taking the same Java class at Berkeley, and when I explained what happened, he gave me the chance to come back. I finished the test, got the job, and the rest is history.

In subsequent jobs, I transitioned fully from C++ to Java, which became my primary programming language for many years. I eventually got the hang of speaking English more confidently, but for a while, it still felt like Russian was my first language, Java was my second, and English was only my third.

Going deeper in the stack and taking on leadership roles

At various times, my career often felt all over the place. But looking back, I see a trajectory I wasn’t aware of at the time. I started with a brief stint in hardware diagnostics, but after that, I worked in the frontend and, over time, descended the software stack from frontend to backend to the deeper infrastructure I work with today.

Parallel to this trajectory down the stack was an upward trajectory in responsibility. Leadership wasn’t an obvious path for me at first — I had to be pitched multiple times — but the more I tried it, the more interesting and enjoyable it felt.

When I worked at Caymas Systems, a telecom startup, my manager was quick to recognize my leadership potential. He was really encouraging, but even more encouraging was witnessing the difference between teams with good leaders and those without.

After Caymas Systems, I worked at Comcast, where I eventually switched from an IC to an engineering manager. Once I experienced the joys of coaching people, building cool software together, and developing high-performing teams, I knew this was the path I wanted to take.

Fail whales and distributed systems

This path took me through a formative time in my career: the almost nine years I spent working at Twitter. I began as a first-line manager and, over time, worked through some of Twitter’s biggest events, including the “fail whale” era and the Ellen DeGeneres “selfie that broke Twitter” moment.

This was an exciting time. I was working at the heart of Twitter’s tech stack, supporting teams that powered its consumer and revenue verticals. This is where I grew into a senior manager and, eventually, a director. Looking back over nearly a decade of work, two major lessons stand out: one technical and one cultural.

The technical lesson was about failure — namely, its inevitability.

Over my tenure, the Twitter stack transitioned from a monolith to a microservices architecture. This resulted in a set of robust, high-scale, low-latency distributed systems, and it was here that I learned that, when building resilient distributed systems, you need to design for failure, not hope to avoid it.

I often think back to How Complex Systems Fail — the more complex a system, the more likely it is to fail. I remembered that lesson every time we were called to the Twitter command center to deal with an incident; it was all hands on deck until everything was back online.

The cultural lesson was about adoption and what it takes to fuel great ideas.

Today, almost two-thirds of enterprises use GraphQL in production, but in its early days, it was a new, largely untested idea. During a hack week, a couple of engineers laid the groundwork for using this technology at Twitter. I worked closely with them and bootstrapped the team that eventually built Twitter’s GraphQL API, replacing the legacy REST services.

I still think about this experience today. It required convincing leadership and building consensus across numerous teams and stakeholders, but once we did, the payoff was significant: this one technical choice accelerated the velocity of product feature teams across the company.

Why I picked Airbnb

When Airbnb reached out in 2022, I realized my time at Twitter was coming to a close. By that point, my organization was well-run and high-performing — a success, but also a sign that I was ready for my next adventure.

Airbnb immediately stood out because the company offered, for the first time in my career, a true alignment between my personal and technical interests. I love traveling, and I have been a long-time Airbnb guest since 2013. I had always wanted to work for a company that built a product I truly cared about, and this was my chance.

I only got more excited when I learned about the people and teams I’d be working with. The Developer Platform organization, which was responsible for supporting all of Airbnb’s engineers, faced challenges I’d seen before. There was a lot of good work happening in silos, and folks were longing for a clear strategy and direction. Also, I saw an opportunity to not only improve developer experience but also build trust with the rest of the engineers and stakeholders.

So, I started at the beginning. We focused on setting up the organization, coaching leadership, and building internal alignment within the team, as well as external alignment across all the teams we supported. Fundamental questions like “Why are we here together?” and “Where are we going?” all had to be answered.

After a year or two of this work, we had a high-performing team with a clear strategy and strong execution, consistently delivering business value and improving the developer experience and productivity at Airbnb. Even more importantly, we earned the rest of the engineers’ trust, and we enabled our technical teams to perform better.

We saw this reflected in the bi-annual DevX surveys (which we built out), and the results showed overall developer satisfaction increasing about 10% year over year during my time on the team.

Solving new problems while working from anywhere

Today, I’m Senior Director of Engineering for Application & Cloud Infrastructure, which includes compute, networking, core services, and the GraphQL application platform. Our mission is to deliver reliable, secure, and efficient platforms for building, operating, and scaling applications, services, and workloads at Airbnb.

My primary users are still the engineers at Airbnb. When they need to compute, they don’t wrangle AWS themselves — we provide a layer of abstraction that helps them use low-level infrastructure. Similarly, if they need authorization, authentication, configuration management, and a host of other services, they come to us rather than starting from scratch.

I’m excited to come to work every day because of the people I get to work with and the opportunities we face together. The culture is excellent, the people are smart and collaborative, and the engineers we support appreciate the work we do.

The setup is empowering, too, and as you solve problems, you can grow and expand to tackle bigger problems that span teams and organizations. Add in the ability to work from anywhere, and for me, it feels like the sky is the limit.

As I look back on my career, and really, my entire life, I tend to see it now through the lens of long-distance trail running — a major hobby of mine.

After working at a startup, having twins, and running my first marathon, I felt like I could do anything. At work and on the trails, I think about how to prepare for the journeys ahead and how to maintain a pace that allows me and the people around me to thrive in the long run. Recovery is necessary, but so is strategy, drive, discipline, and finding the people who will go with you as well as cheer you on along the way.

I’m happy this path, as unpredictable as it has been, has taken me to Airbnb. Airbnb is in that ideal position between a startup and a long-established company. The systems and workflows are mature, but there are still many interesting problems to solve and opportunities to pursue. If that’s of interest to you, I encourage you to check out openings at Airbnb.

Peter Coles was born and raised in Milwaukee, Wisconsin. He studied math at Princeton, earned his PhD in economics at Stanford, and taught at Harvard Business School before joining eBay and becoming a Data Science leader at Airbnb.

As you’ll see from his story, Peter has a deep interest in how marketplaces work. By transitioning from academia to the business world, he not only gets to study first-hand data about millions of guests and hosts, but also to influence product and policy decisions. And he still gets to hang out with academics. Check out all the research Peter and his team are doing here.

My fascination with marketplaces goes back a long time.

Sometime around second grade in Milwaukee, Wisconsin (where I grew up), my friends and I had the great idea to run a rock stand. It was like a lemonade stand, but instead we would sell rocks. Rocks we found in the street. Neighborhood kids could find their own rocks and sell them, and we’d take 25%. Nobody got any sales. Fortunately I’ve learned a bit more about marketplaces since then — more about that shortly.

From kindergarten through high school, I was a public school kid. My parents valued education, and helping others — my father was a doctor, my mother a nutritionist — and those are values I still hold dear.

While I played soccer and tennis and was moderately social, this period was probably best defined by an obsession with competitions. Math, Chess, Science Olympiad, Quiz Bowl, Academic Decathlon, puzzle races with my younger brother — there was hardly a nerdy competition offered where I didn’t compete. Time well spent? Let’s just say I missed a lot of high school parties while studying to become the five-time Wisconsin State Rocks, Minerals, and Fossil Identification champion — so you can be the judge.

By the time college came around, it seemed time to rebel. I wanted to be done with the nerdy stuff. I applied and was accepted to Princeton, and started studying ancient history. After one semester and (in my view) an underappreciated essay on the Hittites, I was back to majoring in math. At least I was good at that! I figured I could work on practical skills later.

After graduating, I accepted a fellowship in Germany and continued to Stanford for a PhD in economics — a somewhat more applied science, though I focused on game theory, at the intersection of math and strategy. I had the good fortune there to be the second graduate student of Jon Levin, now the President of Stanford University, who taught me the importance of simplification in research — even when the subject matter itself is complex.

Even while in this still-theoretical space, I kept my feet on the ground — or at least on the pedals. During a monthlong break in my classes in Germany, I biked around Europe, crashing with friends and family members of my classmates — people I had never met before staying with them. In a sense, I was prototyping Airbnb well before it existed!

My time in graduate school was Silicon Valley in the 2000s, after the dot-com crash, so tech was in the midst of a renaissance. Many of my friends were at growing companies like Google and Amazon. It was very tempting to stay in California to be a part of this, but I ended up with one more stop in academia.

Markets in theory, markets in practice

Harvard Business School, known for its focus on managerial science, was perhaps the most compelling place in the academic world that would allow me to stay close to the tech industry. I got a double stroke of good fortune: not only was I offered an assistant professorship there, but I also got to co-teach with Al Roth, a founder of the field of Market Design. Al is still an important mentor to me, and later won a Nobel Prize!

In my time researching and teaching graduate students, I was exposed to many examples of market design, conducting research on the topic of “Matching”; that is, mechanisms to pair users from two groups, often when price cannot be used to clear the market. This covered strategy of participants, signaling in markets, and I even had a chance to improve the market for PhD economists. I also wrote a number of case studies, including on Zillow, Microsoft, Craigslist, and more. The teaching and writing was a lot of fun, but I also came to realize I wasn’t a fit for academia in the long term. My attention span was too short to dedicate most of my time to research papers (and especially peer reviews), but I was enormously appreciative of this phase of my career.

By this point it was 2013, and two simultaneous and interrelated phenomena were exploding in tech: mobile and the sharing economy. It was a perfect time to head back west and finally enter the tech world.

I landed at eBay, which for a student of marketplaces was an ideal company: just about everything is for sale, and it was ripe for market design. Steve Tadelis, a mentor from my Stanford days, had created one of the first economics teams inside a tech company, which I took over when Steve left. At the same time, eBay was getting on the data science train — this was before every company had a DS team — and my group joined another to form eBay’s Data Labs. One of my favorite projects there was a project called “What’s it Worth” (which I worked on with Airbnb colleague Dean Chen), where we developed a methodology for determining the fair market value of items. Some hands-on practical work, some modeling — this was just what I was hoping for.

In 2015 Riley Newman, one of Airbnb’s first employees and then its Head of Data Science, presented an even more enticing opportunity. The Airbnb platform was growing quickly, and for the first time attracting substantial regulatory attention. They needed an economics team to partner with the growing policy team, to jointly address the question of Airbnb’s relationship to cities. This was a new way for me to apply economics. I was all in.

Addressing Airbnb’s critical questions with data

When I think back to my eight and a half years so far at Airbnb, I view this as entailing three “phases.” In the first, I worked to address economic questions by establishing a global team of data scientists and economists to analyze the relationship of short-term rentals to the world.

Meanwhile, as Airbnb continued to grow, execs were asking big questions that couldn’t be answered by any specific data science team. They needed a group with visibility across the whole organization. So in this second phase, Jackson Wang and I founded a team called Central Strategy & Insights, or CSI.

The acronym was no coincidence: we saw ourselves as forensic investigators, piecing together stories as we collected evidence. One important period of CSI’s work addressed changes brought on by the pandemic — in particular a major adjustment in where guests were looking to stay, and the supply we’d need to accommodate them. We also led the company’s business reviews, and generated analyses to describe the business to shareholders ahead of the IPO.

My third phase at Airbnb started a lot like the first, but supersized: developing models to inform a well-considered approach to policy considerations, this time as travel rebounded after the pandemic and governments were no longer fully occupied with a public health emergency. Our newly expanded group of economics PhDs and analysts also came up with ways to evaluate Airbnb’s impact on guests, hosts, and society, including via our US Economic Impact Report.

A great balance of academic and applied science

Almost all of my first several years at Airbnb had been internally facing. That’s changed in recent times, as we’ve spun up and expanded a program to collaborate with academic researchers to analyze Airbnb’s data and improve the experience for users.

The first step was to figure out how to collaborate with external researchers, while respecting privacy and legal limitations. Collaboration interest then came quickly. We have now published well-received papers with professors from MIT, Berkeley, Stanford, UCLA, NYU and more, with others in progress. One paper I wrote with colleagues and academic partners develops foundations for what “quality” means in platforms, from an economic perspective.

We’ve also launched a monthly seminar where we invite our academic collaborators to discuss research with Airbnb data scientists and technologists. Developing research is great, but there’s nothing like live discussions to cross-pollinate and foster ideas. This builds on a strong collaborative learning tradition at Airbnb, with internal classes and reading groups to grow our skills and keep up with tech developments.

Alongside engaging with academia, I’m so excited my data science colleagues and I have a mandate to be innovative and proactive. We have the space and encouragement to work on big ideas, even if they might take a year or two to prove out — and perhaps more importantly, even if some of the ideas fail. But nothing is more important than the people. I am proud of the students, scientists, and even professors I have hired here over the years, and love seeing them grow and find success.

A license to tackle big topics, continual education, research on the product as well as its relationship to the outside world, amazing colleagues, and a direct connection to academia all make Airbnb a unique place to be a market designer, economist, and data scientist. Whether or not you spent your free time in eighth grade studying rocks.

If you want to learn more about the research happening at Airbnb you can read our published papers here. If this type of work interests you, check out our open roles.

Producing valid and realistic mock data for testing and prototyping with GraphQL has been a persistent challenge across the industry for years. Mock data is tedious to write and maintain, and attempts to improve the process, such as random value generation and field-level stubbing, fall short because they lack essential domain context to make test data realistic and meaningful. The time spent on this manual work ultimately takes away from what most engineers would like to focus on: building features.

In this post, we’ll explore how we’ve reimagined mocking GraphQL data at Airbnb by combining GraphQL validation, rich product and schema context, and LLMs to generate and maintain convincing, type-safe mock data. Our solution centers around a simple new GraphQL client directive — @generateMock — that engineers can add to any operation, fragment, or field. This approach eliminates the need for engineers to manually write and maintain mocks as queries evolve, freeing up time to focus on building the product.

Key challenges

After meeting with Airbnb product engineers and analyzing results from internal surveys, we distilled the most common pain points around GraphQL mocking down into three key challenges:

1. Manually creating mocks is time consuming. GraphQL queries can grow to hundreds of lines, and hand-crafting mock response data is extremely tedious. Most engineers manually write mocks as either raw JSON files or by instantiating types generated from the GraphQL schema, while others modify copy-and-pasted JSON responses from the server. Although both of these methods can yield realistic-looking data that can be used for demos and snapshot tests, they require significant time investment and are prone to subtle mistakes.
2. Prototyping & demoing features without the server is hard. Typically, server and client engineers agree on a GraphQL schema early on in the feature development process. Once the schema has been established, however, the two groups split off and start working in parallel: Server engineers implement the logic to back the new schema and client engineers build the frontend UI, logic, and the queries that power them. This parallelization is particularly challenging for client engineers, since they can’t actually test the UI they’re building until the server has fully implemented the schema. To unblock themselves, client engineers often hardcode data into views, leverage proxies to manipulate responses, or hack custom logic into the networking layer locally, resulting in wasted time and effort.
3. Mocks get out of sync with GraphQL queries over time. Since most mocks are hand-written, they are not tightly coupled to the underlying queries and schema they are supposed to represent. If a team builds a new feature, then comes back a few months later to add new functionality backed by additional GraphQL fields, engineers must remember to manually update their mock data. As there is no forcing function to guarantee mocks stay in sync with queries, mock data tends to shift further away from the production reality as time passes — degrading the quality of tests.

These challenges are not unique to Airbnb and are common across the industry. Although tooling like random value generators and local field resolvers can provide some assistance, they lack the domain knowledge and context needed to produce realistic, meaningful data for high-quality demos, quick product iteration, and reliable testing.

Goals

When setting out to solve these challenges at Airbnb, we established three north-star goals:

1. Eliminate the need to hand-write mock data. Mock data should be generated automatically to free up engineers from needing to hand-craft and maintain mock GraphQL data.
2. Create highly realistic mock data. Mock data should match the user interface designs and look like real production data in order to support high-quality demos, which are highly valued at Airbnb for early feedback.
3. Keep engineers in their local focus loops. Our solution should seamlessly integrate into engineers’ current development processes so they can generate mocks without context-switching to a website, separate repository, or unfamiliar tool.

@generateMock: Schema + context + LLMs = magic

To generate mock data while keeping engineers in their local focus loops, we introduced a new client GraphQL directive called @generateMock, which engineers can use to automatically generate mock data for a given GraphQL operation, fragment, or field:

This directive accepts a few optional arguments that engineers can use to customize the generated mock data, and the directive itself can be repeated with different input arguments to generate different mock variations:

• id: The identifier to use for the mock, as well as for naming generated helper functions. Useful when repeating the @generateMock directive to produce multiple mocks.
• hints: Additional context or instructions on how the mock should look. For example, a hint might be “Include travel entries for Barcelona, Paris, and Kyoto.” Under the hood, this information is fed to an LLM and heavily influences what the generated mock data looks like and how densely populated its fields are.
• designURL: The URL of a design mockup of the screen that will render the mock data. Specifying this argument helps the LLM produce mock data that matches the design by generating matching names, addresses, and other similar content.

At Airbnb, engineers use a command line tool we call Niobe to generate code for their GraphQL queries and fragments. After modifying a .graphql file locally, engineers run this code generator, then use the generated TypeScript/Kotlin/Swift files to send GraphQL requests. To generate mock data using @generateMock, engineers simply need to run Niobe code generation after adding the directive — just as they would after making any other GraphQL change.

During code generation, Niobe produces both a JSON file containing the actual mock data for each @generateMock directive, as well as a source file that provides functions for loading and consuming mock data from demo apps, snapshot tests, and unit tests. As shown in the Swift code below, the mockMixedStatusIndicators() function is generated on the InboxSyncQuery’s root Data type. It provides access to an instantiated type that’s populated with the generated mock data for mixed_status_indicators, allowing engineers to use the mock without having to load the JSON data manually:

Engineers are free to modify the generated mock JSON data as well — as we’ll see below, Niobe will avoid overwriting their modifications on subsequent generation invocations.

What does mock data look like?

The context that we provide to the LLM is vital to generating data that is realistic enough to use in demos. To this end, Niobe collects the following information and includes it in the context passed to the LLM:

• The definitions of the query/fragment/fields being mocked (i.e., those marked with @generateMock and their dependencies).
• The subset of the GraphQL schema being queried, as well as any associated documentation that is present as inline comments. This information enables the LLM to infer the types that are used by the query being mocked. Importantly, this isn’t the whole schema, because including the full schema would likely overload the context window — Niobe traverses the schema and strips out types and fields that are not needed to resolve the query, along with any extra whitespace.
• The URL for the image representation of the design document specified within designURL, if any. Niobe integrates with an internal API to generate a snapshot image of the provided node in the design document. The API pushes this snapshot to a storage bucket and provides a URL that Niobe feeds to the LLM, along with specialized instructions on how to use it.
• The additional hints specified in @generateMock.
• The platform (e.g., “iOS”, “Android”, or “Web”) for which the mock data is being generated (for style specificity).
• A list of Airbnb-hosted image URLs that the LLM can choose from if needed, along with short textual descriptions of each. This prevents the LLM from hallucinating image URLs that don’t exist and ensures that the mock data contains valid URLs which can be properly loaded at runtime when prototyping or demoing.

All this information is consolidated into a prompt we fine-tuned against Gemini 2.5 Pro. We chose this model because of its 1-million token context window, plus the fact that in our internal tests this configuration performed significantly faster than comparable models while producing mock data of similar quality. Using this approach, we’re able to produce highly realistic JSON mocks which, when loaded into the application, yield very convincing results as shown below:

The data in the screenshot on the right looks quite realistic, but if you look closely you may notice that the data is indeed mocked — all the photos are coming from the seed data set that we feed the LLM.

How it works

When an engineer uses the Niobe CLI to generate code for their GraphQL files, Niobe automatically performs mock generation as the final step of this process, as shown in the flowchart below:

• If the @generateMock directive includes a designURL, Niobe validates the URL to ensure it includes a node-id, then uses an internal API to produce an image snapshot of that particular node. The API, in turn, pushes this snapshot to a storage bucket and provides Niobe with its URL.
• Next, the CLI aggregates all the context described in the section above — including the URL of the design snapshot — and crafts a prompt to send to the LLM. This prompt is then sent to the Gemini 2.5 Pro model, and results are streamed back to the client in order to show a progress indicator in the CLI.
• Once the mock JSON response has been received from the LLM, Niobe performs a validation step against this data by passing the GraphQL schema, client GraphQL document, and JSON data to the graphql NPM package’s graphqlSync function.
• If the validation produces errors (for example, if the LLM hallucinated an invalid enum value or failed to populate a required field), Niobe aggregates these errors and feeds them back into the LLM along with the initial mock data. This retry mechanism is used to essentially “self-heal” and fix invalid mock data.
  – This step is critical to reliably generating mock data. By placing the LLM within our existing GraphQL infrastructure, we’re able to enforce a set of guardrails through this validation step and provide strong guarantees that the mock data produced at the end of the pipeline is fully valid — something that wouldn’t be possible by using a tool outside our GraphQL infrastructure like ChatGPT.
  – Finally, once the mock data has been validated, Niobe writes it to a JSON file, alongside a companion source file which provides functions for loading the mock from application code.

@respondWithMock: Unblocking client development

In addition to generating realistic mock data with @generateMock, we also wanted to empower client engineers to iterate on features without waiting for the backend server implementation. A second directive, @respondWithMock, works alongside @generateMock to make this possible:

When this directive is present, Niobe alters the code that’s generated alongside the mock data to include extra details about this annotation. At runtime, the GraphQL client uses this to load the generated mock data, then seamlessly returns the mocked response instead of using data from the server. This effectively allows client engineers to unblock themselves from waiting on the server implementation, since they can easily use locally mocked data when querying unimplemented fields. The screenshot of the inbox screen earlier in this post is actually a real screenshot that was taken by generating with these two directives and running the Airbnb app in an iOS simulator — no manual mocking, proxying, or response modification needed!

@respondWithMock can also be specified on individual fields. When used on fields within a query instead of on the query itself, the GraphQL client will actually request all fields from the server except those annotated with @respondWithMock, then patch in locally mocked data for the remaining fields — producing a hybrid of production and mock data, and making it possible for client engineers to develop against new (unimplemented) fields in existing queries. Engineers can even repeat this directive and use query input variables to decide if and when to return a specific generated mock at runtime, as shown below:

Schema evolution: Keeping mocks truthful

The final challenge we addressed was the issue of keeping mocks in sync with queries as they evolve over time. Since Niobe manages mock data that is generated via the @generateMock directive, it can be smart about maintaining that mock data. As part of mock generation, Niobe embeds two extra keys in each generated JSON file:

1. A hash of the client entity being mocked (i.e., the GraphQL query document).
2. A hash of the input arguments to @generateMock.

Each time code generation runs, Niobe determines whether existing mocks’ hashes differ from what their current hashes should be based on the GraphQL document. If they match, it skips mock generation for those types. On the other hand, if one of the hashes changed, Niobe intelligently updates that mock by including the existing mock in the context provided to the LLM, along with instructions on how to modify it.

It’s important that Niobe doesn’t unnecessarily modify existing mock data for fields that are unchanged and still valid, since doing so could overwrite manual tweaks that were made to the JSON by engineers or break existing tests that rely on this data. To avoid this, we provide the LLM with a diff of what changed in the query, and tuned the prompt to focus on that diff and avoid making spurious changes to unrelated fields.

Finally, each client codebase includes an automated check that ensures mock version hashes are up to date when code is submitted. This provides a guarantee that all generated mocks stay in sync with queries as they evolve over time. When engineers encounter these validation failures, they just re-run code generation locally — no manual updates required.

Conclusion

“@generateMock has significantly sped up my local development and made working with local data much more enjoyable.” — Senior Software Engineer

By integrating highly contextualized LLMs — informed by the GraphQL schema, product context, and UX designs — directly into existing GraphQL tooling, we’ve unlocked the ability to generate valid and realistic mock data while eliminating the need for engineers to manually hand-write and maintain mocks. The directive-driven approach of @generateMock and @respondWithMock allows engineers to build clients before the server implementation is complete while keeping them in their focus loops and providing a guarantee that mock data stays in sync as queries evolve.

In just the past few months, Airbnb engineers have generated and merged over 700 mocks across iOS, Android, and Web using @generateMock, and we plan to roll out internal support for backend services soon. These tools have fundamentally changed how engineers mock GraphQL data for tests and prototypes at Airbnb, allowing them to focus on building product features rather than crafting and maintaining mock data.

Acknowledgments

Special thanks to Raymond Wang and Virgil King for their contributions bringing @generateMock support to Web and Android clients, as well as to many other engineers and teams at Airbnb who participated in design reviews, built supporting infrastructure, and provided usage feedback.

Does this type of work interest you? Check out our open roles here.

Before we talk about OSS, here’s a quick update on Viaduct’s adoption and evolution at Airbnb over the last five years. Since 2020, traffic through Viaduct has grown by a factor of eight. The number of teams hosting code in Viaduct has doubled to 130+ (with hundreds of weekly active developers). The codebase hosted by Viaduct has tripled to over 1.5M lines (plus about the same in test code). We’ve achieved all this while keeping operational overhead constant, halving incident-minutes, and keeping costs growing linearly with QPS.

What’s the same?

Three principles have guided Viaduct since day one and still anchor the project: a central schema served by hosted business logic via a re-entrant API.

Central schema
Viaduct serves our central schema: a single, integrated schema connecting all of our domains across the company. While that schema is developed in a decentralized manner by many teams, it’s one, highly connected graph. Over 75% of Viaduct requests are internal because Viaduct has become a “one‑stop” data-oriented mesh connecting developers to all of our data and capabilities.

Hosted business logic
From the beginning, we’ve encouraged teams to host their business logic directly in Viaduct. This runs counter to what many consider to be best practices in GraphQL, which is that GraphQL servers should be a thin layer over microservices that host the real business logic. We’ve created a serverless platform for hosting business logic, allowing our developers to focus on writing business logic rather than on operational issues. As noted by Katie, an engineer on our Media team:

“As we migrate our media APIs into Viaduct, we’re looking forward to retiring a handful of standalone services. Centralizing everything means less overhead, fewer moving parts, and a much smoother developer experience!”

Re-entrancy

At the heart of our developer experience is what we call re-entrancy: Logic hosted on Viaduct composes with other logic hosted on Viaduct by issuing GraphQL fragments and queries. Re-entrancy has been crucial for maintaining modularity in a large codebase and avoiding classic monolith hazards.

What’s changed?

For most of Viaduct’s history, evolution has been bottom-up and reactive to immediate developer needs. We added capabilities incrementally, which helped us move fast, but also produced multiple ways to accomplish similar tasks (some well‑supported, others not) and created a confusing developer experience, especially for new teams. Another side-effect of this reactive approach has been a lack of architectural integrity. The interfaces between the layers of Viaduct, described in more detail below, are loose and often arbitrary, and the abstraction boundary between the Viaduct framework and the code that it hosts is weak. As a result, it has become increasingly difficult to make changes to Viaduct without disrupting our customer base.

To address these issues, over a year ago we launched a major initiative we call “Viaduct Modern”, a ground-up overhaul of both the developer-facing API and the execution engine.

Tenant API

One driving principle of Viaduct Modern has been to simplify and rationalize the API we provide to developers in Viaduct, which we call the “Tenant API”. The following diagram captures the decision tree one faced when deciding how to implement functionality in the old API:

Each oval in this diagram represents a different mechanism for writing code. In contrast, the new API offers just two mechanisms: node resolvers and field resolvers.

The choice between the two is driven by the schema itself, not ad‑hoc distinctions based on a feature’s behavior. We unified the APIs for both resolver types wherever possible, which simplifies dev experience. After four years evolving the API in a use‑case‑driven manner, we distilled the best ideas into a single simple surface (and left the mistakes behind).

Tenant modularity

Strong abstraction boundaries are essential in any large codebase. Microservices achieve this via service definitions and RPC API boundaries; Viaduct achieves it via modules plus re‑entrancy.

Modularity in the central schema and hosted code has evolved. Initially, all we had was a vague set of conventions for organizing code into team-owned directories. There was no formal concept of a module, and schema and code were kept in separate source directories with unenforced naming conventions to connect the two. Over time, we evolved that into a more formal abstraction we call a “tenant module.” A tenant module is a unit of schema together with the code that implements that schema, and crucially, is owned by a single team. While we encourage rich graph‑level connections across modules, we discourage direct code dependencies between modules. Instead, modules compose via GraphQL fragments and queries. Viaduct Modern extends and simplifies these re‑entrancy tools.

Let’s look at an example. Imagine two teams, a “Core User” team that owns and manages the basic profile data of users, and then a “Messaging” team that operates a messaging platform for users to interact with each other. In our example, the Messaging team would like to define a displayName field on a User, which is used in their user interface. This would look something like this:

Core User team

type User implements Node {
  id: ID!
  firstName: String
  lastName: String
  … 
}

class UserResolver : Nodes.User() {
    @Inject
    val userClient: UserServiceClient

    @Inject
    val userResponseMapper: UserResponseMapper

    override suspend fun resolve(ctx.Context): User {
        val r = userClient.fetch(ctx.id)
        return userResponseMapper(r)
    } 
}

This is the base definition of the User type that lives in the Core User team’s module. This base definition defines the first- and last-name fields (among many others), and it’s the Core User team’s responsibility to materialize those fields.

Messaging team

extend type User {
  displayName: String @resolver
}

@Resolver("firstName lastName")
class DisplayNameResolver : UserResolvers.DisplayName() {
    override suspend fun resolve(ctx: Context): String {
        val f = ctx.objectValue.getFirstName()
        val l = ctx.objectValue.getLastName()
        return "$f ${l.first()}."
    }
}

The Messaging team can then extend the User type with the display name field, and also indicates that they intend to provide a resolver for it. The code has an @Resolver annotation that indicates which fields of the User object it needs to implement the displayName field. The Messaging team doesn’t need to understand which module these fields come from, and their code doesn’t depend on code from the Core User team. Instead, the Messaging team states their data needs in a declarative fashion.

Framework modularity

A major goal of Viaduct Modern has been to make the framework itself more modular. We want to enable faster improvements to Viaduct — especially regarding performance and reliability — without extensive changes to application code. Viaduct is composed of three main layers: the GraphQL execution engine, the tenant API, and the hosted application code. While these layers made sense, the interfaces between them were weak, making Viaduct difficult to update. The new design is focused on creating strong abstraction boundaries between these layers to improve flexibility and maintainability:

The most significant change is the boundary between the engine and the developer-facing tenant API. In the previous system, that boundary hardly existed. Viaduct Modern defines a strong engine API whose core is a dynamically‑typed representation of GraphQL values (input and output objects as simple maps from field name → value). The tenant API, by contrast, is statically typed: we generate Kotlin classes for every GraphQL type in the central schema. In the new architecture, generated types are thin wrappers over the dynamic representation. The tenant API forms the bridge between the engine’s untyped world and tenants’ typed world.

This separation lets us evolve the engine in relative isolation (to improve latency, throughput, and reliability) and evolve the tenant API in relative isolation (to improve dev experience). Large changes will still cross the boundary, but as Viaduct Modern stabilizes, that should be rare.

Migration without a “big bang”

Viaduct Modern would be a non‑starter if it required a step‑function migration of a million+ lines of code. To enable gradual migration, we’re shipping two tenant APIs side‑by‑side — the new Modern API and the existing Classic API — both on top of the new engine. This lets teams realize performance/cost wins from the engine immediately, while adopting the ergonomic wins of the Modern API over time.

Shipping two APIs has also improved the engine API design: building two tenant runtimes simultaneously forced us to keep the engine’s concerns clean and general. Over time, we expect to build additional tenant APIs on the engine (e.g., TypeScript).

Other improvements

Viaduct has seen a lot of other improvements since 2020. The story is too long to be told in this post, but to list some of the highlights:

• Observability. Hosting software from 100+ teams means our framework has to make ownership and attribution crystal clear. In the old system, there is no clear dividing line between tenant code and framework code, so our instrumentation includes a bit of guesswork in its attribution to parties. The new architecture draws a crisp boundary, which enables deeper, more accurate attribution.
• Build time. We’re schema‑first: Developers write schema as source, and Viaduct generates code to provide a strongly typed, ergonomic surface. At our scale, build time is a constant battle. Over the years we’ve made numerous investments to improve build time, including direct-to-bytecode code generation that bypasses lengthy compilation of generated code. We anticipate that the improved modularity in Viaduct Modern will keep build times in check as the codebase grows.
• Dispatcher. We run Viaduct as a horizontally-scaled Kubernetes app. To mitigate blast radius, we use a dispatcher that routes operations to deployment shards, applying shuffle sharding. It also simplifies isolating offline vs. online traffic and hosting experimental framework builds. (We don’t currently plan to open‑source the dispatcher as it’s tightly tied to Airbnb’s serving framework, but we may talk more about our strategies here in the future!)

Open-sourcing Viaduct

Our intent from the start of Viaduct Modern was to open‑source it. We believe that setting out to build software for the world will result in higher quality software for ourselves. Also, as a significant consumer of open-source software, we feel an obligation to give back. Last but not least, while we’ve learned a lot by working with Airbnb developers, we think Viaduct can be massively improved by incorporating ideas and contributions from the wider community.

We’re open‑sourcing at an interesting moment. Viaduct is mature and battle‑tested and also new and evolving. The new engine is now in full production, while the new tenant API is still in alpha. We’ve implemented a small but robust kernel of the new API and are using it in a few (demanding) use cases. We’re investing heavily in the Modern API and migrating our own workloads to it, but it’s early days. By open‑sourcing early, we hope to grow a community who will shape the API with us together.

Is Viaduct for you?

Although our use case proves Viaduct can scale to massive graphs, we think it’s also a great GraphQL server when you’re just starting. We’ve emphasized developer ergonomics from day one, and we believe Viaduct provides one of the best environments for building GraphQL solutions. Whether you’re operating a super-graph today or just kicking the tires, we’d love for you to try Viaduct Modern and tell us what works — and what doesn’t.

—

Thanks to the entire Viaduct team, and especially Aileen Chen and Raymie Stata, for the tireless work on Viaduct Modern.

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Migration in numbers (4.5 years of work):

• Build CSAT: 38% → 68%
• 3–5x faster local build and test times
• 2–3x faster IntelliJ syncs
• 2–3x faster deploys to the development environment

In this blog post, we’ll discuss the why, share some highlights on the how, and finish off with key learnings.

Why Bazel?

Before the migration, our JVM monorepo used Gradle as its build system. We decided to migrate to Bazel because it offered three key advantages: speed, reliability, and a uniform build infrastructure layer.

Speed

Bazel’s cacheable, portable actions allow us to scale performance with remote execution

In 2021, builds of large services often took >20 minutes locally and pre-merge CI p90 was 35 minutes.

Building with Gradle was near its limit. We had already vertically scaled to high-end AWS machines on CI and remote development machines for developers of large services. In CI, we also used heuristics to split project builds and tests across multiple machines. However, this was inefficient, because of machine underutilization and duplication of shared tasks.

Bazel remote execution allowed us to scale to thousands of parallel actions. This was far more efficient than our sharding heuristics. Remote build execution (RBE) workers are also short-lived, which results in better machine utilization and cost efficiency. In addition, Build without the Bytes allows downloading only a subset of files, greatly reducing download volume (in Gradle, every cached artifact needs to be downloaded). Finally and most importantly, local builds are significantly faster thanks to RBE.

In addition, Gradle configuration of some large projects often took minutes due to it being single-threaded. Bazel analysis, in contrast, runs in parallel, in part because its configuration language, Starlark, is constrained to be side-effect-free.

Reliability

Bazel’s hermeticity ensures reliable, repeatable builds

Gradle tasks have access to the full file system, which can lead to serious unintended consequences at scale. One example we ran into was when a developer updated a task to clean up recent files in the /tmp/ directory. This created a race condition with other Gradle tasks that used the /tmp/ directory and caused CI to fail when thousands of Gradle tasks had to be rerun.

Bazel solves this issue with sandboxing, which ensures that only specified inputs are available to a build action. If a file isn’t declared as an input, it simply doesn’t exist in the sandboxed environment.

Gradle tasks also implicitly depend on the machine’s resources. Gradle builds run on local machines and CI machines of different sizes. This can lead to resource contention when a task is run on a smaller machine or when the cache is cold and thousands of tasks are run on the same machine.

Remote build execution (RBE) solves this by running actions in identical containers with strict resource limits. We also configured both local and CI builds to use RBE, which greatly reduces environment differences.

Shared infrastructure

Airbnb currently has a collection of language- and platform-specific repos, such as web, iOS, Python, and Go, all of which are now on Bazel. Unifying on Bazel enables a uniform build infrastructure layer across repos, which includes:

• Remote caching
• Remote build execution
• Affected targets calculation
• Instrumentation & logging from the Build Event Protocol

How did we migrate?

Proof of concept

As a first milestone, we wanted to show that we could build a service and run its unit tests in Bazel. Because this was a proof of concept, we wanted to minimize disruption to engineers. Therefore, the Bazel build co-existed with Gradle. As a result, developers could choose between using Gradle or Bazel locally.

We needed to prove that developers would choose to opt in to using Bazel over Gradle. It wasn’t enough for Bazel to be faster, developers had to willingly opt in to using Bazel.

For this proof of concept, we chose Airbnb’s GraphQL monolith platform, Viaduct, which had the following important properties:

1. It was one of Airbnb’s largest and most complex services. If we could migrate Viaduct to Bazel, then we could likely migrate the rest of the monorepo.
2. Slow builds were a major pain point, so Bazel could have a large impact.
3. Viaduct has 300 product engineers modifying its code every month, so improving Viaduct’s build speed would be a substantial productivity win.
4. Because of (2) and (3) above, Viaduct’s core infrastructure team was eager to partner with us.

To achieve a working Viaduct build with Bazel, we did two things. First, we had to port much of Viaduct’s build logic from Gradle to Bazel. Second, because we decided to maintain co-existing builds and the Gradle build graph was still changing, we decided to build an automated build file generator (which we’ll cover in detail in a separate section).

Importantly, even though we were able to locally build the service 2–4x faster with Bazel, many developers did not yet want to switch.

In talking with the service’s owners, we discovered a number of missing integrations and bugs. It took us an additional few months to address these pain points, after which Viaduct developers willingly switched from Gradle to Bazel.

Scaling builds and tests with Bazel

The proof of concept showed that Bazel was superior to Gradle for one of Airbnb’s largest services and a large audience of developers. Now we wanted to scale it to the rest of Airbnb’s JVM monorepo.

We decided to scale breadth-first, getting all of the repo compiling and testing in Bazel. Again, to minimize disruption, Bazel builds co-existed with Gradle, which had two important benefits.

First, developers could still use Bazel for local development and get most of its benefits even though their code was still built with Gradle for deployment. Second, we could always disable Bazel if it was negatively impacting developers. For example, when Bazel infrastructure like the remote cache or remote execution cluster experienced an incident, we could and did disable Bazel, letting users fall back to Gradle.

However, a major downside was that both Gradle and Bazel build graphs had to be maintained. Manually maintaining a Bazel build graph would have degraded the developer experience. As a result, we invested further in automated build file generation, so that developers didn’t need to manually maintain Bazel build files.

Automated build file generation

For our build file generator, we were heavily inspired by Gazelle, which generates Bazel build files by parsing source files to build a dependency graph.

Although we considered extending Gazelle to support JVM languages, we had very strict performance requirements and needed to handle dependency cycles. This ultimately led us to build our own automated build file generator.

Because we had to maintain co-existing build graphs, we needed to run the build file generator on every commit before merging into mainline. This meant it had to run as fast as possible to not significantly degrade the developer experience. To achieve this, we implemented external caching to speed up the automated build file generation.

Similar to Gazelle, the build file generator parses Java, Kotlin, and Scala source files for package and import statements and symbol declarations to build a file-level dependency graph.

In CI, we publish a cached index of the repository at each mainline commit. When a user runs sync-configs, it downloads this cache and only re-scans directories which have changed since the merge base. This greatly improves performance for the common case where users only modify a small set of files.

In addition, with this build file generator, we were able to support a more fine-grained build graph, which resulted in >10x more Bazel targets than Gradle projects. This enabled faster builds through more parallel builds and less cache invalidation. However, one challenge of moving to a more fine-grained build graph is the possibility of introducing compilation cycles; sync-configs is able to detect this automatically and merge compilation units when necessary.

Even after the migration, the build file generator remains in use. It improves the developer experience by automatically fixing build file configurations and removing unused dependencies. In contrast, when we were on Gradle, users manually maintained ~4,500 Gradle files, which led to unused dependencies, a bloated dependency graph, and slower builds with fewer cache hits.

Porting build logic

In addition to JVM source files, our repo has a large amount of build logic such as code generation owned by multiple teams. These often took the form of Gradle plugins.

Because we now had automated build file generation, when porting the build logic, we also had to integrate it with the build file generator. Also, because we had more granular targets, a single line of Gradle config now might need to apply to 10+ Bazel build files.

As a result, our build file generator architecture supported plugins similar to Gazelle extensions. These plugins were triggered by the presence of specific files such as Thrift or GraphQL files. These plugins could also generate new build targets such as codegen actions.

In some cases, the Gradle logic was manually added as a one-off or not easily inferred from the file structure. As a result, we also supported generator directives similar to Gazelle, such as adding dependencies or setting attributes.

Initially, our team ported much of this build logic ourselves with minimal help from service owners. As Bazel adoption grew, owners of complex build logic were incentivized to migrate to Bazel, because it was faster and more reliable than Gradle. In the process, they often wrote their own build file generator plugins, highlighting the extensibility of our generator.

Third-party library multi-version support

Another major issue we hit on the road to 100% compilation and testing with Bazel was multiple versions of the same third-party library.

Initially, we specified a single version of each library. The build file generator would add dependencies from this universe.

However, in Gradle, each sub-project within the monorepo could use different versions of third-party libraries. As a result, when compiling against a single version, compilation or testing could fail with a missing symbol.

To bring multi-version support to our Bazel system, we built tooling to generate multiple maven_install rules and added a custom aspect to resolve conflicts at the target level.

Multiple versions of Guava in Bazel before we added conflict resolution

Once we had this capability, we systematically synced library versions from Gradle so that each build target’s classpath more closely matched its Gradle counterpart.

To learn more about our approach, see our BazelCon 2022 talk. Since giving this talk, we have made improvements like moving the resolution to analysis-time for better IDE support and adding more user-friendly tooling for updating libraries.

Migrating the deploys

As we got the vast majority of projects building and passing unit tests with Bazel in CI, we began to focus on migrating deployments. The Bazel-built jars were not identical to the Gradle-built jars. As a result, we needed a strategy to ensure the deployments were safe. We started with services.

Services

To verify the correctness of deploying services with Bazel, we used startup and integration tests. Of ~700 services, ~100 encountered startup or integration test failures. The majority of failures were missing dependencies that were loaded via Java reflection, usually from config files or other files. As a result, we were able to fix a number of these issues by parsing files for classes that would be loaded via reflection, and then adding the required dependency.

Another major source of errors was differing library versions, which could lead to missing symbol errors at runtime. In Gradle, users manually specified dependencies and their versions. However, in Bazel, build files were generated from source and dependencies were inferred from import statements, which didn’t specify the version. We solved many of these errors forcing third-party library versions to match those of the Gradle project.

After taking into account reflected classes and syncing versions, only a single-digit percentage of services hit production runtime issues that required more in-depth manual work to fix.

Data pipelines and other projects

In addition to services, we had 450 data pipelines and ~50 other projects that were deployed to either a Spark cluster or a Flink runtime.

Similar to services, we were able to catch a number of issues using tests. In particular, data pipelines on Airbnb’s data engineering paved path have CI tests that run a small version of the Spark pipeline locally. For these ~400 paved-path pipelines, after passing CI tests, only about 3% had production issues at runtime. As a result, we were able to very quickly migrate the paved-path pipelines.

As with services, we had a few remaining deployables that were a bit more bespoke and had to be individually deployed and monitored to verify correctness.

What did we learn?

Customer partnership

Early in the migration, we identified key pilot services that had large opportunities for impact and an appetite to invest in migrating to a new build system. For example, Viaduct had complex build logic leading to slow builds and reliability issues. In addition, many developers contributed to the service, so improving their builds had a large impact on Airbnb’s developer experience.

Partnering with pilot teams was incredibly valuable. They were early adopters and made significant contributions in the form of reporting and debugging issues, profiling performance bottlenecks, and suggesting features. The pilot teams also became advocates and provided internal support, helping motivate the rest of the Airbnb developer community.

Dangers of premature optimization

This migration took 4.5 years. With the benefit of hindsight we think we could have drastically improved the migration timeline if we had migrated first, before improving.

Although increasing build granularity improved build times, it increased the time to migrate. Specifically, increased build granularity greatly increased the number of configuration files, making it much harder to manage configurations manually. This forced more functionality into automated build file generation, which increased its complexity.

If we had migrated first and then optimized the build granularity, we believe we could have migrated sooner, enabling users to get benefits from Bazel sooner and reducing the time spent maintaining two co-existing builds.

Similarly, build granularity also made it harder to match deploy jars between Gradle and Bazel. This led to spending more time testing deployments and fixing runtime issues.

On a more positive note, we accelerated the migration by deciding to support multiple third-party library versions and implementing version resolution. This enabled us to sync versions from Gradle to Bazel, which fixed a large number of build and runtime issues.

Towards the end, one major takeaway in the migration was, by default, we should try to imitate what was there before. In our case, deviating from Gradle usually added technical risk, and should be carefully considered, especially its downstream consequences.

As engineers, we often want to improve things. However, during migrations, improvements can have non-obvious consequences and potentially significantly slow down the migration.

🏁 Conclusion

After 4.5 years, we fully migrated Airbnb’s biggest repo from Gradle to Bazel, achieving:

• Build CSAT: 38% → 68%
• 3–5x faster local build and test times
• 2–3x faster IntelliJ syncs
• 2–3x faster deploys to the development environment

Finally, now that several Airbnb repos are on Bazel, we’re able to share common infrastructure such as remote build caching, remote build execution, affected targets calculation, and more.

Interested in helping us solve problems like these at Airbnb? Learn more about our open engineering roles here.

Acknowledgments

Additionally, thank you Janusz Kudelka, Kumail Hussain, Meghan Dow, Pawel Lipski, Peimin Liu, Tomasz Pacuszka, and various other internal and external partners.

Istio is a foundational piece of our architecture, which makes ongoing maintenance and upgrades a challenge. Despite that, we have upgraded Istio a total of 14 times. This blog post will explore how the Service Mesh team at Airbnb safely upgrades Istio while maintaining high availability.

Challenges

Airbnb engineers collectively run thousands of different workloads. We cannot reasonably coordinate the teams that own these, so our upgrades must function independently of individual teams. We also cannot monitor all of these at once, and so we must minimize risk through gradual rollouts.

With that in mind, we designed our upgrade process with the following goals:

1. Zero downtime for workloads and users. This is the seamless part of the upgrade — a workload owner doesn’t need to be in the loop for Istio upgrades.
2. Gradual rollouts with the ability to control which workloads are upgraded or reverted.
3. We must be able to roll back an upgrade across all workloads, without coordinating every workload team.
4. All workloads should be upgraded within some defined time.

Architecture

Our deployment consists of one management cluster, which runs Istiod and contains all workload configuration for the mesh (VirtualServices, DestinationRules, and so forth), and multiple workload clusters, which run user workloads. VMs run separately, but their Istio manifests are still deployed to the management cluster in their own namespaces. We use Sidecar mode exclusively, meaning that every workload runs istio-proxy — we do not yet run Ambient.

Upgrade Process

At a high level, we follow Istio’s canary upgrade model. This involves running two versions (or Istio revisions) of Istiod simultaneously: the current version and the new version that we are upgrading to. Both form one logical service mesh, so workloads connected to one Istiod can communicate with workloads connected to another Istiod and vice versa. Istiod versions are managed using different revision labels — for example, 1–24–5 for Istio 1.24.5 and 1–25–2 for Istio 1.25.2.

An upgrade involves both Istiod, the control plane, and istio-proxy, the data plane sidecar, running on all pods and VMs. While Istio supports connecting an older istio-proxy to a newer Istiod, we do not use this. Instead, we atomically roll out the new istio-proxy version to a workload along with the configuration of which Istiod to connect to. For example, the istio-proxy built for version 1.24 will only connect to 1.24’s Istiod and the istio-proxy built for 1.25 will only connect to 1.25’s Istiod. This reduces a dimension of complexity during upgrades (cross-version data plane — control plane compatibility).

The first step of our upgrade process is to deploy the new Istiod, with a new revision label, onto the management cluster. Because all workloads are explicitly pinned to a revision, no workload will connect to this new Istiod, so this first step has no impact.

The rest of the upgrade comprises all of the effort and risk — workloads are gradually shifted to run the new istio-proxy version and connect to the new Istiod.

Rollout specification

We control what version of istio-proxy workloads run through a file called rollouts.yml. This file specifies workload namespaces (as patterns) and the percentage distribution of Istio versions:

# "production" is the default; anything not matching a different pattern will match this.
production:
  1-24-5: 100

".*-staging":
  1-24-5: 75
  1-25-2: 25

# A pinned namespace; our end-to-end verification workload.
istio-e2e:
  1-25-2: 100

This spec dictates the desired state of all namespaces. A given namespace is first mapped to a bucket (based on the longest pattern that matches) and then a version is chosen based on the distribution for that bucket. The distribution applies at the namespace level, not the pod (or VM) level. For example,

".*-staging":
  1-24-5: 75
  1-25-2: 25

means that 75% of the namespaces with the suffix -stagingwill be assigned to 1–24–5 and the remaining 25% will be assigned to 1–25–2. This assignment is deterministic, using consistent hashing. The majority of our upgrade process involves updating rollouts.yml and then monitoring.

This process allows us to selectively upgrade workloads. We can also upgrade environments separately and ensure that only a certain percentage of those environments are on the new version. This gives us time to bake an upgrade and learn of potential regressions.

The rest of this post will describe the mechanism through which a change to rollouts.yml is applied to thousands of workloads, for both Kubernetes and VMs.

Kubernetes

Each Istio revision has a corresponding MutatingAdmissionWebhook for sidecar injection on every workload cluster. This webhook selects pods specifying the label istio.io/rev=<revision> and injects the istio-proxy and istio-init containers into those pods. Notably, the istio-proxy container contains the PROXY_CONFIG environment variable, which sets the discoveryAddress to the Istiod revision. This is how the istio-proxyversion and the configuration for which Istiod to connect to are deployed atomically — entirely by the sidecar injector.

Every workload’s Deployment has this revision label. For example, a workload configured to use Istio 1.24.5 will have the label istio.io/rev=1–24–5in its pod template; thus pods for that Deployment will be mutated by the MutatingAdmissionWebhook for Istio 1.24.5.

This setup is the standard method of upgrading Istio, but requires that every Deployment specifies a revision label. To perform an upgrade across thousands of workloads, every team would have to update this label and deploy their workload. We could neither perform a rollback across all workloads nor reasonably expect an upgrade to complete to 100%, both for the same reason — relying on every workload to deploy.

Krispr

To avoid having to update workloads individually, a workload’s configuration never directly specifies the revision label in source code. Instead, we use Krispr, a mutation framework built in-house, to inject the revision label. Krispr gives us the ability to decouple infrastructure component upgrades from workload deployments.

Airbnb workloads that run on Kubernetes use an internal API to define their workload, instead of specifying Kubernetes manifests. This abstraction is then compiled into Kubernetes manifests during CI. Krispr runs as part of this compilation and mutates those Kubernetes manifests. One of those mutations injects the Istio revision label into the pod specification of each Deployment, reading rollouts.ymlto decide which label to inject. If a team sees any issue with their workload when they deploy, they can roll back and thus also roll back the Istio upgrade — all without involving the Service Mesh team.

In addition, Krispr runs during pod admission. If a pod is being admitted from a Deployment that is more than two weeks old, Krispr will re-mutate the pod and accordingly update the pod’s revision label if needed. Combined with the fact that our Kubernetes nodes have a maximum lifetime of two weeks, thus ensuring that any given pod’s maximum lifetime is also two weeks, we can guarantee that an Istio upgrade completes. A majority of workloads will be upgraded when they deploy (during the Krispr run in CI) and for those that don’t deploy regularly, the natural pod cycling and re-mutation will ensure they are upgraded in at most four weeks.

In summary, per workload:

1. During CI, Krispr mutates the Kubernetes manifests of a workload to add the Istio revision label, based on rollouts.yml.
2. When a pod is admitted to a cluster, Krispr will re-mutate the pod if its Deployment is more than two weeks old and update the Istio revision label if needed.
3. The revision-specific Istio MutatingAdmissionWebhook will mutate the pod by injecting the sidecar and associated discoveryAddress.

Virtual machines

On VMs, we deploy an artifact that contains istio-proxy, a script to run istio-iptables (similar to the istio-init container), and the Istiod discoveryAddress. By packaging istio-proxy and the discoveryAddress in the same artifact, we can atomically upgrade both.

Installation of this artifact is the responsibility of an on-host daemon called mxagent. It determines what version to install by polling a set of key-value tags on the VM (such as EC2 tags on AWS or resource tags on GCP). These tags mimic the istio.io/rev label for Kubernetes-based workloads. Whenever they change, mxagent will download and install the artifact corresponding to that version. Thus, upgrading istio-proxy on a VM just involves updating these tags on that VM; mxagent will take care of the rest.

Our VM workloads are largely infrastructure platforms that don’t typically have code deployed at regular intervals. As such, VMs don’t support a deploy-time upgrade (in the way that Kubernetes workloads can be upgraded when they deploy). Similarly, teams cannot roll back these workloads themselves, but this has been acceptable, given that there are just a handful of such infrastructure platforms.

The tag updates are managed by a central controller, mxrc, which scans for outdated VMs. If rollouts.yml would result in a different set of resource tags for a VM, the controller will update the tags accordingly. This roughly corresponds to Krispr’s pod admission-time mutation — however, with the caveat that VMs are mutable and long-lived, and thus are upgraded in-place.

For safety, mxrc takes into account the health of the VM, namely in the form of the readiness probe status on the WorkloadEntry. Similar to Kubernetes’ maxUnavailable semantics, mxrc aims to keep the number of unavailable VMs (that is, unhealthy VMs plus those with in-progress upgrades) below a defined percentage. It gradually performs these upgrades, aiming to upgrade all the VMs for a workload in two weeks.

At the end of two weeks, all VMs will match the desired state in rollouts.yml.

Conclusion

Keeping up-to-date with open-source software is a challenge, especially at scale. Upgrades and other Day-2 operations often become an afterthought, which furthers the burden when upgrades are eventually necessary (to bring in security patches, remain within support windows, utilize new features, and so forth). This is particularly true with Istio, where a version reaches end-of-life support rapidly.

Even with the complexity and scale of our service mesh, we have successfully upgraded Istio 14 times. This was made possible due to designing for maintainability, building a process that ensures zero downtime, and derisking through the use of gradual rollouts. Similar processes are in use for a number of other foundational infrastructure systems at Airbnb.

Future work

As Airbnb’s infrastructure continues to evolve and grow, we’re looking at a few key projects to evolve our service mesh:

• Utilizing Ambient mode as a more cost-effective and easier-to-manage deployment model of Istio. In particular, this simplifies upgrades by not needing to touch workload deployments at all.
• Splitting our singular production mesh into multiple meshes in order to separate fault domains, provide better security isolation boundaries, and scale Istio further. For upgrades, this would further reduce the blast radius, as some meshes that only run low-risk workloads (such as staging) could be upgraded first.

If this type of work interests you, we encourage you to apply for an open position today.

Acknowledgements

All of our work with Istio is thanks to many different people, including: Jungho Ahn, Stephen Chan, Weibo He, Douglas Jordan, Brian Wolfe, Edie Yang, Dasol Yoon, and Ying Zhu.

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Traditionally, organizations have deployed databases on costly, high-end standalone servers using sharding for scaling as a strategy. As data demands grew, the limitations of this strategy became increasingly evident with increasingly longer and more complex maintenance projects.

Increasingly distributed horizontally scalable databases are not uncommon and many of them are open source. However, running these databases reliably in the cloud with high availability, low latency and scalability, all at a reasonable cost is a problem many companies are trying to solve.

We chose an innovative strategy of deploying a distributed database cluster across multiple Kubernetes clusters in a cloud environment. Although currently an uncommon design pattern due to its complexity, this strategy allowed us to achieve target system reliability and operability.

In this post, we’ll share how we overcame challenges and the best practices we’ve developed for this strategy and we believe these best practices should be applicable to any other strongly consistent, distributed storage systems.

Managing Databases on Kubernetes

Earlier this year, we integrated an open source horizontally scalable, distributed SQL database into our infrastructure.

While Kubernetes is a great tool for running stateless services, the use of Kubernetes for stateful services — like databases — is challenging, particularly around node replacement and upgrades.

Since Kubernetes lacks knowledge of data distribution across nodes, each node replacement requires careful data handling to prevent data quorum loss and service disruption, this includes copying the data before replacing a node.

At Airbnb, we opted to attach storage volumes to nodes using AWS EBS, this allows quick volume reattachment to new virtual machines upon node replacement. Thanks to Kubernetes’ Persistent Volume Claims (PVC), this reattachment happens automatically. In addition we need to allow time for a new storage node to catch up with the cluster’s current state before moving to the next node replacement. For this, we rely on the custom k8s operator, which allows us to customize various Kubernetes operations according to specifics of the application.

Coordinating Node Replacement

Node replacements occur for various reasons, from AWS instance retirement to Kubernetes upgrades or configuration changes. To address these cases, we categorize node replacement events into three groups:

1. Database-initiated events: Such as config changes or version upgrades.
2. Proactive infrastructure events: Like instance retirements or node upgrades.
3. Unplanned infrastructure failures: Such as a node becoming unresponsive.

To safely manage node replacements for database-initiated events, we implemented a a custom check in the k8s-operator that verifies that all nodes are up and running before deleting any pod.

In order to serialize it with the second group initiated by infrastructure, we implemented an admission hook in k8s to intercept pod eviction. This admission hook rejects any attempt to evict the pod, but assigns a custom annotation on the pod which our customer database k8s-operator watches and acts on to safely delete the pod serializing it with any database-initiated node replacements described above.

Node replacements due to unplanned infrastructure failure events like hardware failure, can’t be coordinated. But we can still improve availability by ensuring that any node replacement event from the first two groups will be blocked until the failed hardware is replaced.

In our infrastructure the k8s operator handles both proactive and infrastructure-triggered node replacements, maintaining data consistency in the presence of node replacements and ensuring that unplanned events don’t impact ongoing maintenance.

Kubernetes Upgrades

Regular Kubernetes upgrades are essential but can be high-risk operations, especially for databases. Cloud managed Kubernetes might not offer rollbacks once the control plane is upgraded, posing a potential disaster recovery challenge if something goes wrong. While our approach involves using self-managed Kubernetes clusters, which does allow rolling back the control plane, a bad Kubernetes upgrade could still cause service disruption till rollback is completed.

Ensuring Fault Tolerance with Multiple Kubernetes clusters

At Airbnb, we think the best way to achieve high regional availability is to deploy each database across three independent Kubernetes clusters, each within a different AWS availability zone (AZ). AWS uses availability zones not just for independent power, networking, and connectivity, but they also do rollouts zone by zone. Our Kubernetes cluster alignment with AWS AZ also means that any underlying infrastructure issues or bad deployments have a limited blast radius as they are restricted to a single AZ. Internally, we also deploy a new configuration or a new database version to a part of the logical cluster running in a single Kubernetes cluster in one AZ first.

While this setup adds complexity, it significantly boosts availability by limiting the blast radius of any issues stemming from faulty deployments at every layer — whether database, Kubernetes, or AWS infrastructure.

For instance, recently, a faulty config deployment in our infrastructure abruptly terminated all VMs of a specific type in our staging Kubernetes cluster, deleting most of the query layer pods. However, since the disruption was isolated to a single Kubernetes cluster, two-thirds of our query layer nodes remained operational, preventing any impact.

We also overprovision our database clusters to ensure that, even if an entire AZ, Kubernetes cluster, or all storage nodes within a zone goes down, we still have sufficient capacity to handle traffic.

Leveraging AWS EBS for Reliability and Latency Handling

EBS offers two key benefits for our deployment: rapid reattachment during node replacements and superior durability compared to local disks. With EBS, we confidently run a highly available cluster using only three replicas, maintaining reliability without needing additional redundancy.

However, EBS can occasionally experience tail latency spikes, with p99 latency reaching up to 1 second. To mitigate this, we implemented a storage read timeout session variable, allowing queries to transparently retry against other storage nodes during EBS latency spikes. By default the database we use sends all requests and retries to the leader. To enable retries on storage nodes with healthy EBS, we have to allow reads from both leader and replica reads, but prefer the closest one for the original request. This brings the added benefit of reduced latency and no cross-AZ network costs, as we have a replica in each AZ. Finally, for use cases that permit it, we leverage stale reads feature, enabling reads to be served independently by the replica without requiring synchronous calls to the leader, which may be experiencing an EBS latency spike at the time of the read.

Conclusion: Exploring Open Source Databases on Kubernetes

Our journey running a distributed database on Kubernetes has empowered us to achieve high availability, low latency, scalability, and lower maintenance costs. By leveraging the operator pattern, multi-cluster deployments, AWS EBS, and stale reads, we’ve demonstrated that even open source distributed storage systems can thrive in cloud environments.

We already operate several database clusters in production in the described setup, with the largest one handling 3M QPS across 150 storage nodes, storing over 300+ TB of data spread across 4M internal shards. All this with 99.95% availability thanks to techniques described in this post.

For other companies considering to run open-source databases on Kubernetes, the opportunities are immense. Embrace the challenge, run open-source databases to shape these tools for enterprise use. The future of scalable, reliable data management in the cloud lies in collaboration and open-source innovation — now is the time to lead and participate.

Acknowledgments

Thanks to Abhishek Parmar, Brian Wolfe, Chen Ding, Daniel Low, Hao Luo, Xiaomou Wang for collaboration and Shylaja Ramachandra for editing.

SwiftUI feature architecture at Airbnb

We’ve been leveraging declarative UI patterns at Airbnb for many years, using our UIKit-based Epoxy library and unidirectional data flow systems. When adopting SwiftUI in our screen layer, we decided to continue using our existing unidirectional data flow library. This simplified the process of incrementally adopting SwiftUI within our large codebase, and we find it improves the quality and maintainability of features.

However, we noticed that SwiftUI features using our unidirectional data flow library didn’t perform as well as we expected, and it wasn’t immediately obvious to us what the problem was. Understanding SwiftUI’s performance characteristics is an important requirement for building performant and outside of the “standard” SwiftUI toolbox.

Understanding SwiftUI view diffing

When working with declarative UI systems like SwiftUI, it’s important to ensure the framework knows which views need to be re-evaluated and re-rendered when the state of the screen changes. Changes are detected by diffing the view’s stored properties any time its parent is updated. Ideally the view’s body will only be re-evaluated when its properties actually change:

However, this behavior is not always the reality (more on why in a moment). Unnecessary view body evaluations hurt performance by performing unnecessary work.

How do you know how often a view’s body is re-evaluated in a real app? An easy way to visualize this is with a modifier that applies a random color to the view every time it’s rendered. When testing this on various views in our app’s most performance-sensitive screens, we quickly found that many views were re-evaluated and re-rendered more often than necessary:

The SwiftUI view diffing algorithm

SwiftUI’s built-in diffing algorithm is often overlooked and not officially documented, but it has a huge impact on performance. To determine if a view’s body needs to be re-evaluated, SwiftUI uses a reflection-based diffing algorithm to compare each of the view’s stored properties:

1. If a type is Equatable, SwiftUI compares the old and new values using the type’s Equatable conformance. Otherwise:
2. SwiftUI compares value types (e.g., structs) by recursively comparing each instance property.
3. SwiftUI compares reference types (e.g., classes) using reference identity.
4. SwiftUI attempts to compare closures by identity. However, most non-trivial closures cannot be compared reliably.

If all of the view’s properties compare as equal to the previous value, then the body isn’t re-evalulated and the content isn’t re-rendered. Values using SwiftUI property wrappers like @State and @Environment don’t participate in this diffing algorithm, and instead trigger view updates through different mechanisms.

When reviewing different views in our codebase, we found several common patterns that confounded SwiftUI’s diffing algorithm:

1. Some types are inherently not supported, like closures.
2. Simple data types stored on the view may be unexpectedly compared by reference instead of by value.

Here’s an example SwiftUI view with properties that interact poorly with the diffing algorithm:

struct MyView: View {
  /// A generated data model that is a struct with value semantics,
  /// but is copy-on-write and wraps an internal reference type.
  /// Compared by reference, not by value, which could cause unwanted body evaluations.
  let dataModel: CopyOnWriteDataModel

  /// Other miscellaneous properties used by the view. Typically structs, but sometimes a class.
  /// Unexpected comparisons by reference could cause unwanted body evaluations.
  let requestState: MyFeatureRequestState

  /// An action handler for this view, part of our unidirectional data flow library. 
  /// Wraps a closure that routes the action to the screen's action handler.
  /// Closures almost always compare as not-equal, and typically cause unwanted body evaluations. 
  let handler: Handler<MyViewAction>

  var body: some View { ... }
}

If a view contains any value that isn’t diffable, the entire view becomes non-diffable. Preventing this in a scalable way is almost impossible with existing tools. This finding also reveals the performance issue caused by our unidirectional data flow library: action handling is closure-based, but SwiftUI can’t diff closures!

In some cases, like with the action handlers from our unidirectional data flow library, making the value diffable would require large, invasive, and potentially undesirable architecture changes. Even in simpler cases, this process is still time consuming, and there’s no easy way to prevent a regression from creeping in later on. This is a big obstacle when trying to improve and maintain performance at scale in large codebases with many different contributors.

Controlling SwiftUI view diffing

Fortunately, we have another option: If a view conforms to Equatable, SwiftUI will diff it using its Equatable conformance instead of using the default reflection-based diffing algorithm.

The advantage of this approach is that it lets us selectively decide which properties should be compared when diffing our view. In our case, we know that the handler object doesn’t affect the content or identity of our view. We only want our view to be re-evalulated and re-rendered when the dataModel and requestState values are updated. We can express that with a custom Equatable implementation:

// An Equatable conformance that makes the above SwiftUI view diffable.
extension MyView: Equatable {
  static func ==(lhs: MyView, rhs: MyView) -> Bool {
    lhs.dataModel == rhs.dataModel
      && lhs.requestState == rhs.requestState
      // Intentionally not comparing handler, which isn't Equatable.
  }
}

However:

1. This is a lot of additional boilerplate for engineers to write, especially for views with lots of properties.
2. Writing and maintaining a custom conformance is error-prone. You can easily forget to update the Equatable conformance when adding new properties later, which would cause bugs.

So, instead of manually writing and maintaining Equatable conformances, we created a new @Equatable macro that generates conformances for us.

// A sample SwiftUI view that has adopted @Equatable
// and is now guaranteed to be diffable.
@Equatable
struct MyView: View {
  // Simple data types must be Equatable, or the build will fail.
  let dataModel: CopyOnWriteDataModel
  let requestState: MyFeatureRequestState

  // Types that aren't Equatable can be excluded from the
  // generated Equatable conformance using @SkipEquatable,
  // as long as they don’t affect the output of the view body.
  @SkipEquatable let handler: Handler<MyViewAction>

  var body: some View { ... }
}

The @Equatable macro generates an Equatable implementation that compares all of the view’s stored instance properties, excluding properties with SwiftUI property wrappers like@State and @Environment that trigger view updates through other mechanisms. Properties that aren’t Equatable and don’t affect the output of the view body can be marked with @SkipEquatable to exclude them from the generated implementation. This allows us to continue using the closure-based action handlers from our unidirectional data flow library without impacting the SwiftUI diffing process!

After adopting the @Equatable macro on a view, that view is guaranteed to be diffable. If an engineer adds a non-Equatable property later, the build will fail, highlighting a potential regression in the diffing behavior. This effectively makes the @Equatable macro a sophisticated linter — which is really valuable for scaling these performance improvements in a codebase with many components and many contributors, since it makes it less likely for regressions to slip in later.

Managing the size of view bodies

Another essential aspect of SwiftUI diffing is understanding that SwiftUI can only diff proper View structs. Any other code, such as computed properties or helper functions that generate a SwiftUI view, cannot be diffed.

Consider the following example:

// Complex SwiftUI views are often simplified by
// splitting the view body into separate computed properties.
struct MyScreen: View {
  /// The unidirectional data flow state store for this feature.
  @ObservedObject var store: StateStore<MyState, MyAction>

  var body: some View {
    VStack {
      headerSection
      actionCardSection
    }
  }

  private var headerSection: some View {
    Text(store.state.titleString)
      .textStyle(.title)
  }

  private var actionCardSection: some View {
    VStack {
      Image(store.state.cardSelected ? "enabled" : "disabled")
      Text("This is a selectable card")
    }
    .strokedCard(.roundedRect_mediumCornerRadius_12)
    .scaleEffectButton(action: {
      store.handle(.cardTapped) 
    })
  }
}

This is a common way to organize complex view bodies, since it makes the code easier to read and maintain. However, at runtime, SwiftUI effectively inlines the views returned from the properties into the main view body, as if we instead wrote:

// At runtime, computed properties are no different
// from just having a single, large view body!
struct MyScreen: View {
  @ObservedObject var store: StateStore<MyState, MyAction>

  // Re-evaluated every time the state of the screen is updated.
  var body: some View {
    Text(store.state.titleString)
      .textStyle(.title)

    VStack {
      Image(store.state.cardSelected ? "enabled" : "disabled")
      Text("This is a selectable card")
    }
    .strokedCard(.roundedRect_mediumCornerRadius_12)
    .scaleEffectButton(action: {
      store.handle(.cardTapped) 
    })
  }
}

Since all of this code is part of the same view body, all of it will be re-evaluated when any part of the screen’s state changes. While this specific example is simple, as the view grows larger and more complicated, re-evaluating it will become more expensive. Eventually there would be a large amount of unnecessary work happening on every screen update, hurting performance.

To improve performance, we can implement the layout code in separate SwiftUI views. This allows SwiftUI to properly diff each child view, only re-evaluating their bodies when necessary:

struct MyScreen: View {
  @ObservedObject var store: StateStore<MyState, MyAction>

  var body: some View {
    VStack {
      HeaderSection(title: store.state.titleString)
      CardSection(
       isCardSelected: store.state.isCardSelected,
       handler: store.handler,
      )
    }
  }
}

/// Only re-evaluated and re-rendered when the title property changes.
@Equatable
struct HeaderSection: View {
  let title: String

  var body: some View {
    Text(title)
      .textStyle(.title)
  }
}

/// Only re-evaluated and re-rendered when the isCardSelected property changes.
@Equatable
struct CardSection: View {
  let isCardSelected: Bool
  @SkipEquatable let handler: Handler<MyAction>

  var body: some View {
    VStack {
      Image(store.state.isCardSelected ? "enabled" : "disabled")
      Text("This is a selectable card")
    }
    .strokedCard(.roundedRect_mediumCornerRadius_12)
    .scaleEffectButton(action: {
      handler.handle(.cardTapped) 
    })
  }
}

By breaking the view into smaller, diffable pieces, SwiftUI can efficiently update only the parts of the view that actually changed. This approach helps maintain performance as a feature grows more complex.

View body complexity lint rule

Large, complex views aren’t always obvious during development. Easily available metrics like total line count aren’t a good proxy for complexity. To help engineers know when it’s time to refactor a view into smaller, diffable pieces, we created a custom SwiftLint rule that parses the view body using SwiftSyntax and measures its complexity. We defined the view complexity metric as a value that increases every time you compose views using computed properties, functions, or closures. With this rule we automatically trigger an alert in Xcode when a view is getting too complex. (The complexity limit is configurable, and we currently allow a maximum complexity level of 10.)

Conclusion

With an understanding of how SwiftUI view diffing works, we can use an @Equatable macro to ensure view bodies are only re-evaluated when the values inside views actually change, break views into smaller parts for faster re-evaluation, and encourage developers to refactor views before they get too large and complex.

Applying these three techniques to SwiftUI views in our app has led to a large reduction in unnecessary view re-evaluation and re-renders. Revisiting the examples from earlier, you see far fewer re-renders in the search bar and filter panel:

Using results from our page performance score system, we’ve found that adopting these techniques in our most complicated SwiftUI screens really does improve performance for our users. For example, we reduced scroll hitches by 15% on our main Search screen by adopting @Equatable on its most important views, and breaking apart large view bodies into smaller diffable pieces. These techniques also give us the flexibility to use a feature architecture that best suits our needs without compromising performance or imposing burdensome limitations (e.g., completely avoiding closures in SwiftUI views).

Of course, these techniques aren’t a silver bullet. It’s not necessary for all SwiftUI features to use them, and these techniques by themselves aren’t enough to guarantee great performance. However, understanding how and why they work serves as a valuable foundation for building performant SwiftUI features, and makes it easier to spot and avoid problematic patterns in your own code.

If you’re interested in joining us on our quest to make the best iOS app in the App Store, please see our careers page for open iOS roles.

Impulse is one of our internal load-testing-as-a-service frameworks. It provides tools that can generate synthetic loads, mock dependencies, and collect traffic data from production environments. In this blog post, we’ll share how Impulse is architected to minimize manual effort, seamlessly integrate with our observability stack, and empower teams to proactively address potential issues.

Architecture

Impulse is a comprehensive load testing framework that allows service owners to conduct context-aware load tests, mock dependencies, and collect traffic data to ensure the system’s performance under various conditions. It includes the following components:

1. Load generator to generate context-aware requests on the fly, for testing different scenarios with synthetic or collected traffic.
2. Dependency mocker to mock the downstream responses with latency, so that the load testing on the service under test (SUT) doesn’t need to involve certain dependent services. This is especially crucial when the dependencies are vendor services that don’t support load testing, or if the team wants to regression load test their service during day-to-day deployment without affecting downstreams.
3. Traffic collector to collect both the upstream and downstream traffic from the production environment, and then apply the resulting data to the test environment.
4. Testing API generator to wrap asynchronous workflows into synchronous API calls for load testing.

Each of these four tools are independent, allowing service owners the flexibility to select one or more components for their load testing needs.

Load generator

Context aware

When load testing, requests made to the SUT often require some information from the previous response or need to be sent in a specific order. For example, if an update API needs to provide an entity_id to update, we must ensure the entity already exists in the testing environment context.

Our load generator tool allows users to write arbitrary testing logic in Java or Kotlin and launch containers to run these tests at scale against the SUT. Why write code instead of DSL/configuration logic?

• Flexibility: Programming languages are more expressive than DSL and can better support complex contextual scenarios.
• Reusability: The same testing code can be used in other tests, e.g., integration tests.
• Developer proficiency: Low/no learning curve to onboard, don’t need to learn how to write testing logic.
• Developer experience: IDE support, testing, debugging, etc.

Here is an example of synthetic context-aware test case:

class HelloWorldLoadGenerator : LoadGenerator {
   override suspend fun run() {
       val createdEntity = sutApiClient.create(CreateRequest(name="foo", ...)).data

       // request with id from previous response (context)
       val updateResponse = sutApiClient.update(UpdateRequest(id=createdEntity.id, name="bar"))
       
       // ... other operations
       
       // clean up
       sutApiClient.delete(DeleteRequest(id=createdEntity.id))
   }
}

Decentralized

The load generator is decentralized and containerized, which means each time a load test is triggered, a set of new containers will be created to run the test. This design has several benefits:

• Isolation: Load testing runs between different services are isolated from each other, eliminating any interference.
• Scalability: The number of containers can be scaled up or down according to the traffic requirements.
• Cost efficiency: The containers are short-lived, as they only exist during the load testing run.

What’s more, as our services are cloud based, a subtle point is that the Impulse framework will evenly distribute the workers among all our data centers, and the load will be emitted evenly from all the workers. Impulse’s load generator ensures the overall trigger per second (TPS) is as configured. Based on this, we can better leverage the locality settings in load balancers, which can better mimic the real traffic distribution in production.

Execution

The load generator is designed to be executed in the CI/CD pipeline, which means we can trigger load testing automatically. Developers can configure the testing spec in multiple phases, e.g., a warm up phase, a steady state phase, a peak phase, etc. Each phase can be configured with:

• Test cases to run
• TPS (trigger per second) of each test case
• Test duration

Dependency mocker

Impulse is a decentralized framework where each service has its own dependency mocker. This can eliminate interference between services and reduce communication costs. Each dependency mocker is an out-of-process service, which means the SUT behaves just as it does in production. We run the mockers in separate instances to avoid any impact on the performance of the SUT. The mock servers are all short lived — they only start before tests run and shut down afterwards to save costs and maintenance effort. The response latency and exceptions are configurable and the number of mocker instances can be adjusted on demand to support large amounts of traffic.

Other noteworthy features:

• You can selectively stub some of the dependencies. Currently, stubbing is supported for HTTP JSON, Airbnb Thrift, and Airbnb GraphQL dependencies.
• The dependency mockers support use cases beyond load testing. For instance, integration tests often rely on other services or third-party API calls, which may not guarantee a stable testing environment or might only support ideal scenarios. Dependency mockers can address this by offering predefined responses or exceptions to fully test those flows.

Impulse supports two options for generating mock responses:

1. Synthetic response: The response is generated by user logic, as in integration testing; the difference is that the response comes from a remote (out-of-process) server with simulated latency.
   – Similar to the load generator, the logic is written in Java/Kotlin code and contains request matching and response generation.
   – Latency can be simulated using p95/p99 metrics.
2. Replay response: The response is replayed from the production downstream recording, supported by the traffic collector component.

Here is an example of a synthetic response with latency in Kotlin:

downstreamsMocking.every(
      thriftRequest<FooRequest>().having { it.message == "hello" }
    ).returns { request ->
      ThriftDownstream.Response.thriftEncoded(
        HttpStatus.OK,
        FooResponse.builder.reply("${request.message} world").build()
      )
    }.with {
      delay = latencyFromP95(p95=500.miliseconds, min=200.miliseconds, max=2000.miliseconds)
    }

Traffic collector

The traffic collector component is designed to capture both upstream and downstream traffic, along with the relationships between them. This approach allows Impulse to accurately replay production traffic during load testing, avoiding inconsistencies in downstream data or behavior. By replicating downstream responses — including production-like latency and errors — via the dependency mocker, the system ensures high-fidelity load testing. As a result, services in the testing environment behave identically to those in production, enabling more realistic and reliable performance evaluations.

Testing API generator

We rely heavily on event-driven, asynchronous workflows that are critical to our business operations. These include processing events from a message queue (MQ) and executing delayed jobs. Most of the MQ events/jobs are emitted from synchronous flows (e.g., API calls), so theoretically they can be covered by API load testing. However, the real world is more complex. These asynchronous flows often involve long chains of event and job emissions originating from various sources, making it difficult to replicate and test them accurately using only API-based methods.

To address this, the testing API generator component creates HTTP APIs during the CI stage according to the event or job schema. These APIs act as wrappers around the underlying asynchronous flows and are registered exclusively in the testing environment. This setup enables load testing tools — such as load generators — to send traffic to these synthetic APIs, allowing asynchronous flows to be exercised as if they were synchronous. As a result, it’s possible to perform targeted, realistic load testing on asynchronous logic that would otherwise be hard to simulate.

The goal of the testing API generator is to help developers identify performance bottlenecks and potential issues in their async flow implementations and under high traffic conditions. It does this by enabling direct load testing of async flows without involving middleware components like MQs. The rationale is that developers typically aim to evaluate the behavior of their own logic, not the middleware, which is usually already well-tested. By bypassing these components, this approach simplifies the load testing process and empowers developers to independently manage and execute their own tests.

Integration with other testing frameworks

Airbnb emphasizes product quality, utilizing versatile testing frameworks that cover integration and API tests across development, staging, and production environments, and integrate smoothly into CI/CD pipelines. The modular design of Impulse facilitates its integration with these frameworks, offering systematic service testing.

Conclusion

In this blog post, we shared how Impulse and its four core components help developers perform self-service load testing at Airbnb. As of this writing, Impulse has been implemented in several customer support backend services and is currently under review with different teams across the company who are planning to leverage Impulse to conduct load testing.

We’ve received a lot of good feedback in the process. For example: “Impulse helps us to identify and address potential issues in our service. During testing, it detected an ApiClientThreadToolExhaustionException caused by thread pool pressure. Additionally, it alerted us about occasional timeout errors in client API calls during service deployments. Impulse helped us identify high memory usage in the main service container, enabling us to fine-tune the memory allocation and optimize our service’s resource usage. Highly recommend utilizing Impulse as an integral part of the development and testing processes.”

Acknowledgments

Thanks to Jeremy Werner, Yashar Mehdad, Raj Rajagopal, Claire Cheng, Tim L., Wei Ji, Jay Wu, Brian Wallace for support on the Impulse project.

Does this type of work interest you? Check out our open roles here.

Over the years, Airbnb has invested in conversational AI to enhance customer support. In our previous blog posts Task-Oriented Conversational AI in Airbnb Customer Support and Using Chatbots to Provide Faster COVID-19 Community Support, we explored how AI-driven chatbots streamline guest and host interactions through automated messaging. This post explains how we extend that work to voice-based support, leveraging machine learning to improve real-time phone interactions with our intelligent IVR system.

We’ll take you through the end-to-end IVR journey, the key machine learning components that power it, and how we designed a system that delivers faster, more human-like, and more intuitive voice support for our community.

Reimagining the voice support journey

Traditional IVR systems often rely on rigid menu trees, requiring callers to press buttons and navigate pre-set paths. Instead, we designed an adaptive, conversational IVR that listens, understands, and responds in real time. Here’s normally what happens when a caller reaches out to Airbnb support:

1. Call and greeting: IVR picks up and prompts, “In a few sentences, please tell us why you’re calling today.”
2. Automated speech recognition (ASR): The caller’s response is transcribed with Airbnb-specific ASR. For example, if a caller says, “I need to request a refund for my reservation,” ASR accurately converts this speech into text, preserving key domain-specific terms.
3. Understanding intent: A Contact Reason Detection model classifies the issue into a category like cancellations, refunds, account issues, etc.
4. Decision-making: If self-service is possible, the system retrieves and sends a relevant help article or an intelligent workflow via SMS or app notification. If the caller explicitly requests agent support or the issue requires human intervention, the call is routed to a customer support agent with relevant details attached.
5. Clarifying response: A Paraphrasing model generates a summary of the user intent, which IVR shares with the user before delivering the solution. This ensures that users understand the context of the resource they receive. Continuing our example, the system would respond, “I understand your issue is regarding a refund request. We have sent you a link to resources about this topic. Follow the instructions to find answers. If you need to speak with an agent, press 0 to be connected to our customer service representative.” The underscored Paraphrasing component enhances engagement by bridging the gap between system-generated responses and user comprehension, making the self-service experience more intuitive.
6. Resolution or escalation: The caller receives an SMS or app notification with a direct link to a relevant Airbnb Help Center article. If further assistance is needed, they can press 0 to connect with a customer service representative.

By moving away from rigid menus to natural language understanding, we allow guests and hosts to express their issues in their own words, helping to increase satisfaction and resolution efficiency.

Breaking down our ML-powered IVR system

1. Automated speech recognition (ASR): transcribing with precision

In a voice-driven support system, achieving high transcription accuracy is essential, particularly in noisy phone environments where speech can be unclear. General speech recognition models often struggle with Airbnb-specific terminology, leading to errors like misinterpreting “listing” as “lifting” or “help with my stay” as “happy Christmas Day.” These inaccuracies create challenges in understanding user intent and impact downstream processes.

To enhance ASR accuracy, we transitioned from a generic high-quality pretrained model to one specifically adapted for noisy phone audio. Additionally, we introduced a domain-specific phrase list optimization that ensures Airbnb terms are properly recognized. Based on a sample of hundreds of clips, this significantly reduced the word error rate (WER) from 33% to approximately 10%. The reduced WER significantly enhanced the accuracy of downstream help article recommendations, increasing user engagement, improving customer NPS among users who interacted with the ASR menu, while reducing reliance on human agents and lowering customer service handling time.

2. Contact Reason prediction: understanding the why

After transcribing the caller’s statements, the next step involves identifying their intent. We accomplished this by creating a detailed Contact Reason taxonomy that categorizes all potential Airbnb inquiries, as elaborated in “T-LEAF: Taxonomy Learning and EvaluAtion Framework.” We then use an intent detection model to classify calls into a Contact Reason category, ensuring each inquiry is handled appropriately. For example, if a caller mentions “I haven’t received my refund yet,” the model predicts the Contact Reason as Missing Refund and forwards it to the relevant downstream components.

In production, we deploy the Issue Detection Service to host the intent detection models, running them in parallel to achieve optimal scalability, flexibility, and efficiency. Parallel computing ensures that intent detection latency remains under 50ms on average, making the process imperceptible to IVR users and ensuring a seamless real-time experience. The detected intent is then analyzed within the IVR workflow to determine the next action, whether it’s guiding the user through a self-service resolution or escalating directly to a human agent.

Occasionally, callers prefer to speak directly with a human agent instead of describing their issues, using terms like “agent” or “escalation.” For such scenarios, we use a different intent detection model to recognize when a caller wants to escalate to a human agent. If this intent is detected, the IVR system honors the caller’s request and routes the call to the suitable support team.

3. Help article retrieval: delivering the right information

Many common Airbnb issues can be quickly resolved by providing clear and relevant educational information. To help provide useful information to users and minimize the need for human customer support, we use the Help Article Retrieval and Ranking system. This advanced system automatically identifies the issue in a user’s inquiry and delivers the most relevant help article link via SMS text message and Airbnb app notification. Our process incorporates two machine learning stages.

Semantic retrieval and ranking: We index Airbnb Help Article embeddings into a vector database, enabling efficient retrieval of up to 30 relevant articles per user query using cosine similarity, typically within 60ms. An LLM-based ranking model then re-ranks these retrieved articles, with the top-ranked article directly presented to users via IVR channels. This dual-stage system not only powers IVR interactions but also supports our customer support chatbot and Help Center search. Across these platforms, its effectiveness is continuously evaluated using metrics like Precision@N, facilitating ongoing improvements and refinements.

4. Paraphrasing model: enhancing user understanding

A key challenge in IVR-based customer support is ensuring users clearly understand the resolution before receiving help article links, as they typically lack visibility into the article’s contents or title. To address this, we implemented a lightweight paraphrasing approach leveraging a curated set of standardized summaries.

UX writers created concise and clear paraphrases for common Airbnb scenarios. During online serving, user inquiries are mapped to these curated summaries via nearest-neighbor matching based on text embedding similarity. We calibrated a similarity threshold to ensure high-quality matches. Manual evaluation of end-to-end model outputs confirmed precision exceeding 90%.

The outcome was a finite-state solution delivering the most appropriate paraphrased IVR prompt before presenting a help article link. For example, if a caller states, “I need to cancel my reservation and request a refund,” the model generates a response like “I understand your issue is about a refund request” before sending the retrieved help article link.

Integrating this model ensures users receive clear, contextually relevant summaries prior to accessing help articles. In an experiment targeting English hosts who contacted customer support, we found that presenting a paraphrased summary before sending the article link increases user engagement with article content, resulting in improvement in self-resolution rates, helping to reduce the need for direct customer support assistance.

Conclusion

By combining Automated Speech Recognition and Contact Reason Detection systems with a help article retrieval system, and a paraphrasing model, we have created an IVR system that streamlines support interactions and improves user satisfaction. Our solution enables callers to describe issues naturally, reduces dependency on human agents for common inquiries, and provides instant, relevant support through self-service. When human assistance is necessary, the system ensures a smooth transition by routing users to the right agent with essential context.

Interested in working at Airbnb? Check out our open roles.

Acknowledgements

Thanks to Zhenyu Zhao, Mia Zhao, Wayne Zhang, Lucca Siaudzionis, Lulu Chen, Sukey Xu, Floria Wan, Michael Zhou, Can Yang, Yaolin Chen, Shuaihu Wang, Huifan Qu, Ming Shang,Yu Jiang, Wanting Chen, Elena Zhao, Shanna Su, Cassie Cao, Hao Wang, Haoran Zhu, Xirui Liu, Ying Tan, Xiaohan Zeng, Xiaoyu Meng, Gavin Li, Gaurav Rai, Hemanth Kolla, Ihor Hordiienko, Matheus Scharf, and Stepan Sydoruk who helped bring this vision to life. Also thanks to Paige Schwartz, Stephanie Chu, Neal Cohen, Becky Ajuonuma, Iman Saleh, Dani Normanm, Javier Salido, and Lauren Mackevich for the review and editing.

Thanks to Jeremy Werner, Joy Zhang, Claire Cheng, Yashar Mehdad, Shuohao Zhang, Shawn Yan, Kelvin Xiong, Michael Lubavin, Teng Wang, Wei Ji, and Chenhao Yang’s leadership support on building conversational AI products at Airbnb.

In this blog post, we’ll highlight the unique challenges we faced migrating from Enzyme to RTL, how LLMs excel at solving this particular type of challenge, and how we structured our migration tooling to run an LLM-driven migration at scale.

Background

In 2020, Airbnb adopted React Testing Library (RTL) for all new React component test development, marking our first steps away from Enzyme. Although Enzyme had served us well since 2015, it was designed for earlier versions of React, and the framework’s deep access to component internals no longer aligned with modern React testing practices.

However, because of the fundamental differences between these frameworks, we couldn’t easily swap out one for the other (read more about the differences here). We also couldn’t just delete the Enzyme files, as analysis showed this would create significant gaps in our code coverage. To complete this migration, we needed an automated way to refactor test files from Enzyme to RTL while preserving the intent of the original tests and their code coverage.

How We Did It

In mid-2023, an Airbnb hackathon team demonstrated that large language models could successfully convert hundreds of Enzyme files to RTL in just a few days.

Building on this promising result, in 2024 we developed a scalable pipeline for an LLM-driven migration. We broke the migration into discrete, per-file steps that we could parallelize, added configurable retry loops, and significantly expanded our prompts with additional context. Finally, we performed breadth-first prompt tuning for the long tail of complex files.

1. File Validation and Refactor Steps

We started by breaking down the migration into a series of automated validation and refactor steps. Think of it like a production pipeline: each file moves through stages of validation, and when a check fails, we bring in the LLM to fix it.

We modeled this flow like a state machine, moving the file to the next state only after validation on the previous state passed:

This step-based approach provided a solid foundation for our automation pipeline. It enabled us to track progress, improve failure rates for specific steps, and rerun files or steps when needed. The step-based approach also made it simple to run migrations on hundreds of files concurrently, which was critical for both quickly migrating simple files, and chipping away at the long tail of files later in the migration.

2. Retry Loops & Dynamic Prompting

Early on in the migration, we experimented with different prompt engineering strategies to improve our per-file migration success rate. However, building on the stepped approach, we found the most effective route to improve outcomes was simply brute force: retry steps multiple times until they passed or we reached a limit. We updated our steps to use dynamic prompts for each retry, giving the validation errors and the most recent version of the file to the LLM, and built a loop runner that ran each step up to a configurable number of attempts.

With this simple retry loop, we found we could successfully migrate a large number of our simple-to-medium complexity test files, with some finishing successfully after a few retries, and most by 10 attempts.

3. Increasing the Context

For test files up to a certain complexity, just increasing our retry attempts worked well. However, to handle files with intricate test state setups or excessive indirection, we found the best approach was to push as much relevant context as possible into our prompts.

By the end of the migration, our prompts had expanded to anywhere between 40,000 to 100,000 tokens, pulling in as many as 50 related files, a whole host of manually written few-shot examples, as well as examples of existing, well-written, passing test files from within the same project.

Each prompt included:

• The source code of the component under test
• The test file we were migrating
• Validation failures for the step
• Related tests from the same directory (maintaining team-specific patterns)
• General migration guidelines and common solutions

Here’s how that looked in practice (significantly trimmed down for readability):

// Code example shows a trimmed down version of a prompt 
// including the raw source code from related files, imports, 
// examples, the component source itself, and the test file to migrate.

const prompt = [
  'Convert this Enzyme test to React Testing Library:',
  `SIBLING TESTS:\n${siblingTestFilesSourceCode}`,
  `RTL EXAMPLES:\n${reactTestingLibraryExamples}`,
  `IMPORTS:\n${nearestImportSourceCode}`,
  `COMPONENT SOURCE:\n${componentFileSourceCode}`,
  `TEST TO MIGRATE:\n${testFileSourceCode}`,
].join('\n\n');

This rich context approach proved highly effective for these more complex files — the LLM could better understand team-specific patterns, common testing approaches, and the overall architecture of the codebase.

We should note that, although we did some prompt engineering at this step, the main success driver we saw was choosing the right related files (finding nearby files, good example files from the same project, filtering the dependencies for files that were relevant to the component, etc.), rather than getting the prompt engineering perfect.

After building and testing our migration scripts with retries and rich contexts, when we ran our first bulk run, we successfully migrated 75% of our target files in just four hours.

4. From 75% to 97%: Systematic Improvement

That 75% success rate was really exciting to get to, but it still left us with nearly 900 files failing our step-based validation criteria. To tackle this long tail, we needed a systematic way to understand where remaining files were getting stuck and improve our migration scripts to address these issues. We also wanted to do this breadth first to aggressively chip away at our remaining files without getting stuck on the most difficult migration cases.

To do this, we built two features into our migration tooling.

First, we built a simple system to give us visibility into common issues our scripts were facing by stamping files with an automatically-generated comment to record the status of each migration step. Here’s what that code comment looked like:

// MIGRATION STATUS: {"enyzme":"done","jest":{"passed":8,"failed":2,"total":10,"skipped":0,"successRate":80},"eslint":"pending","tsc":"pending",}

And second, we added the ability to easily re-run single files or path patterns, filtered by the specific step they were stuck on:

$ llm-bulk-migration --step=fix-jest --match=project-abc/**

Using these two features, we could quickly run a feedback loop to improve our prompts and tooling:

1. Run all remaining failing files to find common issues the LLM is getting stuck on
2. Select a sample of files (5 to 10) that exemplify a common issue
3. Update our prompts and scripts to address that issue
4. Re-run against the sample of failing files to validate our fix
5. Repeat by running against all remaining files again

After running this “sample, tune, sweep” loop for 4 days, we had pushed our completed files from 75% to 97% of the total files, and had just under 100 files remaining. By this point, we had retried many of these long tail files anywhere between 50 to 100 times, and it seemed we were pushing into a ceiling of what we could fix via automation. Rather than invest in more tuning, we opted to manually fix the remaining files, working from the baseline (failing) refactors to reduce the time to get those files over the finish line.

Results and Impact

With the validation and refactor pipeline, retry loops, and expanded context in place, we were able to automatically migrate 75% of our target files in 4 hours.

After four days of prompt and script refinement using the “sample, tune, and sweep” strategy, we reached 97% of the 3.5K original Enzyme files.

And for the remaining 3% of files that didn’t complete through automation, our scripts provided a great baseline for manual intervention, allowing us to complete the migration for those remaining files in another week of work.

Most importantly, we were able to replace Enzyme while maintaining original test intent and our overall code coverage. And even with high retry counts on the long tail of the migration, the total cost — including LLM API usage and six weeks of engineering time — proved far more efficient than our original manual migration estimate.

What’s Next

This migration underscores the power of LLMs for large-scale code transformation. We plan to expand this approach, develop more sophisticated migration tools, and explore new applications of LLM-powered automation to enhance developer productivity.

Want to help shape the future of developer tools? We’re hiring engineers who love solving complex problems at scale. Check out our careers page to learn more.

Search plays a crucial role in helping Airbnb guests find the perfect stay. The goal of Airbnb Search is to surface the most relevant listings for each user’s query — but with millions of available homes, that’s no easy task. It’s especially difficult when searches include large geographic areas (like California or France) or high-demand destinations (like Paris or London). Recent innovations — such as flexible date search, which allows guests to explore stays without fixed check-in and check-out dates — have added yet another layer of complexity to ranking and finding the right results.

To tackle these challenges, we need a system that can retrieve relevant homes while also being scalable enough (in terms of latency and compute) to handle queries with a large candidate count. In this blog post, we share our journey in building Airbnb’s first-ever Embedding-Based Retrieval (EBR) search system. The goal of this system is to narrow down the initial set of eligible homes into a smaller pool, which can then be scored by more compute-intensive machine learning models later in the search ranking process.

Figure 1: The general stages and scale for the various types of ranking models used in Airbnb Search

We’ll explore three key challenges in building this EBR system: (1) constructing training data, (2) designing the model architecture, and (3) developing an online serving strategy using Approximate Nearest Neighbor (ANN) solutions.

Training Data Construction

The first step in building our EBR system was training a machine learning model to map both homes and de-identified search queries into numerical vectors. To achieve this, we built a training data pipeline (Figure 3) that leveraged contrastive learning — a strategy that involves identifying pairs of positive- and negative-labeled homes for a given query. During training, the model learns to map a query, a positive home, and a negative home into a numerical vector, such that the similarity between the query and the positive home is much higher than the similarity between the query and the negative home.

To construct these pairs, we devised a sampling method based on user trips. This was an important design decision, since users on Airbnb generally undergo a multi-stage search journey. Data shows that before making a final booking, users tend to perform multiple searches and take various actions — such as clicking into a home’s details, reading reviews, or adding a home to a wishlist. As such, it was crucial to develop a strategy that captures this entire multi-stage journey and accounts for the diverse types of listings a user might explore.

Diving deeper, we first grouped all historical queries of users who made bookings, using key query parameters such as location, number of guests, and length of stay — our definition of a “trip.” For each trip, we analyzed all searches performed by the user, with the final booked listing as the positive label. To construct (positive, negative) pairs, we paired this booked listing with other homes the user had seen but not booked. Negative labels were selected from homes the user encountered in search results, along with those they had interacted with more intentfully — such as by wishlisting — but ultimately did not book. This choice of negative labels was key: Randomly sampling homes made the problem too easy and resulted in poor model performance.

Figure 2: Example of constructing (positive, negative) pairs for a given user journey. The booked home is always treated as a positive. Negatives are selected from homes that appeared in the search result (and were potentially interacted with) but that the user did not end up booking.

Figure 3: Example of overall data pipeline used to construct training data for the EBR model.

Model Architecture

The model architecture followed a traditional two-tower network design. One tower (the listing tower) processes features about the home listing itself — such as historical engagement, amenities, and guest capacity. The other tower (the query tower) processes features related to the search query — such as the geographic search location, number of guests, and length of stay. Together, these towers generate the embeddings for home listings and search queries, respectively.

A key design decision here was choosing features such that the listing tower could be computed offline on a daily basis. This enabled us to pre-compute the home embeddings in a daily batch job, significantly reducing online latency, since only the query tower had to be evaluated in real-time for incoming search requests.

Figure 4: Two-tower architecture as used in the EBR model. Note that the listing tower is computed offline daily for all homes.

Online Serving

The final step in building our EBR system was choosing the infrastructure for online serving. We explored a number of approximate nearest neighbor (ANN) solutions and narrowed them down to two main candidates: inverted file index (IVF) and hierarchical navigable small worlds (HNSW). While HNSW performed slightly better in terms of evaluation metrics — using recall as our main evaluation metric — we ultimately found that IVF offered the best trade-off between speed and performance.

The core reason for this is the high volume of real-time updates per second for Airbnb home listings, as pricing and availability data is frequently updated. This caused the memory footprint of the HNSW index to grow too large. In addition, most Airbnb searches include filters, especially geographic filters. We found that parallel retrieval with HNSW alongside filters resulted in poor latency performance.

In contrast, the IVF solution, where listings are clustered beforehand, only required storing cluster centroids and cluster assignments within our search index. At serving time, we simply retrieve listings from the top clusters by treating the cluster assignments as a standard search filter, making integration with our existing search system quite straightforward.

Figure 5: Overall serving flow using IVF. Homes are clustered beforehand and, during online serving, homes are retrieved from the closest clusters to the query embedding.

In this approach, our choice of similarity function in the EBR model itself ended up having interesting implications. We explored both dot product and Euclidean distance; while both performed similarly from a model perspective, using Euclidean distance produced much more balanced clusters on average. This was a key insight, as the quality of IVF retrieval is highly sensitive to cluster size uniformity: If one cluster had too many homes, it would greatly reduce the discriminative power of our retrieval system.

We hypothesize that this imbalance arises with dot product similarity because it inherently only considers the direction of feature vectors while ignoring their magnitudes — whereas many of our underlying features are based on historical counts, making magnitude an important factor.

Figure 6: Example of the distribution of cluster sizes when using dot product vs. Euclidean distance as a similarity measure. We found that Euclidean distance produced much more balanced cluster sizes.

Results

The EBR system described in this post was fully launched in both Search and Email Marketing production and led to a statistically-significant gain in overall bookings when A/B tested. Notably, the bookings lift from this new retrieval system was on par with some of the largest machine learning improvements to our search ranking in the past two years.

The key improvement over the baseline was that our EBR system effectively incorporated query context, allowing homes to be ranked more accurately during retrieval. This ultimately helped us display more relevant results to users, especially for queries with a high number of eligible results.

Acknowledgments

We would like to especially thank the entire Search and Knowledge Infrastructure & ML Infrastructure org (led by Yi Li) and Marketing Technology org (led by Michael Kinoti) for their great collaborations throughout this project!

Most of the existing literature on lifetime value focuses on traditional sales channels in which a single seller transacts with many buyers (e.g. a retailer selling clothing to a customer). In contrast, this blog post explains how we model lifetime value in a platform like Airbnb, with multiple sellers and buyers. In the first section, we describe our general listings lifetime value framework. In the second section, we discuss relevant challenges when putting this framework into practice.

Our Listing Lifetime Value Framework

Our listing lifetime value (LTV) framework estimates three different quantities of interest: baseline LTV, incremental LTV, and marketing-induced incremental LTV.

(1) Baseline LTV

To measure LTV, we need to define what we mean by “value” and what time horizon constitutes a “lifetime.” Simplifying slightly for the purposes of this blog post, we define and estimate our baseline listing LTV as the total number of bookings that a listing will make on Airbnb over the next 365 days.

We rely on machine learning and the rich information we have about our listings to estimate this quantity for each individual listing. In practice, we also follow financial guidance to arrive at present value by projecting outcomes into the future and applying a relevant discount rate to future value.

Table 1 shows some hypothetical baseline LTV estimates. As you can see from the examples, LTV is not static, and can evolve as we improve the accuracy of our estimates, observe changes in our marketplace, or even develop a listing (e.g., by providing guidance that helps hosts improve the listing to get more bookings).

We use baseline LTV estimates to segment our listings and identify which types of listings resonate best with our guests. This informs our supply expansion strategy. We also use baseline LTV to identify listings that are not expected to reach their full booking potential and may benefit from additional guidance.

(2) Incremental LTV

When estimating lifetime value, we face a challenge that is common across multi-sided marketplaces: the transactions made by one listing might come at the expense of another listing’s transactions. For example, when a new listing joins our marketplace, this listing will get some bookings from guests who were previously booking other listings. We need to account for this dynamic if we want to accurately measure how much value is added by each listing.

We address this challenge by creating “incremental LTV” estimates. We refer to the additional transactions that would not have occurred without the listing’s participation as “incremental value,” and the transactions that would have occurred even without the listing’s participation as “cannibalized value.” We estimate the incremental LTV for a listing by subtracting cannibalized value estimates from the baseline LTV. We explain this adjustment in more detail when discussing measurement challenges.

(3) Marketing-induced incremental LTV

Lifetime value is not static, and our LTV model needs to tell us how our internal initiatives bring additional listing value. For example, suppose we run a marketing campaign that provides hosts with tips on how to successfully improve their listings. To understand the return from the campaign, we need to measure how much value is accrued due to the campaign, and how much value would have been organically accrued without our marketing intervention. We calculate “marketing-induced incremental LTV” to measure how much additional listing LTV is created by our internal initiatives.

Having outlined our measurement framework (summarized in Figure 2), we now cover some of the technical challenges we faced when putting this framework into practice.

Challenges when measuring Listing Lifetime Value

Challenge (1): Accurately measuring baseline LTV

The most important requirement for our framework is accurate estimation of baseline LTV. Figure 3 illustrates our estimation setup. First, we leverage listing features snapshotted at estimation time t. This data includes rich knowledge we have about each listing and host (availability, price, location, host tenure, etc). We then use these features to train our machine learning model. As a value label, we use the number of bookings made within the next 365-day period, which is observed on date t + 365.

This setup has two important implications that impact accuracy and evaluation:

• We have to wait 365 days to fully evaluate the accuracy of a prediction.
• Our initial training data might not allow us to make accurate predictions if we observe shocks between the time when the training data was captured, and the time when we score the model.

In practice, we felt the full consequences of these implications during the COVID-19 pandemic, when travel came to a halt and marketplace dynamics changed drastically. Our model’s training data from before the pandemic had dramatically different characteristics relative to the scoring data we collected after the pandemic. When dealing with this shock, we implemented various strategies that helped us improve model accuracy:

• Reducing training windows, allowing us to reduce model drift.
• Feeding the model with granular geographic data and human-provided information about external factors as borders closed and reopened due to the pandemic.
• Adopting LightGBM, which handles high cardinality features like the geographic variables mentioned previously.

Challenge (2): Measuring incrementality

Accounting for incrementality is challenging because we never observe the ground truth. While we observe how many bookings are made per listing, we cannot tell which bookings are incremental and which bookings are cannibalized from other listings.

Since we don’t have an incrementality label to estimate this outcome directly, we instead estimate a production function. Intuitively, incrementality is heavily dependent on our ability to connect both sides of our marketplace. Production functions allow us to identify when our supply of listings and demand from guests connect and provide incremental value. Incrementality estimates will be high when a segment has high guest demand and relatively low listing supply. In contrast, incrementality will be low when segments have a large volume of listing supply and relatively low demand, meaning guests have an easy time finding a place to stay and a new listing is more likely to cannibalize bookings from other listings.

Specifically, we model how our total supply of listings (S) and total demand from guests (D) impacts our target outcome bookings (O), as in equation (1):

We estimate this model with historical supply, demand, and outcome data aggregated across internally-defined segments that have little overlapping demand. Having estimated model (1), we calculate how extra supply of listings results in additional bookings in the given segment: this is our estimate of incrementality.

Challenge (3): Handling uncertainty

To handle the uncertainty we experienced during the pandemic, we began updating our LTV estimates as listings received greater or fewer numbers of bookings than initially expected. This approach has helped us capture any shocks that occur after making our initial predictions.

To show how this can be useful, let’s go back to our marketing campaign example. Assume that we run this campaign for six months, and that we measure the success of this campaign by comparing marketing-induced incremental LTV against our total marketing investment in the campaign. As a first approach, we could use the initial baseline LTV figures (which feed into marketing-induced LTV) estimated at the time when the listing was first targeted by our initiative. However, listings targeted on day 1 of the marketing campaign will have six months of booking history by the time the campaign ends and we evaluate success. A more accurate approach uses realized bookings after the initial prediction to start correcting for model error.

Table 2 illustrates how this works. Suppose that on 2024–01–01, we expect that Listing A will get a total of 16 bookings by the end of the year. If six months into the 365 day period, Listing A has received 16 bookings, we should adjust its expected value upward to, say, 21 bookings. In fact, every day for 365 days after 2024–01–01, we can look at the bookings that Listing A has accrued and adjust the expected bookings accordingly. By construction, the expected and accrued bookings converge to the final bookings 365 days after the initial booking date. Going back to our marketing example, if Listing A ultimately receives 20 bookings, updating the initial estimate means we went from 20% underprediction on day 0 to a more reasonable 5% overprediction as of month 6.

In practice, we make daily adjustments to a listing’s expected value based on the listing’s accrued value, updated listing features, and value arrival patterns for similar listings estimated using historical data.

Conclusion

In this blog post, we explained how we approach listing lifetime value at Airbnb. We covered our measurement framework, including baseline LTV, incremental LTV, and marketing-induced incremental LTV. We also zoomed into measurement challenges, like when travel patterns changed drastically during the COVID pandemic and accurately estimating LTV became more difficult.

Estimating the lifetime value for each listing is important because it helps us serve our community more effectively. Use cases include:

• Identifying unique listing segments through which new hosts can showcase their hospitality to a large guest audience.
• Pinpointing locations where listings have an opportunity to get more bookings, and might benefit from additional demand.
• Identifying which internal marketing initiatives bring the most value to our community.

It’s also worth noting that our measurement framework may extend to other applications, such as the lifetime value for Airbnb Experiences listings, where the value of an experience listing will heavily depend on travel trends and on guests’ ability to discover these experiences.

We continue to solve interesting problems around LTV every day (and as more insights come up, we’ll keep sharing them on our blog). Can you see yourself making an impact here? If so, we encourage you to explore the open roles on our team.

Acknowledgments

Finally, we need to give special thanks to Airfam and alumni Sam Barrows, Robert Chang, Linsha Chen, Richard Dear, Andrey Fradkin, Ruben Lobel, Brian De Luna, Dan T. Nguyen, Vaughn Quoss, Jason Ting, and Peng Ye. Without their foundational work, these LTV models would not have been possible.

Thanks as well to Rebecca Ajuonuma, Carolina Barcenas, Nathan Brixius, Jenny Chen, Peter Coles, Lauren Mackevich, Dan Schmierer, Yvonne Wang, Shanni Weilert, and Jane Zhang for their valuable feedback when writing this blog post.

Doctor, lawyer, or engineer?

Anybody who grew up in Kenya around when I did is probably aware that medicine, law, and engineering were the only options for an ambitious student. And at least in my family, while all three careers were highly regarded, nothing was quite as prized as becoming a doctor.

Needless to say, I made my parents very proud when I was accepted to medical school. This is the story of the more than 20 years since, during which I’ve learned so much and have found my niche — as a software engineering leader.

Choosing my own adventure

It may help to describe how I arrived at that medical school acceptance in the first place, and how by that time, I had already started developing an interest in computers. As the child of an entrepreneur father and an engineer mother, I am grateful to have had two amazing role models from a young age. My mother’s grit has been particularly inspiring. She was the only female in her university cohort, and time and time again she has had to work harder than everybody else to prove herself. She became a leader at Kenya’s major telecom company and directly contributed to bringing the country high-speed Internet, giving my generation and ones after access to a world of information.

Always proponents of education, my parents sent me to one of the country’s premier institutions, the Starehe Boys’ Centre and School. The school’s mission is to develop youth into better human beings and leaders, with an emphasis on a holistic education beyond just academics. To this day, I live by the school’s values: integrity, leadership, and service.

I was lucky to be one of the first students in Kenya to take a computer studies course in school. While most students used the computer lab to play video games, myself included, I spent a lot of my time learning how to code. Seeing my deep interest in programming, my classmates would joke that I was going to be the next Bill Gates. I think those formative experiences instilled in me a dream to one day work at Microsoft, one of the biggest technology companies in the world. I wanted to make an impact through technology, which I saw and continue to see as an engine for leveling the global playing field. At the time, however, that was all just a dream and Redmond, Washington couldn’t have felt further away.

For a while, I put chasing that dream on hold as I applied to medical school and was later accepted. Everything was set for me to matriculate in six months’ time, and during that gap I took the opportunity to get some more hands-on coding experience. I realized that medicine wasn’t for me and that not only did I want to study software engineering, but I wanted to come to the United States to be as close as possible to Silicon Valley.

It took a lot of courage to admit this change of heart to my parents, and their initial reaction was a pragmatic one — it was too expensive, too far, and just not feasible to go abroad. Still, I was motivated to give it a try, so I took the SAT on my own and did everything else needed to apply to American colleges. I received some acceptance letters and, by showing my initiative, convinced my parents that my dream might be within reach after all.

Dreams do come true, but then what?

I remember being extremely excited leading up to my move to the US, until about halfway through the plane journey when it hit me that I was leaving my family and everything I knew behind. I had to overcome that fear and the imposter syndrome that came from doubting whether I picked the right path. To add to that, there’s a lot of culture shock that accompanies moving continents. For the first time in my life I was an ethnic minority and had to grapple with what that meant.

For me, the culture shock extended into the classroom, too. I enrolled at the Florida Institute of Technology to study software engineering. It was my first time seeing students challenging teachers and engaging in open discussions. Putting myself in a new environment exposed me to an entirely new outlook.

The amount of effort I put into adjusting to my new home paid off more than I could have ever imagined. I used to be a kid in Kenya with a vague sense that coming to America was the right move for me. With a lot of hard work, I got a job at my dream company: Microsoft!

Microsoft had so much support and mentorship, along with growth and learning opportunities that kept me busy for 11 years. To an extent, I was still following the mindset I grew up with that values loyalty. The path was clear: I could have stayed at the company indefinitely, then gotten married, and soon after started a family. What I learned from all the friends I made over those years, however, is that Microsoft is just one of many companies doing amazing things. Once you get to a point in your career where you’re not growing the same way, you’re not learning the same way, or you just want a different challenge, it’s okay to change.

I had focused on infrastructure during my time at Microsoft, and as much as I enjoyed it, I wanted to keep exploring. I joined Uber to lead the team building the company’s customer support platform. This is where I discovered my niche for building platforms at the sweet spot between product and infrastructure. I love being able to shape systems that directly affect millions of people and translate into features that people can see and feel.

Why I picked Airbnb

After a bit over three years at Uber, I made the switch to Airbnb, which felt right for so many reasons. Airbnb’s mission around building belonging and connection really resonated with me. The company has an ambitious vision, and I believe promoting belonging and connection are fundamental to solving so many other societal problems. This, in addition to travel being a passion of mine (I’ve been to 55 countries across 6 continents!) made me very excited about Airbnb.

The way Airbnb works toward its mission is unique. We have a creative touch when it comes to technology, something our CEO, Brian, encourages a lot. We care deeply about the details and chasing perfection in a healthy way. Of course, there’s no such thing as actual perfection, but to strive for that the way we do at Airbnb produces great results, something our users see every time they interact with the product.

Arguably the biggest factor in my decision, though, was Airbnb’s culture. This can be hard to fully put into words; what I love about Airbnb’s culture comes through in all the little things you experience day to day. There’s a genuine warmness between people who seek to build belonging and connection everywhere they go. People are welcoming, particularly to new hires. Even the interview process at Airbnb feels more human and conversational, which is different from so many other companies. Culture starts with the details and adds up into bigger things too: I think Airbnb truly excels at work-life integration and as somebody who recently started a family, I’m very glad I came here.

Integrity, leadership, and service at Airbnb

At Airbnb, I lead the Marketing Technology (Growth Platform) team responsible for Canvas, an internal platform that enables marketing and product teams to effectively engage with customers. Our overarching goal is to drive business growth and product engagement. Canvas has tools for creating, managing, and measuring content that gets published both to Airbnb and offsite channels such as emails, notifications, and ads. I’ve reaffirmed how much I enjoy my role as a platform owner since I get to be at the nexus of so many areas that are important to the overall business. I get to think about everything from notifications, to personalization using machine learning, to the underlying infrastructure powering it all.

On a daily basis, I put into practice the three values I’ve lived by since my days in school: integrity, leadership, and service. My philosophy around leadership is that it’s not about power or being in charge. Rather, leadership is a form of service. I make a point to be empathetic, and my mission as a leader is to unlock the best in others through coaching and mentorship. My own mentors and coaches have played a large part in getting me to where I am, and I seek to pay that forward.

Of course, integrity also remains at the heart of every decision I make as a leader. Privacy and compliance are key focus areas for my team right now, which I enjoy because of the strong alignment those goals have with my value of integrity. To me, integrity means handling user data with the same care I’d want for my own data.

Currently, we’re also doing cutting-edge work on personalizing our marketing. Instead of blasting out the same email campaign to every user, we want to identify the journey a particular user is on and customize the content they see to be more relevant. Not only is this an interesting technical problem, it’s a nuanced issue of respecting user privacy while offering a more tailored experience.

In the last couple years, Airbnb has been undergoing an incredible transformation from a startup into a mature company. There’s a huge modernization effort within the company to scale our tech stack to match the scale at which we have a global impact. If that’s interesting to you, I encourage you to check out openings at Airbnb. If there’s one thing I’ve learned from the wild ride I’ve had so far is that there’s no set path you need to follow — take a chance, and you’ll be amazed by what you might achieve.

Here’s Veera’s story:

Learning and exploring

I grew up in Tamil Nadu, in the South of India. I was always a curious kid, trying to understand how everything worked, so when it came to choosing a course of study, engineering was a natural fit. I feel lucky that I had a lot of education opportunities in front of me, and I was able to choose the path I wanted to take.

My first exposure to computers came when I was in 8th grade. These were still the relatively early days of the computer, and my dad brought one home so he could learn to use it. I found it fascinating and learned BASIC and Logo. These are simple languages, but I was excited that I could use them to make drawings and text appear on the screen. Those rudimentary programs opened me to the world of what’s possible with computer science.

Studying and practical experience

I went to the College of Engineering, Guindy to study Computer Science and Engineering. My studies covered a lot of subjects, but the one that excited me most was networking. I was really curious to understand how data moves from one place to another.

My first practical networking experience came while still in school. There were a bunch of gamers I knew, and they wanted to set up a LAN to play Age of Empires and Quake together. I got together with a couple of my friends and built out an inexpensive networking solution for them, covering everything from cabling to routers to setting up configurations. I’m actually not that big of a gamer myself, but building out a network was really exciting for me. I love to understand things on a practical level, because while theoretical understanding is important, I always find it most meaningful to see how things actually work.

The power of engineering

After graduation, I got a job in networking. There were several interesting companies in the space at the time, and I ended up joining one run by a group of IIT (Indian Institute of Technology) professors. It was a great opportunity to learn from some of the brightest minds in the field. I always tried to make sure I was learning, so I sought out whatever would help me continue to grow. Eventually, I moved on to opportunities at larger internet companies where I could use my networking knowledge but also expand into topics like large-scale, distributed systems.

Being a software engineer has always been exciting to me because it gives you the power to solve so many problems. When my daughter was born, my wife and I were looking for names that had to fit multiple constraints–e.g. it had to start with a specific letter–and it was a struggle to come up with viable options. As an engineer, I realized there was an easier way to find all our choices. I wrote a quick program that downloaded a list of millions of names and then ran them through our criteria. From that, I was able to narrow it down to a list of a few thousand options. My wife was amazed that I could generate so many names with just a couple hours of work.

The challenges of engineering are also interesting. You have to work hard to keep yourself informed. The industry moves so fast. When I started, I was using Java 4, and now we’re on Java 18. The way you would solve a problem in either of these versions is so different. All these newer languages have also emerged, and you can apply each to different situations. It feels like every day new machine learning research pushes the boundaries in unimaginable ways. I don’t know what’s going to come next, but I know it’s going to evolve quickly.

Finding impact at Airbnb

After a while in my previous role, I began to feel like my learning curve was getting saturated, so I wanted to look for a new challenge somewhere I could have a larger impact. I heard Airbnb was opening a tech center in Bangalore, and I was excited by the opportunity to be one of the first engineers there.

I joined in September 2021 as the first engineer in the Hosting org in Bangalore. I focus on tools for compliance, which is a complex problem space. Every region has their own laws on short-term rentals that hosts have to follow, and the laws can vary at different levels — the US has their laws, and then California might customize some of them, and San Francisco might have their own on top of that, and so on. These laws can also change quickly, like they did during Covid, so our products need to be versatile and adapt to new conditions.

Airbnb has been a great place to work. It’s startup-y in that there are challenging technical problems to work on, but the job is stable and the company respects your work-life balance. As a technical leader, there’s a great opportunity to be part of the evolution of our technology roadmap. The architecture recently transitioned from a monorepo to a Service Oriented Architecture, so we’re still figuring out the best approaches for the problems we’re solving.

I’ve also appreciated Airbnb’s culture, especially the focus on inclusivity and belonging. My coworkers want to make everyone feel comfortable. When they introduced themselves to me, they all included their pronouns, making it easier for anyone else to share theirs. The people here live the culture and make everyone feel included.

Building our office in Bangalore

One of my favorite things about working as an engineer in the Bangalore office is the ownership and accountability. We’re not just a delivery center, where we’re being passed requirements from elsewhere and building that one piece of software. We like to call ourselves a capability center. Our team is tasked with the whole span of product development, from identifying what user problems exist all the way through to delivering a solution for them. We work on the same roadmap, codebase, and tech stack as Airbnb HQ.

Our team is growing quickly, both in Bangalore and remotely across India. With the team being spread out, trust and team-building have been important. We have a social meeting every Friday, and the whole team shows up so we can get to know one another. It’s great for connecting with teammates, and the trust we’re creating helps us build more successful products.

Airbnb leadership has a clear roadmap for the future of the Bangalore Tech Center, and the team is growing quickly. It’s been exciting to build our first tech center outside of headquarters. We’re hiring for a number of teams and we’d love to hear from you!

Check out our open roles here.

Though it can be hard and intimidating to find your place in the “big tech” industry in Silicon Valley, Florian has relied on curiosity and openness to establish a successful career at Airbnb. Read on for Florian’s own words on working at the intersection of business and software engineering, transferring to the US and scaling tech programs from zero to 10x for Airbnb, and more.

A global citizen finding a home for his career

Many years ago, Airbnb had a tagline that really inspired me: “Don’t Go There. Live There.” I’ve tried to embrace that idea as much as I could. I’ve had the chance to live and work in several different countries — Germany, the UK, Singapore, China, and now the US, where I’m currently based in San Francisco.

I grew up in Southern Germany, specifically a smaller city called Ulm (which is famous for Albert Einstein, and for having the tallest church building in the world). Growing up, my dad was a great inspiration for me. In his twenties, he started his own business. His journey taught me a lot about perseverance, dedication, and visionary thinking. I’m still inspired by how he pioneered the intersection of hardware and software engineering in the very early days of computers.

In my career, I’ve broadly moved from hardware to software engineering across different industries. (My first job was folding pizza boxes at a friend’s restaurant.) In Germany, there’s a large hardware and manufacturing presence: cars, industrial machines, and appliances in general. I started off at Airbus, building airplanes with advanced plastics and fiber materials, and then moved into the automotive industry and strategy consulting in the mobility space.

Along the way, I became really interested in software companies and internet technology. In Singapore, where I studied for my MBA, I joined a fast-growing tech startup to manage their partnerships across APAC. I loved it. I was flying around Southeast Asia to speak at startup events, pitch to investors, explain the idea and what problem we were solving — it was all very exciting to me.

I ended up attending a fireside chat with Mike Curtis, Airbnb’s VP of Engineering at that time, hosted by a local startup co-working space, where he shared lessons he picked up on his journey. I was inspired and ended up connecting with Mike and some folks from Airbnb after the talk, and the opportunity for me to join Airbnb grew organically from there.

Pioneering projects at the intersection of tech and business

TPMs are involved in every major release at Airbnb. Some TPMs are more product and business-focused (that’s me), and others are more infrastructure and platform-oriented. The potential for impact is high because you’re often very close to engineering leaders. You have conversations with the CTO, and many senior leaders in engineering — and on our product teams, as well.

There are two projects I have been involved in recently that demonstrate the breadth of our programs. One is the adoption of our Design Language System (DLS), a repository of prebuilt components that designers and engineers can reuse instead of building something new. So whenever an engineer at Airbnb has to implement a button, they don’t have to build this from scratch. Another is reducing our AWS costs. We introduced a new attribution model at Airbnb that directly associates AWS costs to different teams, and my role was monitoring our organization’s consumption and championing a lot of cultural change to think about AWS cost-efficiency whenever we build products.

Strategies for remote program management

Recently a big focus has been, naturally, keeping teams engaged and collaborative during remote work. Communicating a clear program vision and using frameworks to keep projects on track are two strategies that are more important than ever.

One other thing that I consider critical is celebrating wins and key milestones. Before, when we were in the office, it was much easier to celebrate big wins. At Airbnb, we have a tool where you can send appreciation to others. I use it all the time, because when I work with multiple stakeholders across design, product, engineering, and QA, people are generally contributing at different stages. When the project ends, you might not have this big forum anymore, because most people have moved on already. But it’s important to still give credit to the work that everyone contributed along the way and make sure their managers have visibility into their contributions.

What makes the TPM role unique at Airbnb

TPM is relatively new as a function itself, and every company approaches it slightly differently. Even at Airbnb, it’s continuously evolving and progressing. One thing that’s always the case, though, is that TPMs need to use influence to lead programs, teams, and products — without exercising direct authority. As a TPM, you have to champion ideas, be a great communicator, and work with your partners to align priorities and push projects forward.

You don’t have to study software engineering to be a technical program manager — I think you need to show that you have the ability to grow and learn on the technical side, and this can happen before the job or on the job. For instance, at first I didn’t have much context about AWS optimization, frontend design language systems, or open source, but people at Airbnb are super open to sharing knowledge which allowed me to quickly onboard and take on these programs. You just need to be curious and open to learn.

I love working at Airbnb because of the people and the mission-driven nature of the company. And I think what makes being a TPM at Airbnb extremely unique is that there’s still a lot of “greenfield” areas and unexplored territory, where you can really help define a new strategy and vision.