How Coinbase Uses PuppyGraph to Investigate Service Failures

5 min read

Summary

Coinbase needed to understand how service failures could affect its customer products. By June 2025, the team had combined distributed tracing data with service ownership information to build a service dependency graph. PuppyGraph provided graph queries over the data in Databricks, with access managed through Unity Catalog, without loading the data into a separate graph database. Coinbase's root cause analysis bot, RCA Bot, used dependency queries to expand incident investigations and assemble monitoring signals into candidate root cause recommendations. Engineers could also query service callers directly, helping them understand potential impact and identify teams to contact.

The Challenge: Connecting Service Failures to Customer Impact

A low-severity incident in Coinbase's product eligibility service initially looked like an intermittent problem. When the customer experience incident response team asked whether users were affected, the answer was still unknown. Other incidents soon followed: customers could not access staking or borrowing, and Coinbase One was also affected. Teams were investigating symptoms across products without immediately recognizing that they were connected.

It took a little over an hour and a half after the initial incident was opened to understand and report the customer impact. Teams identified a recent configuration change and fixed the issue, but the delay in understanding its reach exposed a gap: Coinbase needed a quicker way to connect service failures with their causes and effects.

Coinbase already had detailed traces in Datadog. Each trace recorded the steps in a request, including the services involved and their calls to one another. Those records contained the relationships needed to build a service graph, but following individual traces could become overwhelming. The team needed to connect those calls to the services and projects engineers were responsible for.

The Data Foundation: Reusable Tracing Data

Coinbase's trace collection supported multiple observability analyses, including service dependencies and critical user journeys. Its first collection approach used an Airflow job to retrieve spans from Datadog twice a day and store them in Snowflake. It produced a heavily sampled dataset, with too few examples of some of the metadata the team needed. Collection at specific times also missed activity outside those windows. The spans API had not been designed for the bulk export Coinbase was attempting.

For its common Kubernetes web-service path, Coinbase changed how it collected the data. Agents continued sending traces to Datadog while also shipping them to a facade API built by the data platform team, called Datadog Shadow. That second path carried the data through Kafka into Databricks. Shipping from the agents avoided additional sampling stages on this collection path, while retaining client-side trace sampling.

In Databricks, Coinbase organized the trace data by hour or day, subdivided it by trace ID prefix, and used Z-ordering on trace and span IDs. The layout was designed to help queries narrow the data they needed to read. This collection and storage path made trace data available for multiple analyses while the existing Datadog monitoring path continued to operate. The service graph was one use of that shared data.

Access to the Databricks data used Unity Catalog's credential vending and role-based access control. PuppyGraph requested temporary credentials from Unity Catalog for the tables it needed to query. Those credentials were limited in scope and duration, allowing PuppyGraph to read the underlying data files with access managed through the catalog.

The Solution: A Service Graph over Databricks with PuppyGraph

Building the service dependency graph

To turn the records into service relationships, Coinbase combined trace data with its ownership registry. The registry described projects and services, including their owners, teams, and on-call information. Trace IDs, span IDs, and parent IDs provided the tree structure of each request. Joining the records with ownership information allowed the team to extract distinct pairs of one service calling another, together with their associated projects.

PuppyGraph provided graph queries over the Databricks data without requiring that data to be ETLed into a separate graph database. Engineers and RCA Bot could traverse those relationships to retrieve service dependencies.

"With PuppyGraph, we can have deeper controls of augmenting metrics such as failure, throughput, number API calls within latency SLA, data lineage ... to the service map topology."

Eric Sun, Coinbase

Using the graph to guide incident investigations

For incident response, Coinbase rebuilt the service graph daily, with one current version available to RCA Bot. The bot used that dependency map to identify projects to investigate, then gathered additional monitoring signals.

An incident in Slack starts RCA Bot's project selection, with alert mapping or team ownership as a fallback. The bot queries PuppyGraph for dependencies, combines signals from Datadog and Snowflake, and returns candidate projects, confidence scores and evidence links to engineers in Slack. — PuppyGraph supplies the dependency context. Coinbase's RCA Bot combines it with monitoring signals to recommend projects for engineers to investigate.

Coinbase engineers opened incidents through Incident Bot in Slack. When RCA Bot detected a new incident, its first task was to find a starting project. It checked whether the person who opened the incident had a recent PagerDuty page linked to a Datadog monitor, and whether that monitor was tagged with a project in the ownership registry.

When that chain was incomplete, the bot used ownership as its fallback. Projects owned by the incident creator's team became the starting set. This made the registry useful even when an alert did not identify the project directly.

RCA Bot then queried PuppyGraph for the starting projects' dependencies, up to a selected depth. That expanded the investigation beyond the initial projects to the services they relied on. For the resulting set, the bot fetched information from Datadog and other data held in Snowflake: recent configuration changes and deployments, changes in service-level objective (SLO) performance, and changes in error rates.

The bot combined these signals to recommend projects that might be related to the root cause. Its Slack response included confidence scores and links to supporting information, including SLOs, error rates, and recent events. Engineers could follow those links to investigate a candidate or rule it out. The graph supplied the dependency context; the bot assembled the signals into a starting point for debugging.

The Results: A Starting Point for Investigation and a Clearer View of Impact

Engineers had a starting point for investigating possible causes beyond the initial project or team. RCA Bot's candidate projects and evidence links in Slack helped them decide what to investigate next.

Dependency queries also helped engineers understand potential impact. Starting with the eligibility service, for example, they could use PuppyGraph to identify services that called it and decide which teams to contact.

Further resources

Start free. Pay when it goes to production.

Free download Book demo

All case studies

* No payment required to start either one.

$ docker run -p 8081:8081 \
  -p 8182:8182 -p 7687:7687 \
  puppygraph/puppygraph:latestPulling puppygraph/puppygraph:latestStarting PuppyGraphWeb console listening on 0.0.0.0:8081Gremlin endpoint listening on 0.0.0.0:8182PuppyGraph is running# no pipelines · no second database · no data movement