On-call and incidents

From the alert to the root cause.

When something breaks, TraceX lines up the errors, the logs and the deploys, finds the change that caused it, and writes the fix, while you're still reading the alert.

The problem

Sound familiar?

Today

Five tabs at 2 a.m.

Sentry, Datadog, CloudWatch, GitHub and the database are all open, and none of them talk to each other.

Today

Which deploy was it?

Matching the first error to the change that caused it is slow, manual work.

Today

The fix waits for morning

Even once the cause is clear, someone still has to write and test the change.

Errors

Every error traced to the change behind it.

TraceX reads the Sentry issue and its stack trace, finds the line in your code, and matches the first occurrence to the deploy that shipped it.

• Stack traces mapped to your code
• Matched to deploys and commits
• Read-only, from the first minute

checkout-web • issues
Error TypeError: Cannot read properties of undefined (reading 'amount')
chargeCard • first seen Oct 22, 14:02
node_modules/stripe/lib/charges.js create
src/checkout/payment.ts:88 chargeCard
src/checkout/submit.ts:41 onSubmit
It started with deploy a1c9e2: on a retry the payment intent is undefined.

Logs

The lines that explain it, picked out.

TraceX searches Datadog and CloudWatch around the moment it started and highlights the log lines that show what went wrong.

• Datadog and CloudWatch Logs Insights
• Queries written for you
• The relevant lines highlighted

Logs • service:checkout
14:01:52 INFO POST /charge 200 order=48188
14:02:03 WARN card_error retry=1 order=48190
14:02:04 INFO POST /charge 200 order=48190
14:02:04 INFO POST /charge 200 order=48190
14:02:31 WARN card_error retry=1 order=48197
14:02:32 INFO POST /charge 200 order=48197
Each card error retried the charge without an idempotency key, so 6 orders were charged twice.

The fix

A pull request, ready to review.

TraceX writes the smallest change that fixes the cause, with its risk and how to validate it. You review it and merge it.

• You approve every change
• Validation steps included
• Draft pull requests if you prefer

app/workers/refunds.py +1 −1
def pending_refunds(db): − return db.refunds.filter(state="pending")

• return db.refunds.filter(state="awaiting_payout")

Fix ready Low risk Open pull request Opened
Validate run the refund worker against staging; 34 refunds move to payout

Timeline

The incident, in order.

TraceX lines up the deploy, the first error, the alert and the fix on one timeline, ready for the postmortem.

• Deploys, errors and alerts in one place
• Times taken from your own systems
• Ready to paste into the postmortem

14:00 Deploy a1c9e2
Retry card errors in checkout
14:02 First error
TypeError in chargeCard, 128 events
14:04 Alert fires
Checkout error rate above threshold
14:07 Root cause found
Retries run without an idempotency key
14:11 Fix opened
Pull request #322, ready for review

How it fits

Set up once. Useful every day.

01

Connect your signals

Sentry, Datadog or CloudWatch, GitHub, and a read-only database.

02

Ask when it breaks

Ask in the dashboard, in Slack or with tracex -c from your terminal.

03

Review and ship

Check the root cause and its evidence, then review and merge the fix.

What you get

Built for how you work.

Signals in one place

Errors, logs, deploys and data, lined up on one timeline instead of five tools.

The cause, not the symptom

TraceX follows the evidence back to the change that started it.

Shorter incidents

The investigation runs in parallel with you, from the moment you ask.

Ask follow-ups

Who was affected? Since when? Ask in plain English and get cited answers.

From the terminal too

Run tracex -c and ask without leaving your shell.

No risk to production

Read-only database access, and write queries refused before they run.

FAQ

Questions

Does TraceX replace our alerting?

No. Your alerts still tell you something is wrong; TraceX helps you find out why, and fix it.

Can it roll back a deploy?

No. TraceX never changes production. It tells you which change caused the problem and writes a fix for review.

Does it work from the terminal during an incident?

Yes. tracex -c gives the same answers, with the same evidence, without leaving your shell.

Find the root cause. Ship the fix.

TraceX is free during the private beta. Connect a database and a repository and ask your first question in minutes.