Sentry

tags : Infrastructure, Observability

Concerns with sentry §

• The way we’re using sentry is per project which is fine for error tracking
• But when distributed tracing is in play, it crosses project-boundaries. Which seems to be a business plan feature. So you can’t really do distributed tracing across your different projects in sentry unless you’ve a business plan?
• The sentry performance UI expects each project to have a Frontend, Backend and Mobile filter. Which doesn’t fit nicely into how we’ve structured our services.
• Discover page has limitations with the team plan

Some terms specific to sentry §

Some of these terms are specific to sentry, some are general distributed tracing terms. See this for sentry specific definition of these terms. Following is summary.

• Trace
  • identifier: trace_id
  • Contains Transaction
• Transaction
  • identifier: transaction_name (Eg. api/v2/users/<user_id>/)
  • Contains spans
• Spans
  • Contained in a Transaction

Cheetsheet §

See https://docs.sentry.io/product/discover-queries/query-builder/

Writing Query (Discover page) §

• OR cannot be applied across aggregated and non-aggregated filters
• If we want to match multiple values in a query we can use []
• You can click on tags (facet box/map) in the side to add them to the query
• With transaction based on your search criteria and sample rate, result events may be limited.
• Concepts
  • Graph options
    • Interval: How you group things within the date/time range selected.
    • Display
      • Top Period & Top Daily need an aggregation function like p95 to be applied. When we use this, we get another option to set limit
      • Total Period, ~

Gotchas §

• Avoid using multiple project view that sentry allows us to do from the UI. This gets confusing because we use different
• With some projects we have fields such as user_org and in other sentry project that field might be missing. So Discover Page queries don’t span across orgs.
• Because of how we’ve been instrumenting things historically, we’ll need to have to have dashboards of the name <org_name>:<service_name>:<env_name>
• At the service level, we have distinction of the same services with <service_name>-poc and <service_name>-production.
  • While we can differentiate them in sentry, we choose not to do so. We’ll always split at the highest tenant when organizing queries in sentry, which is the customer in this case.

Project specific issues §

• Different healthchecks endpoint naming convention (Prefer to use /healthz)
• Different ports, some allow config some don’t
• The key for project env is different for diff projects, some have MANHATTAN_ENV, some have PROJECT_ENV etc.
• In some projects it’s DATABASE_NAME, in some projects its POSTGRES_DB_NAME etc.