Tracing at Slack: Thinking in Causal Graphs
&gt; Zipkin and Jaegar are two of the most popular open source tracing projects that follow the above model<p>Jaeger. This error happens multiple times in the article.<p>&gt; While the current APIs work very well for their intended use cases, using those APIs in contexts where there is no clear start or end for an operation can be confusing or not possible.<p>I don’t understand this part. At Klarrio, we are using Jaeger to trace a multi-tenant microservice system. We ingest data over mqtt and forward to kafka. Contexts are created on ingest and forwarded via kafka to further services. Each service adds its own spans to the trace. A message can be sent from one tenant to another, we don’t know when the trace is completed as the tenant can add spans at any time in the future. There isn’t clear end to the process. Traces can span over days. We use it to report latencies in a real time environment to a governmental body.<p>&gt; whose event loops call into application code (inversion of control), we need complex workarounds in existing APIs that often break the abstractions provided by these libraries<p>I’d like to know more how a custom solution solves this problem. At the end of the day, the trace id and parent span id still has to be somehow forwarded to the next service.<p>* edit: there is an example of a curl query towards the end of the article. Jaeger has a built in reporter and it’s already possible to send a context over http to another service. It’s just data?<p>So it also has to somehow plug into the mentioned frameworks&#x2F;IoC? What’s different?<p>In worst case, this is a perfect example of where a contribution for opentracing support would be probably appreciated.<p>&gt; Further, we also found that in those use cases there may be multiple request flows (producer-consumer patterns via kafka, goroutines, streaming systems) happening at the same time or we may want to track a single event across multiple request flows. In either case, having a single tracer for the execution of the entire application can be a limiting factor.<p>It’s possible with Jaeger. We do this at Klarrio. Span context can be sent over the wire. Furthermore, one is not limited to a single tracer per application.<p>&gt; Raw spans are internal implementation details in tracing systems.<p>Yes, implementation detail.<p>&gt; These internal implementation details are hidden from users, who interact with these spans via an instrumentation API at creation time and via the trace UI at consumption time.<p>Consumption, how? In the UI? Yes, they are visible. Through an API? They’re in tags and logs fields. They’re definitely not hidden.<p>&gt; For example, I’d like to be able to ask a question like “Am I missing my SLA because of slow DB queries on a specific http endpoint?” Even though traces contain this information, users often have to write complex Java programs, a multi-day effort, to answer their questions. Without an easy way to ask powerful analytical questions, users have to settle for the insights that can be gleaned from the trace view. This drastically limits the ability to utilize trace data for triage.<p>Jaeger has pluggable storage back ends. One of them is kafka. With kafka, it’s one step to such a solution. Either store them in a database allowing for such query (presto, druid... hello?) or write a kafka app managing alerts. Why reinvent the wheel completely?<p>I’m not convinced but if it works for Slack, great. So... can I try it?