Distributed tracing - .NET

An introduction to .NET distributed tracing.

https://learn.microsoft.com/en-us/dotnet/core/diagnostics/distributed-tracing

An activity is a trace that represents a single operation happening within a service.

Typically you use it to measure how fast parts were what happened.

Basic Setup

using System.Diagnostics;

namespace MyService.Tracing;

public static class MyActivitySource
{
    public const string Name = "MyService";

    public static readonly ActivitySource Instance = new(Name);
}

Within the worker

using var activity = MyActivitySource.Instance.StartActivity(
    "rabbit.process_message",
    ActivityKind.Consumer);

Set Tags

activity?.SetTag("messaging.system", "rabbitmq");
activity?.SetTag("messaging.destination.name", queueName);
activity?.SetTag("messaging.rabbitmq.routing_key", routingKey);
activity?.SetTag("app.transaction_id", transactionId);

Set Success

activity?.SetTag("app.message_outcome", "completed");
activity?.SetStatus(ActivityStatusCode.Ok);

Set Failure

activity?.SetTag("app.message_outcome", "dead_lettered");
activity?.SetTag("app.message_failure_reason", "retry_exhausted");
activity?.SetStatus(ActivityStatusCode.Error, "retry_exhausted");

Activity Current

Just allows you to grab activity from the current async context

Activity.Current?.SetTag("app.message_outcome", "sent_to_retry");
Activity.Current?.SetStatus(ActivityStatusCode.Error, "service_error");

Nesting Activities

You can nest activities, and the child activities will be linked to their parent.

Measure Speed

Note that the timer only gets set after the activity was disposed.

using (var activity = TransactionTaggingActivities.Start(ActivityName))
{
    val = await SomeMethod();

    activity?.SetTag(TagName, val.someMeaningfullValue);
}

Testing

For local testing you will need to add an activity listener. Without this the activity will be null and not populate.

ActivitySource.AddActivityListener(new ActivityListener
{
    ShouldListenTo = source =>
        source.Name == ActivityNames.SourceName,

    Sample = (ref ActivityCreationOptions<ActivityContext> _) =>
        ActivitySamplingResult.AllDataAndRecorded,

    SampleUsingParentId = (ref ActivityCreationOptions<string> _) =>
        ActivitySamplingResult.AllDataAndRecorded,

    ActivityStarted = activity =>
    {
        Console.WriteLine($"START {activity.OperationName}");
    },

    ActivityStopped = activity =>
    {
        Console.WriteLine($"STOP {activity.OperationName}");
    }
});

Start your service, then in a separate terminal run.

dotnet-trace ps

dotnet-trace collect --process-id <PID> --providers "Microsoft-Diagnostics-DiagnosticSource:0xFFFFFFFFFFFFFFFF:5"

Then open PerfView, and open the generated .nettrace file.

What tags should I use?

Messaging

General Messaging Attributes Azure Event Hubs Attributes GCP Pub/Sub Attributes Kafka Attributes RabbitMQ Attributes RocketMQ Attributes Azure Service Bus Attributes Deprecated Messaging Attributes General Messaging Attributes Attributes describing telemetry around messaging systems and messaging activities. Attributes: Key Stability Value Type Description Example Values messaging.batch.message_count int The number of messages sent, received, or processed in the scope of the batching operation. [1] 0; 1; 2 messaging.client.id string A unique identifier for the client that consumes or produces a message. client-5; myhost@8742@s8083jm messaging.consumer.group.name string The name of the consumer group with which a consumer is associated. [2] my-group; indexer messaging.destination.anonymous boolean A boolean that is true if the message destination is anonymous (could be unnamed or have auto-generated name). messaging.destination.name string The message destination name [3] MyQueue; MyTopic messaging.destination.partition.id string The identifier of the partition messages are sent to or received from, unique within the messaging.destination.name. 1 messaging.destination.subscription.name string The name of the destination subscription from which a message is consumed. [4] subscription-a messaging.destination.template string Low cardinality representation of the messaging destination name [5] /customers/{customerId} messaging.destination.temporary boolean A boolean that is true if the message destination is temporary and might not exist anymore after messages are processed. messaging.message.body.size int The size of the message body in bytes. [6] 1439 messaging.message.conversation_id string The conversation ID identifying the conversation to which the message belongs, represented as a string. Sometimes called “Correlation ID”. MyConversationId messaging.message.envelope.size int The size of the message body and metadata in bytes. [7] 2738 messaging.message.id string A value used by the messaging system as an identifier for the message, represented as a string. 452a7c7c7c7048c2f887f61572b18fc2 messaging.operation.name string The system-specific name of the messaging operation. ack; nack; send messaging.operation.type string A string identifying the type of the messaging operation. [8] create; send; receive messaging.system string The messaging system as identified by the client instrumentation. [9] activemq; aws.sns; aws_sqs [1] messaging.batch.message_count: Instrumentations SHOULD NOT set messaging.batch.message_count on spans that operate with a single message. When a messaging client library supports both batch and single-message API for the same operation, instrumentations SHOULD use messaging.batch.message_count for batching APIs and SHOULD NOT use it for single-message APIs.

https://opentelemetry.io/docs/specs/semconv/registry/attributes/messaging/

If using OTel I recommend using as much as possible from the list above, it is great because it points you for specific things like GCP and Rabbit.

Then anything that is specific to your service.

Example

using var activity = MyActivitySource.Instance.StartActivity(
    "rabbit.process_message",
    ActivityKind.Consumer);

activity?.SetTag("messaging.system", "rabbitmq");
activity?.SetTag("messaging.destination.name", queueName);
activity?.SetTag("messaging.rabbitmq.routing_key", routingKey);

try
{
    await ProcessMessage();

    activity?.SetTag("app.message_outcome", "completed");
    activity?.SetStatus(ActivityStatusCode.Ok);
}
catch (Exception ex)
{
    activity?.SetTag("app.message_outcome", "failed");
    activity?.SetTag("exception.type", ex.GetType().FullName);
    activity?.SetTag("exception.message", ex.Message);
    activity?.SetStatus(ActivityStatusCode.Error, "unexpected_exception");

    throw;
}