Pusher failed to consume trace data err="DoBatch: InstancesCount <= 0

[chazapp]

I'm trying to implement OpenTelemetry tracing in my Django DRF Application, using OTEL Django Auto Instrumentation. I ultimately want to push my traces into Tempo to visualize them into Grafana later on. I'm using OTLPSpanExporter to push my traces via gRPC on port 4317. I'm doing this locally with Tempo running in a Docker container. When the traces are pushed, I get this error Pusher failed to consume trace data err="DoBatch: InstancesCount <= 0 in Tempo logs. What am I doing wrong ? Here's the full context:

Incidents-API Settings.py

# settings.py
if os.getenv("TRACING_BACKEND_HOST"):
    DjangoInstrumentor().instrument()
    trace.set_tracer_provider(
        TracerProvider(
                resource=Resource.create({SERVICE_NAME: "incidents-api"})
        )
    )
    tracer = trace.get_tracer(__name__)
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("TRACING_BACKEND_HOST"), # localhost:4317
    )

    # Create a BatchSpanProcessor and add the exporter to it
    span_processor = BatchSpanProcessor(otlp_exporter)

    # add to the tracer
    trace.get_tracer_provider().add_span_processor(span_processor)

# docker-compose.yml
version: "3"
services:
  tempo:
    image: grafana/tempo:latest
    command: "-target=distributor -config.file=/etc/tempo.yml"
    volumes:
      - ./tempo.yml:/etc/tempo.yml
    ports:
      - "3200:3200/tcp"
      - "4317:4317/tcp"

# tempo.yml
compactor: null
server:
  http_listen_port: 3200
distributor:
    receivers:
      otlp:
        protocols:
          grpc:
ingester:
    trace_idle_period: 1s
    lifecycler:
        ring:
            replication_factor: 3
storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

$ python manage.py runserver --noreload
Performing system checks...

System check identified no issues (0 silenced).
September 02, 2022 - 15:45:33
Django version 4.0.6, using settings 'incidents.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[02/Sep/2022 15:45:47] "GET /health HTTP/1.1" 301 0
[02/Sep/2022 15:45:47] "GET /health/ HTTP/1.1" 200 2
Failed to export traces, error code: StatusCode.UNKNOWN

level=info ts=2022-09-02T13:49:20.242055169Z caller=main.go:200 msg="initialising OpenTracing tracer"      
level=warn ts=2022-09-02T13:49:20.242850817Z caller=main.go:98 --CONFIGURATIONWARNINGS--=(MISSING)
level=warn ts=2022-09-02T13:49:20.242880617Z caller=main.go:104 msg="compactor.compaction.compacted_block_timeout < storage.trace.blocklist_poll" explain="Queriers and Compactors may attempt to read a block that no longer exists"
level=warn ts=2022-09-02T13:49:20.242892887Z caller=main.go:104 msg="c.Compactor.Compactor.RetentionConcurrency must be greater than zero. Using default." explain="default=10"
level=warn ts=2022-09-02T13:49:20.242900037Z caller=main.go:104 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2022-09-02T13:49:20.242944507Z caller=main.go:115 msg="Starting Tempo" version="(version=, branch=main, revision=525b0265)"
level=info ts=2022-09-02T13:49:20.243261566Z caller=server.go:306 http=[::]:3200 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-09-02T13:49:20.244034503Z caller=memberlist_client.go:436 msg="Using memberlist cluster label and node name" cluster_label= node=2c1ae38e1f4f-ae516793
ts=2022-09-02T13:49:20Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=info ts=2022-09-02T13:49:20.24541415Z caller=module_service.go:82 msg=initialising module=server     
level=info ts=2022-09-02T13:49:20.24544886Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-09-02T13:49:20.24545973Z caller=module_service.go:82 msg=initialising module=overrides  
level=info ts=2022-09-02T13:49:20.24547574Z caller=module_service.go:82 msg=initialising module=ring       
level=info ts=2022-09-02T13:49:20.24548481Z caller=module_service.go:82 msg=initialising module=usage-report
level=info ts=2022-09-02T13:49:20.245730601Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"   
level=info ts=2022-09-02T13:49:20.24579096Z caller=module_service.go:82 msg=initialising module=distributor
ts=2022-09-02T13:49:20Z level=info msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=tempo     
level=info ts=2022-09-02T13:49:20.245946935Z caller=app.go:195 msg="Tempo started"
level=error ts=2022-09-02T13:50:26.634686845Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="DoBatch: InstancesCount <= 0"

For reference, I've tried swapping Tempo with OpenTelemetry collector and got the exact same error, but I still fail to find what is missing to my code to finally get traces ingested.
Thank you for your help

[joe-elliott]

It looks like you're trying to run Tempo in "single binary" mode, but also setting RF=3 which won't work.

Try reducing replication_factor to 1.

[mapno]

It's here now https://github.com/grafana/tempo/blob/main/example/docker-compose/shared/tempo.yaml

[MrMegaMango]

Hi @joe-elliott
I'm getting the same error looks like Tempo distributor can't push log into ingestor:
level=error ts=2024-09-20T10:27:51.33477459Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="DoBatch: InstancesCount <= 0"

my setup
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: tempo
namespace: argocd
spec:
project: default
source:
repoURL: https://grafana.github.io/helm-charts
chart: tempo-distributed
targetRevision: 1.18.0
helm:
releaseName: tempo-distributed
values: |
service:
type: ClusterIP
config: |
storage:
trace:
backend: local
local:
path: /var/tempo/traces
querier:
frontend_worker:
frontend_address: tempo-distributed-query-frontend.monitoring.svc.cluster.local:9095
server:
http_listen_port: 3100
distributor:
receivers:
otlp:
protocols:
grpc:
http:
traces:
otlp:
grpc:
enabled: true
http:
enabled: true
ingester:
replicas: 1
config:
replication_factor: 1

  # optional: use storage config to use other blob storage like s3

destination:
server: 'https://kubernetes.default.svc'
namespace: monitoring
syncPolicy:
automated:
prune: true
selfHeal: true

[MrMegaMango]

➜ ~ kubectl get pods -n monitoring
tempo-distributed-compactor-6d88b49f4d-n5qvm 1/1 Running 0 131m
tempo-distributed-distributor-65b6ddcbbc-tcd4f 1/1 Running 0 131m
tempo-distributed-ingester-0 1/1 Running 0 22m
tempo-distributed-memcached-0 1/1 Running 0 6d20h
tempo-distributed-querier-68f77ccd76-22dvn 1/1 Running 0 131m
tempo-distributed-query-frontend-745fc85494-6qhnp 1/1 Running 0 131m
➜ ~ kubectl get deployments -n monitoring
NAME READY UP-TO-DATE AVAILABLE AGE
grafana 1/1 1 1 27d
prometheus-kube-state-metrics 1/1 1 1 28d
prometheus-prometheus-pushgateway 1/1 1 1 28d
prometheus-server 1/1 1 1 22d
tempo-distributed-compactor 1/1 1 1 6d20h
tempo-distributed-distributor 1/1 1 1 6d20h
tempo-distributed-querier 1/1 1 1 6d20h
tempo-distributed-query-frontend 1/1 1 1 6d20h
➜ ~ kubectl get statefulsets -n monitoring
NAME READY AGE
prometheus-alertmanager 1/1 28d
tempo-distributed-ingester 1/1 6d20h
tempo-distributed-memcached 1/1 6d20h

➜ ~ kubectl logs tempo-distributed-ingester-0 -n monitoring -f
level=warn ts=2024-09-20T10:19:14.201408415Z caller=main.go:131 msg="-- CONFIGURATION WARNINGS --"
level=warn ts=2024-09-20T10:19:14.201557317Z caller=main.go:137 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2024-09-20T10:19:14.201691083Z caller=main.go:226 msg="initialising OpenTracing tracer"
level=info ts=2024-09-20T10:19:14.210035738Z caller=main.go:119 msg="Starting Tempo" version="(version=2.6.0, branch=HEAD, revision=e85bbc57d)"
level=info ts=2024-09-20T10:19:14.215153596Z caller=cache.go:55 msg="caches available to storage backend" footer=false bloom=false offset_idx=false column_idx=false trace_id_idx=false page=false
level=info ts=2024-09-20T10:19:14.217668706Z caller=server.go:249 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=info ts=2024-09-20T10:19:14.220923723Z caller=memberlist_client.go:439 msg="Using memberlist cluster label and node name" cluster_label= node=tempo-distributed-ingester-0-733121bd
level=info ts=2024-09-20T10:19:14.222809751Z caller=module_service.go:82 msg=starting module=internal-server
level=info ts=2024-09-20T10:19:14.223039502Z caller=module_service.go:82 msg=starting module=cache-provider
level=info ts=2024-09-20T10:19:14.223237049Z caller=module_service.go:82 msg=starting module=store
level=info ts=2024-09-20T10:19:14.223463581Z caller=module_service.go:82 msg=starting module=server
level=info ts=2024-09-20T10:19:14.223839836Z caller=module_service.go:82 msg=starting module=overrides
level=info ts=2024-09-20T10:19:14.224002525Z caller=module_service.go:82 msg=starting module=memberlist-kv
level=info ts=2024-09-20T10:19:14.224162823Z caller=module_service.go:82 msg=starting module=usage-report
level=info ts=2024-09-20T10:19:14.224415749Z caller=module_service.go:82 msg=starting module=ingester
level=info ts=2024-09-20T10:19:14.224526233Z caller=ingester.go:364 msg="beginning wal replay"
level=warn ts=2024-09-20T10:19:14.224828576Z caller=wal.go:94 msg="unowned file entry ignored during wal replay" file=blocks err=null
level=info ts=2024-09-20T10:19:14.224938521Z caller=ingester.go:402 msg="wal replay complete"
level=info ts=2024-09-20T10:19:14.22514529Z caller=ingester.go:416 msg="reloading local blocks" tenants=0
level=info ts=2024-09-20T10:19:14.225304216Z caller=lifecycler.go:624 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2024-09-20T10:19:14.225452563Z caller=app.go:208 msg="Tempo started"
level=info ts=2024-09-20T10:19:14.226459259Z caller=lifecycler.go:649 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2024-09-20T10:19:14.226874116Z caller=lifecycler.go:493 msg="auto-joining cluster after timeout" ring=ingester
^C
➜ ~ kubectl logs tempo-distributed-distributor-65b6ddcbbc-tcd4f -n monitoring
level=warn ts=2024-09-20T08:30:04.661324744Z caller=main.go:131 msg="-- CONFIGURATION WARNINGS --"
level=warn ts=2024-09-20T08:30:04.661385345Z caller=main.go:137 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2024-09-20T08:30:04.661402395Z caller=main.go:226 msg="initialising OpenTracing tracer"
level=info ts=2024-09-20T08:30:04.750142643Z caller=main.go:119 msg="Starting Tempo" version="(version=2.6.0, branch=HEAD, revision=e85bbc57d)"
level=info ts=2024-09-20T08:30:04.762858433Z caller=server.go:249 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=info ts=2024-09-20T08:30:04.766822967Z caller=memberlist_client.go:439 msg="Using memberlist cluster label and node name" cluster_label= node=tempo-distributed-distributor-65b6ddcbbc-tcd4f-069451a0
level=info ts=2024-09-20T08:30:04.77447654Z caller=module_service.go:82 msg=starting module=internal-server
level=info ts=2024-09-20T08:30:04.774575747Z caller=module_service.go:82 msg=starting module=server
level=info ts=2024-09-20T08:30:04.775104608Z caller=module_service.go:82 msg=starting module=overrides
level=info ts=2024-09-20T08:30:04.775158529Z caller=module_service.go:82 msg=starting module=memberlist-kv
level=info ts=2024-09-20T08:30:04.775192203Z caller=module_service.go:82 msg=starting module=usage-report
level=info ts=2024-09-20T08:30:04.775321704Z caller=module_service.go:82 msg=starting module=ring
level=info ts=2024-09-20T08:30:04.775463161Z caller=ring.go:297 msg="ring doesn't exist in KV store yet"
level=info ts=2024-09-20T08:30:04.775498627Z caller=module_service.go:82 msg=starting module=metrics-generator-ring
level=info ts=2024-09-20T08:30:04.775817635Z caller=ring.go:297 msg="ring doesn't exist in KV store yet"
level=info ts=2024-09-20T08:30:04.776133808Z caller=module_service.go:82 msg=starting module=distributor
level=info ts=2024-09-20T08:30:04.776832451Z caller=app.go:208 msg="Tempo started"
ts=2024-09-20T08:30:04Z level=info msg="OTel Shim Logger Initialized" component=tempo
ts=2024-09-20T08:30:04Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning." component=tempo documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks feature gate ID=component.UseLocalHostAsDefaultHost
ts=2024-09-20T08:30:04Z level=info msg="Starting GRPC server" component=tempo endpoint=0.0.0.0:4317
ts=2024-09-20T08:30:04Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning." component=tempo documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks feature gate ID=component.UseLocalHostAsDefaultHost
ts=2024-09-20T08:30:04Z level=info msg="Starting HTTP server" component=tempo endpoint=0.0.0.0:4318
level=error ts=2024-09-20T08:48:43.771195394Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="DoBatch: InstancesCount <= 0"
level=error ts=2024-09-20T08:48:44.806389801Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="DoBatch: InstancesCount <= 0"
level=error ts=2024-09-20T08:48:46.84448721Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="DoBatch: InstancesCount <= 0"

[joe-elliott]

Your distributor is not seeing any ingesters to push to. Check the ingester ring endpoint on the distributors. Every ingester should be listed with its IP.

If they are not listed it means the ring is not propagating correctly.

Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend.

I would also pay attention to this warning. If you are running in distributed mode, you should be using object storage.

[MrMegaMango]

Your distributor is not seeing any ingesters to push to. Check the ingester ring endpoint on the distributors. Every ingester should be listed with its IP.

If they are not listed it means the ring is not propagating correctly.

Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend.

I would also pay attention to this warning. If you are running in distributed mode, you should be using object storage.

Thanks man turns out I need to manually add

          memberlist:
            join_members:
              - dns+tempo-distributed-gossip-ring:7946