Skip to content

OpenTelemetry Collector: Write log data with OTLP API

Last updated on

STACKIT Logs supports ingesting log records through the OpenTelemetry (OTLP) protocol over HTTP. You can configure the OpenTelemetry Collector’s OTLP/HTTP exporter (otlp_http) to forward logs directly to your STACKIT Logs instance.

To authenticate requests, STACKIT Logs supports two authentication methods:

  • STACKIT Service Account (recommended): Uses the oauth2client extension to exchange service account credentials for short-lived JWT access tokens automatically.
  • Instance Access Token: Uses the bearertokenauth extension with a static bearer token created in your Logs instance.

Section titled “Authenticate with a STACKIT Service Account (Recommended)”

Using a STACKIT Service Account is the recommended approach for automated and production workloads. The oauth2client extension automatically obtains short-lived JWTs from the STACKIT IAM endpoint, removing the need to manage and rotate long-lived access tokens.

/etc/otelcol-contrib/config.yaml
# 1. Define OAuth2 client extension
extensions:
oauth2client:
client_id: "<sa-credentials-sub>"
client_certificate_key_id: "<sa-credentials-kid>"
client_certificate_key_file: "<path-to-sa-private-key.pem>"
grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer"
token_url: "https://accounts.stackit.cloud/oauth/v2/token"
signature_algorithm: "RS512"
iss: "<service-account-id>" # Service account ID or email
audience: "https://accounts.stackit.cloud"
expiry_buffer: "2m"
# 2. Define OTLP/HTTP exporter
exporters:
otlp_http:
# NOTE: The collector internally appends '/v1/logs' to this endpoint
endpoint: <data-source-url-of-logs-instance>/otlp
auth:
authenticator: oauth2client
sending_queue:
enabled: true
sizer: bytes # How the queue and batching is measured
num_consumers: 4 # Number of consumers that dequeue batches
queue_size: 41943040 # Buffer size e.g. 40 MB buffer
batch:
sizer: bytes
flush_timeout: 1s # Keeps log tailing latency ≤ 1s
min_size: 524288 # the minimum size of a batch e.g. 512 KB - Enough payload for high gzip compression (~5-7x)
max_size: 1572864 # the maximum size of a batch, enables batch splitting e.g. 1.5 MB - Caps any single request to at most six 256 KB lines
# 3. Configure pipeline processors according to your use case
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 15
# 4. Configure the collector service and log pipeline
service:
extensions: [oauth2client]
pipelines:
logs:
receivers: [...] # Configure receivers according to your use case
processors: [memory_limiter, ...] # Configure processors according to your use case
exporters: [otlp_http]
  1. Open or create your collector configuration file (for example, /etc/otelcol-contrib/config.yaml).
  2. Add an oauth2client extension under the extensions block with your service account credentials:
    • client_id: The sub field from your service account key JSON file.
    • client_certificate_key_id: The kid field from your service account key JSON file.
    • client_certificate_key_file: Path to the private key PEM file associated with the service account key.
    • iss: The service account ID or service account email.
    • token_url: https://accounts.stackit.cloud/oauth/v2/token
    • grant_type: urn:ietf:params:oauth:grant-type:jwt-bearer
    • signature_algorithm: RS512
    • audience: https://accounts.stackit.cloud
  3. Define the otlp_http exporter under exporters:
    • Set endpoint to <data-source-url-of-logs-instance>/otlp.
    • Set auth.authenticator to oauth2client.
  4. Register oauth2client in the service.extensions list.
  5. Add otlp_http to the service.pipelines.logs.exporters list.

Authenticate with an Instance Access Token

Section titled “Authenticate with an Instance Access Token”

You can also authenticate using a static access token generated directly from your STACKIT Logs instance. The bearertokenauth extension attaches this token to the Authorization: Bearer <token> header of every exported HTTP request.

/etc/otelcol-contrib/config.yaml
# 1. Define Bearer token authentication extension
extensions:
bearertokenauth/stackit_logs:
scheme: "Bearer"
# Provide the token directly:
token: "<Token value from Logs instance Access Token>"
# Or load it from an external file:
# filename: "/etc/otelcol-contrib/logs-access.token"
# 2. Define OTLP/HTTP exporter
exporters:
otlp_http:
# NOTE: The collector internally appends '/v1/logs' to this endpoint
endpoint: <data-source-url-of-logs-instance>/otlp
auth:
authenticator: bearertokenauth/stackit_logs
sending_queue:
enabled: true
sizer: bytes # How the queue and batching is measured
num_consumers: 4 # Number of consumers that dequeue batches
queue_size: 41943040 # Buffer size e.g. 40 MB buffer
batch:
sizer: bytes
flush_timeout: 1s # Keeps log tailing latency ≤ 1s
min_size: 524288 # the minimum size of a batch e.g. 512 KB - Enough payload for high gzip compression (~5-7x)
max_size: 1572864 # the maximum size of a batch, enables batch splitting e.g. 1.5 MB - Caps any single request to at most six 256 KB lines
# 3. Configure the collector service and log pipeline
service:
extensions: [bearertokenauth/stackit_logs]
pipelines:
logs:
receivers: [...] # Configure receivers according to your use case
processors: [...] # Configure processors according to your use case
exporters: [otlp_http]
  1. Open or create your collector configuration file (for example, /etc/otelcol-contrib/config.yaml).
  2. Add a bearertokenauth extension under extensions:
    • Set scheme to "Bearer".
    • Specify your access token using either the token field (inline string) or the filename field (path to file containing the token).
  3. Define the otlp_http exporter under exporters:
    • Set endpoint to <data-source-url-of-logs-instance>/otlp.
    • Set auth.authenticator to bearertokenauth/stackit_logs.
  4. Register bearertokenauth/stackit_logs in the service.extensions list.
  5. Add otlp_http to the service.pipelines.logs.exporters list.

STACKIT Logs enforces a per-stream rate limit. When ingesting logs through the /otlp endpoint, sending non-batched or overly large payloads for a single log stream (a unique set of indexed resource attributes/labels) can trigger 429 Too Many Requests errors.

To prevent hitting the per-stream rate limit, configure the sending_queue with byte-sized batching in your otlp_http exporter:

exporters:
otlp_http:
sending_queue:
enabled: true
sizer: bytes # How the queue and batching is measured
num_consumers: 4 # Number of consumers that dequeue batches
queue_size: 41943040 # Buffer size e.g. 40 MB buffer
batch:
sizer: bytes
flush_timeout: 1s # Keeps log tailing latency ≤ 1s
min_size: 524288 # the minimum size of a batch e.g. 512 KB - Enough payload for high gzip compression (~5-7x)
max_size: 1572864 # the maximum size of a batch, enables batch splitting e.g. 1.5 MB - Caps any single request to at most six 256 KB lines
  • sizer: bytes: Measures both the queue and batches by byte size rather than item count, allowing precise control over payload sizes against byte-based rate limits.
  • min_size and max_size: Groups logs into batches of at least 512 KB for efficient compression while splitting batches larger than 1.5 MB so individual requests stay within the 3 MB/s per-stream limit.
  • flush_timeout: Flushes batches at least every 1s to smooth out traffic bursts and maintain low latency for lower-volume streams.