OpenTelemetry Collector: Write log data with OTLP API
Last updated on
STACKIT Logs supports ingesting log records through the OpenTelemetry (OTLP) protocol over HTTP. You can configure the OpenTelemetry Collector’s OTLP/HTTP exporter (otlp_http) to forward logs directly to your STACKIT Logs instance.
To authenticate requests, STACKIT Logs supports two authentication methods:
- STACKIT Service Account (recommended): Uses the
oauth2clientextension to exchange service account credentials for short-lived JWT access tokens automatically. - Instance Access Token: Uses the
bearertokenauthextension with a static bearer token created in your Logs instance.
Prerequisites
Section titled “Prerequisites”- STACKIT Logs instance: Create your first STACKIT Logs instance
- Data source URL: Retrieve API URLs for your STACKIT Logs instance
- Credentials:
- For Service Account authentication (recommended):
- A STACKIT Service Account: Create a service account
- A Service Account Key pair: Create a service account key
- An Access Policy configured on your STACKIT Logs instance granting the Service Account write permissions.
- For Access Token authentication:
- An access token with write permissions: Retrieve credentials for STACKIT Logs instance
- For Service Account authentication (recommended):
Authenticate with a STACKIT Service Account (Recommended)
Section titled “Authenticate with a STACKIT Service Account (Recommended)”Using a STACKIT Service Account is the recommended approach for automated and production workloads. The oauth2client extension automatically obtains short-lived JWTs from the STACKIT IAM endpoint, removing the need to manage and rotate long-lived access tokens.
Configuration example
Section titled “Configuration example”# 1. Define OAuth2 client extensionextensions: oauth2client: client_id: "<sa-credentials-sub>" client_certificate_key_id: "<sa-credentials-kid>" client_certificate_key_file: "<path-to-sa-private-key.pem>" grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer" token_url: "https://accounts.stackit.cloud/oauth/v2/token" signature_algorithm: "RS512" iss: "<service-account-id>" # Service account ID or email audience: "https://accounts.stackit.cloud" expiry_buffer: "2m"
# 2. Define OTLP/HTTP exporterexporters: otlp_http: # NOTE: The collector internally appends '/v1/logs' to this endpoint endpoint: <data-source-url-of-logs-instance>/otlp auth: authenticator: oauth2client sending_queue: enabled: true sizer: bytes # How the queue and batching is measured num_consumers: 4 # Number of consumers that dequeue batches queue_size: 41943040 # Buffer size e.g. 40 MB buffer batch: sizer: bytes flush_timeout: 1s # Keeps log tailing latency ≤ 1s min_size: 524288 # the minimum size of a batch e.g. 512 KB - Enough payload for high gzip compression (~5-7x) max_size: 1572864 # the maximum size of a batch, enables batch splitting e.g. 1.5 MB - Caps any single request to at most six 256 KB lines
# 3. Configure pipeline processors according to your use caseprocessors: memory_limiter: check_interval: 1s limit_percentage: 80 spike_limit_percentage: 15
# 4. Configure the collector service and log pipelineservice: extensions: [oauth2client] pipelines: logs: receivers: [...] # Configure receivers according to your use case processors: [memory_limiter, ...] # Configure processors according to your use case exporters: [otlp_http]Setup steps
Section titled “Setup steps”- Open or create your collector configuration file (for example,
/etc/otelcol-contrib/config.yaml). - Add an
oauth2clientextension under theextensionsblock with your service account credentials:client_id: Thesubfield from your service account key JSON file.client_certificate_key_id: Thekidfield from your service account key JSON file.client_certificate_key_file: Path to the private key PEM file associated with the service account key.iss: The service account ID or service account email.token_url:https://accounts.stackit.cloud/oauth/v2/tokengrant_type:urn:ietf:params:oauth:grant-type:jwt-bearersignature_algorithm:RS512audience:https://accounts.stackit.cloud
- Define the
otlp_httpexporter underexporters:- Set
endpointto<data-source-url-of-logs-instance>/otlp. - Set
auth.authenticatortooauth2client.
- Set
- Register
oauth2clientin theservice.extensionslist. - Add
otlp_httpto theservice.pipelines.logs.exporterslist.
Authenticate with an Instance Access Token
Section titled “Authenticate with an Instance Access Token”You can also authenticate using a static access token generated directly from your STACKIT Logs instance. The bearertokenauth extension attaches this token to the Authorization: Bearer <token> header of every exported HTTP request.
Configuration example
Section titled “Configuration example”# 1. Define Bearer token authentication extensionextensions: bearertokenauth/stackit_logs: scheme: "Bearer" # Provide the token directly: token: "<Token value from Logs instance Access Token>" # Or load it from an external file: # filename: "/etc/otelcol-contrib/logs-access.token"
# 2. Define OTLP/HTTP exporterexporters: otlp_http: # NOTE: The collector internally appends '/v1/logs' to this endpoint endpoint: <data-source-url-of-logs-instance>/otlp auth: authenticator: bearertokenauth/stackit_logs sending_queue: enabled: true sizer: bytes # How the queue and batching is measured num_consumers: 4 # Number of consumers that dequeue batches queue_size: 41943040 # Buffer size e.g. 40 MB buffer batch: sizer: bytes flush_timeout: 1s # Keeps log tailing latency ≤ 1s min_size: 524288 # the minimum size of a batch e.g. 512 KB - Enough payload for high gzip compression (~5-7x) max_size: 1572864 # the maximum size of a batch, enables batch splitting e.g. 1.5 MB - Caps any single request to at most six 256 KB lines
# 3. Configure the collector service and log pipelineservice: extensions: [bearertokenauth/stackit_logs] pipelines: logs: receivers: [...] # Configure receivers according to your use case processors: [...] # Configure processors according to your use case exporters: [otlp_http]Setup steps
Section titled “Setup steps”- Open or create your collector configuration file (for example,
/etc/otelcol-contrib/config.yaml). - Add a
bearertokenauthextension underextensions:- Set
schemeto"Bearer". - Specify your access token using either the
tokenfield (inline string) or thefilenamefield (path to file containing the token).
- Set
- Define the
otlp_httpexporter underexporters:- Set
endpointto<data-source-url-of-logs-instance>/otlp. - Set
auth.authenticatortobearertokenauth/stackit_logs.
- Set
- Register
bearertokenauth/stackit_logsin theservice.extensionslist. - Add
otlp_httpto theservice.pipelines.logs.exporterslist.
Batching to avoid per-stream rate limits
Section titled “Batching to avoid per-stream rate limits”STACKIT Logs enforces a per-stream rate limit. When ingesting logs through the /otlp endpoint, sending non-batched or overly large payloads for a single log stream (a unique set of indexed resource attributes/labels) can trigger 429 Too Many Requests errors.
To prevent hitting the per-stream rate limit, configure the sending_queue with byte-sized batching in your otlp_http exporter:
exporters: otlp_http: sending_queue: enabled: true sizer: bytes # How the queue and batching is measured num_consumers: 4 # Number of consumers that dequeue batches queue_size: 41943040 # Buffer size e.g. 40 MB buffer batch: sizer: bytes flush_timeout: 1s # Keeps log tailing latency ≤ 1s min_size: 524288 # the minimum size of a batch e.g. 512 KB - Enough payload for high gzip compression (~5-7x) max_size: 1572864 # the maximum size of a batch, enables batch splitting e.g. 1.5 MB - Caps any single request to at most six 256 KB linessizer: bytes: Measures both the queue and batches by byte size rather than item count, allowing precise control over payload sizes against byte-based rate limits.min_sizeandmax_size: Groups logs into batches of at least512 KBfor efficient compression while splitting batches larger than1.5 MBso individual requests stay within the 3 MB/s per-stream limit.flush_timeout: Flushes batches at least every1sto smooth out traffic bursts and maintain low latency for lower-volume streams.