Failed to export logs. Server is UNAVAILABLE
Problem Statement
The error message "Failed to export logs. Server is UNAVAILABLE. Make sure your collector is running and reachable from this network. Full error message: data refused due to high memory usage" indicates that the OpenTelemetry Collector (or a similar data collection agent) is attempting to send logs to a destination server, but that server is refusing the data because it is experiencing high memory usage.
Possible causes and solutions:
Collector Configuration Issues:
- Incorrect exporter configuration: Verify that the OTLP exporter in your collector's configuration is correctly pointing to the intended destination server and port.
- Compression settings: Consider enabling compression on the exporter to reduce the size of the exported payload, which might alleviate memory pressure on the receiving server.
- Batch processing settings: Adjust the settings of your batch span processor (if applicable) to send data less frequently or in smaller batches, reducing the load on the receiving server.
Destination Server Issues:
- High memory usage on the destination: The core of the problem is the high memory usage on the server receiving the logs. This could be due to:
- Insufficient resources: The server might simply not have enough allocated memory to handle the incoming log volume. Increase the allocated memory for the receiving server or collector instance.
- Inefficient processing: The receiving server or collector might be struggling to process the incoming data efficiently, leading to a build-up in memory. Investigate the configuration and performance of the receiving component.
- Memory leaks: There might be a memory leak in the application or collector running on the destination server.
- Unreachable destination: While the primary error points to high memory, ensure the destination server is indeed running and network-reachable from where the logs are being exported. Check firewalls, DNS, and proxy settings.
- High memory usage on the destination: The core of the problem is the high memory usage on the server receiving the logs. This could be due to:
OpenTelemetry Collector Specifics:
- Memory limiter: If a memory limiter processor is configured in the receiving OpenTelemetry Collector, it might be actively dropping data when memory usage exceeds a defined threshold. Adjust or remove the memory limiter if appropriate for your use case.
- Backpressure: The collector might be experiencing backpressure, where it cannot export data as quickly as it is ingested. This can occur if the exporter is slow or the destination is unavailable, leading to data buffering and increased memory usage.
Troubleshooting Steps:
Monitor memory usage:
Use monitoring tools to observe the memory consumption of the destination server or collector process to confirm the high memory usage.
Review logs:
Examine the logs of both the exporting collector and the receiving server for more detailed error messages or insights into the memory issue.
Adjust resource allocation:
If running in a containerized environment (e.g., Kubernetes), increase the memory limits for the pods running the receiving collector or application.
Optimize configuration:
Review and optimize the configuration of both the exporting collector and the receiving component to improve efficiency and reduce memory consumption.
