We have a Docker container running InfluxDB v2.7.8, where we experience issues Sunday evening at 00 AM, with a recurrence interval of 2 to 3 weeks between occurrences.
We’ve noticed that memory usage suddenly begins to increase steadily, and InfluxDB struggles to write data to disk. While some data is still being written, most of it is not. I've reviewed the logs, including Docker logs, syslog, and docker.service logs, but I haven't found any relevant information. There are no error entries suggesting that Influx is having some sort of problem. It is compacting and doing its normal tasks, and it's also logging to docker like normal.
Our first attempt to resolve this issue was upgrading InfluxDB from version 2.7.5 to 2.7.8, as there was a changelog entry addressing an infinite write loop bug. Unfortunately this didn't resolve the issue, as we experienced the same problem again today.
We have Telegraf running on the server which gathers metrics from the Influx, where we've seen a spike in the queue active:
Image may be NSFW.
Clik here to view.
Below is the memory usage graph for the server:
Image may be NSFW.
Clik here to view.
At first we suspected that the ulimit or the inotify limits were reached, but that sounds weird as it would result in a error message in Influx where it is unable to open a file or something.
The servers we have experienced the issue is VMs, and it has happened on both Proxmox and VMware. The error has also occurred on both a standalone Docker server, as well as a Docker swarm cluster.
I have tried opening an issue on their GitHub, but I did not receive any help there, so I am now grasping at straws trying to get some help here.
We have tried upgrading the Influx from 2.7.5 -> 2.7.8. The only thing solving the issue is to restart the container. Then it releases all the memory and works fine afterwards.
