Hello. We are using akka actors and akka http client, java over Linux.
Recently we’re experiencing errors which caused by having too many open files. The exceptions we get are in akka TcpListener
What could be the root cause? What possibly we are doing wrong?
Is it the http client or it could be also actors communicating between them?
By looking at the file descriptors which remain open too long, most of them are either pipes or anon_inode[eventpoll]. Are they related to akka in any way?
l-wx------ 1 64 May 17 05:56 1001 -> pipe:[1161688358]
lrwx------ 1 64 May 17 05:56 1002 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1003 -> pipe:[1161688359]
l-wx------ 1 64 May 17 05:56 1004 -> pipe:[1161688359]
lrwx------ 1 64 May 17 05:56 1005 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1006 -> pipe:[1161688360]
l-wx------ 1 64 May 17 05:56 1007 -> pipe:[1161688360]
lrwx------ 1 64 May 17 05:56 1008 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1009 -> pipe:[1161688361]
l-wx------ 1 64 May 17 05:56 1010 -> pipe:[1161688361]
lrwx------ 1 64 May 17 05:56 1011 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1012 -> pipe:[1161688362]
l-wx------ 1 64 May 17 05:56 1013 -> pipe:[1161688362]
lrwx------ 1 64 May 17 05:56 1014 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1015 -> pipe:[1161688363]
l-wx------ 1 64 May 17 05:56 1016 -> pipe:[1161688363]
lrwx------ 1 64 May 17 05:56 1017 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1018 -> pipe:[1161688364]
l-wx------ 1 64 May 17 05:56 1019 -> pipe:[1161688364]
lrwx------ 1 64 May 17 05:56 1020 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1021 -> pipe:[1161688365]
l-wx------ 1 64 May 17 05:56 1022 -> pipe:[1161688365]
lrwx------ 1 64 May 17 05:56 1023 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1024 -> pipe:[1161688366]
l-wx------ 1 64 May 17 05:56 1025 -> pipe:[1161688366]
lrwx------ 1 64 May 17 05:56 1026 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1027 -> pipe:[1161688367]
l-wx------ 1 64 May 17 05:56 1028 -> pipe:[1161688367]
lrwx------ 1 64 May 17 05:56 1029 -> anon_inode:[eventpoll]
lr-x------ 1 64 May 17 05:56 1030 -> pipe:[1161688368]
l-wx------ 1 64 May 17 05:56 1031 -> pipe:[1161688368]
lrwx------ 1 64 May 17 05:56 1032 -> anon_inode:[eventpoll]
The exceptions looks like this:
akka.actor.default-dispatcher-10..akka://system/IO-TCP/selectors/$a/2] a.i.TcpListener - Accept error: could not accept new connection
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:1.8.0_212]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) ~[?:1.8.0_212]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) ~[?:1.8.0_212]
at akka.io.TcpListener.acceptAllPending(TcpListener.scala:112) ~[akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.io.TcpListener$$anonfun$bound$1.applyOrElse(TcpListener.scala:85) ~[akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.io.TcpListener.aroundReceive(TcpListener.scala:34) ~[akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.actor.ActorCell.invoke(ActorCell.scala:557) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [akka-actor_2.12-2.5.17.jar:2.5.17]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [akka-actor_2.12-2.5.17.jar:2.5.17]
