Hi,
on a “moderate” cluster of 60 jvm, we got, for a day, multiple
WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] - [outbound connection to, control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
occurred.
We use akka cluster on two DC, but without “cluster DC”. Connection are pretty fast between DC.
example for a day :
2020-01-27 01:16:14,224 -1.15.26 [Cluster-akka.actor.default-dispatcher-5001] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server2-dc1:33649], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 03:09:18,065 -1.15.26 [Cluster-akka.actor.default-dispatcher-5113] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc1:35745], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 03:09:18,065 -1.15.26 [Cluster-akka.actor.default-dispatcher-5113] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc1:35745], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 07:39:00,734 -1.15.26 [Cluster-akka.actor.default-dispatcher-5395] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc1:35951], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 07:39:00,734 -1.15.26 [Cluster-akka.actor.default-dispatcher-5395] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc1:35951], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 11:05:38,602 -1.15.26 [Cluster-akka.actor.default-dispatcher-5612] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server5-dc1:2552], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 11:05:38,602 -1.15.26 [Cluster-akka.actor.default-dispatcher-5612] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server5-dc1:2552], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 11:56:11,783 -1.15.26 [Cluster-akka.actor.default-dispatcher-5655] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc2:30115], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 11:56:11,785 -1.15.26 [Cluster-akka.actor.default-dispatcher-5655] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc2:30115], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 12:41:55,584 -1.15.26 [Cluster-akka.actor.default-dispatcher-5711] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server1-dc2:25873], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 12:41:55,585 -1.15.26 [Cluster-akka.actor.default-dispatcher-5711] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server1-dc2:25873], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 15:11:47,983 -1.15.26 [Cluster-akka.actor.default-dispatcher-5862] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server1-dc1:35901], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 15:11:47,983 -1.15.26 [Cluster-akka.actor.default-dispatcher-5862] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server1-dc1:35901], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 15:36:29,761 -1.15.26 [Cluster-akka.actor.default-dispatcher-5888] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc2:38309], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 15:36:29,761 -1.15.26 [Cluster-akka.actor.default-dispatcher-5888] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc2:38309], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 16:59:10,164 -1.15.26 [Cluster-akka.actor.default-dispatcher-5962] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc1:38853], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 16:59:10,165 -1.15.26 [Cluster-akka.actor.default-dispatcher-5962] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc1:38853], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 20:16:07,863 -1.15.26 [Cluster-akka.actor.default-dispatcher-6141] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc2:2552], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2020-01-27 20:16:07,864 -1.15.26 [Cluster-akka.actor.default-dispatcher-6141] WARN akka.stream.Materializer [akka.stream.Log(akka://Cluster/system/Materializers/StreamSupervisor-1)] [] - [outbound connection to [akka://Cluster@server3-dc2:2552], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
We use Akka 2.6.1, Artery on TCP.
Everything seems normal and good, but this warning is a little bit strange !