1. <property name="communicationSpi"> <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> <property name="socketWriteTimeout" value="5000"/> <property name="usePairedConnections" value="true"/> </bean> </property>
2. 조치방법
2.1. version up (2.7)
2.2. cluster discovery 방식 변경
2.2.1. zookeeper discovery
2.2.2. ip discovery
2.2.2.1. zookeeperFinder
2.2.2.2. ip
2.3. cluster node 최소화
2.4. failure detection time out 시간 늘리기
2.5. Cluster 2개로 운영
3. issue(1/24)
3.1. local cache를 사용하여 값이 이상하게 나옴 504에러가 남 로그인 시 에러남
3.1.1. error msg
3.1.2. 원인
3.1.2.1. 1) [ERROR] [cl-rms-svc-02.OAS] [cacheGrid%] o.a.i.s.c.tcp.TcpCommunicationSpi Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3, bytesRcvd=140836, bytesSent=55692, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3, ..... java.io.IOException: Connection timed out at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1250) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748)
3.1.2.1.1. <property name="communicationSpi"> <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> <property name="socketWriteTimeout" value="5000"/> <property name="usePairedConnections" value="true"/> </bean> </property>
3.1.2.2. 2) [WARN ] [cl-rms-svc-02.OAS] [cacheGrid%] o.a.i.s.c.tcp.TcpCommunicationSpi Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/112.107.147.89:47103, failureDetectionTimeout=20000]
3.1.2.3. 3) [INFO ] [cl-rms-plat-02.HTTP] [Thread-2 ] o.a.ignite.internal.util.typedef.G Invoking shutdown hook...
3.1.2.4. 4) [WARN][cl-rms-plat-02.HTTP] [cacheGrid%] o.apache.ignite.internal.diagnostic Pending transaction deadlock detection futures:
3.1.2.5. 5) [WARN ] [cl-rms-plat-02.HTTP] [cacheGrid%] o.apache.ignite.internal.diagnostic Found long running cache future
3.1.2.6. 6) [ERROR] ClusterTopologyCheckedException: Remote node has left topology
3.1.2.7. 7) [ERROR] [cl-rms-plat-02.HTTP] [cacheGrid%] o.a.i.s.c.tcp.TcpCommunicationSpi Failed to process selector key
3.1.2.8. 8) [INFO ] [cl-rms-plat-02.HTTP] [Thread-2 ] o.a.ignite.internal.util.typedef.G Invoking shutdown hook...
3.1.2.9. 9) transaction dead lock인 상태이므로 cluster에 join되지 못하고 4~8 반복됨
4. 근본원인
4.1. System critical errors (e.g. OutOfMemoryError).
4.2. Unintentional system worker termination (e.g. due to an unhandled exception).
4.2.1. DMZ <> 내부 네트워크 감시프로그램?
4.3. System workers hanging.
4.3.1. springContext에 기생하기 때문에??