ThingWorx High Availability 環境では、ignite サーバーが停止され、ThingWorx がシャットダウンされます

 [L: ERROR] [O: c.t.s.ChangeWatcher] [I: ] [U: SuperUser] [S: ] [P: platform2] [T: pool-9-thread-1] Unable to update cache with model changes after 3 attempts: encountered exception class org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed to perform cache operation (cache is stopped): default
 [L: ERROR] [O: c.t.t.e.EntityEventProcessor] [I: ] [U: SuperUser] [S: ] [P: platform2] [T: pool-11-thread-1] EntityEventProcessor failed
[L: ERROR] [O: c.t.c.s.e.RoutingGroupServiceImpl] [I: ] [U: ] [S: ] [P: platform2] [T: pool-21-thread-1] Error executing syncRoutingGroups

Unable to update cache with model changes after 415823 attempts: encountered exception class org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed to perform cache operation (cache is stopped): default

28-Oct-2022 14:10:25.355 SEVERE [tcp-disco-msg-worker-[####:47500 crd]-#2%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.error Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=sys-stripe-18, threadName=sys-stripe-18-#19%twx-core-server%, blockedFor=25s]
28-Oct-2022 14:10:25.358 WARNING [tcp-disco-msg-worker-[####:47500 crd]-#2%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.warning Thread [name="sys-stripe-18-#19%twx-core-server%", id=56, state=WAITING, blockCnt=2, waitCnt=178649]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56c0ccfc, ownerName=null, ownerId=-1]

28-Oct-2022 14:10:25.360 WARNING [tcp-disco-msg-worker-[####:47500 crd]-#2%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.warning Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-18, igniteInstanceName=twx-core-server, finished=false, heartbeatTs=1666937399489]]]
	class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-18, igniteInstanceName=twx-core-server, finished=false, heartbeatTs=1666937399489]
		at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1803)
		at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1798)
		at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
		at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
		at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2858)
		at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7759)
		at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946)
		at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
		at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697)
		at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
28-Oct-2022 14:10:25.383 WARNING [tcp-disco-msg-worker-[####:47500 crd]-#2%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.warning Page locks dump:

WARNING [tcp-disco-msg-worker-[b04731d2 ##.##.##.##:47500 crd]-#2%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.warning Node is out of topology (probably, due to short-time network problems).
WARNING [jvm-pause-detector-worker] org.apache.ignite.logger.java.JavaLogger.warning Possible too long JVM pause: 1127 milliseconds.
INFO [disco-notifier-worker-#100%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.info Received activate request with BaselineTopology[id=0]
INFO [disco-notifier-worker-#100%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.info Started state transition: true
INFO [disco-notifier-worker-#100%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.info Received state change finish message: true
INFO [disco-notifier-worker-#100%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.info Received activate request with BaselineTopology[id=0]
INFO [disco-notifier-worker-#100%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.info Started state transition: true
INFO [disco-notifier-worker-#100%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.info Received state change finish message: true
SEVERE [disco-event-worker-#101%twx-core-server%] org.apache.ignite.logger.java.JavaLogger.error Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SEGMENTATION, err=null]]
SEVERE [http-nio-8080-exec-20] org.apache.ignite.logger.java.JavaLogger.error Failed to send message to remote node [node=TcpDiscoveryNode [id=3b14f882-3cd2-4023-bb5c-7bd7fb9771aa, consistentId=0:0:0:0:0:0:0:1%lo,##.##.##.##,127.0.0.1:47500, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, ##.##.##.##, 127.0.0.1], sockAddrs=HashSet [/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /##.##.##.##:47500], discPort=47500, order=3, intOrder=3, lastExchangeTime=1686133068840, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1686172951927, key=UserKeyCacheObjectImpl [part=155, val=507311674B4E187754C954D6671606F3, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], subjId=a442896f-c10b-4dc1-a53d-32f05af09d0e, taskNameHash=0, createTtl=86400000, accessTtl=86400000, txLbl=null, mvccSnapshot=null]]]
       class org.apache.ignite.IgniteCheckedException: Failed to send message to remote node (node has left the grid): 3b14f882-3cd2-4023-bb5c-7bd7fb9771aa
       at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2924)
       at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865)
       at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
       at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:349)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:241)
       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync0(GridDhtAtomicCache.java:1457)
       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1600(GridDhtAtomicCache.java:141)
       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:477)