Article - CS321821

Updating lastConnectionTime property of Remote Things can create a deadlock scenario if the associated queue is full on ThingWorx Platform

Modified: 22-Nov-2022   


Applies To

  • ThingWorx Platform 8.1 to 8.5

Description

  • When many Remote Things connect at once the ThingWorx Platform becomes unstable
  • lastConnectionTime property updates seem related to ThingWorx performance
  • ThingWorx Platform performance is poor
  • ThingWorx Platform has stopped working and Composer is no longer accessible
  • Thread dumps show numerous thread waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync:
    • Blocked Thread
      "https-jsse-nio-8443-exec-85" tid=0x7746d in WAITINGBlocked: 2696[-1ms], Waited: 879194[-1ms]
      User CPU: 35m26s
          - synchronizer <0x143fa594> (a java.util.concurrent.ThreadPoolExecutor$Worker)
          at sun.misc.Unsafe.park(Native Method)
          - waiting on <0x37c62d09> (a java.util.concurrent.locks.ReentrantLock$NonfairSync), held by https-jsse-nio-8443-exec-84
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
          at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
          at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
          at com.thingworx.communications.platform.modules.DispatchingServerCommunicationModule.processBindMessage(DispatchingServerCommunicationModule.java:471)
          at com.thingworx.communications.platform.modules.DispatchingServerCommunicationModule.processCompletedBindMessage(DispatchingServerCommunicationModule.java:348)
          at com.thingworx.communications.platform.modules.DispatchingServerCommunicationModule.processBindRequestMessage(DispatchingServerCommunicationModule.java:267)
          at com.thingworx.communications.platform.modules.DispatchingServerCommunicationModule.process(DispatchingServerCommunicationModule.java:146)
          at com.thingworx.communications.server.endpoints.DispatchingServerEndpoint.messageReceived(DispatchingServerEndpoint.java:116)
          at com.thingworx.communications.server.connection.jsr356.Jsr356ServerConnection.onMessage(Jsr356ServerConnection.java:338)
          at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43
      • Locking Thread:
        "https-jsse-nio-8443-exec-84" tid=0x77467 in WAITINGBlocked: 2755[-1ms], Waited: 881187[-1ms]
        User CPU: 38m20s
            - synchronizer <0x37c62d09> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
            - synchronizer <0x3f476c55> (a java.util.concurrent.ThreadPoolExecutor$Worker)
            at sun.misc.Unsafe.park(Native Method)
            - waiting on <0x6f66f64> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
            at java.util.concurrent.LinkedBlockingDeque.putLast(LinkedBlockingDeque.java:396)
            at java.util.concurrent.LinkedBlockingDeque.put(LinkedBlockingDeque.java:649)
            at com.thingworx.persistence.provider.StreamEntryProcessor.queueEntry(StreamEntryProcessor.java:142)
            at com.thingworx.system.managers.ThingManager$PersistentPropertyCache.put(ThingManager.java:796)
            at com.thingworx.things.Thing.rawSetPropertyVTQ(Thing.java:4103)
            at com.thingworx.things.Thing.rawSetPropertyVTQ(Thing.java:4063)
            at com.thingworx.things.connected.RemoteThing.updateLastConnection(RemoteThing.java:322)
            at com.thingworx.things.connected.RemoteThing.setConnected(RemoteThing.java:291)
            at com.thingworx.things.connected.RemoteThing.setCommunicationEndpoint(RemoteThing.java:359)
            at com.thingworx.things.connected.RemoteThing.bindEndpoint(RemoteThing.java:419)
        	...
      This is a printer-friendly version of Article 321821 and may be out of date. For the latest version click CS321821