ESI publishing takes a long time in Windchill Enterprise Systems Integration (ESI)

• Windchill Enterprise Systems Integration (ESI) publishing takes a long time
• Publishing a small structure has got acceptable performance but larger structure (100s components) is too slow
• Windchill queues are stuck and waiting for the db-connections. The existing logs show the db-connections are consumed by some ESI requests
• Following thread issue those queries:

at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1782)
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:4838)
- locked com.microsoft.sqlserver.jdbc.TDSReader@c8e8bd3
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:6150)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:402)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:350)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)
- locked java.lang.Object@748cc31
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:155)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:285)
at wt.pds.AbstractResultCursor.executeQuery(AbstractResultCursor.java:559)
at wt.pds.AbstractResultCursor.executeQuery(AbstractResultCursor.java:523)
at wt.pds.BasicResultCursor.getNextResultSet(BasicResultCursor.java:282)
at wt.pds.BasicResultCursor.advance(BasicResultCursor.java:234)
at wt.pds.BasicResultCursor.next(BasicResultCursor.java:82)
at wt.pds.SQLDatabasePds.query(SQLDatabasePds.java:801)
at wt.pom.PersistentObjectManager.refresh(PersistentObjectManager.java:1544)
at wt.pom.PersistentObjectManager.query(PersistentObjectManager.java:1317)
at wt.fc.StandardPersistenceManager.restore(StandardPersistenceManager.java:1932)
at wt.fc.StandardPersistenceManager.restore(StandardPersistenceManager.java:1193)
at wt.fc.StandardPersistenceManager.restore(StandardPersistenceManager.java:1187)
at wt.fc.cache.ReferenceCache.restore(ReferenceCache.java:305)
at wt.fc.cache.ReferenceCache.get(ReferenceCache.java:296)
at wt.fc.cache.StandardObjReferenceCacheService.getObject(StandardObjReferenceCacheService.java:149)
at sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at wt.services.ServiceFactory$ServerInvocationHandler.invoke(ServiceFactory.java:399)
at com.sun.proxy.$Proxy26.getObject(Unknown Source)
at wt.fc.DefaultInflator$CorCache.inflate(DefaultInflator.java:409)
at wt.fc.DefaultInflator$CorCache.inflate(DefaultInflator.java:397)
at wt.fc.DefaultInflator$AbstractCache.get(DefaultInflator.java:291)
at wt.fc.DefaultInflator$AbstractCache.get(DefaultInflator.java:259)
at wt.fc.DefaultInflator.get(DefaultInflator.java:135)
at wt.fc.CachedObjectReference.getObject(CachedObjectReference.java:201)
at wt.workflow.engine.WfActivity.getParentProcess(WfActivity.java:780)
at wt.workflow.engine.StandardWfEngineService.synchRobotToEvent(StandardWfEngineService.java:9673)
at wt.workflow.engine.StandardWfEngineService$SynchEventListener.notifyVetoableEvent(StandardWfEngineService.java:447)
at wt.events.StandardKeyedEventBranch.intDispatchEvent(StandardKeyedEventBranch.java:312)
at wt.events.StandardKeyedEventBranch.dispatchVetoableEvent(StandardKeyedEventBranch.java:250)
at wt.events.StandardKeyedEventDispatcher.intDispatchEvent(StandardKeyedEventDispatcher.java:286)
at wt.events.StandardKeyedEventDispatcher.dispatchVetoableEvent(StandardKeyedEventDispatcher.java:215)
at wt.services.StandardManagerService.intDispatchVetoableEvent(StandardManagerService.java:406)
at wt.services.StandardManagerService.dispatchVetoableEvent(StandardManagerService.java:302)
at com.ptc.windchill.esi.txn.ESIDispatchReleaseCompleteEvent.dispatchEvent(ESIDispatchReleaseCompleteEvent.java:77)
at com.ptc.windchill.esi.txn.ESIDispatchReleaseCompleteEvent.run(ESIDispatchReleaseCompleteEvent.java:121)
at wt.session.SessionThread.run(SessionThread.java:225)

• On one of the cluster node high CPU utilization observed by following thread:

"SessionThread-2320" Id=14563 daemon prio=5 RUNNABLE (in native)
      Blocked (cnt): 1531; Waited (cnt): 41
      CPU nanos: 128103733936; User nanos: 103860000000; Bytes allocated: 27956684704
      Method context: 794xrp9jq;malhrj4i;24089;m8avap;60004; DB session: 644
      % of 1 CPU used by context: 3.628668579854804 (128103330532/3530312226451); Bytes allocated by context: 27956682984
  at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
  at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
  at sun.nio.ch.IOUtil.read(IOUtil.java:197)
  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    - locked java.lang.Object@33b32402
  at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:178)
  at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:544)
  at oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:234)
  at oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:174)
  at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:122)
  at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:100)
  at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:86)
  at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:762)
  at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:427)
  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:394)
  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:255)
  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:610)
  at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:253)
  at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:86)
  at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:928)
  at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:950)
  at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1099)
  at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3640)
  at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1384)
  at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3687)
    - locked oracle.jdbc.driver.T4CConnection@2e58a4f4
  at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165)
  at wt.pds.SQLDatabasePds.lock(SQLDatabasePds.java:1172)
  at wt.pom.PersistentObjectManager.lock(PersistentObjectManager.java:905)
  at wt.fc.StandardPersistenceManager.lock(StandardPersistenceManager.java:1303)
  at wt.workflow.engine.StandardWfEngineService.synchRobotToEvent(StandardWfEngineService.java:10038)
  at wt.workflow.engine.StandardWfEngineService$SynchEventListener.notifyVetoableEvent(StandardWfEngineService.java:450)
  at wt.events.StandardKeyedEventBranch.intDispatchEvent(StandardKeyedEventBranch.java:320)
  at wt.events.StandardKeyedEventBranch.dispatchVetoableEvent(StandardKeyedEventBranch.java:255)
  at wt.events.StandardKeyedEventDispatcher.intDispatchEvent(StandardKeyedEventDispatcher.java:304)
  at wt.events.StandardKeyedEventDispatcher.dispatchVetoableEvent(StandardKeyedEventDispatcher.java:228)
  at wt.services.StandardManagerService.intDispatchVetoableEvent(StandardManagerService.java:410)
  at wt.services.StandardManagerService.dispatchVetoableEvent(StandardManagerService.java:303)
  at com.ptc.windchill.esi.txn.ESIDispatchReleaseCompleteEvent.dispatchEvent(ESIDispatchReleaseCompleteEvent.java:84)
  at com.ptc.windchill.esi.txn.ESIDispatchReleaseCompleteEvent.run(ESIDispatchReleaseCompleteEvent.java:145)
  at wt.session.SessionThread.run(SessionThread.java:225)