Document a Test Strategy

As a current licensed ThingWorx owner, you have access to download the latest version and replicate your production instance on another server. Piloting the upgrade ahead of time provides the following assurance:

1. End-user buy-in and support of new functionality
2. Firsthand experience with new functionality

While piloting the upgrade, we recommend making the environment as close to production as possible. If you are interested in having PTC help establish your ThingWorx's foundation, check out the Upgrade ThingWorx Foundation to 9+ service. PTC starts by doing a full assessment of your ThingWorx solution and infrastructure in the ThingWorx Upgrade Assessment service.

When you initially deployed ThingWorx or the last time you upgraded, a test strategy was created. Decide how you will apply that strategy to this new upgrade. Refer to the Determine Changes to Architecture, Select an Upgrade Option, When to Upgrade to modify any test scripts. Developers and device engineers work together to create a testing plan.

In general, we recommend the following testing:

• Unit testing:  build in a unit testing concept to allow code to be tested separately from UI testing. This type of testing is only necessary if you modify code during this upgrade. We recommend making changes to code or making backlog updates after completing the upgrade.
• System testing: test the features and functionality of a collection of components defined by an acceptance criteria document.
• Security and penetration testing: test for vulnerabilities, risks, and threats (always test with different uses that represent real uses)
• Connectivity testing: test the edge/Kepware, products, devices/machines, and platform integrations
• Migration Validation: verify the data from the previous version of ThingWorx matches in the upgraded instance.
• Stress or load testing: test the stability of the solution and how it handles errors and load
• User acceptance testing (UAT): test with select end-users. Very little UAT is necessary for this upgrade.
• Deployment testing: verify the package works as expected on a staging server that represents a production configuration. This type of testing is required if you modify code during this upgrade.

It would help if you did additional testing that was not mentioned based on your specific solution or industry. Remember to consider security and compliance requirements at every stage for each instance.

Create a testing plan for each instance. Be sure to align testing with your existing DevOps processes. Plan to test in a QA environment that replicates your production environment as closely as possible.

Verify user permissions, logins, and data

Make sure users have access to the right components within the application. Have several users login to ThingWorx in a test/QA environment. They should verify they have access to everything they need. Test the user permissions for each user group. If someone can't access a mashup or if they have too much access, change their permissions.

Also, verify that each environment shows the appropriate data for each user group. You may need to strip production data from some environments. Do this to prepare for testing, especially user acceptance testing.

Monitor the data flows, integrations, and connectivities

• Implies on remote connectivity and integration
• Data flow is continuous and integral

Observe and monitor the application run for 1-2 days

• Measure performance impacts, load tests
• Ensure smooth running of software

Verify logs, entities count, and system stats

• Regular monitoring of logs for any errors/issues
• Entities count to ensure if anything not missed
• Overall system stats, health, and configurations

Verify all customizations & extensions

• Ensure all customizations and extensions are correctly moved to the test server and are in place

Take some time to review your existing scripts and make sure they include the following:

• Value statement  
• Summary of what the solution does (refer back to your use case) 
• Data sources and data flow 
• Design specification 
• Acceptance criteria 
• Test procedure 
• Environments and who has access to what 
• Security requirements 
• Naming conventions 
• ThingWorx version or release numbers 
• Results of quality assurance (QA) test 
• Performance figures - Results of Stress or load test
• Revisions 
• Software or any third-party app 

Keep your documentation brief and definitive. Your documentation provides a quick reference to help you solve issues and make faster decisions.