Business Expansion with Roche Diagnostics’ Remote Services
00:01 Unpredictable events happen. What counts most in these situations is quick and easy issue resolution, but often it is difficult to find immediate help with Roche remote service. Our experts are available at your fingertips, only clicks away. Whenever you need help, you can request assistance from a Roche expert. Upon your request. they troubleshoot with you to resolve issues quickly and conveniently.
00:31 Roche remote service offers connected labs, a range of services, and dedicated support staff. For your lab, this means technical problems are addressed without long waiting times, and you don't have to compromise on productivity and efficiency. Roche remote service includes remote support via screen sharing, software updates and digital updates of vital test data to run core analyzers. Digital uploads from lab systems to Roche also helps to improve and control
01:05 the test environment. Roche remote service helps you to reduce downtime and immediately boost lab performance. The secure and certified digital infrastructure also sets up your lab for the future of connected services and diagnostics. Roche is currently strengthening the software of its gateways at customer premises and updating the back end infrastructure with the PTC ThingWorx Industrial Iot Solutions platform,
01:35 one of the leading solution providers in digital infrastructure. PTC ThingWorx is implemented according to the specific needs of Roche and will be seamlessly integrated into the existing ecosystem. This new solution also helps to provide increased cyber security to keep your data secure and protected. Want to know more. Set up a meeting with your Roche service representative.
02:08 All right. With that brief video introduction, I'm happy to welcome you to this breakout session. We're going to talk about how we're expanding business and how we're supporting Roche Diagnostics in upgrades and modification to their remote service. My name is Matthew Riedel. I'm a technical director with Kalypso Rockwell Automation and I'm happy to be joined today with my colleagues from Roche Diagnostics that are working on the project together with me, Eva. Hi, I'm Eva Mszanka.
02:41 I’m a program lead for remix, the upgrade. And Thomas. Hello. Welcome everyone. My name is Thomas. I’m a technical product manager at Roche doing this journey together with Kalypso. Thank you. And what's the better way of telling the story if just not inviting the clients and they can tell the story themselves? So in today’s session, we are going to look at a brief introduction into Kalypso and Roche. So for people who maybe do not know who Kalypso is or who Roche is,
03:11 and you will have that opportunity to know that, and then we'll look into the remote services and the use cases supported on the new platform. You will also understand how transformational the project is and what kind of challenges we had to overcome in terms of complexity of the solution. And then we'll share with you some takeaways and best practices, some lessons learned that we've experienced on the project. So a little bit first about Kalypso,
03:41 because we always need to self promote ourselves a little bit. So Kalypso has a very long story when it comes to supporting the life science projects, and it's everything from PLM, from the QMS systems to connected products and to smart manufacturing. We can support our clients in the life science domain across the full spectrum of digital thread. Right now, across of the 50 top life science companies in the world.
04:11 We are involved in active projects or we've been involved in some projects with 30 of them. We are we are helping the clients to be competitive, staying on top of the technology and making sure that we deliver the value through digital technologies. What is Kalypso? Kalypso is part of the Rockwell Automation. We are the consulting and implementation arm of Rockwell Automation, fully owned by Rockwell since 2020
04:42 So it’s a three years now, but at the same time, Kalypso and Rockwell, we are a very strong partner with the PTC. We are working across the full spectrum of technology that PTC provides. That's why we are here. That's why we are Pinnacle sponsor of the event as well. And we've been awarded many awards through the PTC Partner Network Recognition program. Some of them you've seen even yesterday in the Global Global Summit. And we deliver deliver outcomes, we deliver digital using a lot of PTC technology.
05:18 But today you will learn more around remote service and ThingWorx based solutions. With that. I'll let Thomas introduce Roche. So Roche Diagnostics, a leading health care provider around the globe. So we provide multiple solutions in different areas, meaning analyzers, but also IT solutions to complement the analyzer portfolio. And you see here on the slide,
05:48 there are a lot of different areas which we are providing solutions and more or less in all areas we have an integration with our remote service solution. While we do this since more than 20 years, starting with an own solution, then going to the world in 2009. Now we are ahead of our next journey with the ThingWorx solution. So you see, we have a very big ecosystem where we have around 60,000 devices around the globe connected to more than 40,000 gateways.
06:22 We have physical gateways. We also have software gateways integrated into the analyzer, but also for IT systems. We use these software gateways. So it's really a global solution. Having more than 140 countries where the solution is up and running and we have more than 100 different types of assets which we connect to this platform. So the operation of that is rather complex. All of these connected systems, they produce a lot of data which we transport through the platform,
06:55 through the current one, and then the future one with ThingWorx and you see it's more than 200GB a day, which is collected telemetry data to improve efficiency of the analyzers and to ensure higher uptime at the customer. We have more than 7000 users globally using our solution at Rush, providing services to customers. And I can tell you we have around 70,000 plus remote sessions a month, which we execute through the solution.
07:27 And we have an increasing install base of around 5 to 10% . When you look what we do with this solution, it all starts with trust because we need to ensure that only devices which are allowed to connect to the platform can really connect and make use of the other workflows. So the registration and trust process is always the foundation. Here we now have a CRM integration with our Salesforce system to improve the usability
07:57 for the users. We use the master data and reuse the data in the ThingWorx solution. One of our main use cases I mentioned it before is the remote support where we have more than 70,000 sessions a month. So this is, for instance, remote error analysis, screen sharing, but also other options we use. Like remote task execution. We also provide digital updates to our systems, to the gateways, but even to the medical devices we provide updates, software updates,
08:30 information associated with the analyzers, but also product information which you need to operate the analyzers. Collecting data as you have seen before. We collect a lot of data from these thousands of system is very important and we have multiple different systems to analyze this data and feed it back to the R&D and also for other service purposes. With this, we ensure higher uptime and reduce downtime in order to laboratories having a good time
09:01 and can do what they really need to do. It is executing tests like corona tests and orders. Very important is also the security aspects that we have a certified solution. We have multiple different ISO certifications and now very new is also the hop compliance, which is extremely important for the business in the US. So together with Kalypso and PTC, we now achieved that. The new ThingWorx solution will be HIPPA compliant. All right. So for the technical folks in the room,
09:33 that's the slide that you're probably interested in, how that solution really looks like from a technical architecture perspective. Of course, you have a lab, you have a instruments where the agent runs, and that agent is responsible for telemetry gathering, for file collection, for establishing connection to the cloud and the platform where the remote service application is deployed. This diagram here is very, very high level. You can just imagine that behind that,
10:04 actually there is a spider spider web of diagrams that we unfortunately couldn't show you. But this is the high level architecture. So what happens, the data coming from the from the instrument is being obviously connected to ThingWorx. You can see that ThingWorx is running on Microsoft Azure Cloud that's managed and hosted by PTC. We obviously have a lot of integration to the backend systems that are responsible for transferring data to other systems
10:35 for further analysis, for reporting. We have an integration to CRM system that manages the whole order process flow and also informs us about any changes to the configuration on on the on the client side. What I want to also highlight is that we have separate environments for global infrastructure and global client base versus China due to regulations and compliance. So we have to deal with that complexity.
11:06 On top of that, working in a validated environment or invalidated industry requires a lot of validation and verification steps. That means you have a lot of environments in your portfolio to take care of. What does it mean? It means that you need to establish your CI and CD or DevOps operations quite early, otherwise you will be endlessly deploying manually, which is error prone and time consuming. So that was one of our first initiatives to actually ensure that we have
11:38 a proper DevOps in place and we can deploy with a single click to multiple environments depending on the current situation of testing activities. Let's move a little bit forward where I can talk to you about the migration approach. Okay. So what you see here is a simplified diagram that shows you as ease and transition state and to be state. Okay. Of course, right now Axeda and service link was the source system where the existing Axeda agents were connected.
12:09 So what we did, we built a dedicated module on top of ThingWorx that actually is integrating Axeda and is responsible for validating when the asset is ready for migration, once the asset is ready for migration. And there is a lot of things going in the background, by the way, right? We need to check that there is connectivity to the new system that all the hardware, software preconditions are in place and so on. But once the asset is ready for migration, the operator or an affiliate, they can say,
12:40 I want to take all these assets in this laboratory and migrate them to the new system. What we do, we actually use the existing infrastructure of to to send down the new agent that will automatically connect to a new platform. And then once we can once we validate that the migration is successful, we can send additional cleanup packages that will essentially remove the old, old software running on the instrument because we want to ensure that we are running on the full ThingWorx stack.
13:15 So that's kind of an overview of the migration approach in a high level. I'll move it now to Eva. Yes. So, so far you could see somehow how big is the solution and the complexity and the numbers behind it. So how to make it happen? You need to have a team. And our really first focus was how to create the team that can work together in a seamless way and how we also collaborate with all our stakeholders. So as you can see, we have overall layer of the governance,
13:46 but below we have three streams that we are working and running. We are working and running in parallel. We have a development stream which is which is combining different providers. So we have PTC colleagues, we have Kalypso, but we have also Nagarro who is helping us to develop on the gateway side and also on the backend. And we have also a lot of lot of collaboration with internal IT team and the objective is always to work in a seamless way. So those teams are really always focusing on the solution and not the problem.
14:19 And there is no ping pong between different, different. When different problems comes. We always focus on the solution and have a good collaboration between us. The second stream is very also important, which is the testing and integration. So this solution is running for a lot of different types of assets and we need to make sure that from the beginning, on the component side, we start the testing, but then we also ensure the testing from the end to end perspective with the gateway ThingWorx and also together with all
14:49 the instruments. So here we have a strong collaboration with all the business area and, and they are proper teams also from their side who are from the beginning engaged with us and they are starting in the early phases. We started the testing together with them. And the first stream, which is extremely important. It's the deployment stream. So all the change management, it's it's really crucial when it comes to the volumes that we have as we are heavily deployed right now in in one more than 100 countries.
15:21 And we need to collaborate with all the affiliates. So we have here the consulting company who is helping us and we have dedicated stream that is from the beginning creating our community. On one side we are preparing for the migration. So and affiliates needs to ensure that the data is ready. Once we are ready to to move and to go live. And on the other side, we also, from the very beginning started to build our community with of the users. So we have early adopters group that is helping us to engage with the user from
15:51 the beginning and and, and advance on the adoption of the solution from the moment that we started developing it. And this is somehow how our story looks like. Here you can see how we were, how we were progressing with the development from the beginning. So to be successful, what we what we what we did, we selected the safe framework. So we are working in the in the Agile way, but in the safe framework as we have different components and we need to collaborate with different stakeholders.
16:21 So we are planning on more or less three more spaces to be able to sync with everyone, have a focus and then to, to approach, to have a step wise approach and and validation on the way. So safe was was one of the key parts. And here we needed to engage with everyone. So we trained all the team. No matter if we have Roche people or people from our from from our providers. So everyone were trained so we could have a good start from the beginning
16:51 and everyone was on boarded. And this also helped us really to create the team with the good inclusive environment. When everyone is really free to speak up because this is the key. You can just pretend that everything is perfect, but the reality is different. So without this one, and without being able to really have a good, good and healthy environment where you can collaborate, things are not possible from the other thing, which is which is very important. It's that things will happen. Yeah. So, so it's not only a happy story.
17:23 There are many different things that that happen during the our way. So on one side the key things here key learnings and insights that from the from the very beginning, once we started to have the the end to end connection, we realized that that we need to we need to also look on all the capabilities also on the PTC product. So we are collaborating together with the with the PTC colleagues on, on enabling all the functionality that we need on different components that we are using, including platform and the edge.
17:55 And right now we are we are we are getting the somehow state of art, I would say. But what for what we need. But this is already one year collaboration. So so we need to do it step by step and you need to have a partners that collaborates with you on the other side. Yeah, we learned that how important it is to go to production as soon as we can. So in this journey you can see also that we have two steps approach for the migration, whereas a big enabler for the migration,
18:26 which is the migration of the gateways from Windows 7 to Windows 10. So we went in November, we started in November and we thought that we could go to production. We we started the activity, but it's called the first customer monitoring. So it's somehow like a small deployment in release in, in the limited number of customers. And here we we we figure out that things were not working as the environment was slightly different. But the lower environment that we were testing and this was one of our biggest learnings and you could see then later
18:57 on how we learn from it to to not repeat the same story when we need to go with ThingWorx to production. Right now we are in a good shape because we started once again the FCM with Windows 7, Windows 10 and now it's working. But we need to to learn to adapt. On the other side, we also have the topics related to to compliance. So as, as Tomas was saying, regulations are different in different countries and it's not everything is super global. So HIPPA was a big challenge for us, but together we made it.
19:30 So right now we are in a good in a good shape, but we also needed to pass through all the all the complexity of setting up a compliance environment. On the other side, we have to cover all over the world. Not everything can be global. So we needed to collaborate with with China team and we needed to set up locally China ThingWorx instance. But right now we are also finishing the setup and we want to start the validation and all the certifications that are needed. And there are also new challenges that coming
20:01 on our way that are related to this directive. So this is more European new regulations that we need to add another enhancement to it. And and the last thing or the not, not the less important is the user focus. So one thing is the build product, right? So here we have sales save, but to build the right product, we really needed to have a user focus from the very beginning.
20:31 As I was mentioning a bit before, we created our early adopters group. So from the Pi one, I would say when we were already designing the first solution concept, we were collaborating with other subject matter experts from different regions representing the key users to make sure that the solution that will bring will cover their needs. So. So from the very beginning we have their early feedback and we could adopt to it. So early adopters was one part and the other key part was that
21:02 the part of our developers, it's it's also the UX lead. Yes. So we have, we have the full UX people assigned to our solution, even though it's more a backend solution, but it's still important as the usability of the solution. It's a key for, for quick adoption. And during this this journey with the early adopters group. So on one side it was the validation, but then on the other side what was very important is to is to collaborate with them
21:33 through the different phases. So we have a phase which is called user experience phase. So in the moment when we were ready and have the end to end connections, we already gave the access to our users to play with the environment because we needed to know the showstoppers that will come on our way. And we needed also to know where we should have a focus for for enhancement and improvements. And the second step is the first customer monitoring. So this is, I already mentioned with the Windows 7 to 10. So this is the activity that that we are doing once we are in production.
22:05 And to have this group helps us also with the adoption of the solution. Here you can see some example. Yeah. So how the wireframe look like and how then the final implementation goes after it and, and you can just, just see on some samples what, what was the word. This was super important to make sure that the users are heard and we can implement what they want. So the time was spent to to create those wireframes
22:35 and then to ensure that we actually implement something that look exactly like the wireframes. So now let's have a look on how we proceeded with the rollout of the solution, the different phases. So Eva mentioned it before that we were looking for user feedback from the very beginning. But one thing is the user feedback. Another thing is good testing, but there's no such thing like going into production. So what we did is you see this so-called early access phase. This is the remote service edge agent.
23:06 We decided in the very beginning of the project we do not want to have any legacy anymore. We want to have the latest and greatest software, meaning the older agent was out of scope for sure, and we decided to go with the SDK and build our own edge agent. This gives a great user experience, especially for the remote sessions, because they are now instant no longer waiting for a ping or missing a ping because if the supporter is on the phone with the customer, every second feels like a minute.
23:37 Therefore, this was one of the main decisions to go with this new technology and we are super happy with this one. Now, this early access phase was really a phase where we deploy this agent on customer system in a very controlled way because we know that connectivity is key and you can check a lot of connectivity and also software in general in your test environment. But the custom environment is always a bit different. They may have different proxy servers, VPN setups, deep packet inspection,
24:08 whatever they have, and therefore we selected a couple of customers knowing they do not have the easiest setup and we deployed the agent and were able to confirm, yes, we can have a proper connectivity. This was happening the first quarter this year and now you see the second quarter where we are in. Now we do this internal and affiliate user experience phase. So again, we use the production system, we give access to some of our key users, the subject matter expert, they use the system
24:38 and they give us direct feedback which you can then tackle in the upcoming development cycle. So we also ensure that we have a very short turnaround time as much as possible, giving the right prioritization to the different topics. Very soon we go into the first customer monitoring where we have a couple of hundred customers using the full scope and this is then the real proof that everything is running smooth with the new solution. And afterwards we then go into the global rollout for this 100,000
25:11 plus assets we have in the system throughout 2024. And we always said to our user community, we will give you one year to do this transition because there is a lot of organizational changes going on. Many people need to be retrained, but also we need to ensure that the customers that's they allow us to do the transition because even though we say this is a simple upgrade from an older technology to
25:41 a new technology, our customers are very sensitive these days about privacy and security. Therefore, when we announce this change, there was a, Oh, by the way, what are you doing exactly? And then we need to explain in detail what this is all about. How do we secure the connectivity, how do we ensure privacy of the data and so on. Therefore, this is a very big task for our affiliates, all the customer engagement and therefore this one year is required. Here are somehow the key closing notes.
26:11 So our our main approach is always focusing on the team, focus on the people. It's not only the backlog because usually we think we have product features, but the team it's that can make it happen. So creating this open environment when people can collaborate, this is a key for for for success. And also the team needs to work for a purpose. So business needs to be very clear. What is our objective, what we are doing together with the customers, why customers need it, and why this is heavily used because then they can provide
26:44 the solution that we really need and they can work together with us. And knowing where where the critical path is is is on our way also scale. It's very relevant, at least for us. As you could see from the numbers, it's not about the small deployment we need to roll out. We need to change from to ThingWorx in one year. So it's not a not a long period of time for for this post numbers. So here testing the. Requirements and testing with volume. It's a key to engaging with users.
27:15 Without it, it's mission impossible from the beginning. They need to be onboarded so they can adopt the solution and they can prepare because once we finish the development and we go to them, this is absolutely too late. We would we would then spend too much time and having this local community will help us also then to spread the word about what's coming and and how good it is. Established DevOps in the early part. This is, I think Matthew was already mentioning that having everything deployed
27:45 and being able to do it with with one click in different environments. It's a key for success. If I can add also on this topic, with such a big organization, with so many asset types, with so many stakeholders involved, it's just natural that you will have a lot of changes during the project. So automate whatever you can, configuration uploads some, some things around the environment configuration, but also the asset base, etcetera. So invest in the automation early on. This is super important.
28:19 And the last point, but not, not the less important. Have fun. It's a big pressure. It's a long, long journey. It will be for us I think in overall three years. So first building the team that knows each other, that creates the relation. It really helps. Yeah. And then later on, as we have the name, internal name of our project, 3D, so we can really have a lot of fun and even working on some remix till the end of the project. Absolutely. So that's, that's all the content we wanted to present to you.
28:53 Thank you so much for coming. Thank you. Thomas. Thank you Eva, for actually doing my job and presenting the the overview of the project. I'm extremely privileged to work with you on this and having having your support today and in the Project daily. I have two questions, one technical and one operational. The technical question is we I saw that your architecture diagram had exceeded agent as well as ThingWorx agent exceed does support connectivity to ThingWorx.
29:25 Why did we go for the ThingWorx agent? That's question one. The question two would be China has an internal server. I know that Azure also operates on China. Why did we not go with the Azure itself? All right. Do you want me to take this? Okay, so the first one, why we didn't stay on a hybrid hybrid installation. Like the agent talking to fingers, we decided that going native ThingWorx or the stakeholders decided. But we supported that decision, and we explained the technical rationale behind it.
29:58 Going with the native ThingWorx, you will realize those benefits that you can't really do with the old technology that's being based the speed of getting the tunnel open. That was life changing experience for the users. They love it right now. So that's one one thing. And then for the other question, I think you mentioned about hosting the ThingWorx in China that's hosted by third party and it's not part of the PTC hosting, is that right? You want to add something to.
30:32 Be hosted locally in China because of the laws? Yeah. So so here we somehow have been separated, small projects with the China police for they are responsible, for instance, and they are managing them, deploying the cloud with the cloud preferred solution. And then on top we provide them the code with ThingWorx and then calypso and supporting them. For for the role. Yes. And one last thing for the agent, where we went for it, the main driver was the technical possibilities. You only have if the new one.
31:02 But we also have to admit in the old one we have some customizations and we did not want to port them again to ThingWorx. We want to leave them behind us and only go with the new technology stack.