Automotive Trailblazer: How Volvo Group is Reimaging Service and Training with Augmented Reality

00:00 Welcome, everybody. I'm Vincent Banu from Volvo Group, and I'm leading the augmented reality development in the field of service around diagnostic service instructions, standard times and so on and so on. And today, I will try to bring you what we have been developing lately, and it's actually displayed on the floor just on the Xtropolis around first responder application. And together with me I have Mike. Do you want to.

00:33 Hi, I'm Mike Gaubatz. I'm leading product planning and advanced research in Vuforia and I will cover the second part of it, which is basically explaining a little bit what the digital thread is doing in relation to the to the application that that Watson will present to you. And just before we start, I'm really thrilled to be here this morning. We just released a press. We just released the press release about this application. So it's really new. It's just been launched a few weeks ago, but it's officially communicated today. So it's really a great moment today.

01:07 But let me start with a small story there. I'm pretty sure it's natural with all of you to wear seat belts in the taxis in your car. It's it's natural. It's been natural for all of us and it's saving life. We know that the front story here or the true story is that the seat belt has been created by Volvo in 59. So that shows the safety matters for us in the Volvo group and is really part of our core value. You see that statement?

01:40 Basic principle of all engineering is and must be safety. This is also a statement from nobody else than one of the founders from Volvo Group. So you can see once again how important it is. And as a matter of fact, the picture that you see here is the research accident team that has been founded in the 60s. It's a team that is still active and that is analyzing all the accident, doing research, doing analysis around people, behaviors,

02:13 all all what can happen around road road accident in order to improve our safety. We also have examples like we are testing a lot, a lot, a lot around crash test and especially with the late battery electric vehicles. They are important. The architecture is pretty different. We have other types of danger, so we are really testing all those vehicles by doing crash tests and simulating or real condition crash tests. Now at Volvo Group, we also have other long term goals and where we try to solve customer's problem and customer's business challenge first.

02:51 The first one is we want to be 100% fossil free. So by 2030, we want to sell 35% of all the trucks without any emissions. So completely battery, battery, energized battery electric vehicles or fuel cells. And by 2040, we want to be 100% fossil free as a company. So not only the product, but also in everything that we do. The second big pillar is also that we want to be 100% more productive. And here the example is that we don't want to sell just trucks as product,

03:25 but we want to sell complete end to end solution, including service, including uptime and so on, where we will sell payload, we will sell kilometers to our customers. We have example here in the US in California where we have teamed up with the government and the state in order to create a partnership there called Light, and it's operating battery electric vehicles between the two ports of Los Angeles City and coming back on the 100% safe that

03:59 I just mentioned before, we are always thinking that safety is about the driver, about the vulnerable road users and so on. But not only. It's also all the people that are interacting with our product, of course. Service technician. I will come back on it. But in that case, we wanted to be sure that also the first responder can be safe interacting with our vehicle. So introducing those new type of technology, we will introduce new type of risk.

04:33 There are some documentation about it and this is what we are creating today. It's called the Emergency Response Guide. It's an ISO standard and it's describing the architecture of the vehicles. Basically, the the thing is that the first responder, the main problem is to identify what type of vehicle they have and what type of energy they have. So here on the left side, you see. What this document is about. So first, you providing help to identify the vehicle.

05:04 Then show some rescue information like schematic of the vehicle and how it's laid out. And then some procedural information to immobilize and disable the hazard. What we have proposed there, taking this opportunity of those new technology and because it's really new, the battery electric vehicles for them, they are really not used to it. And you should just bear in mind that it's we have more than seven Tesla,

05:34 if I may quote that, in our truck that is down there. So you can imagine the number or the amount of power and energy that is inside. So we really need to help them to understand how those products are made. And we have decided first to embed everything , all documentation into an app. So that's not really new. Some others apps are existing, but the added value that we are bringing here is first helping them to map or to search by chassis ID or Vin numbers and so on.

06:05 So they are sure to get the correct documentation and we are adding on an augmented reality features where for the rescue information it will precisely overlay on the vehicle all the components and the hazardous information and then also for the immobilization and disabling hazard, we can guide them precisely to where they need to act. For example, they have the chassis switch on the chassis to cut off or in worst case scenario, they need to cut cables.

06:36 What is important for having this augmented reality feature on the technological side or the technical side is that in order to make that work, we have the we have made it work for each and every vehicle. So it means that we need it to have digital twins of each and every produced vehicle, each and every configuration. And to date the first version or the launch version of the app is covering 1500 vehicles already. So each and every unique configuration is able to be called by the app

07:07 and directly overlay the correct components on the app. Why we have chosen Augmented Reality and I will switch on the video. So here is just an abstract of the application. If you want to have the full experience, you can go down in the Xtropolis. You will have the full flow of the app, but here it's just showing augmented reality. So first we have used this x ray kind of vision. It's really self training. You can easily understand where where the components are and the how

07:43 the vehicle is. Architecture, as I said, it's pretty natural and self training. And at the end of the video, the the key thing with the this x ray vision and being able to have precise overlay of the components and especially in dangerous situation to remove hazard in that particular case. Here on the screen you can see the first cables. They are orange, of course.

08:13 So it's it's pay attention to that cable. Yeah. If I stop here, you can see that the other one at the back is blinking. So it's really notifying that you need to cut the one that is blinking and not the one that is orange. So it's really adding on information on the physical object to clearly understand what to do or not to do next. A bit of why we have done it now. We have introduced a brand new architecture with battery electric vehicles.

08:44 And for those type of new technology and new introduction, it's always easier to start from the beginning. So that's why we have decided to take the leap and get to to use this cutting edge technology with R. We have also done a staggered introduction on those products. So it means that on all, not all configuration are available at Start. So it's always a way to easily start somewhere, start small,

09:14 learn and continue to improve. And at the end we are following the production so it will just continuously grow. Another important aspect that has been mentioned this morning in the opening keynote is our common architecture and shared technology that we are using. It's really a motto here in the in the Volvo group and is allowing us really to reuse a lot of components, reduce the lead time to create new product and so on,

09:44 and in a sense also improve the the availability of the CAD and the geometrical pieces of the of the truck and for us for creating those digital twins and managing those digital twins, it makes it more easy. There is other aspect why we have done it now. We have because of this new start and this new product line that we are launching. We have used a pragmatic approach. And here I must say that it's it's not only just a technological thing,

10:16 it's also a thing about people where you need to have people that understand your product, understand your product configuration, understand how the the cards are made at your company, understand really how we can approach this because we really need to create this. It was new for us and it was a learning curve. And also take those decisions, know the product to take those decisions. Decision finally to say, okay, for that kind of use case, do I really need to have a full fledged digital twin?

10:47 Each and every screw, each and every part of the truck? Probably not. And that's what we've done. We have focused only on few components that are part of this emergency response guide and started with that. That is also why we have done it now and why we have taken this this approach. And the last thing on on that for that slide is it has been a way also for us to prepare more augmented reality enablers by working with PTC.

11:19 Also so built a best practices like getting to know and use the Vuforia library so the best effects possible to understand how AR is you can visually look at are also get to know and how to use model targets at this scale. We have also automatized all the conversion process and ordering of model target automatically because as I said before,

11:51 1500 trucks to start with is not a small piece of cake. So we wanted to not manually do it each and every, each and every truck. So we have built that. So it was, as I said, the learning curve for us also to prepare the next, the next evolution. And there I'm coming back to where I'm coming from, the service field where we will of course use AR, where it needs to be used or where it can bring value, of course,

12:23 for safety for same reason than the first responder. But we will probably need much more detailed digital twin in order to create those good instructions. Also more advanced diagnostic information to locate the components and so on and so on to build user autonomy and, and also start to reduce downtime. Really downtime is key. You must know that the the trucks, they are tools today.

12:53 It's not like passenger cars. They are really in the need like machines and like manufacturing. They really need to be on the maximum uptime and maximum availability. And we are working today to find where and if augmented reality can really bring value in certain cases. And that's why all this work on this, let's say more simpler, simpler case on the. First responder has enabled us to really build upon that and and know what we could use in the future for those more advanced use cases.

13:28 Why we need to prepare in the correct way as well is that we will not face some people just like that. That is really the number of service point and number of people that we are potentially facing when we are rolling out solutions. They are as much as we are and at the Volvo Group, so we must do it right first time. We must have the correct solution, correct use case, correct information,

13:58 because if we do not if it's useless or if we do not do it right, then the word will spread and we will not. The change management will be much more difficult. And what we are working then for that when it comes to augmented reality, it should not replace everything. First of all, it should be really useful to the point and get to know our service technician, how they are working and and what is really important for them that will help them.

14:30 I can just quote or say that, for example, being able to identify a precise connector or a precise fuze or precise control unit, probably it's the same control unit across the product line, but it's never placed at the same place or never exactly at the same place, depending on the configuration. And if we are able to do that, then we will gain a lot of or start to gain around on the the uptime. It should be easy to use and really to the point,

15:02 easy understanding as we saw earlier on the this X-ray vision and context management and context understanding we should find and should leverage on the the correct way to display information. The second thing, it should be integrated, and that's a topic that we are highly discussing with PTC because our service technicians, they don't want to they don't want to jump in between application and it's already the case today. So we should go the other way around and try to reduce as maximum

15:33 as possible application. And this is a challenge in the AR domain, how to find the correct data model to be integrated and seamlessly integrated in our existing ecosystem that we have today. Quite a lot deployed, as I said, referring to the slide before, it's quite significant amount of technician that we have to manage. So we can't change just like that. And finally, we need to make it robust because the guys, they are working on some time on minutes, they cannot just last and wait for days.

16:06 The solution just need to work like that and that's why we are, I think, collaborating quite a lot with PTC and they are bringing us this added value and this strong experience and our technology. And I will let Mikael go on and explain a bit more on what we have been working. Thank you very much. So I would like to take the chance to talk a little bit about the digital thread.

16:37 You heard that term several times. I would I would guess during the during this LiveWorx and you might have also seen this diagram several times now. So, so so this is this is the infinity loop that you should be familiar with. But there is this add on down there and up there that basically says what roles plays. Vuforia Vuforia takes data from the digital side and displays it on the on the physical side. And it also is able to inform

17:07 the digital side from from different informations. And I would like to dig with you a little bit deeper into this green arrow that basically says I can take data from my backend system, from my production system, from my service system and use it in R, just like Vincent has just pointed out, 1500 trucks is nothing that you would put manually into an R application. That needs to be something that that that that is automatable.

17:38 So I've chosen as a as an example a video from LiveWorx four years ago, I think, where we showed where we showed a inspection procedure just as an example, to say, Hey, this is general, this is general generalizable for a lot of applications. You see, we are kind of acquiring what we call the model with a model target. We are showing a blue outline. We are showing a procedure.

18:08 We are showing what we call hero parts. These are these orange parts that need to be inspected. So. And we're showing additional product information. And you would you would say all that data resides somewhere in my backend system. Doesn't matter if it's a manufacturing process or if it's a service process. That data is somewhere. I have most of that already there. So so if you remember, if you remember from that slide,

18:40 we would need some some kind of geometry that describes the object that we are talking about. We would like to have the procedure and there is various various representations that we that we need to extract from from that data. I mentioned the part hero objects. That's an invention that we use internally. But what you could imagine is that there is this one part that you want to attract the attention to. This is in most of the cases, we want to find a part that needs to be inspected.

19:11 You might want to find the part like Vincent just mentioned in all these configurations, this specific sensor, this specific connector might be on a on another side. Sometimes it's even inside. So in my little case here, it's this yellow axle that we want to visualize, and that's if it's internal, it's not visible. So we need to have a 3D model of that hero part. Obviously, that that that needs to come from somewhere and and obviously from the from

19:41 the backend system. There is another thing where a 3D model plays a big role and that might not be so obvious and that's called occlusion. People who are working with AR know what that means, but I think the static images shows shows what I mean, if I would want to display the the screw driver appearing behind the real physical object, I would need to make sure that because the physical object is represented in the in the in the video image,

20:11 I need to make sure that only the visible part of the of the screwdriver that I'm rendering or that I'm displaying is is shown. If I would just display the screw driver on top of the screen, it would like look like the upper image, but it should look like the lower image. So I need something that is hiding the part of the screw driver behind my physical object that is that is already there. So I don't need to to render anything because it's it's already in the image.

20:41 And, and, and what we need for that is a model, a 3D model that is basically rendered like a ghost that that covers the, the screw drivers so that it appears to be to be hidden behind behind the object. So that's occlusion. So another 3D object, 3D model that we need for AR in a specific version. You can imagine that is a very rough, a rough representation. It doesn't need any structure or things like that, but it's a, it's a derivative of,

21:14 of what we need. Then you've seen this blue effect before. But usually if we are detecting an object, we want to show that to the user somehow we would now, now it's moving. Okay, so it doesn't behave like I was expecting to. But, but anyhow, in the middle image, you saw basically the, the, the acquisition effect. So now the computer vision has recognized the object. It's now clear to everybody in this room, hopefully, that the computer vision knows where it is. But again, we are displaying something on top of

21:46 the physical object that represents that, that is the geometry. And then if we want to make sure that the user understands that all is the computer vision is still working, we want to have maybe a little outline. So again, we need a 3D model of the object just to for the user interface to make sure that that we are communicating correctly. So this is the first motivation. We need 3D models from the back end system that you have in your production system

22:16 usually and, and want to use them in. AR But, but they cannot be used 1 to 1. It's they are too complicated in most of the cases and, and they might have too much information in there that that is that is not useful. So so we this is the typical things that we do in this digital thread for AR. If you want, we do model conversion, conversion and cleanup. So if something is from a geometric point of view, not not clean enough,

22:47 we might want to repair that. And that's the important thing. We do model simplification and we do what we call a model target generation. I will talk about that a bit later. So simplification, I've shown you, I've shown you the occluder already. That was this ghost object that we are displaying. And I and but there's other things like the hero objects that you want to have crisp and clear so that people understand what what the part is that they.

23:17 They should look at. So there's different ways of of simplifying these things. And and this is this is what the digital thread is, is able to do. It knows that you want to you want to recognize an object. So it creates the occluder for it. And if you it knows that because in a procedure you want to point to a specific part, it would basically simplify that. Here's an example of a of a car that was basically 16,000

23:49 polygons when it came out of the engineering software of of of Creo. And it had 4500 draw calls, whatever that means. And and the simplification is basically reducing it down to 200,000 polygons, which is a size that is capable so that a mobile phone can display it in real time. And you see visually there's almost no difference between those two things. So if you do it right, if you are if you're doing the simplification correctly,

24:20 then then you the visual quality is, is is is perfect for the kind of application that we are that we are doing. Here are two additional aspects that I that I want to mention. Ryan Orvell showed already a state based model targets in his talk. State based model targets are mainly used if you have an assembly process. So in your factory, your products might be assembled like this.

24:53 This pump here has different assembly states, and in each of these assembly states, we want to recognize the object. So we need to basically create something that computer vision can understand the object in this different states. So what we're doing is basically we are training neural networks or I if you want, in a general term that is basically distinguishing in a first step between these different states. And then it's basically giving us also the position that that the object has.

25:25 As soon as it recognizes the the specific state, it would basically provide us the correct position. All that is derived from from the from the model geometry, but also from the model states. So again, something that we need to provide from the back end system. So for example, an MPM link process plan would be a perfect input for creating a state based model target because in the process plan you have you would have all these production states already kind of

25:58 included. You would feed that into this process automatically, create a state based model target that basically simplify is simplified and can be used in VR. And the result would look like that. You might have seen the video already in in Ryan Orwell's talk, but that's that's basically how we track or and recognize an object while it is assembled so so the AI is recognizing the different states

26:28 and we'll always make sure that we tracking from the right from from from the right part while the worker is is doing its job. A last aspect that I want to mention in some cases like in trucks, it's it's a parent model. Targets are recognizing the object but from where. That's always a bit of a of a question if this is a little thing on on the table. Okay it's clear that I want to recognize it from everywhere around it.

26:58 But a truck and the service technician stands kind of in arm length in front of of the engine and recognizing the whole the whole truck from from that short distance is something completely different. Also from a especially from the computer vision point of view. So what's what's important here is that there should be some positions defined where the service technician would be. And that's usually something that that a good service engineer would basically be be aware of.

27:29 So they would they would figure out can a service technician, technician even be there and do the repair. So so it's it's already again data that might be in your in your back end data. And I tell you that's an important information that we can use for the application because if we if we transport it through the digital thread, these are the what we call the user boxes. These are the definitions that the AI is then using to train the neural network to recognize the object from from these specific views.

28:01 As a result, you see, we are we are starting the tracking immediately. If the if the cab is open of the truck and the service technician is supposed to stand very close, we can recognize the truck even. Even if. If. If. The service technician is is standing where he's supposed to stand. It doesn't make sense if he needs to go four meters back just to to to start the experience, that that doesn't make sense. And even even more obvious is if the service technician needs to go into

28:34 the service pit. So it's under the track. There's no way to go four meters back. There's unless he's digging a hole, maybe. But so, so, so clearly there is something that we need to make sure that are is recognizing the truck. Even if you are down there and look at at a part of the part of the truck and and to prove that this is possible. Here is an example of a Volvo truck that is basically basically ready for for have

29:04 as service procedure being displayed from from from that position. So as soon as we have these positions and they should be in your in your back end data somehow we can we can use them to to to generate the correct things. So to sum up, I don't know how how are we doing with time to sum up. I think I think the conclusion that you should take away is I believe that all the data is there that we need for for our in your back end system.

29:35 It's there is and we can use a lot of of that. We use the geometry, we use the procedures that you are creating. We're using the service positions that that that you are creating. I think it's important to understand that I would work as good as your data is. So if you if your data is not in in in a in a good situation, if there is a general purpose CAD model and there are so many variations, but you you have not represented them then with, with basically be in lower quality.

30:09 But as soon as as as the data is in good shape, I think we have the big chance to basically automatically convert it to the data that we need for our and and create scalable applications, as I showed you today. Thank you very much.