Updating Your Creo More Frequently

00:00 My name is Matt Burr. I work for solar turbines and you are at the updating your Creo more frequently session. So in the world of SaaS, I know there's a lot of interest in SaaS. This is not all about SaaS. This is for anybody that is on prem and this is just kind of a journey that we have been down since 2018 timeframe. So this is our agenda. A little bit about myself, about solar turbines, what a gas turbine is. I'm going to recap a little bit of what what I presented in 2019, and then I'll go over some what our environment is and then any questions at

00:34 the end. So about myself. I graduated from Suny Morrisville in 1992. I've been using PTC software since 1993, so I've been using it for a very long time. Pro Engineer ten was the first version I've been on. I've been introduced to a bunch of different applications, design programs, worked as a design engineer for nine years and then have been in system administration for the last 21. So I have a little bit of an experience of how a user uses a software plus how we administrate the software for the users. So solar turbines itself, we are headquartered in San Diego, California.

01:07 We are a subsidiary of Caterpillar, Inc. We are one of the world's largest manufacturers of industrial gas turbines. There are 16,000 units and over 3 billion operating hours in more than 100 countries. We build some small stuff. We build some big stuff, but our small stuff is still pretty big. And we have about 8000 employees worldwide. So what is the gas turbine? I just want to make sure I explain this really quick, but I'm not going to do this. There's a video that follows on. We have nothing to do with solar panels. We have nothing to do with wind turbines. We are a industrial gas turbine company. So what we do.

02:54 So that's a little bit about our company and you can kind of see the product that we do. It's not really small, simple parts. There's a lot of complexity to our geometry. And so we've struggled with how do we manage these big complex things and how do we manage with Windchill and some other things, and how do we do our upgrades for Creo? Because we got tied up in our in our implementation. So legacy, this was our process that we did prior to 2018. So we would do long project updates, 9 to 12 months,

03:25 so we would decide we're going to do an upgrade. It would take us 9 to 12 months to do that actual upgrade. We do lots and lots of testing and I do the quotes we did thorough administration administrator and user testing and as thorough as you could possibly get because how many here have missed something when you did your thorough testing? Okay. So we'd select a build, we'd be on that one. It might be nine months old by the time we select that build because you have to wait for PTC software to be super stable and you pick that one and it's nine months old before you get to it.

03:57 You might identify some bugs during testing and that might scare you from going to that build. And so you don't go to that build. So you might delay it a little bit longer and you might wait for a fix, you might not wait for a fix. And so that was kind of our strategy that we had for many, many years. Once we did get it in production, we had that set it and forget it. You would put it in production and then you wouldn't touch for 18 months because you might break something again. So we didn't change anything. So we would leave it like that. And we may never put PTC bug fixes, so we'd report them,

04:27 but we'd never put them in production. So we'd go through all that work, report them, we'd hammer PTC. Why can't you fix this? Your software is terrible. It'd come in and then we'd get thanks and then we'd never put it in production. So we did lots of incremental customizations, so we had no problem customizing our code. We had no problem breaking our stuff. We had no problem trying to fix our stuff, but we couldn't do it from the main CAD software. So in 2018, we kind of stepped back and said, Let's look at this from a different manner. So we went down a process where we started doing upgrades in every 6 to 9 months.

04:59 So a major upgrade was 6 to 9 months when we decided we were going to go. It was a 6 to 9 month project, total length, again, lots of testing, again, that thorough administrative testing. But once in production we were proactively updating. So every three months we were doing maintenance releases. So unlike that, set it and forget it that we had. We were constantly moving. We were on the move constantly. When PTC was releasing something, we were testing it, we were building it. We were trying to make it work as quickly as we possibly could,

05:29 and we transitioned from those random ad hoc changes. So we would do a change here, We do a change there and we might do 1 in 3 weeks. We might wait four months for the next one. We started scheduling and said we're going to do things together. Our enhancements, the CAD enhancements, we were doing them all at once. So we really changed how we did things. But why did we get there? What was the precipice for us doing this? Because we could have. It had been working for years. Why would we change so near the end of our Creo three journey?

06:01 We were really happy with what we had. It was stable, it was mature. We weren't making a whole lot of changes. And then use your starting reporting issues. And I don't mean like, hey, you know, this is a little bit buggy. It was, Hey, Creo is taking 5.5 minutes to launch. Hey, this model is taking a really long time to regenerate. And it didn't do this last week. And we we did all you know, we're calling PTC. We're looking at different things. Some people could recreate it, some couldn't. Well, the root cause was actually a Windows update. A Windows update had come in and it caused a performance issue with Creo three.

06:35 We didn't make the change. Microsoft made the change. It got deployed. We never tested it because we never thought we needed to look at it and it actually caused an issue. At that point we were at we had to make a decision, okay, do I do a maintenance release of Creo three or do I go to Creo four because other versions of Creo three that we had smattered around because again, we had that stable environment where we would still give, well, you know, this isn't working, but it works in this build here. You use this build and this works in this build, so you use that build.

07:06 So we wound up with this ecosystem with all these different builds throughout our system because it would fix one nuanced issue. So we had to make a decision, What are we going to do? So we we identified that, hey, that's stable environment. Yeah, it's not that stable. So think about all the things in your environment that are changing around your Creo install windows, updates, hardware changes, graphics drivers. You're getting a new piece of hardware in your network, changes something in your system is changing that is totally outside of your control.

07:37 You can't change any of that. So we went back and said, You know what? There's no real value in us doing it the way we've been doing it. So we had to fix other issues. Multiple builds were in place and we said, What we thought we had as good was not good. So this is what put us on the path of how do we do this differently? How do we get to that stage where it's 6 to 9 months? How do we do the three months? And that was why we did it. So when we first did this, lots of people saying,

08:08 why wouldn't we, you know, why would you adopt it? You know, what's working is working. Yeah, you ran into this one issue. Why does it matter? What if we miss something in testing? That was the first question we get asked. It's the first question we get asked in everything. What if you miss something in testing? Don't we already? What's going to be different? If I upgrade every three months, what's different than me upgrading in three months and missing something than me upgrading, missing something and then never updating? So what's the value in not doing that change ? Slow and steady wins the race.

08:40 Remember, you just kind of set it. You forget it and everything works well. The problem is agile methods are everywhere. Why on that? Why on our primary tool that everybody's relying on, Why aren't we using that? Why aren't we trying to do a new process that why not our primary design tool? We're not doing that. But what about all the changes we would need to do because we'd have to change every three months? But again, you're stable. Environment is changing around you constantly. So what you think is stable and you think you have control over you don't.

09:10 And I'm sure everybody here has had cybersecurity things and changes that you've been forced to make in the last year. Raise your hand if you've never had to make a cybersecurity change in the last six months. What if it broke your CAD software? What do you do? You're scrambling to make it work. So this is why we kind of went down this route of let's look at this from a different perspective. So we needed to be more proactive in upgrading because our stable environment, we realized, wasn't that stable.

09:41 So how have we made it happen? So we took inventory of everything. So when we first went down this journey, that's 6 to 9 months, it was 12 because the first three months we took inventory of everything and I mean everything. What are the config files We reviewed all of our internal documentation Was our launch mechanism working? Nothing was off the table. And this is an important key phrase. Nothing was off the table because we all have this mindset of there's something in our company that we can't change. Why? And that was the thing we said.

10:16 Nothing is off the table. We had to be looking at this as nothing is off the table. So the first thing that we did that we struggled really bad with was we had no way of really testing anything. So we actually had to go down to the fundamentals and define a QA or a development, a QA and a production environment. So we knew that we had a QA, an environment that our developers could test in. I have the meme in my cube of I don't do testing, but when I do it's always in production and we had to get out of that.

10:49 We had to get people testing in and into environment. We then had to test it in a QA environment and the reason why we had a QA environment is we weren't just testing our changes, we were testing our promotion process because you would inevitably go from into QA and you test and it's not working well, why is it not working? Oh, I missed a file. Well, how many times have you promoted to production and gone? Oh, I forgot a file. I'll add this one. Oh, I forgot another file. Yeah, this one. Oh I forgot another file. And we had gotten to the point where we had done that so many times we had to define

11:19 a process to do that. So we went back to that point. We combined our build customization updates with product increment changes and we improved our planning for updates. So that's what we did back then, and we're still using it today. So this whole mechanism that we've been using, this whole thing we've been using and it's worked fantastic. We did Creo seven and Covid was here. And one of the important things to understand is when we did Creo seven,

11:49 we were work from home. We were 100% work from home. And we were we were had enough confidence that we could go from Creo four to Creo seven with nobody in the office and they would be working the next day. So it's an important it was a real thing for us to to see that value. So is it working? Yes, it is definitely working. It's a learning curve and it's always and it isn't always easy. That's one of the key things to keep in mind. It creates more work for system administrators, and that's probably tough for a system administrators to hear.

12:21 But there's a reason why there's value in it, because it's smaller chunks of work, because you're doing smaller incremental changes. You're not trying to do it all at once. You've suddenly pared things down into a much smaller list. It's on a calendar. If you do any projects, you talk to the business about any projects they do. It's all on a calendar. When's this going to occur? How is this going to impact? And because it's on a calendar, you can actually start having communications saying, Hey, I'm planning to do an upgrade. What do you have going on?

12:51 Is there a release you have going on? Is there something else going on? You start talking to other IT organizations. Do you have a server outage at this point and it suddenly starts creating this this world of communication. So this says we're scheduled out to 2021. This is from my presentation in 2019, but we were scheduled out to that point. It helps to set attainable goals and incremental updates to our launch program. So we continue to improve our launch program. As we started going through that, we added multiple language support. We've added multiple sessions connected to Windchill.

13:21 Again, we're learning how to leverage Agile methodologies more and more and more, and now we're scheduled out to 2025. So today my users can go to our calendar and they can see what's our at least tentatively planned release schedule for the builds of software so they know what's coming and they can look at it and go, Oh, that's going to conflict with whatever we have going on. And we actually have moved our schedule around or added stuff into production based on their requirements from the business. So what have we seen since making the change?

13:55 So we have actually wound up with more stability in the system. Yes, we have more changes, but we're seeing fixes and action, high ticket counts. When we go to initial build, that's our initial release. That's normal. But they drop off. But we've been getting better user feedback. They're actually reporting issues. When you're on that set it forget it mentality. Why are they going to report something to you? You're not going to do anything about it, right? So why why do they report it? They just complain that PTC doesn't do things or it doesn't do stuff

14:25 or sysadmin doesn't care. We've actually had better engagement from the users. We even see we've even seen improved bug fix timeline because remember that whole thing we beat PTC fixes, fixes, fixes. They hand it to you, you never put it in production. Why would they bother putting in production if you're never going to put it in production? So we've been able to work with our with the the TCS, our Tsam to kind of say if you get us this, here's when it's going to go in production. When they send us something, they know we're going to install it, we're going to test it, we're going to confirm that it's fixed,

14:55 we're going to put it in production. And it has helped our relationship with PTC. So before I move on, are there any questions Can you use a mic, if you don't mind, please Yeah. How did you. Overcome the OpenGL slash Win32 GDI issues when you're working offline remotely? Uh, which issues are you talking about specifically?

15:26 Large assemblies have a hard time transporting the graphics over. Oh, we do. Most of our users use remote or have their own computers at home. A small laptop and onto a workstation. Or they'll either RDP into a workstation or we give them technical workstations and they work from home. Okay. Yep. Thank you. Yep. All right. So let me go into our environment right now. So we are on Windchill 112 and 210. It is a global deployment. We have our PTC career view 9.1 We're

15:58 on Creo Parametric 7.0.1.0. It is a global deployment. This isn't just a San Diego thing. We actually deploy across 11 different servers around the globe and we have 1000 unique users. So in 2019, if you'd come up to me and said, How do you how are you guys deployed? We had a network install. That was what we did. We had it sitting on a server and everybody loaded it from the server. We had our customizations on there. We had a small launch script that was sitting on the user's machine and then we had some session log data,

16:28 but all everything was pretty much server based, so our architecture looked pretty much like this. So we had our file server, our Windchill server, the user's computer. When they launched Creo, it would actually bring the data down. They'd run it on their computer. Data was shared back and forth through PLM and then it was put into the log. Data was put back into the server and I'll explain why we do log data, log data in a little bit. So this is what we were doing. Everything worked fantastic. And then I don't know, around 2020, something happened. Oh yeah, Covid hit.

16:59 So all of a sudden we went from, Hey, this is working fantastic. Oh, now what do we do? Remember that comment where I said, we're all working on local laptops? It doesn't work too well over VPN when somebody is working from home, performance is pretty bad. So then we started getting the questions, How do I work from home? Oh, there's a cybersecurity issue. Okay, what's your cloud strategy? What's going on? I got this issue and I got this issue, this issue and this issue. And you just keep hearing all this stuff and eventually you just you can't. What do you do? I mean, you're at that point and you're struggling.

17:30 And if you're trying to figure out how do you do it? And we went back to what is our strategy? So our strategy was we started breaking them down into small chunks. So we worked on one one set of issues. And then we worked on our next set of issues. And we worked on our next set of issues and we just slowly drained the issues into these different sprints that we were doing over a three month period. And these are reflective of some of the changes that we made.

18:02 So we did enhancements. We did build changes. We did a major release over this this time period, things that you would think you couldn't do. But because of the new strategy that we had, the issues didn't go away. They just became more manageable. When you break them into smaller chunks, they become a lot more manageable. Those big maintenance releases that seem like you can't do, they are manageable. So based on that, we changed. We changed everything. We now use local Creo installs.

18:35 We would we were never going to do that before. That was if you talk to my manager in 2016. We can't do it now. We do everything through a local install because that mindset of nothing was off the table. So since we've deployed it, these are some statistics from 2019. So we've had 7 or 11 different incremental changes. We transitioned from network to local installs. That was a forced one we weren't real comfortable with, but we did it. We figured out a way to do it and we and we did it in a very quick process.

19:05 It took it took us three weeks to get everybody moved to a local installation. So 1000 users globally all switched to local installs. We did three Creo four maintenance releases. We did Creo four to Creo seven. We did three maintenance releases of Creo. Seven We did bug fixes, we did a Windchill upgrade. We did a Windchill upgrade. We've done cyber security updates. All of these things we've had to do. We figured out how to do because we were more comfortable with a more rapid changes.

19:36 So we've even gone back or gone to look at what's our dream environment. So cloud is the word. Everyone loves the word cloud. Our first thing we need to get in cloud is our Creo and how do we get our Creo into the cloud to make it effective? And so we're still going to plan on leveraging our local installs. We're not planning on moving everything to the cloud initially, but what we want to do is figure out how do we get our customizations into the cloud. But our first thing that we did, which will go in production in a few weeks,

20:07 we're actually moving our local cache to one drive. Something that we didn't think we could do, but that's our first step. We're going to try dabbling using one drive for our local cache and our users config profile we've struggled with because we're working from home. Somebody gets a new computer. Oh, I forgot to upload everything, I didn't get my config pro. All my customizations are gone. We're going to get rid of that by going to one drive. So again, we're just we're thinking outside the box. We've had more flexibility to think outside the box.

20:37 So we're going to continue where things are. We're going to have all of our system config files on the server. We're going to have a home directory and all those things. All that system stuff is going to remain the same, but we're going to start moving little bits and pieces over time. And that's what we've learned is don't do it all at once, kind of break it down into little pieces and you will slowly get there. That whole agile mentality. So back in 2019, again,

21:07 we have three unique major environments. That was it. We had a TNP and MFO and a tool design environment and they were for business needs and they had to have what they had. So again, nothing was off the table. So it became a transition for us. And we're going down to one unique environment. If you want to have some fun, go and ask your users to give up their customizations. It is not an easy thing to do, but there's a real value to your company to do that because what do you spend

21:40 the most of your time checking when you do an upgrade your stuff? Your stuff that you've created. And how often does it get used? Ultimately, we came to an agreement on many of these areas. There are still a few areas where we have disagreements, and so we're allowing them to do those those few items. But I'm down from 252 differences down to 32, which is a significant reduction in the amount of work I have to do and I have to manage. We have 11 unique license configurations.

22:14 So how do you manage all that? So that's another one of those how do you manage all the variances? So we've started to leverage the BSF file. So for those of you that have multiple licenses, you can leverage an environment variables instead of having it in your BSF file, you can actually leverage things as environment variables. It will make your process simple. Us doing this is again what allowed us to move things more rapidly. I'm not I don't have to have multiple installations of Korea with different license configurations. I have one and I define the license on launch.

22:50 So again, however, one size isn't going to fit all. We know that. So every system has a standard config option. This is what we consider our out of the box or base install. We have a common enterprise configuration. We have our optimizations that we allow people to do and then the users personal config. All of this stuff is things that we allow them to do. But again, if I should have brought up the graph that I had from 2019, there were three additional boxes on there from 2019.

23:20 We've simplified down to a much smaller subset. And so the last thing that we've been looking at is our launch script. So we had a what I felt was a stellar launch script. And you'll see it's an image on the left hand side and that little dialog box is whatever user saw. And the first time they started there, the habit was, Oh, I got to pick the box. And so they would check every single box. Whether they needed it or not. It just human nature is. I got to check the box if you ever do the.

23:52 Has anybody here read a Eula before? How many people have checked that box on a website? Get out of my way. I just want to continue moving. This became our Eula for them. And so they just checked whether they needed it or not. And we stepped back and said, Wait, that doesn't make sense. Most of our users don't need that. So we've done a redesign of our launch script. So instead of them getting that entry point you see on the left, they're now getting the entry point you see in the middle. We've spent a lot of time on our launch script. It's been a year project to go from there to there to make it work.

24:25 So our launch script is set up in such a way. We do data collection on start up. From there, we launch an application that we have which is called our Solar Agent. Our agent is our tool that does lots and lots of little things. And I'll show you a little bit of what it does. So it's a monitoring app that we created, that little monitoring app. It manages the local installs, it goes in and it makes sure that that local install is on the computer. If it's not there, it goes and installs it. If I'm going to go to the next build, it's going out and saying, is there a newer build that they're going to

24:56 Yes, I'm going to download it to the user's machine before they run it. It's going to be there. So this little application is doing all that development. It's doing all that pull down of information. It's monitoring to make sure that the user hasn't done anything they shouldn't do and that Creo hasn't done anything. It also goes in and it says, Did somebody go in and delete parametric X out of their install directory? It will go and grab it off the gold copy and put it back on the user's computer. It's our way of kind of monitoring each local install for each user and doing it from a simple utility.

25:31 And all. Once they exit out, we have a little questionnaire that the application does. Did it crash? If it does, we'll have a little pop up. We ask the user why did it crash? So we collect information and metrics for why we have crashes that we do and we have data collection on exit. So why do we do why do we have a monitoring agent? What started in the first place? Number one request we had when we went to Creo three is I need to be able to launch two sessions of Creo connected to two different workspaces.

26:01 You can't do it from Creo three. You cannot do it. If you're connected to the same cache. It's not. You cannot do it. So we had to come up with a way to allow people to run multiple caches but not go crazy and start, Oh, I was in this cache and I did all my work and then I went to this cache, connected the same workspace and all my work is missing. So we had to come up with a way to manage all that. So we created a utility that does it. So it's a very complex issue, but that was why we did it. And so we refer to it as our I refer to it

26:31 as our little clippy helper that sits there in the background. And it's just making sure that everything is working for the for the user. So some final observations and suggestions. So we install the software, install it all. Disk space is cheap. That was the other thing we realized. We kept saying, how can we make it smaller and smaller and smaller? Why just install the whole thing whether you need it or not? Install it because you might need it someday. The launch script is important. It's one of those that if you can take the time to develop a launch script,

27:03 it will help you in the end. Our launch script, our agent. It does all the work for me. I don't have to sit there and monitor every single user's computer. I let the application do it for me. It's 6000 lines of code. Collect data if you can. We collect it on launch, and that includes their trail file. And why do I collect their trail file? So remember my comment of we had 252 customizations we're trying to do. I actually wrote a data miner that went into every single trail file and looked

27:33 for each of the individual map keys and reported back their usage. And it was really nice to be able to go and see that, hey, in this last three years, these 27 map keys have never been used. I can get rid of them. I'm not relying on a user telling me whether or not they use it, because if you ask them, they're going to say, Oh yeah, I use it. I actually had data and metrics I could go back and look to. So collecting data is really, really important for that sort of stuff. In the end, make sure it's flexible so we test and evaluate different maintenance releases. We are constantly looking at making changes.

28:04 So we have 710 in production. 711 is set up so we can start testing as soon as it's available. It's downloaded. I have 711, 904 zero and 10.0. All of them are installed. So I can test any way and do all the different permutations that I want. From my perspective, I can test them against an environment, make sure everything's working and I can start testing early on. So right now our code is now 8000 lines with that little increase. But that's where we're at. So any questions?

28:41 So intrigued with your the dual session situation. We just told our users no, we couldn't do it because of the cash situation. And I kind of understood that at least from PTC. Maybe it's a just a rumor that they're not really supporting that any longer. I'm curious, in your situation, do you ever have users that actually deal with data corruption issues if they're actually using dual dual sessions at Creo? No, because they're in two separate cash folders. Okay. So we actually set them in parallel cash folders.

29:12 And at one point in time we had limited to the number of caches they could have with our new launch script that was moving to one drive. We're letting them have the they can do whatever they want. They can do as many as they want because each cache is tied to their computer. That makes sense. So we have a computer named Cash one, computer ID cash two, and then they can do whatever they want to do. But our our actual launch script and gives them information on what what server was connected to. So if it was connected to our production or our test server and then every single workspace that was ever connected to or so you know,

29:44 which workspace they had connected to, so they from that interface before they even start, they can see whether or not they had connected the same workspace to multiple caches. Okay. All right. Thank you. Yep. Do you put the cash on one drive as well? We are right now, yes. Doesn't that impact I mean, does the download of a large amount of data. To your cache and then uploading it to OneDrive. Impact Performance. No. So. So keep in mind one drive is just a folder.

30:16 It's just it's still on your C drive. It's just whether you synchronize it up to one drive or not. We're not it's not in the cloud. When I do a download, it's downloading to my local C drive and then it will cache it or it will synchronize its way back up to the one drive location. We're still testing it. I may come back in 2024 and say, Yeah, don't ever do that again. That was a bad idea. But we've been testing it for about six months now. We've had three instances where data corruption has occurred and it was 95% of them was one user.

30:50 Like one user was able to corrupt it on a daily basis. And so we have said we need to sit down and understand what you're doing. But right now we needed a solution that would allow us to be more portable. And we've we've just struggled with this work from home and people getting new computers and making sure they didn't lose something. So this is our first attempt to see if we can improve that. So I guess that I'll report back and let everybody know if it's if it's working or not. Time is up.

31:21 Sorry. You okay? You got time for another one? Yeah, I do. Your startup script. You said you. Update configuration with the startup script. Does that happen prior to the running of Creo ? So they wait for that to happen. Well, as far as like the config files and things like that or so, the config files are almost instantaneous. They don't, they don't even notice that one. The when we stage a new version though.

31:53 So we're on 710 right now. In about two months time I'll change our launch script. There's an XML file that it reads to tell it that stage 711 and it'll use robocopy and just start slowly trickling it down to their computer. So as they're running 710, it's just going to update in the background. They won't even know it's they don't even know it's installed until the next time they launch. We've switched to 711 and it's immediately on 711. It gives us that flexibility.