Tricky Bits with Rob and PJ

Insomniac Leak - What Happened and Where Do We Go?

• Rob Wyatt and PJ McNerney • Season 1 • Episode 2

Share episode

Enjoying the show? Hating the show? Want to let us know either way? Text us!

On December 20th, Insomniac Games was hacked and had games, financial, and HR files leaked to the world. Having both worked there, Rob and PJ dive into the particulars of the situation and analyze where things are and where they might go from here.

Ierengaym. com ierengaym. com

PJ:

So some hackers got a hold of the latest files, latest game files from insomniac. For Spider-Man, I believe, as well as Wolverine. And then put up a bid to say, Hey, if you want these files, Sony included, you wanna pay us$2 million? As far as I can tell, no such,, ransom was paid and the files got leaked out to the public. which was representative of, I recall correctly, of about 98% of the total files for the game. It's a story we've heard before in terms of, uh, people getting access to content and then running this kind of, uh, ransom. I think it hits home for us a little bit because we know a whole bunch of the people that are there.

Rob:

We worked there

PJ:

We worked there.

Rob:

I actually went on the tour and had a look at what was leaked and it's far more than just the files for Spider-Man and Wve Reed.

PJ:

Oh, okay.

Rob:

those files are there. I went to look because it's, there was a lot of personal information leaked and I just wanted to see if my information was in there and it isn't, and neither is yours.

PJ:

Oh, thank goodness.

Rob:

there is some old photographs of us. And is offer letters which go back that far,

PJ:

Oh wow.

Rob:

most of it tends to go back to about 2010 ish. of it's later, some of it's a little bit earlier, we were and in your case gone by then. Not really anything for us, but the entire source code's there, all the source code for the engine, the builders, the tools, everything is there, all the assets are there. Maybe 98%. There's a lot it seems like everything or nearly everything is there. that's bad in itself. And this happens with a few games. one's much worse because all the financial data and all of the HR data is also leaked, and

PJ:

Ouch.

Rob:

there's contracts there. There's contracts for my company that's now defunct. Uh. contracts go back all the way too. So of detail, but not a huge amount. For some people it's real bad because there's I nine forms with photocopiers of passports and things like that.

PJ:

Oh.

Rob:

salaries, bonus calculations, bonus payments. So., way worse than just we leak the game. frankly, I'm surprised the game site doesn't happen more often. It's, you think so many people have access to this and so many people from home or VPN access and I'm just surprised it, it doesn't happen more with these big games. As an ex insomnia, like I'm kind of disappointed that it took them 20 minutes to get domain access from when they

PJ:

That.

Rob:

when they got domain access, it was 20 minutes. So there was always a lot of smoke and mirrors for like, this is secure, that's not secure. Um. You can access this, you can't access that. And it was more frustrating than anything'cause you need access and I can't get access to it. And it was a whole process to get access it. It's a shame that from the top, it was wide open. Seems to be from the start. I don't know whether they left something wide open or whether they got a password. The details of how they got hacked may never get,. aau, Sony Insomniac in internal investigation, which is going to give us any details. It's like, what actually happened? Like how did they get in? Because that's probably damage in itself and could give away more information. a doubt the, uh, the Recit group is gonna tell us because they probably use the same thing on multiple places. So. I doubt we're ever gonna know how they got access in 20 minutes.

PJ:

Well, let's start taking this, this stuff a piece at a time, starting that last point. Sony has owned Insomniac for quite a number of years now. If I recall correctly. I think they were bought, yeah.

Rob:

Joe in the Spider-Man game or

PJ:

Yeah,

Rob:

So it's been a few years. Yeah.

PJ:

it's been a few years. Very typically when you have an acquisition of that nature, there tends to be this, uh, hey, we're gonna like pull all of your it into the centralized place, which is gonna, in this case gonna be Sony. I don't know one way or another, but typically when you do that, you kind of get put under the umbrella of whatever that security is going to be, which includes maybe at least all of like Sony Computer Entertainment, uh, as well as possibly if they are more centralized and any of the Sony Pictures stuff as well. So, I'm, I'm curious, like, is this actually a factor of They got brought under the umbrella and there was some missing bit that was not covered or, you know, were, they kept at arm's length for a long time and kept all of the same systems running, which could have exposed in, a security issue that way.

Rob:

Yeah, that's the sort of thing we're not gonna know. I don't think

PJ:

No.

Rob:

No one's gonna be like, this is exactly what happened. And it's a shame. I wish it was almost law that you had to detail how it happened. It's like, look at like an NTSB report for an airplane crash. an independent body that just looks at the facts of like, this is what happened with point fingers as we need to at anybody, even ourselves. it would be nice if there was some sort of independent body that would do that for. Sort of hacks because it still seems like smoke and mirrors without saying what happened and what they did to, uh, fix it. You still have to suspect that there's gaping holes there still, because if you'd have asked insomnia two weeks ago, what's your security like? They'd have been typical PR speak of like, it's like WorldCast, security encryption, blah, blah, blah. And it's like, you know, all, everything that everybody always says before they hacked. and it's all just buzzword bullshit if you ask me, because if you don't use the tools properly, it's not secure at all.

PJ:

Going back to the other point of why doesn't this happen more often, uh, I would say that, I'm surprised this doesn't happen more often in the case of movies, especially, uh, with games, if you have, a royalty agreement where it's like, Hey, you're gonna get paid based off of the sales of this game. You're actively disincentivized to put the game on the open market. Because you're not gonna get a financial benefit from it. And if they catch you, you know, let's say you'll go to jail or something like that, or at least you'll get sued. But I, I know that there is a lot of sectors of the entertainment industry that don't have any sort of rev share or, or a poultry one at that. And I'm, I'm actually always surprised that even though we like spoilers and things come out. It's actually not, it's actually a lot less widespread than one would expect. Like we don't hear a story about this happening every year even. And when it happens, it is a big deal. Partially'cause it happens so rarely.

Rob:

I, I agree. And most of the leaks have been through external. Efforts to gain access to companies like Project Red was that and, uh, the Capcom leak, Nintendo Now Insomniac, none of that's been internally leaked and internal leaks are incredibly rare.

PJ:

Which is kind of amazing in, in the best of ways, really. I mean, either it's because of like incentivization or people are just acting in, in good faith, not to like Leak these things out.

Rob:

disks and things like that. Uh, things get leaked that way or attach the wrong file and things get leaked. But it's always little tiny details. Somebody working on this game could quite easily just go, there you go. And, and maybe they lock things down, but that never works. Yeah, it's, I was doing, some work for game development. I was on site and couldn't plug a USB driver in, and I'd got files on a USB driver. I couldn't get off. my machine and I have to go through it and get them to do it, blah, blah, blah. And I asked like, what's the point? And he's like, well, with no, no source code leaves the building at all,

PJ:

Hmm.

Rob:

And I made the IT guy challenges like, I bet you when I leave in a month, I can give you a hard disk with copy of all your source code on it. And it's like, I bet you can't. It's like no one's ever got a file out of this place. When I left in a month, I handed him a hard disk. I was like, Bye-Bye. Enjoy. So again, smoke and mirrors. This is what I'm getting at. Like IT or IT departments do this think we're golden. We fixed the problem and they really haven't. They've left a giant gaping hole and that was a perfect example. have no idea what changed after I did that little stunt. well that's the, the essence of this problem of like the IT guys are like, yes, we do all these things and none of it really affects. It's just, it's just an obstacle. It's just gonna slow people down. And in this case, it didn't even slow them down because 20 minutes is ridiculous.

PJ:

Now that's a small time.

Rob:

to know how they got in. It's like, was it a password hack? Did they get a key somewhere or was it just unpatched software? Was it just wide open? I have no idea. And we probably, like I said, we'll never know. It's unfortunate.

PJ:

Is it the case effectively that they're doing that to prevent vent, you know, kind of the casual employee from trying to just like do something silly

Rob:

for sure. And it, it mostly works, but it doesn't work for a determined actor

PJ:

Yeah, clearly.

Rob:

and for external security, you have to assume everyone is determined. Actor casual, security doesn't work. It's, is a perfect example. My little stunt was a perfect thing. Example as to if you're determined actor, nothing's gonna step in your way unless it's truly closed off. Then you've got that balance of like how usable, it's like convenience versus security. Is that sliding scale of like how convenient is it for employees secure it is, uh, is it, and what tools are at the backend to help with this?

PJ:

Yes.

Rob:

Some ways, I think it's a little knowledge is dangerous. Like I'm sure they thought they'd locked these machines down super, super solidly, but my stunt and the dev cake were giant holes. but how much inconvenience does it cause and at what point does that inconvenience become detrimental to development? And that's where, uh, these companies set the line and everyone sets it kind of differently. And it also comes down to like. Culture of the company, how much do they trust their, their employees and don't trust you at all? Nothing. It's where you draw that line for versus hard security comes down to that trust.

PJ:

What are the changes that are gonna be made in order to prevent this kind of hack? And again, in the future, are they gonna lock down things more? Is that gonna slow down development? Is this gonna be like, kind of a big pain in the butt or you know, they just fire somebody and Keep on trucking.

Rob:

I don't think anyone's gonna get fired. Maybe they will, and think there'll be some sort of knee-jerk reaction, which will lock things down in an inconvenient way and not fundamentally solve the problem. The best thing that can happen to these companies that get hacked is they get hacked again next week after something has to make them be like, look, we need to fix this. And just having a knee jerk reaction, locking down employees' machines, which won't pull of the problem is uh, not gonna help. Aftermath is the interesting part of like, first of all, does a chief security officer actually do of like, they're not actively involved in penetrating their own systems. They're not actively involved in like white hat hacking and like that. All they're doing is sitting there going, okay, we'll use this vendor, that vendor, and we'll see the big security picture. You're not helping. Those people are not helping the problem. It's just obfuscation smoke and mirrors.'cause now there's layers removed and you're putting just in somewhere else. The other thing as well is. What's the recourse from this? For people? It's like, yeah, I should be looking at these files, but my personal information could be in there. So in my eyes, that gives me full access to go look and see what's in there because I know damn well in Somnia Sonia are not gonna contact me and say, your information was publicly out there. I don't think even know where I live these days. So that's not gonna happen. So I'll take it on my own self and go look and. The other thing is, well, is the disorganization of these files. It's literally a dump of a network server just a random layout of files. There's some I nines for some people and there's some others in another place. And just the general disorganization of all these files is interesting and the fact that they're just files at a hard disk, it's, it's like, should that change, is it encrypted? What happens if you recycled the machine and someone pulls the drive out there's many ways that data can get out. I just think it's a little, we don't care about your information'cause it's just dumped on a hard disk with a single access password group or domain to get access to that folder.

PJ:

I mean, if some of this stuff was going back over a decade, they might've thought, oh. We're past some sort of statute of limitations. Were in the clear, but clearly that's not the case. And I, I think this is actually the most tragic thing of the whole affair, is that people's passports have been leaked. And that's actual personal information. That's not, if you're going after the company, then sure, I get it. You are deciding you're gonna be. You know, you know, crafting some larger story.'cause you want to get some money out of like a large corporation. If you've got personal files that are actually getting out there, then you're actually harming individuals and that's really irresponsible.

Rob:

I don't blame the hackers though

PJ:

Oh, I, I hear what you're saying. I hear what you're saying. Yeah.

Rob:

just took an image of what they could access. This is access, this p, there's copies of user folders including in all that is just the HR and finance data.'cause it looks like they got access to The full network domain, so they just dumped the entire thing. that happens to include personal information and I don't blame the hackers'cause they just went cut and paste, drag and drop basically to get all those files out.. It's why are they not in some secure or in some secure database where there's another level of, of, of security, even just an encrypted drive in a drive. So it's just like a multiple image, which has very encrypted personal information like make an effort to protect your employee's information. There's no effort made here at all.

PJ:

So this may be a clue then about how well integrated they were into the larger corporation, because I have to imagine Sony does keep all of this stuff. Most, most large corporations will use something for retaining all of these HR things that aren't gonna be co-located with the source code. Now that this has happened, guess what?, Sony's coming in now. We're gonna lock you down.

Rob:

Plenty of games have been leaked and been crazy successful. Uh, I don't think leaking, the source code's a big deal. There's nothing crazy proprietary in there. I mean, it is a cool engine and it does cool things, but they'd have probably talked about it at GDC and or released the code if you asked anyway. So, and I agree with that. I think code is just text in a file. It's, it's not that important. It's the mindset and the architecture and the ability to use the tools you're given, in this case, the engine to make a, uh, core game and leaking that bad enough. Well, like I said, plenty games have been leaked and been super successful afterwards. And will they change the game? Will they take out what this, something's there? people try to rebuild the engine from source?

PJ:

That was actually gonna be my question, whether anyone was gonna either replicate the game itself or would they attempt to actually compile the engine, compile the tools, and make their own game, like effectively use it as their own game engine. Um,

Rob:

certainly could. All the assets are there in usable form too. So there's like, if you want a Spiderman mesh,

PJ:

I

Rob:

one. And so there's lots of things people are gonna pick at these. Data divers are gonna have a field day with this

PJ:

Sure.

Rob:

It'll be interesting to see what happens,One more thing I wanna bring up on this too is that's the way the press is handling it. seems to be like they're doing it different for insomniac because insomniacs the Golden Child and other companies, when had these leaks, the press have been all over it and they've looked at it and the various game press. websites and things have been very damning. Well, this one, they're like, oh, we're not gonna talk it It leaked. We're not gonna be part of it. And it's like, why are you playing favorites? It's, it's like it's a leak. They should dive into it. They should see what's there because it is newsworthy and that's their job. It's almost like the game press now is just part of the PR machine. The next thing we'll hear of this leak is when insomnia have to do a press release and it'll be all over the news as like, this is what's happening. It's like journalist's job is to get on that onion and get the to browser and go and look through it to see what's there, the numbers, to add up things, and do real reporting. Now you have the chance to, rather than just company line.

PJ:

Here's the, uh, cynical take I'll have on it. If you are a journalist and you are really hoping for a big scoop, you are maintaining a close relationship with a company and especially a company where. It's known for developing big AAA games. It's known for working with High-end ip. In this case Marvels and it has a track record that's been stellar. You are one journalist in a C of journalists. You are probably terrified about losing your in with the company. even though you're right, someone should be diving in deep to see what actually is going on here and asking the hard questions. But that might be all well and good in that moment, but then you've burnt your bridge with that company and now they're like, well, I don't want to use you anymore. You're not gonna get in any kind of like early scoop on this. So you're right. It's not, it's not investigative journalism. It is PR ery.

Rob:

You're absolutely right. And that happened 20 years ago when I was an insomnia act doing the ratchet and clan games. It was like on press day. It was like very carefully pick the people who you want to come. It's like that guy doesn't give us good reviews, so we're not gonna invite him and. It wasn't just like everybody come, everyone a fair game. It, it's very much a big PR machine. Like the game of press is not free and independent in any way, shape, or form for exactly what you just.

PJ:

Yeah. Uh, I mean the, the economic incentives are not there. That's, uh, just is what it is. Game news is not going to really be mainstream news. It, it's, it's not gonna get. A big splash on the, any of the large publications. Um, and then unfortunately the publications that are close to it that understand it, which are the game publications, they're not incentivized to, really dive in deep because it's, it's not just about the reputation with you and Insomniac, it's then gonna be all the other studios that Sony has as well. Because they'd be like, oh, that guy did a hit piece on Insomniac. We don't want him talking to Naughty dog either. So I think there's this very delicate balance that plays itself out in that market. But at the end of the day, does it mean that any of the reviews matter? Like it.

Rob:

it's, the whole thing is just pr. someone needs to just be like the guy. I'll buy my own copy. I'll say what I want I'll deep dive into this, uh, leak and see what's in there.

PJ:

which implies then that they have a different revenue source than basically the game publication.

Rob:

That's entirely true and that's what it all comes down to, isn't it? I Think we have to wait and see what the aftermath is what changes come about and that's all we can say about it for now. And unfortunately, we do have a lot of friends who still work at Insomniac who

PJ:

Yeah.

Rob:

affected by this. So certainly feel for all of those folk, hopefully comes from this and they still make a killer game and

PJ:

yeah, I think the game will still do well. I mean, it's an amazingly popular IP and, and insomniac has proven its ability to, you know, produce great games so that I don't have too much of a doubt on, uh, my, my money is basically on Sony will tighten the reins around them more. the security stuff that they'll put in, you know, will likely be separating out hr, separating out the finance side. The source code bit is gonna be a really interesting thing because I wonder how much of that, if it gets changed, will affect the development cycles for Insomniac itself. Because if you make development harder, which is. What you would typically do when you're trying to impose more security. That either means that your game times take longer to develop, or you spend less time in development with them because it's just is more difficult to actually do things. So my money basically is on that because that's a. Fairly standard response from larger corporations. Just let's lock stuff down and you know, run our playbook.

Rob:

And that's exactly what's gonna happen here. Uh, here. They might not change development. It doesn't look like they got access to peripherals server. It looks like all the code that was leaked is from people having copies of the peripheral server on their machine and. So there is no Perforce database that was leaked as, as far as I can tell, that isn't anywhere to be found. Obviously if you are working with a source control, you do have a local copy of it. Um, some of those local copies are on servers and some are on machines. And like I I said in the leak, user folders have been leaked and entire copies of hard drives are in there too. Obviously you can look at the code and see that they, all the, the tools integrated with Perforce but that server didn't get leaked. And if it did, I can't find it.

PJ:

Let's wait for the response. As I said, I think it's gonna be via the corporate playbook.

Rob:

Oh, for sure. There's gonna be lots of PR and like press releases and it's going. None of it's gonna mean anything. And again, the best thing that can happen is you get hacked again in a week to be like, didn't fix anything.

PJ:

and then the real changes began.

Rob:

Yeah, but it, it does open up the question of like, what can you do if you have 200 people all working on the same project? Like you can't lock everything down.

PJ:

I mean, but this is, this is kind of the thing, which is that you don't need to lock everything down. Uh, they might because it's just a really standard response, but it wasn't everyone who Who leaked this thing? It was basically like, you know, just the domain admin and then that once they got effectively access to root or its equivalent, like they were able to just go to town on it.,

Rob:

And I think there's a bigger conversation here too, as to. Blockbuster games as a whole and the security around them. And that's, is that security from the fact that it's a blockbuster and it's gonna be announced on one day. So everything's kept secret, like the Apple style, or would this be less of a problem if we just release things gradually, like, oh yeah, the game's progressing. Look, he's a new level, blah, blah, blah, rather than blockbuster demonstrations at certain events to get that hype up. I get why they do it, but it doesn't help with making yourself a target. You're effectively painting a target on your back. and look at us

PJ:

​Next time.

Rob:

For sure