covert creations » MMORPGs, Security, and the Grand Promise of Middleware

MMORPGs, Security, and the Grand Promise of Middleware
October 6, 2006 on 8:50 am | In games programming, games design, games industry, mmo, mmorpg, WoW, security |

WoW and SecurID

A big congratulations goes out to Neardeath Studios on the 10th year of Meridian 59. What a fantastic accomplishment. M59 is the first, the longest-running, and most respected MMORPG of them all.

This article is in response to M59 co-creator Brian “Psychochild” Green’s post, “Why middleware will not save us“. He hits pretty hard, and sets his sights on the “middleware market” in the MMORPG space. I’ll say I agree with the bulk of it. Yet, some of the specifics cause me trouble. Thus this post.

His argument noted two levels of the MMORPG industry, the indies and the AAAs (”the blockbuster games”). The gist of his article is that, as a technological cure-all, MMORPG middleware companies fail in their promise. They will make little impact on game development in MMORPG games. A gross-oversimplification on my part, so I’d encourage you go read Psychochild’s post.

First off, how does one define middleware?

There are plenty of types of middleware that provide an invaluable solution to laborious game dev issues, notably asset creation. Take the L-System utilized in SpeedTree (an automated tree creation and placement system). This is a tech solution at its purest. And yes, it’s middleware, and it sells very well. Then there’s PhysX/Havok. Then there’s all number of behind-the-scenes stuff that you’d never hear about. All handily employed in MMORPG production.

Forget all those. Forget pipeline and asset coordination stuff. Let’s focus on the “client-server mmorpg system”. By implication, I suspect Psychochild means client-based graphics engines, and also client-server engines designed for MMORPG diku-style games.

Vendors
So in this space, we have a few significant players. BigWorld, Hero, Kaneva, Gamebryo. And there’s a small army on the horizon, getting ready to enter the fray, including Google (but that’s another post). We even have indie-level alternatives : Nevrax, Crystal Space, Ogre and the eminiently popular, GarageGames.

Do these companies cut in terms of MMORPGs? Psychochild believes that it’s unlikely. According to him, their wild claims of solving game development woes is tantamount to fluffy vapour, largely because they’re unproven. It’s true that nobody really knows the real answer yet. Experience counts, but the games are the real proof. I agree.

Yet, I take issue with his minimization of middleware’s value proposition. The claim that these vendors’ “wild promises” seems far-fetched, simply because press release fodder doesn’t emerge when you’re in the throes of selling serious products. From my experiences, at least in the security industry, it’s often quite the the opposite. What I can tell you is that on some levels, at my former company (RSA Security), we were faced with the same types of challenges as the folk in the MMORPG middleware business, which I’ll get into in a moment.

Expectations

In our business, trumpeting any technology as the end-all is, most certainly, a recipe for disaster. We’d often say, “No, don’t buy our stuff for a catch-all solution”. It’s really process that trumps technology every time. So Brian’s definitely bang-on with respect to MMORPG solutions. How could they possibly solve the problems of game development automatically? Well, much like this fictional MMORPG vendor, we did not automatically prescribe tech for all manner of security requirements. Yet, the customers would actually fight us on this. Our team would sell them two products, and then I’d whiteboard twenty things for the process side. They’d install the stuff while paying lip-service to those “other things”. Then, rarely do them. Sometimes it would come back to bite everyone in the ass.

The problem lay partially in customer expectations. The technological quick-fix is what they desperately wanted! In many cases, we had to “down-sell” in order to actually sell. We had to educate on the capabilities, on what was possible. No, our users were not stupid - they were just extremely busy. They were relying on us, the experts.

I wonder, are MMORPG developers busy too?

Sometimes it’s more about education and consultation, long before we even spoke about tech.

Tech Time Savers

The principles that Neardeath Studios applied in creating M59 are golden : good engineering, good process, good creativity. Rolling up the sleeves was not an issue. They knew what they were doing, and very few others did (they couldn’t, Neardeath/3D0 was the first!).

Yet, if we applied the same can-do approach today, it would take years to execute. All of those principals are still good, but there is literally no denying that the space and complexity have blossomed for MMORPGs recently. Even worse on the client graphics engine side. We just don’t have enough time to build the thing. I cannot imagine anyone reflexively shunning a product that helps you get to market faster, even if it’s not perfect.

So, where does security fit in to a conversation about MMOs?

Well, to understand that is to understand a bit of general security. I’ll be brief.

Security is obviously a huge topic, and it’s generally acceptable to distill it into three main areas : Integrity, Availability and Confidentiality. From these three, most software vendors minimize the quick-fix approach (or they should). Yet, again and again, we’d get calls saying “We’ve suffered a [breach | hard down | bug]. It was your fault.” Oft-times, it may have been a deployment issue, something else, or bad system design : all of them having nothing to do with us. Yet, we’d work our analysis with diligence, hopefully solve or rectify the immediate problem and then kick it back to legal should there be any mitigating issues. Yes, we had to cover our asses.

This situation is likely no less true in the MMORPG middleware market. In fact I’d bet it’s worse.

Marketdroid-speak isn’t going to convince attentive devs.

These middleware companies of which Psychochild speaks likely don’t bill themselves as the “final stop” in the process of game creation. That’s because everyone knows it’s not true. The promises may sound adequate as bullet points on a product page, but when you sit down, face-to-face, with a potential customer, the gloves have to come off. It would actually harm them to promise more than they can deliver.

So, here’s where the gloves come off.

Confidentiality

Take the confidentiality aspects of MMORPG-style games. This means that your data is secured, shielded in transit, hidden from prying eyes, and all that spy stuff. If you could build your own identity management system for your game, would you want to? Or would you go with a vendor that concentrated almost wholly on doing that correctly? Would you design your own protocol like Blizzard did (and a dubious one at that)?

I had a brief look at Blizzard’s protocol and auth system. It was extremely dismal, and I found 3 clear vulnerabilities within a few minutes of tracing and other little prods. See that nice image of the SecurID token? Lets just say, it is a very good idea.

So are you going to implement your own hashing scheme in the protocol? Should you use a nonce during that interaction? What about secure coding practices? Do you have a lot of time to analyse the thousands of security risks and choose to accept, mitigate or eliminate your exposure to risk?

These are things that have nothing to do with games, they’re aspects of ensuring the integrity of a transactional system. With proper abstraction, the concepts could be applied to any system.

Today, MMORPGs are beginning to deal with real-world currency and micro-transactions. These factors are not something you fuck around with in a garage. No amount of gaming-level expertise will fix things when you realise that your re-authentication packet was vulnerable to replay just because developer #3 didn’t read the spec right. Or becuase some idiot used the system time as a salt for your hash. Been there, fixed that.

Availability

This area concerns itself with providing a reliable and fast system for whatever interaction you require. Performance analysis, tuning and capacity-planning is an entire career for some people. Gonna wing that? What metrics are you defining as reasonably characteristic of your MMORPG? What tools do you have to measure whether you were successful?

So, do you purchase a product that had been vigorously stress-tested, stood up on big-iron, tested again, then tuned from a code level? If you’re a serious developer, you are looking at these things. If you’re a serious developer, you’d like someone else to give you something that you could tweak, rather than starting from scratch. So that you can concentrate on writing your game logic and (secure, haha) SQL procedures.

Sure, there’s no tech quick-fix for this stuff. But, damn, it helps!

Integrity

This is actually a really tough one. The ability of a system’s data not to be changed mid-stream or without authorization is extremely difficult in games. You want to prevent cheating. You want to prevent “gaming the game”. I’ve had a hand in providing a little bit of market research for RSA here (Valve’s VAC, Punkbuster, Quake3, risks, problems, etc.), specifically with regards to the cheating aspect prevalent in online gaming.

We didn’t touch it with a ten foot pole.

There are some tech that can mitigate your risk to these exploitive behaviours, but it really does come down to good design. Some gaming security experts prescribe allowing the client to be somewhat open, because motivated hackers will get into it anyway. It’s classic tit-for-tat. WoW installs a “watcher service” on your computer. You run your hack-bot-cheat in a VM. Or the classic use of OGLE to “see” data about adjacent world or game elements. Don’t get me started on hacking the client. I noted Grimwell posted in your thread. He knows this stuff inside and out. This endless war is a classic security truism (see Bruce Schneier, expert on cryptography and security).

MMORPG middleware is NO more the white knight than say, RSA’s SecurID system. But you are saving yourself time and risk by going with one of these vendors whose sole concentration is working on this stuff and perfecting it. I hope I’ve shown how so many of these client-server issues have nothing to do with games. They exist in payment systems, nuclear waste disposal management… and MMORPGs.

Graphics Engines != Server Engines

He says that graphics engines are a worthy place for purchase because the issues are so specialized and complex. I’ll definitely agree with that. Those guys in garages are becoming less and less these days. They exist, but they’re somewhat of an anomaly and are rightfully celebrated.

Then, extending this same idea into the server space. I’ll agree, there is a distinct advantage to buying tech from an established game-maker versus a relative neophyte. But this is where we part ways. Don’t dismiss middleware entirely! The world has somewhat changed since the days of John Carmack. Risks are inherently greater, time is inherently shorter, and the competition is right beside you. There are reasons why you go to IBM and buy a Blade Server. You’re not going to find your MMO Billing Centre at CompUSA. Likewise, established game-makers may know very little about cryptography, authentication, performance testing or secure network programming. This stuff is middleware’s bread and butter (or should be).

Ahh Finally

I will definitely agree with Psychochild on the tools and experience front. If a vendor doesn’t grok workflow in asset creation/instantiation, then forget it! This is where games essentially get made. He was also correct on the service and consultative approach - if you’re an MMORPG middleware vendor without this sales tack, then you are not going to succeed. Research, good software engineering, not to mention creativity. Yes. A good work ethic. Excellent management. Selectivity in tools. Yes, yes, yes.

All these are good things, both from the developer and the vendor side. But don’t dismiss what middleware can buy. It might not write your game for you, but the time and security payoff is too great to be ignored.

In sum, beware the white knights in any industry.

Technorati tags: security, world of warcraft, middleware, bigworld, kaneva, gamebryo, hero

48 Comments »

RSS feed for comments on this post. TrackBack URI

Justin
October 6, 2006

Interesting use of security to illustrate the infrastructural parts of MMOG middleware that are usefully 'outsourced'.

A while ago, I was discussing middleware with a friend who's shot an entire feature length film on greenscreen. Rather than saving time on his production, he's found himself in a rathole, since he failed to anticipate the amount of work filling in the green backgrounds with 3D models.

He had sent me a link to Multiverse, a service oriented MMOG middleware company, and I sketched out for him the idea of a virtual film studio. Imagine, I told him, that your art director builds sets in a MMOG world, allowing you to virtually tour it and approve it or change it; further, if the set is well made, virtual actors can later be brought in to block the scenes and set up camera angles; the entire production can be storyboarded in real time, even. At the implausible end, if the quality of the 3D models is high enough for film, the greenscreened actors can be matted into the same virtual world in post-production, turning the pre-production assets into the finished film. All with minimal meatspace activity.

I sketched this out because I remembered a story about the F-22 Raptor being designed virtually: Engineers in one location designed parts for the engine or the cowling or the wings, and created virtual models; engineers in another location entered the 3D world and fit the pieces together, both validating the design and testing the assembly and maintenance procedures--a piece could be rejected because they couldn't figure out how to insert it. By the time the design was done, the maintenance procedures had already been written, without the cost of a lot of expensive mockups and machined pieces.

The point of these scenarios is that virtual reality is useless as a gimmick--that's why a 3D web never caught on. But there are real advantages to be had that go beyond not needing everyone to be in the same location. What this has to do with middleware is that there's a fertile market here for MMOG middleware building virtual environments, especially if they're service oriented. Multiverse especially is aiming outside the gaming market, to academics and businesses that are already looking at virtual reality for things like distance education. I think this is a wise move, because it's a more stable financial environment that avoids the scale problems of AAA game titles ("will we have 50,000 users or 5,000,000?").

These guys use the Half-life 2 engine for architectural visualization. Imagine buying a middleware engine, hiring some artists, and contracting to the architecture community to create virtual walkthroughs. Middleware is definitely not a white-knight, but it's solid market proposition that goes far beyond gaming, and will (must) incidentally work out a lot of the infrastructural issues that are exactly what a game company shouldn't worry about.
Psychochild
October 6, 2006

Let me clarify one thing: I'm not one of the original co-creators of Meridian 59. I started work on it just over a year after it launched. I have been working (on and off) with the game for over 8 years of its 10 year history, though, so I feel comfortable asserting some moral ownership over it.

One thing I guess I didn't make clear in the original article is that I was talking to a specific audience. I'm not saying all middleware is fraud, but rather than it's not the silver bullet some people think it is.

Often there's discussions about the high costs and even higher risks of online game development, usually in response to someone lamenting a lack of innovating. "Yet another DIKU clone!" someone wails, so the usual suspects come along to say that the costs make it difficult to justify anything else. "Well, just wait until we have good middleware!" someone will retort. "Then anyone will be able to make an online game on the cheap!" My article was largely to disabuse these people of that assertion.

I also didn't go into some of the beneficial aspects of middleware because it didn't fit into this theme. I think there are some really good potentials for middleware. Security is one good example. So is social analysis, data analysis, etc. But, as you point out, these aren't plug-n-play technologies that handle things automagically. You still need to know what the hell is going on to use them effectively.

So, yeah, I agree with some of your points and clarifications. They didn't fit within the original purpose of the post, though.

Some more thoughts in another comment.
Psychochild
October 6, 2006

Let me discuss a bit about security in online games, since you brought it up.

Security, like other things, is a question of tradeoffs. The basic thing to keep in mind here is that we're talking about access to a game account. This isn't banking access, hospital records, or any other thing that has huge legal protection and penalties for faulty security. Not to say that game accounts aren't important, but I'd rather someone hack my player accounts rather than my bank account. (My admin characters, on the other hand....)

The main problem we run into is that many players are not that adept with technology. It wasn't uncommon for the M59 customer service reps to a call on a regular basis from someone that didn't understand you had to have an internet connection to play the game. And this was at a point where M59 was primarily available by download! So while the SecureID tokens might be good security, it would confuse a lot of customers. And given how many "new" (and likely technologically inept) players WoW brought, that would be murder on the CS side of things.

Further, I think that while it's good to have some level of paranoia, and while having paranoid security people is generally good, sometimes you can be a bit too paranoid. A great example was talking to someone about making a MUD, and he recoiled in horror at the thought of using raw Telnet. Yet, I argued the cost of making people install an SSH client would be too large for some people. Not to get too much into a "back in my day..." argument, but I played text MUDs for years in a school filled with devious and intelligent computer hacker types. Yet, after years of playing these games I didn't lose access to any accounts. The only time I had some problem was when someone hacked my account *from the server side*. That was one of the dangers of an LP-MUD where people had editing powers.

And, I think this is another area where technology is not a silver bullet. As a security expert you should know what the greatest security vulnerability is: social engineering. Most game passwords aren't stolen by Trojans, sniffers, or anything else. They're willingly shared with other players the owner thinks he or she can trust. Yet, some people can't resist the temptation and do something to screw the owner over. All the security precautions in the world mean nothing if the owner willingly shares access.

Some more thoughts on this topic.
covert.c.
October 6, 2006

"Rather than saving time on his production, he’s found himself in a rathole,"

Is your friend George Lucas?

"He had sent me a link to Multiverse, a service oriented MMOG middleware company,"

You missed my blog post earlier this year talking about Multiverse! This is an entry on the field by, among other, James Cameron and will be tied into his next film.

As you know, they're just getting started as an indie-offering and hosting service. They are likely modelled after the Neal Stephenson notion of a metaverse, crossing between genres with your digital avatar. It's an ambitious vision, even if they are smartly going into the serious games genre (education, simulation, etc.).

It's unfortunate that Chan was flummoxed by that nasty scorpion, the famous content creation problem. It's the killer of all A-A-A, destroyer of worlds.

Ryzom Ring, a Nevrax creation, tacks user-created content onto an MMORPG. Of course, I'm skeptical about the end result. Will we see yet new and cunning examples of Headcrash's famous game?

Build the content creation into the game, and then you've got something.
covert.c.
October 6, 2006

Psychochild:
“Well, just wait until we have good middleware!” someone will retort. “Then anyone will be able to make an online game on the cheap!” My article was largely to disabuse these people of that assertion.

This directly ties into my observation that the "users" (i.e. the customers) are desperate for that technological quick-fix. Expectations are very high in both security and gamedev.

Security, like other things, is a question of tradeoffs. The basic thing to keep in mind here is that we’re talking about access to a game account. This isn’t banking access, hospital records, or any other thing that has huge legal protection and penalties for faulty security.

Don't get me started, Brian. I can go on for days. Joking! (sort of)

You are absolutely right about tradeoffs. There is security versus usability, security versus expedience, security versus performance, and sometimes there's even security versus security. UGH. How these trade-offs are relevant is up to the experience of the developer. I didn't touch on that, good point. If you're an experienced game developer, you'll already know how far to push the threshold of the security-performance curve.

There are three interesting nuggets, if you will, we can pull out of this quagmire:
- Security can actually be binary. It's either secure, or it isn't. There's no such thing as "pretty secure". This can be a handy rule of thumb when assessing risk for a system, a process, or even a policy.
- Security is, foremost, a process. If your procedures and policies are not oriented towards preventing undesireable things, then no amount of technology will ever help you.
- Trust is a chain. Users are the weakest link.
It's interesting that you talk about SSH in a MUD. Yes, there's overkill. Even from my background, I'd somewhat shrug at the benefits of SSL. The threat that it mitigates is actually extremely rare. Spending tons of money, doing programmatic acrobats in securing the channel against man-in-the-middle attacks is almost superfluous sometimes. Or perhaps those attacks are so rare because SSL is so damned good.

As far as tokens, the image I used was more illustrative than literal. There are variants (e.g. SoftID) that allow you to eliminate the interaction and still gain the security...but this goes off course significantly from our original discussion...

One final note on security - you're right it's not a bank account. But as I mentioned, we are getting closer and closer to micropayment systems involving real monetary transactions. The ramifications of this are enough to send security people whimpering into the nearest bar.
Psychochild
October 6, 2006

Security can actually be binary.

Sure. I'm sure you know as well as I do that the answer to "Can this system be cracked?" is almost always "Yes, it can." The second question is, "What is the likelihood?" And that's where you get the shades of gray. You can do things to make the systems less of a target. You can also look at the work required.

In online games, man-in-the-middle attacks require a lot of sophistication. You have to know your target or hope you find someone foolish, and each success only gains you one account access. The more tempting targets are ones where you get more bang for the buck: pulling passwords off the server, for example. So, it pays to take a bit more care securing information on your server rather than worrying about the rare man-in-the-middle attacks. Again, being smart and experienced is better than any middleware, IMHO.

But as I mentioned, we are getting closer and closer to micropayment systems involving real monetary transactions.

Again, it pays not to be a dumbass. Rule #1: Don't store credit card info on the game server. Make them enter it every time. Yes, it makes it harder for impulse purchases, but it also makes it more difficult for you to be on the wrong end of a lawsuit when credit card numbers get leaked.

Honestly, I don't think microtransactions are going to make security any more of a problem. My WoW account has had a few hundred dollars spent on building and maintaining it, but no one has tried to hack it as far as I know. Of course, I'm probably a bit smarter about security than your average game player. But, when it comes to microtransactions you just have to be smart about it: be careful in what you allow, and log everything. And, be prepared to spend some of those earnings hunting down the "bad guys", just like you have to do with subscription-based games.

More thoughts,
covert.c.
October 6, 2006

The second question is, “What is the likelihood?”

One thing I've always assumed is that crime follows the money. If your system provides a mechanism for players to engage in RMT, then your only assumption from that point forward is that the "likelihood" moves to 100%.

You can do things to make the systems less of a target.

Contrary to popular thinking, you cannot reduce a threat. I don't mean to quibble semantics, but in the digital world, you have absolutely no control over how your service/presence is perceived by motivated hackers (well, unless you take yourself offline, or make your game unpopular). You can only reduce your risk of being harmed by these threats. Honeypots are popular, but I am personally not an advocate of them.

Rule #1: Don’t store credit card info on the game server.

That is good advice. My thinking in microtransactions would have assumed that the whole innards of the game would be bent towards making credit card purchases easier. I'm ravenously curious as to how those Korean MMORPGs (particularly Kart-Racer) approach this issue.

I don’t think microtransactions are going to make security any more of a problem. My WoW account has had a few hundred dollars spent on building and maintaining it, but no one has tried to hack it as far as I know.

WoW doesn't support microtransactions, so your anecdote is unconnected to the first statement. Microtransactions do affect security because now you have a clearer relationship between the game activity and money. There is no tastier target anywhere for enterprising evil do'ers.

You also mention logging. This is aboslutely critical. IMHO, security through audit is a failure in security but at least you'll be able to protect yourself in court.

You probably cannot comment, but I have to ask. Have you ever had to actually "hunt down" someone for some breach or exploit in M59?

BTW, thanks very much for your insights and this discussion.
Steven "PlayNoEvil" Davis
October 6, 2006

So much to talk about...

First, on Middleware (speaking as a biased maker of middleware and a long time developer)

Is middleware a magic bullet. No. The biggest problem with middleware is that it is not used properly. Too many developers want to code. The hard lesson of software development is to write as little code as possible. For middleware users, this means working with what you bought, not trying to make it "perfect". It makes much more sense to make your application work with the middleware. This is where most middleware projects fail.

The game industry is not alone in this confusion. Everyone takes purchased software and "tweaks" it. If they start changing its guts, then they have missed the point of the product. There are so many hidden design assumptions in any product that going and improving it is often an endless, fools errand.

Game developers are not in the software business (usually). They are in the game business. So, focus on the game, not the code. Accept middleware as good enough or take on the task of writing, maintaining, and supporting its functions yourself. Recognize that when you do this, you must have the staff to write, maintain and support that code and that, for a typical game, you only get to spread those costs over one (or maybe a couple of) projects. A middleware developer is focused solely on this one product. They can spread its development and support costs over many customers (hopefully)... and, again hopefully, they know something about the subject matter - you can call and yell at them if there is a problem. If it is your own team, hopefully your "expert" hasn't left and gone on to bigger things and left you with an undocumented lump of stuff.

Middleware is risk mitigation, not optimal solutions.

Our software, SecurePlay, solves several security problems associated with cheating very well. It is based on certain assumptions about how networked games can, may, or should work and where security problems exist. When I talk to people about the product, I am 100% open and have told several potential clients that it will not solve their problem based on the nature of their project or where they are in their development process or their security needs. This should be true of any project considering middleware - the tool will fit and help or it will not. No harm, no foul.

The leaders of any development projects need to find a way to not write software and deliver a product or service that reaches as much of their business objective as possible.

Game developers need to realize they live in a world of hard choices and where do they want to focus their dollars to mitigate risk. The biggest risks are business and creative risks. Middleware should be a cheap way to allow game developers to focus on these areas by putting resources where they are likely to increase project success.

Separate comments on security to follow.
Steven "PlayNoEvil" Davis
October 6, 2006

(Beware, rant follows)

First, a disagreement over some language:

Security is Never Binary. You are never totally secure. This is the fallacy of the war on terrorism as being practiced today. Once you recognize imperfection, you focus on getting the most "bang for your buck" rather than the illusion of perfection*.

Also, we (security folks) should probably stop talking about Confidentiality, Integrity, Availability and instead we should start talking about company's security requirements in their language**. (Something noted above in the focus on process, not the security products).

This is one of the great things about the game industry. Cheating and griefing costs in terms of customer service, customer acquisition, and retention. Piracy's costs are also pretty clear. They all are intimately tied to the game's business model. Protecting customer data is not just the law, it is a key business asset.

Game security is much easier than arguing about firewall features and IDS's.

If you are not making or saving money with security for your business, don't do it. If there is no security business case, then don't bother. This is the same for everything else (including middleware).

However, as a person who has gamed his entire life, the neglect of security by the computer games industry is puzzling. The integrity of the game experience is at the core of any good game. Breaking trust destroys the "magic circle" that everyone is so fond of talking about. Poor security has been proven repeatedly to be very expensive for games.

* Talking about the "war on terrorism" is useful. If we think about the "liquids" issue, the question is whether this is a sufficiently serious threat for our security resources to spend time on. If you have traveled recently, probably 30% of the dollars/people/time that the available security folks had was focused on this single issue. Is 30% (or whatever the number is) of our security vulnerability tied to toothpaste? is this the most cost-effective way to manage the problem?

Since security only exists in context of the operational environment, is this security effort preserving the aircraft industy more than the risk associated with a "liquids" based attack?

** Too many times people think Encryption = Security. So they encrypt the data links and have the "protected" database available by use of anyone directly via its API or raw file download. Even the recent breaches of millions of US veterans records don't "get it". Encrypting the data (the proposed solution) means nothing if a malicious individual can get functional access to the protected data.
covert.c.
October 7, 2006

For middleware users, this means working with what you bought, not trying to make it “perfect”. It makes much more sense to make your application work with the middleware.

There are a couple of issues with this :
- Games can vastly differ in their requirements and execution from one to another. If your middleware solution is the game server code itself, it seems inevitable (to me) that a degree of customization will be required.
- This also assumes that the software in question does the job perfectly and is well-designed. In the early days of another company I worked at, we would commit a grave sin and use our customers as part of the development process (a no-no, but was necessary to grow the product properly). We're in the same boat in the MMORPG space - the industry is just too young to expect a perfectly shrink-wrapped solution that does everything that you need it to do (especially in the face of my first point).
There are so many hidden design assumptions in any product that going and improving it is often an endless, fools errand.

Again, I have a few issues with this assertion.

Brian touched on this, and I agree - professional services and consultation should go hand in hand with the middleware offering. Even the most complex security solutions that we offered at RSA would sometimes require significant gluecode, customized business rules or even whole features to satisfy the myriad of surprises that our customers would throw at us. I can think of many times where we slapped our heads in surprise, "Wow we did NOT even think about that."

It may have been our jobs as designers, marketers or professional services to anticipate the market in these ways, but I suspect that games are even worse for this! I don't care how smart your marketing team is, you're gonna get blindsided at some point or another. You're going to have to adjust that code for a particular customer situation. A fact of life in software development.

And in the worst case (as you describe it) even the game company itself sometimes wants to take something and tweak it to their requirements. It's a young industry here, one middleware developer may not have everything the customer needs. This need not be a travesty, looking at the penetration of opensource in commercial software.

Game developers are not in the software business (usually). They are in the game business. So, focus on the game, not the code.

Notwithstanding my previous assertions, this struck me as an excellent point. All sorts of hacky magic goes on in game development that should never appear in commercial software. Offloading the arduous task of creating a sexy, secure and reliable authentication system should never be in the balliwick of a game developer. As you say later on, they have the harder task in assuming the technical and creative risks of creating an entertainment product, so they should never have to bear the burden of creating infrastructure elements like that. It would be like asking a casual developer to write their own browser so people can play their games - it's not within their problem space and shouldn't be.

I think both Raph and Brian would agree that this is another staring example of how young this industry really is. The most basic of frameworks are simply not available that MMO game developers can rely on.

I believe the current players in the middleware space have a huge responsibility to address this. That's clearly an opportunity.

A middleware developer is focused solely on this one product. They can spread its development and support costs over many customers (hopefully)...

Moreover, the middleware guys have a laser-like focus on their area of expertise.

When I talk to people about the product, I am 100% open and have told several potential clients that it will not solve their problem based on the nature of their project or where they are in their development process or their security needs.

Security guys always try to be honest (or they should).

The biggest risks are business and creative risks.

Absolutely true.

Security is Never Binary. You are never totally secure.

In retrospect, I'll have to concede this to you. In the grand scheme of things, very true. However within the realm of possibility, total near-security is attainable. For instance, we can say a 2048 bit RSA key is not totally secure either. However, it's clearly not worth worrying about whether someone is going to brute force your private key. In this sense, Psychochild was correct. It's a grey area within a frame of likelihood.

Also, we (security folks) should probably stop talking about Confidentiality, Integrity, Availability and instead we should start talking about company’s security requirements in their language**.

Point taken. I find it illustrative for a general discussion since I related each of these to specific issues within MMORPG security. Still, you are aboslutely correct - no one cares about high falutin' security concepts. They want to stop their software from being mercilessly copied (or whatever it is that most concerns them).

However, as a person who has gamed his entire life, the neglect of security by the computer games industry is puzzling.

I'm somewhat surprised by this statement. My thinking was that middleware vendors took security very seriously. I guess I'm naive in that assumption and share your puzzlement.

...the question is whether this is a sufficiently serious threat for our security resources to spend time on.

I realise you're using this example for illustration, but coming from my area, I'll say that airport security is theatre, pure and simple. The hysteria involved in these types of issues almost prohibits a methodical and proper response.

I honestly don't believe that the war on terror is a particularly good example. It is so far from the real aims of security planning that it's more useful as a bad example than a good one. Theatre, politics and security make very poor companions.

Too many times people think Encryption = Security. So they encrypt the data links and have the “protected” database available by use of anyone directly via its API or raw file download...

The recent breach in "Second Life" is a case in point. Again, this is blind reliance on tools and tech versus a holistic approach to security (from policy right on down to the locks on your doors). It's maddening to see it in action. Even worse as a result of stark laziness on behalf of the service provider.

Your blog contains innumerable references to breaches and exploits right across the spectrum of online gaming. Have you quantified any loose trends in your survey that you could share here? The problem must be increasing!

I insisted earlier that RMT* makes MMORPGs extremely tempting targets for hackers. The problem is only going to become more serious as virtual worlds become more like our own. I strongly believe that your effort in this milieu offers a profound service to companies that finally "get it". And, I'm hoping that the MMORPG middleware vendors take it more seriously (if they aren't already).

* For those that don't know, RMT = Real Money Transactions
Steven "PlayNoEvil" Davis
October 7, 2006

A quick response for game industry readers, other comments to follow.

Growth of Security Incidents

I have been tracking game security incidents since the late 1990's. At that time there were usually a half dozen incidents per year. By 2004, the pace was up to about 1 per month. In 2005, there as about 1 incident every 2 weeks. This year (2006), I have said the pace was about 1 per week. This is actually very conservative. The pace is closer to an average 2 or 3 per week. Part of this is attributable to my closer monitoring since I started my blog, so 1 per week is a good/bad-enough number.

Methodology - My threshold for an "incident" is something that is announced via a press release, a regular news article, or a major online site. Occasionally, I'll pick up something smaller if I find it particularly interesting or informative. I do not go hunting into warez or hacking sites to find attacks though I will use borderline hacker sites if they provide useful details on the nature of the attacks (usually, I find these sites via major online sites as their reference for the story).

Generally speaking, Game publishers own the security problem in this industry. This has serious implications. Developers are compensated for delivering a product on-time and typically get very little compensation from royalties. Therefore, they do not have any incentives for good security design or practices unless such incentives are written into their contracts by the publishers. Also, because game publishers still see themselves in the "publishing" business as opposed to a service business with a long tail, they have allocated security into the QA or distribution side of the business where there is little power or incentive to address security strategically.

MMOs are similar. At best, security is seen as an operations, not design, problem. Therefore, it does not typically get considered by the game design or business team. There is still a view that security consists of encryption and firewalls.

Finally, don't criticize PunkBuster and AhnLabs and GameGuard. They are well-optimized for the current business environment. They are a pretty good solution when security is an afterthought.

RMT is only part of the motivation to take security seriously. Now that computer games have returned to gaming's roots by becoming multi-player (much less massively multi-player), they have stopped having a quick one-month sales cycle and have turned into long product tail / service businesses. Cheating matters in this environment. It is essential to make that long tail work. Look at Blizzard's other games. Diablo II, Starcraft, and Warcraft are consistent sellers because they have the Battle.net service to keep fueling new players. This is painfully obvious also for multi-player casual games where game play is all that matters.
Steven "PlayNoEvil" Davis
October 7, 2006

On Middlware, again -

"Glue-code" is different than interfering with the core functions of the middleware. If (or IF) the middleware is properly designed, the expectation is that the game developer will have to do some coding. Glue-code / customization is expected for any application, otherwise there is no need for a coder (which would be a perfect solution, but highly unlikely).

Good middleware should do its job and allow the developer to do his. Otherwise it is not middleware, but a poorly optimized implementation of the user's application.

Professional services should be available, but if the middleware is well-designed and documented, it SHOULD stand on its own*. * Assuming a skilled customer developer team.

I absolutely agree that I learn from every customer engagement that I have. There are "generalizable" solutions that I can create based on specific customer problems. In this case, I should take on the task of expanding or improving the middleware and the developer should stick with the glue/custom code. Reality is never perfect, but that should be our collective goal.

On terrorism, politics, and security, I agree that what we see is theatre. Unfortunately, or realistically, security is theatre and politics and rhetoric and depressingly little real engineering or business. ;(
covert.c.
October 7, 2006

So many interesting things to comment on, but I was particularly tweaked by the publisher side of the equation. I would have thought (perhaps naively) that they would be all over this issue.

As we've both sort of stated already, it's not just a battle of technology (interesting as it is), but also one of mindshare.

If what you say is true, it seems the gaming industry has a lesson coming. This is exactly the type of battle we fought in the online security industry a few years ago.

Excellent food for thought.
Steven "PlayNoEvil" Davis
October 7, 2006

And, by and large, look at the sorry state of online security. If it wasn't for California's and a couple of other places disclosure laws, there would be no info available about the numerous security problems out there.

This is a *good thing* in the game industry, because game play security is so central to the online experience, problems get outed faster than with bank thefts, cybertheft, etc.
Adam MacDonald
October 7, 2006

the *only* people at the AGC in Sept. who had anything semi-worthwhile technical to say were middleware vendors: Emergent, IBM, BigWorld and Simutronics. They understood scale. The Emergent guys in particular were impressive.

For some reasons MMO providers regard their architectures as somehow something that needs secrecy. They don't benchmark, they don't support open standards, and there's a culture IMO that they are solving scale and availability that no one else deals with (lol!). SOE actually had the best things to say about performance, but by and large the gaming industry doesn't seemd to understand that the difficult problems they face in regards to data integrity or uptime etc. have already been solved by other people. And all they need to do is cherry pick those providers who can offer them the best deal. Bioware seems to be doing this. Who needs to write a new chat client or a presence technology when what they really need to focus on is the game and art assets?
UNO
October 7, 2006

Hey not to sound obtuse but is there a dummed down version of this blog post? Could we tone this down a bit? maybe with some simple diagrams to help explain to Joe Gamer WTF you are talking about?
Psychochild
October 7, 2006

Interesting discussion. A few points:

Contrary to popular thinking, you cannot reduce a threat.

Perhaps a better way to put it is that you can deter some threats. Leaving unprotected ports on your server could encourage someone scanning ports to give your server a double-take. It's like locks on the doors of your car. Does it stop the dedicated thief? No, but it does discourage the opportunist. Even a car alarm doesn't make your car 100% unstealable, it just discourages people from continuing with the theft.

WoW doesn’t support microtransactions, so your anecdote is unconnected to the first statement.

Sorry, I got distracted by something shiny. My point is, why is an micropayment-based account that I've spent $200 on more appealing than the WoW account I've spent $200 on? The same people interested in hacking the first account to get the microcurrency would probably be just as interested in hacking my WoW account, selling off the assets, and scooping the gold into a stooge account. So, I don't think that microtransactions will make individual accounts more appealing, unless developers do something stupid...

My thinking in microtransactions would have assumed that the whole innards of the game would be bent towards making credit card purchases easier.

Yes, you are right. And, I'm sure someone will eventually get caught with their pants down and lose a lot of credit card information. It's already happened with companies like Amazon. Personally, I doubt I'd make enough extra money to cover my liability for when the credit card info does get stolen off my server.

The way to solve this is to have people buy points, similar to how XBox Live does it. The points are stored on the account, and then spent by the player. No credit card information needs to be saved.

My thoughts,
covert.c.
October 7, 2006

And, by and large, look at the sorry state of online security.

True, however compared to 1998 we don't have to spend our whole time explaining why tech X or procedure Y is important. It sounds like you're faced with this old battle right now.
covert.c.
October 7, 2006

But by and large the gaming industry doesn’t seemd to understand that the difficult problems they face in regards to data integrity or uptime etc. have already been solved by other people.

Could it be because it requires specialist knowledge? Even the language of uptime ("five nines", etc.) is a less proliferate sub-dialect of system architecture. As you said, there's so much on the radar of a game developer already.

Who needs to write a new chat client or a presence technology when what they really need to focus on is the game and art assets?

Totally agree. Is it a question of money?

I think Steve made a great point when he said that mw* vendors should stop using specialist language and start speaking in terms of the problem space.

* got sick of typing middleware.
covert.c.
October 7, 2006

My point is, why is an micropayment-based account that I’ve spent $200 on more appealing than the WoW account I’ve spent $200 on?

Because your WoW account isn't able to spend more of that money. An RMT system with micro-transactions, at least how I imagine it, is.
digi
October 13, 2006

Because what Justin was talking about is the only thing in this thread that I have any experience with I'm going back a bit here but:

You don't need to go as elite as the F-22 Raptor to see full blown 3D design in effect. We do it here in our office where the entire marine vessel is done (structural, piping, etc) lock stock and barrel in "virual" space. The engineers can run all kinds of tests on it from hull performance to stability. The database even spits out all the sheets of metal that are needed to be cut to the shipyard to maximize space per sheet. Although it's pretty amazing, it's also fairly commonplace. That's all my point is.

But as far as using these tools as a pre-vis for prospective design or even as a low-end final result I think they are more than capable.

My experience with security is pretty humble. The best solution to any question I have is... if you want a secure computer system it's as easy as: Don't put it online.
covert.c.
October 15, 2006

Finally, don’t criticize PunkBuster and AhnLabs and GameGuard...they are a pretty good solution when security is an afterthought.

Actually, that wasn't my intention at all. For RSA the available solutions weren't the troubling aspect, the business of helping solve it was!

Thanks everyone for the excellent discussion.
Pingback GameTomorrow » Security expert weighs in on game security
December 5, 2006

[...] For interested readers, a related discussion took place in early October over at covertcreations. The article is titled, ‘MMORPGs, Security and the Grand Promise of Middleware’. [...]
unbeliever
January 15, 2007

How does one define middleware?

Microforte's Bigworld. Period.
covert.c.
January 15, 2007

I don't agree. Even if I adjusted your sentence to say, "MMORPG Middleware" it still isn't true.
unbeliever
January 15, 2007

Ideals aside, what other MMOG "middleware" infrastructure is there?
covert.c.
January 15, 2007

I'm not sure what you mean by ideals, seeing as the gist of the article argues against blind adherence to any catch-all solution.

But to seriously address your question, meaning that I'll assume you're legitimately asking and not just trying to play some gotchya game of literalism, is that there are indeed several MMO middleware products on the market (or on their way).

There's HeroEngine, Bigworld (as we've agreed) and Gamebryo. A looser definition of MMO middleware would include the Multiverse and Kaneva offerings, which include the hosting service as well. They're all linked at the top of the article.
Pingback covert creations » My adventures in ASCII
February 5, 2007

[...] Most widely read post : MMORPGs, Security, and the Grand Promise of Middleware Got some good traction, thanks to Steve Davis and the good folks at IBM. [...]
John1004
June 4, 2009

Very nice site!
John1761
June 5, 2009

Very interesting site. Hope it will always be alive!
John1596
June 5, 2009

It is the coolest site,keep so!
John1761
June 5, 2009

Great work,webmaster,nice design!
John1596
June 5, 2009

Excellent site. It was pleasant to me.
John1761
June 5, 2009

Great site. Good info
John1596
June 6, 2009

Perfect work!
John963
June 6, 2009

Great .Now i can say thank you!
John1761
June 6, 2009

Perfect site, i like it!
John1761
June 8, 2009

Great site. Keep doing.
John1596
June 8, 2009

Beautiful site!
John963
June 8, 2009

Perfect work!
John1596
June 8, 2009

Nice site! Thank you!
John963
June 8, 2009

Incredible site!
John1596
June 8, 2009

I bookmarked this guestbook. Thank you for good job!
John963
June 8, 2009

I want to say - thank you for this!
John963
June 8, 2009

Great work,webmaster,nice design!
John1761
June 8, 2009

Great .Now i can say thank you!
John1596
June 8, 2009

Very interesting site. Hope it will always be alive!
John1596
June 8, 2009

If you have to do it, you might as well do it right

MMORPGs, Security, and the Grand Promise of Middleware

48 Comments »

what say ye? leave a comment

commendably covert

recent forum posts

Archives:

Categories:

Meta:

go covert!