Forum Settings
       
Reply To Thread

Raising server Capacity vs Just Creating new worldsFollow

#1 Sep 02 2013 at 8:55 AM Rating: Good
Like many of you I wondered about just upping the Capacity of each world against just adding new worlds. In my research I came across a really good comment on reddit from an IT guy. Note this is NOT my own works just something I am passing along. It is rather lengthy but well written and might interest some.


Quote:
I wrote this comment in reply to a post asking 'They refuse to add new NA servers, why? Can anyone explain this?'. I felt as though I could explain this, at least in part, and so I did. And here it is.

I only had about ten minutes to spare to write this up before I went back to scaling our own systems, so I'd love to hear some other people who work in the industry chime in with their experiences, examples, and opinions. Maybe we can get rid of this 'MOAR SURVURS DUH' attitude that some people seem to have.

Edit: Oh, and this post is intentionally dumbed-down, not because I doubt my fellow redditors but because I had to rush through writing it. Obviously the issues are significantly more complex than the examples I lay out here.

As someone who works in infrastructure/servers/networking/IT/etc. for a company that does large-scale multiplayer games, I might actually be able to.

First, not everything scales linearly. Within a given 'world', in the servers that handle people online, you may have one 'server' that can handle 5000 people online at once, but adding a second server may not get you to 10000; two servers together may only be able to handle 9000 people; three servers might only get 12000, four might allow for 14000, and five might support 16000.

This can be due to the overhead of managing multiple characters and multiple interactions. If you have two people in one zone, and you're updating their positions every 1 second, you have to send 2 updates every second per person (updating each person with their official location and everyone else's location), so 4 updates total. If you have 4 people (twice the number of people) you have to send 16 updates per second. If you have 8 people, you're sending 64 updates per second (each 8 person getting 8 updates). This is a really simplified example, but it shows how 2+2 can be a lot more than 4.

So if you have a case like that, you have the option of spending those 'five servers' on one world to handle 16,000 people, or five worlds to handle 25,000 people. This is why the solution to capacity problems in MMOs is usually to open new worlds, and not just to grow existing ones.

Obviously these systems are more complex. A single 'world' is made up of dozens of components (character servers, combat servers, chat servers, instance servers, dispatch servers, login servers, etc.), and each one of these systems could have the same issues, and completely different load profiles and scaling issues. Because 'instance servers' appear to be shared across all worlds, they have to handle the capacity of every single dungeon, instanced fight, etc. for every single NA/EU world concurrently, which means that they scale completely differently than the rest of the worlds.

Another problem those servers had was that because no one could create an instance, a lot of people got backed up at the same points; before Sastasha, before Ifrit, before your level 20 class quest, etc., so now instead of having players spreading out across the level curve, you have huge clusters of people catching up to each other like something out of Amazing Race.

Then they bring the instance servers back up, and everyone rushes to do their instances. All the Ifrit fights, all the lv5 quests, all the Haukke Manor runs. Now suddenly instead of having instances spread out because levels are spread out, you have a huge proportion of players all trying to get into instances at once, and your load spikes, and now no one can get in. Now it's a completely different problem; instead of being unable to handle the common case of instance requirements, you can't handle the case of a large proportion of people online trying to run an instance, all happening at once.

One of the problems with servers is that if your servers are overloaded, it's easy for your monitoring tools to start failing (because the system won't run them because there's too much else going on), and you can have problems logging in. In those cases, you can have servers which hit their capacity in unexpected ways, suddenly, before you have a chance to spot the problem and figure out what's happening. For example, a memory leak that only happens sometimes can take down a server rapidly, and make it extremely difficult to track down because once the server has died you can't log in to debug it.

It wouldn't surprise me if some of their downtime was trying to work around those issues while also adding a lot of debugging information so they could track down what exactly was happening on the server and find the source of the problem (instead of just trying to mitigate it).

So for that instance issue, it doesn't matter how many servers they add to run instances, if they're still going to have them die off too quickly because there's a software bug they need to fix, or because each server adds less and less capacity because of non-linear scaling.

That's just my two cents though. It could be a dozen other reasons.

#2 Sep 02 2013 at 9:02 AM Rating: Excellent
Scholar
****
4,511 posts
Alright, good read. Kudos (Kupos?) for finding it.

Still, i remember reading that 5000 was their absolute max due to all the interactions and updates on locations and movements that get send through the server. It's what makes things move smoothly, it wasnt that they didnt -want- to increase it, it was said that they -couldn't- increase it since it would affect the quality of all the other players playing.

Sure, games like WoW can handle more, that is true. But they also dont send half the information that XIV does. Back when i played that for a bit, i could see characters shoot around on my screen all the time. Player location updates and actions didnt get updated quite as often as they do now in XIV.

I've yet to see anyone stutter in moving around, outside of my own graphical card limits ofcourse. I am pretty sure SE would love to uphold this quality that they're currently offering, or they would have yet another problem to deal with, that they couldn't get rid of.

While i'd love for more than 5000 connections at one point, i really dont want it to affect the quality of my playing experience. Simply making more and more worlds and hope (or offer freely) that people will transfer to that seems like the only option. I know it's not nice to hear, but still.
____________________________
[XI] Surivere of Valefor
[XIV] Sir Surian Bedivere of Behemoth
http://na.finalfantasyxiv.com/lodestone/character/2401553/
#3REDACTED, Posted: Sep 02 2013 at 9:08 AM, Rating: Sub-Default, (Expand Post) Quality they're currently offering?
#4 Sep 02 2013 at 10:13 AM Rating: Good
***
2,536 posts
Good read.

If expanding current servers yields diminishing returns, the obvious choice would be to add new servers after a certain point.

What I don't get, however, is the reason why they haven't implemented an auto-kick for afk peoples already. I'm no programmer, but it seems easy enough to do.

Every other game I've played has this feature. This is the only game I've played that doesn't have it...and it's the only game that actually needs it right now.

Smiley: dubious

Edited, Sep 2nd 2013 11:14am by Threx
#5 Sep 02 2013 at 11:32 AM Rating: Excellent
****
5,745 posts
Threx wrote:
What I don't get, however, is the reason why they haven't implemented an auto-kick for afk peoples already. I'm no programmer, but it seems easy enough to do.

An AFK timer wouldn't fix the underlying issue. It won't even mitigate it. People stay logged in while AFK because of the difficulty logging in during prime time / weekend / holiday. If they only played during off peak hours, they wouldn't need to stay logged in. So it would stand to reason that these players will play during peak times.

If there was an AFK timer, those players would simply be replaced by someone else. Since the player losing his spot would have been playing during peak hours, you aren't increasing the number of people who are playing during peak hours. You're just slightly changing which people are able to play.

You would still wind up with not enough server capacity for all the people that want to play. You would still wind up with lots and lots of upset players who can't get in. There would simply be one less scapegoat for people to blame.
#6 Sep 02 2013 at 11:36 AM Rating: Good
**
576 posts
svlyons wrote:
Threx wrote:
What I don't get, however, is the reason why they haven't implemented an auto-kick for afk peoples already. I'm no programmer, but it seems easy enough to do.

An AFK timer wouldn't fix the underlying issue. It won't even mitigate it. People stay logged in while AFK because of the difficulty logging in during prime time / weekend / holiday. If they only played during off peak hours, they wouldn't need to stay logged in. So it would stand to reason that these players will play during peak times.

If there was an AFK timer, those players would simply be replaced by someone else. Since the player losing his spot would have been playing during peak hours, you aren't increasing the number of people who are playing during peak hours. You're just slightly changing which people are able to play.

You would still wind up with not enough server capacity for all the people that want to play. You would still wind up with lots and lots of upset players who can't get in. There would simply be one less scapegoat for people to blame.


While true, you're forgetting that it also increases congestion for the off peak hours, limiting the number of people who can play during those times.
#7 Sep 02 2013 at 11:55 AM Rating: Excellent
**
448 posts
Point of an AFK timer goes to slaying the 1017 monster, or fear of it. If everyone gets rotated off by an afk timer, it opens up spots, we'd hope consistently. Once people realize they no longer have to fear getting 1017'd for hours, there is no point in remaining logged in once you are done playing. Sure, you can do stuff to get around it and stay logged in with a macro or whatever, but at that point, you don't need to. Your character is not needed in the world for anything if you are not playing. They can't bazaar anything, they are merely placeholders to bypass the login routine. If they is no/little problem logging in, then people wouldn't devolve into the me-first-only savages we have become.
#8 Sep 02 2013 at 11:58 AM Rating: Decent
Scholar
****
4,511 posts
And lets not forget an AFK timer is going to anoy the crap out of us. I wouldnt mind if it was set at an Hour or something. It would drive me to quit if it was set to 5 minutes or so. Knowing SE, they would set it to a few minutes. You alt+tab to look something up, come back and you've disconnected...
____________________________
[XI] Surivere of Valefor
[XIV] Sir Surian Bedivere of Behemoth
http://na.finalfantasyxiv.com/lodestone/character/2401553/
#9 Sep 02 2013 at 12:02 PM Rating: Good
**
448 posts
KojiroSoma wrote:
And lets not forget an AFK timer is going to anoy the crap out of us. I wouldnt mind if it was set at an Hour or something. It would drive me to quit if it was set to 5 minutes or so. Knowing SE, they would set it to a few minutes. You alt+tab to look something up, come back and you've disconnected...


20-30 minutes would be fine. If you are afk for that long, why not just log off? ATM, because you can not log back on, but if you can, then really there is no issue with logging out for a half an hour while you are in the bathroom.
#10 Sep 02 2013 at 12:04 PM Rating: Excellent
**
576 posts
Refews wrote:

20-30 minutes would be fine. If you are afk for that long, why not just log off? ATM, because you can not log back on, but if you can, then really there is no issue with logging out for a half an hour while you are in the bathroom.


I recommend more fiber in your diet. ^.~
#11 Sep 02 2013 at 1:21 PM Rating: Good
Scholar
***
1,098 posts
Nice read, it almost blames the playerbase. A few days after launch an evil thought popped into my head that hasn't left. What if the tech ppl knew what the server load was for digital download for all the 1.0 gamers and the new people that they expect to buy the physical copy of ARR the game and built the servers specs for that. The evil part comes with the infinite digital download for new players how many to sell who would limit that? It's like a marketing cash grab with virtual goodies for the player but no one told the server networking IT people how many they are selling. I think someone in corporate is doing a study of player payment retention to the cost benefit analysis of adding new servers and the gamers are losing. Smiley: frown
____________________________




[ffxivsig]459049[/ffxivsig]
#12 Sep 02 2013 at 1:24 PM Rating: Default
Judging by the price of SE stocks, I think some people are gonna lose their jobs over this FAILED launch.

#13 Sep 02 2013 at 1:40 PM Rating: Good
**
576 posts
The stock is up over 40% since April. Even in the past weeks, it's only dropped a few percent.
#14 Sep 02 2013 at 1:51 PM Rating: Default
Launch Hype.

http://www.marketwatch.com/investing/stock/sqnxf

#15 Sep 02 2013 at 2:10 PM Rating: Good
**
576 posts
Jaimierom wrote:
Launch Hype.

http://www.marketwatch.com/investing/stock/sqnxf



I'm not sure what you're getting at. Overall, the stock is up for the year and has only dropped a small amount after the launch. Nowhere do I see enough of a swing to get people fired (over the stock price, specifically).
Reply To Thread

Colors Smileys Quote OriginalQuote Checked Help

 

Recent Visitors: 226 All times are in CST
Anonymous Guests (226)