Wah, I don't want to give up that feature...

Popularity and performance

Published on May 19, 2006 By Draginol In Websites

Anyone who's been around Stardock's various websites knows the pattern. We find a new way to make things faster or upgrade some piece of hardware to make the sites run smoother and the traffic instantly jumps which brings back the problems for the regulars.

The #1 performance hogging feature on all of Stardock's sites (whether it be JoeUser or WinCustomize or GalCiv2.com or TotalGaming.net) is the same thing -- the referrals. That is, all items actually look at where a visitor is coming from and then adds to a massive table in a massive database on where that referral came from and what the URL was to get back to it.

When you have, literally, hundreds of thousands of people coming to the site every day (which is thousands per hour) it's a significant strain. There's no OFF button for it, it would be a lot of changes in a lot of places in code.

But people feel the pain of that database being hit because it's tied to the article database -- commenting timing out being one of the most common.

We have a new server for the database that is a real monster. It has 12 gigabytes of memory, 4 dual core CPUs, you get the picture. But it will take some weeks to migrate things to it. The same person doing the migration could be asked to go and turn off the referrals which would make things nicer but that would delay the migration which will make everything faster. Hence the dilemma.

I am tending to just let it suffer so that the migration goes quicker and refrain from writing articles like this one (that is why today was so slow) until the migration is done.

Comments

WebGizmos

on May 19, 2006

Thanks Draginol! This is the kind of information we need to see more of around here. Its so much better knowing whats going on in the background than being left in the dark.

Incoherent

on May 19, 2006

ok, call off the pitchfork wielding mob.

WolfmanZ

on May 20, 2006

Yes, thank you for keeping us informed. Can you shed any light on 2 issues I'm having?
1. The tab bar at the top of the site will show that I'm logged it but sometimes I still get the ads, smartwords, and am unable to complete tasks as a member (such as posting a comment to this article).

2. Quite often various libraries (most commonly Screenshots) are completely missing from the Gallery page. The page seems to render fine otherwise, just some of the librarys are left off.

Thanks again!

PurrBall

on May 20, 2006

Do what you feel is best, Brad.

Zoomba

on May 20, 2006

any thoughts to clustering a series of smaller database servers and load balance connections out to them? How about a cache engine on the front end that people hit, and the cache engine goes back in and refreshes every minute or two? This could reduce load when viewing the main pages of any of the sites, tossing back a static page that links into dynamic data. The subpages such as forums and individual blogs could remain dynamic. There's also the possibility of farming off the collection of referral data to a secondary database that syncs into the main one periodically. These measures reduce the amount of "immediate change" users could possibly see, but it would spread the load a bit and make the site a little less reactive to spikes in traffic.

The DB you're putting in is huge and should handle a ton of transactions per second, but it looks like it's still a single point of failure. Just tossing out some ideas on spreading the load out and reducing the reliance on the one box.

Frogboy

on May 20, 2006

We're doing much of all of the above already. If we didn't, things would have long since grounded to a total halt.

The gotcha is in the places where we didn't design for clustering early on and so as traffic increases you eventually run into some issue that would have seemed minor in the design but shows up when you have millions of hits a day.

In the long term, absolutely, we need to cluster things that aren't clustered and break things up into more pieces. But it is a very painful, slow process to do that. Just migrating a database to another machine can take many days.

singrdave

on May 20, 2006

Honestly, your efforts are not going unappreciated.
I look forward to reading and interacting on JU every day.
Thanks for your efforts in trying to make it (and your whole company) better.

singrdave

on May 20, 2006

BTW, was that the same Brad Wardell I saw in Friday's FOX Dayside audience? Right towards the end, they said that somebody named "Brad Wardle" (pronounced it like "turtle") had won something. Was that you?

APM0138

on May 26, 2006

Sounds like fun. Any chance we can see the code? Is it a mess? Seriously, I need to learn ASP.NET. Uh, by the way, what does this say about Stardock's code in general?

Kevin_C

on May 26, 2006

what does this say about Stardock's code in general?

I'm sure if it says anything, it says that it's holding up pretty well considering it was probably written before these issues were even comprehended. If you're trying to infer that the code on this site is sub-standard, I think you need to re-evaluate your position. Yes, the site has some problems, yes it can be annoying/frustrating at times, but I think if you worked on a site that got even 1/10th the traffic that this site gets you'd appreciate the technical issues that go into coding for it a bit more. Oh, and I doubt you'll be seeing the code any time soon. If you need to learn ASP.net buy a book, or go to the numerous community sites out there for .NET development.

Brad,
The info is appreciated and I've gotta say, the site has been pretty snappy for me recently so whatever you're doing keep it up!! It really is appreciated!

DJBandit

on May 26, 2006

What?!!! More problems? With all the money I pay to hang out in JU?............ Wait, this site is free, oh never mind. Take your time, anything to make this place better.

(Disclaimer: Just in case no one gets it I was only joking)

seldomseen

on May 27, 2006

Just ticks right along at the moment

APM0138

on May 29, 2006

If you're trying to infer that the code on this site is sub-standard

Well, it obviously must be substandard, as all good code is written in C.

Yes, the site has some problems, yes it can be annoying/frustrating at times

FRUSTRATING? The twelve times I've tried to visit, I might have had a problem! What if I did? Don't you think that would be worse than annoying? I NEED the information on this site, my entire day revolves around it!

I think if you worked on a site that got even 1/10th the traffic that this site gets you'd appreciate the technical issues that go into coding for it a bit more.

Huh? I don't get it? Don't you just slap on another processor to halve the workload?
1 processor: work
2 processor: work / 2
3 processor: work / 4

And so on...

Oh, and I doubt you'll be seeing the code any time soon. If you need to learn ASP.net buy a book, or go to the numerous community sites out there for .NET development.

Wouldn't that require a purchase? I mean, if StarDock has nothing to hide, why not a full disclosure policy? Or is the code that bad?

~A/PM

P.S. FLAME: You don't seem to understand tongue-in-cheek humor, so in the hope of appealing to the lowest common denominator, this one is blatant sarcasm.

Jafo

on May 29, 2006

You don't seem to understand tongue-in-cheek humor,

That would be 'wry humour'.... Spell checker

Often it will only appear funny to the author.

To the rest, the necessity for its explanation belies its worth....

Kevin_C

on May 30, 2006

You don't seem to understand tongue-in-cheek humor

Of course, labeling your previous post as humour implies it has even the slightest element of humour for those of us who aren't sitting with you at your desk as you type it to pick up on. Which your statement had not an inkling of humour to it.

so in the hope of appealing to the lowest common denominator

You're already posting the comment, so why attempt to appeal to yourself. Oh, and your comment speaks volumes about your character and maturity but then again I guess we shouldn't expect too much.

That would be 'wry humour'

More like 'Dry humour'!

Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:

Richer content, access to many features that are disabled for guests like commenting on the forums.
Access to a great community, with a massive database of many, many areas of interest.
Access to contests & subscription offers like exclusive emails.
It's simple, and FREE!