Our achilles heel. Stardock's server farm has a lot of redundancy. But it has a bottle neck that until the last few months wasn't really a bottleneck.
It's a server called Marvin. Marvin is our Stardock.net account server. It traditionally hasn't been that busy, but as more and more sites start using it, it's gotten bad. Very bad.
The long term solution is to cluster it and use replication to mirror accounts onto other machines to load balance. But that's going to take time.
The short term solution is to get a mongo server in there to do the work. And that mongo server has arrived. It's got 6 hard drives in it to do both RAID 0 and RAID 1. It's got SMP Duo Core 64-bit processors. It's got (I think) 10 Gigabytes of RAM in it.
Once we have that up, the sites will be faster and definitely more reliable. Then we can work on the long term solution which is via replication and mirroring.
Update: The relief has arrived! No more unscheduled down time anticipated.