It seems our hosting facility (very expensive hosting facility, let me add) screwed us up yesterday when a typo in a maintenance script started a chain reaction of failures that has led to some unavailability of Myxer services. None of our physical web, database, or file servers were directly affected, rather it’s that the internet addressing and routing systems that allow your web browser (or your phone’s) to reach our servers were misconfigured.
The details of how this mistake worked its way through our infrastructure are gritty, but the end result is that:
- The MyxerTones.com website was unavailable for approximately two hours yesterday morning (8/15/06, 6-8AM ET) . We discovered this fairly quickly (you think?) and were able to reconfigure to repair all requests for myxertones.com.
- During the MyxerTones.com downtime, MyxerTags placed on websites around the world were “dark.” The alternate text was displayed during this time (”Send this ringtone to your phone…”), but attempts to activate MyxerTags would have failed. Similarly, any of our partners using our APIs or data feeds would have experienced failures during that time. At this point, MyxerTags and all other Myxer services are accessible and functioning correctly.
- Download links sent anytime before about 11PM ET last night (Wednesday, 8/15) do not currently work. If you resend any ringtone you’ve made in the past, you will be sent a new SMS message with a functioning link. But if you follow an old link, you’ll either see a “not found” or “404″ message, or you might actually see an “under construction” page (which isn’t even ours!). [Note: we're expecting the old links to start working again for most users within a couple of hours, so when you read this the problem will likely already be fixed.]
A lot of websites don’t have a big problem with going offline for a couple of hours every month or so for maintenance, but we do. Our website is built on the Myxer platform, which is relied on by our partners around the world to deliver something like two or three ringtones every second of every day, so you can imagine how pissed off we are for enduring this downtime. As we strive to make Myxer synonymous with “radically simplified mobile”, any downtime for our website or our partners is unacceptable.
Having identified and corrected this specific problem, we will now be spending some time re-evaluating our hosting environment and plans for future growth. Despite the fact that we spend a lot of time having fun with our website and technology, we are seriously committed to delivering worldclass service whether you’re trying to download a single ringtone to your own phone, or you’re integrated with our platform to mobilize your entire catalog.
Myk (with CTO hat on)
