To business hell and back.
Still recovering.

Leipzig, Germany - Imagine this: You're running a business over the internet. You're rather new to the game, a small company, so you're depending on news sites to publish your releases. Also, most of your customers are residents of the United States.

One day in June 2004, you're sending out a news item which you believe will easily make it to the front page of MacNN. But the day goes by, unnoticed, nobody seems to care. "Well", you think, "better luck next time. But then... did Apple release a G5 Powerbook, or what? Whatever..."

Several months later, you're sending out another newsletter, which again should be of great interest, but again nobody cares. Coverage in Germany only, US-sites seem to ignore you. You start to wonder: What exactly did you do before to raise interest in your company and your products. How did you get Macworld, the PowerPage and all those guys turn their heads and publish your stories?

And most of all: Why on earth are you seeing less and less customers? No news, no business? Sucks.

Then, one rainy winter morning, your mail.app inbox shows 12 unread messages. Most are regular stuff; feature requests, licensing issues, support questions. But there's this one mail, header tentatively titled "Have you seen this?", body only featuring a link to one person's weblog. You click it in minor anticipation. Only four paragraphs. Makes your heart jump. You can almost taste the paleness your face has adopted:

“As of writing this, [blue-tec.com] hasn’t worked in over a month. The website is down, forums are down, and email support down likewise.”

So... what are you going to do, now that you just quit smoking?

At first, this looks like one single guy's problem. Misconfiguered his router. Well... maybe. But then again, for all you know, and for all you've learned, this life features but a few certainties. One of them says "there's more where that came from". You just know that if there's one case, there's got to be others. You don't even need to consult Murphy for this. It's the laws of nature. Singularities only exist in outer space.

After recovering from a row of nervous breakdowns, you begin to understand. Suddenly, you see the bigger picture, pieces start falling, start fitting together. That one guess, this one, haunting idea, that's been lurking in the back of your head for the last several months, leaps forward and starts occupying your every thought: Are we getting blocked? Could it be as... devestating, as... simple as that?

And again: What do you do now?

Well, we left a comment on the weblog and started to have a short conversation over there (eMail didn't work, of course). No results, but at least that one customer knew that we were still alive. While we were further trying to investigate (you don't know how helpless you can possibly feel, until you've experienced something like this — no clues, completely tapping in the dark), another posting popped up. This time on VersionTracker:

“I feel as though I’m stuck in the Twilight Zone. I’ve not been able to access the blue-tec site for over 3 months [...] the site never comes up.”

Dang!

Trying to stay calm, trying to get in touch via VT's comments (no response), trying to solve the puzzle without any hint whatsoever. IP says they don't know, offering a forced DNS update. Accepted, no success. Are we spamming? Somehow? Did we get hijacked? Doesn't look like that. What the... You got mail. What's it reading? "Check 43folders.com!!!" This better not...

“it’s kind of funny to see the Ulysses developer posting [...] ‘What good are 20 dollars if the dev-team doesn't answer your questions?’
Now the site is down and the app has apparently been unavailable for months. so everyone who shelled out $100+ must be enjoying all those ’upgrades‘ now, huh?”

All of a sudden, this one man's problem has turned into a marketing disaster. People start to believe we're out of business. For good. And we all know about the power of blogs. Post a response, quick. Let them know we're online, alive, kicking, hard at work.

On top of this, the situation now calls for a radical priority shift. Stall all development, solve the problem!

Now... do you remember that one line in one of the first paragraphs? The one about singularities, and how there's always more? There's another certainty: "You are not alone."

Turns out another one of our IP's clients found out that he got blacklisted by a US-provider, but he managed to get removed and reported this upon success. Starting from there, our IP worked the night-shift and all of a sudden reported success in finding what seemed to be the major culprit for our alleged downtime: As was the case with the other client, a certain internet provider over in the United States had us blacklisted — and boy was that a big name, or what?

Verizon.

As in: "verizon.com" As in: "No wonder they can't see us." As in: "WTF?" As in: "What now?" As in: "Momma, help. I'm lost..."

We still don't know exactly what happened, or how it came to this. Verizon has apologized and assured that they took us from the list late May. Of course, there'll be no compensation. Of course, there could be hundreds of potential (and, what's even worse: actual) customers out there, believing that we have given up, that we are out of business, that development was cancelled, that they invested in software that turned out to be DoA.

This story has been written in part to tell what has happened to us, and to tell that we're still alive and that it has never been different. And maybe it pops up as a top hit on google when somebody performs a search for "blue-tec down" or some such.

But this story has also been written to show how easy it is to be kicked off the face of the internet. One single switch, one line. This is still all-news to us, we still can't believe what has happened or how to deal with it properly, and we're not finished investigating, but if there's one thing I surely wish existed: A centralized database of blacklists and blocks, or maybe even better a notification of some sorts (a good old letter will do), sent out to either the IP or better yet the domain registrant.

How many websites, how many small companies out there are experiencing the same thing? How do they know what's happening to them? How do they stand a chance of ever knowing?

Fighting evil is great and all, and spam... yeah... spam sucks big time, awright, but this practice sucks worse! Don't nobody reject provisional conviction anymore? I do. We do. Word! Yeah. Really, I do, how come? Damn...

Whatever, though, let's end this, and let's hope it doesn't happen again. Let's also hope that we can recover from this (pretty severe) technical knock out. Uhm, but wait! There's just one more thing: Spread the word, please, would'ya? We're still up, never been down, we got big things up our sleeves, and you really, really, really ain't seen nothing yet.

Greetz
Marcus Fehn
Creative Director