[wikka-news] When it rains...

[Marjolein Katsma]

The old say "when it rains it pours" has been very applicable to Wikka the 
last week or so...

We were just recovering from the shock of about a day downtime as a result 
of our DNS problems ( about a day outage 11-12 November, see our blog at 
http://blog.wikkawiki.org/2007/11/12/dns-problem-solved-wikkawikiorg-is-back/ 
) and traffic was picking up again but not quite back to its old level even 
on the 13th, due to the common delay in worldwide DNS propagation (a change 
can take 72 hours to reach all DNS servers in the world)...

And then another disaster struck. While throughout the DNS problems we were 
still reachable from some locations (due to the propagation delay), as of 
the 14th we suddenly were not reachable at all. Not even via our IP 
address. Our host (through an obviously very junior tech support person) 
wasn't very helpful in initially diagnosing the problem as a domain 
expiration citing from obviously still outdated data instead of going to 
the authoritative DNS server to check. Finally we got the real diagnosis: 
another domain hosted on the same IP address had been hacked, and now had a 
page used for that most abominable of Internet scams: phishing. Our host's 
"upstream" (the service provider they get their connectivity from) had - 
after receiving an abuse report about it -  null-routed the IP address.

Surely that was the right action to take. But we were victims in that we 
were not only not able to right the wrong, but our sites were as cut off 
from the world as the hacked domain was. And our host wasn't exactly 
forthcoming with information.

In the midst of the panic (what do you do when you're helpless?) we took 
some emergency measures: we started by grabbing all installed Wikkas (the 
main domain and the subdomains) and the corresponding databases, which 
luckily we could still access via an administrative login. A temporary 
location was found to put up all this data. During the scare of the DNS 
problems we had also already registered a few extra domains. The plan is to 
set up a system where - should disaster strike again - we can have a mirror 
up and running with a recent version of the databases, practically at the 
drop of a hat.

Luckily the offending phishing site has now been removed and our 
connectivity restored.  Phew!
Keep an eye on our blog http://blog.wikkawiki.org/ for more...

Meanwhile we continue to work on our backup plan - we were not ready yet, 
it's a somewhat complicated operation - so we will be able to quickly 
migrate to another server should anything happen to our main domain, or our 
connectivity on our main host. We will also set up an independent 
"information page" where you can go for information should anything seem 
amiss with our site.

We hope we'll never need our backup plan!

Many thanks to our members who quickly alerted us of the problems, and to 
all of you for your patience.  We're still alive and kicking!

--
JavaWoman
Web Standards Compliance Officer, Wikka Development Crew
http://wikkawiki.org/JavaWoman
Skype: callto://goneagain