a quick word of explanation for MFO "blinking" in the past week

David_Snowball

Dear friends,

We try to keep folks informed about developments at the Observer, even when they're not strictly fund-related. A number of folks have noticed that MFO has been "blinking" lately; we've been going offline for between one and five minutes. I know that's frustrating and worrisome to folks, those of us behind the scenes more than most.

The short version is this: there's a storm raging on the internet as bad guys become more sophisticated and powerful. They're targeting some of the upstream servers, the places that smaller servers like Green Geeks feed into. While we're not at all a target, the backwash from those attacks affects us and, in some cases, causes our server to restart. The momentary outages we experience are largely a result of those failures and the time it takes to redirect our traffic or get our home server back online.

We're trying to add another layer of redundancy to minimize the effects and there's some hope that the upstream folks are getting better control of the situation.

For folks whose level of technical sophistication dwarfs mine (uhhhh ... that would be more junior high kids with a smart phone), here's the official word that we received from Green Geeks:

I do apologize I do see a network upstream problem that is effecting the cabinet that your server is stored on. Our failover has already switched and things are stable at this time, however may take a few minutes for DNS propagation to fully kick in for you.

Unfortunately as you may have been aware, the internet as a whole is currently experiencing some pretty significant DDOS based attacks. These attacks however are not targeted at your server or even our data center locations but instead to NTP server relays. This has generated a large amount of pressure for upstream providers and they have been scrambling over the last few weeks to patch this. As a result, unfortunately we have to continually cycle through our backup upstreams as a network is congested. Based on feedback from our upstream providers this should fully stabilized by the end of the week!

We'll keep working at it and I'll share any news as it becomes available.

Back to writing about the nice folks at Dreihaus and another cool one-star fund.

David

Old_Joe

I'm not overly impressed with explanations that use the non-word "effecting" instead of "affecting".

Snarl!

JohnChisum

Thanks David for the update. I was wondering why the site was going down especially since it's a new location.

catch22

Hi OJ,
You noted:
I'm not overly impressed with explanations that use the non-word "effecting" instead of "affecting".

I do believe we had the same thought, at the same time.

I'll have to let it ride though; if'an they are very good techies.

Hey, only -16 degrees with a strong wind at Thursday evening. Almost March 1 and things should change any day now.....
Glad to read your area is receiving some rain.

Lastly, I performed the "click" upon the quote icon; but I suspect not the same as the old "reply". Don't know that you receive any type of notice about the reply.

Take care,
Catch

David_Snowball

Be nice to the geeks or you'll force me to note that "effecting" is not a "non-word" (normally rendered as "nonword"), it's merely a misspelled one - hardly worse than when senior members of the Observer family (ahem) mistake "peak" for "pique."

Still riting!

David

Accipiter

as we segue from blinking to ...

http://www.darkreading.com/vulnerability/no-easy-solution-to-stop-amplification-a/240165528

http://www.computerworld.com/s/article/9246230/Attackers_use_NTP_reflection_in_huge_DDoS_attack

Old_Joe

Hey! I are also a geek. That common error sort of peaks my pique. I really don't think that it's a misspelling... it's my observation that lots of folks don't even know that there are two different words involved.

Don't mind me, anyway. I'm grouchy because a major geek project here is giving me fits. And besides I can't spell nonword. Or maybe I can?? As I was typing this the computer just flagged "nonword" as a misspelling and suggested either "non word" or "non-word". But when I look up "non-word" in the computer's dictionary, that says "non-word" aint even a word and says to use "nonword". Go figure. Enough already!!

I really like the clean look of the new software, you guys. Nice job! We'll have to see about a rewrite on the user guide after (if) I get things under control at this end.

Take care... OJ

InformalEconomist

We all ("W'all") B geex.
No harrum, no fowl.
See wut I'm sayin'?

Anyhoosumever, thanks awfully for writes about upstream DDOS attax.

cman

I am skeptical of the explanation. There is no continuing internet "storm". Seems to me like they are using that as an excuse to hide over/poor provisioning of their virtual servers where a few high activity sites keep starving others out on the same physical server. This is a common problem with virtual servers especially with cheap hosting places that attract a disproportionately larger number of abusing and questionable clients.

David_Snowball

You're certainly welcome to your skepticism. That having been said, this month's 400 Gbps and 350+ Gbps DDoS attacks - the largest in internet history - feel like storms to many. "This style of attacks has grown dramatically over the last six months and poses a significant new threat to the web" (CloudFlare, 13 February). Arbor Networks, a security firm, reports four subsequent attacks in the 100+ Gbps range and a dramatic rise in the size of mega-attacks. What's been described as "a massive and concerted attack" is partly responsible for the bitcoin collapse. For a broader take, see, for example, this article from PC World.

We selected Green Geeks because of an exceptional record for reliability, responsibility and customer support. They're not cheap, at least by the standards of our size and budget, but they have been remarkably responsive.

David

cman

Note I said continuing storm. There were episodes you mentioned earlier in Feb that brought certain sites down and even those were managed in short amount of time. The problems here are continuing sporadically over weeks and for very short periods. This morning was terrible. This is not a typical signature of such storms. No such massive attacks are currently being reported by any site. The NTP server attacks require specific sites to be targeted by spoofing one or more of its IP addresses.

I am not trying to criticize your choice. There are a lot of parameters that go into these decisions but it would be wise to validate explanations from ANY provider. You can easily test this by trying to access another site hosted by GG with a different IP address. If other sites are working fine, then it is more likely a resource starvation in your assigned physical server and unlikely that only this particular server IP address was targeted.