DGN Down - Attack of the Robots II

[Troy Spiral (13)]

DGN Continues to be popular with the search engines (which 99% of the time is a good thing, something that websites want and work very hard to get) but in our case we seem to be more popular than is desired (from a cost standpoint). Several of them index our website so much, that just the act of "looking" burns up all of our bandwidth, thus causing the website to go down.

This wouldn't be a problem if we had a larger budget and could just absorb the additional traffic, but such is life we barely have enough donations to cover the "normal" traffic.

Last time around it was Yahoo that (primarily) over-did its searching, this time it is MSN and a smaller search engine called Yandex (but still a monstrous tentacled behemoth compared to tiny little DGN) that tag-teamed us and finally pushed us over the edge, sucking down all our bandwidth for the month in less than a week.

There is no easy solution to this problem as there are zillions of different web spiders and we cant disallow them all as the only place that can really do that effectively is our web host, and the search engines aren't actually doing anything abusive, we are just a bit to big for our britches.

I've reluctantly disallowed the specific MSN and Yandex bots. We'll see how it goes, hopefully we dont go broke.

[prick]

Thanks for the clarrification Troy!

[phee]

We need John Conner.

[Epic_Fail_Guy]

Dude... it's just some fucking punk that's spamming yahoo or google with search results a million times over with you to eat up your bandwidth... and this board's company is too herp derp to realize this or have any system set up to prevent this??

[JonnyDark]

For maybe a future reference. I switched over to globat a couple years ago, it's unlimited bandwidth/traffic, mailboxes etc, except for the databases which is up to 50. They always have new customer deals, which right now is just around/below $54 total for the first year inc private domain name registration and/or transfer. After that it runs me around 80ish per year inc private domain re-registration . Maybe worth the look sometime.

I ended up staying with them to host all my sites/forums, after all the hosting i've used and searched through it's still the cheapest for unlimited usuage with all the extras inc. And their customer service is top notch. Maybe this could help in the future. I understand the $ issues though

~Jonny

http://www.globat.com/

[n0Mad]

I think it's Eevee's fault. She breaks things.

[Troy Spiral (13)]

Dude... it's just some fucking punk that's spamming yahoo or google with search results a million times over with you to eat up your bandwidth... and this board's company is too herp derp to realize this or have any system set up to prevent this??

That would be easy to fix if that was the case. Honestly though it is "legit" traffic, not any hacker type stuff. We just sort of crossed a threshold or some such a few months ago were we get more attention from the search engines, more akin to a medium sized website as opposed to a low-budget one like we are.

Normally that would be a good thing, as typically better placement in search engines = more "real" website traffic = more donations, just in our case, probably mostly my fault, im loathe to push for donations very much the way most "larger" (dont websites do.

[Troy Spiral (13)]

For maybe a future reference. I switched over to globat a couple years ago, it's unlimited bandwidth/traffic, mailboxes etc, except for the databases which is up to 50. They always have new customer deals, which right now is just around/below $54 total for the first year inc private domain name registration and/or transfer. After that it runs me around 80ish per year inc private domain re-registration . Maybe worth the look sometime.

I ended up staying with them to host all my sites/forums, after all the hosting i've used and searched through it's still the cheapest for unlimited usuage with all the extras inc. And their customer service is top notch. Maybe this could help in the future. I understand the $ issues though

~Jonny

http://www.globat.com/

Unfortunately they dont support the IPB/G software directly, which is the main feature I need over any other. Not "just a webhost" so to speak. Going with IPS itself vs more budget solutions had been worth it. Hard to put a pricetag on guys that actually write the software and half the time fix problems before we even know they are problems.

Its a good suggestion and I'm glad you think enough about DGN to offer it, its just hard to compare the two types of services.

[JonnyDark]

Ahh gotcha, I've only used PHPBB , e-commerce, wordpress etc etc since using Globat hosting, after IPB started to charge for the software.

IPB is one of the very few that they don't support.

Take it easy,

~Jonny

[Pulse State]

Yandex is one of the biggest, if not the biggest, Russian search engines. Since this board is not in the Russian language, nor does it really cater to anyone there or in the former Soviet sphere of influence, I think it's safe to say that you could disallow hits from that one without affecting your overall rankings and traffic too much.

[Eevee]

I think it's Eevee's fault. She breaks things.

!?

Since when?

[phee]

!?

Since when?

She breaks boys hearts.

[prick]

DaMN ROBOTS!!!!!!!!!!!!!

[Eevee]

She breaks boys hearts.

[n0Mad]

She breaks boys hearts.

And some girls too.