Spiders again?
- Thread starter zombie
- Start date
-
Hey - turns out IRC is out and something a little more modern has taken it's place... A little thing called Discord!
Join our community @ https://discord.gg/JuaSzXBZrk for a pick-up game, or just to rekindle with fellow community members.
well i dont think there's any problem with spiders atm.. spiders make search engines find utas more and more 
i dont think martz wants to get rid of em coz he can easily stop em afaik ..
needs a few additions and they would stop "spidering"
thx for the notice i'll let martz read this and he can judge himself
i dont think martz wants to get rid of em coz he can easily stop em afaik ..
needs a few additions and they would stop "spidering" thx for the notice i'll let martz read this and he can judge himself
Last edited:
Google has changed tactics, it's spiders used to go out indexing ever 30 days to a site, now they visit once a day. We got a lot of threads googled already from it's initial trawl of the forums when we moved to vB3 (vB2 doesn't spider very well because of the way links and URLS used to be made up). This hit us hard on bandwidth, especially when AskJeeves, Inktomi (Yahoo, MSN and many others) were all on at the same time (about 30 guest bots).
This isn't so much of an issue now since we are saving lots of bandwidth of the improvements made a week or so ago to compress html in each page from around 100k to 15-20k resulting in quicker browsing or spidering with reduced overhead
So yeah bots are currently our friends. I feel it's very imporant the rest of the world can find our LOL thread.
This isn't so much of an issue now since we are saving lots of bandwidth of the improvements made a week or so ago to compress html in each page from around 100k to 15-20k resulting in quicker browsing or spidering with reduced overhead
So yeah bots are currently our friends. I feel it's very imporant the rest of the world can find our LOL thread.
Ok maybe i am stupid but woot the fuck are spiders 
? ... i guess u dont talk about this lil creapy things
? ... i guess u dont talk about this lil creapy things
RighteousFury said:Ok maybe i am stupid but woot the fuck are spiders
Spiders are bots from search engines which go out and find websites (such as this one) and downloads everything with text in it. They follow links to other pages and forums, and all the links on those pages.
Google for example will view/download the entire LOL thread and put each word of text into its database and give it a rating depending on lots of things. So when you search google for LOL - you should find the LOL thread in the results, but there are 4.9 million matches/pages. Search for LOL utassault and the LOL thread is first. Without indexing (the process described above) you wouldn't be able to find any of our content through a search engine.
