Sign up to save your podcasts
Or
Share 013 Spiders Eating Your Servers? Impact and How to Fix, with Charlie Arehart
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
013 Spiders Eating Your Servers? Impact and How to Fix, with Charlie Arehart
Charlie Arehart talks about “Are spiders eating your servers? The impact of their unexpected load and how to counter it” in this episode of the ColdFusion Alive Podcast, with host Michaela Light.
These days a lot of people are concerned about site security and they'll … Either they or someone in the organization will run some sort of a scam against their server, but when they run that scan, again what are they doing? They're trying to call lots of requests with different forms of potentially bad break-in type code. And those tools, sometimes they're not as careful and they might make a request every second." - Charlie Arehart
Show notes
Why your ColdFusion app being slow might not be your SQL, JVM, or even your CF code
What exactly is unexpected load
What kind of unexpected traffic have you seen
Is it a lot of traffic?
Is it common to find on CF servers?
What can be the crazy problems with unexpected traffic?
But, but, what about intranet-only sites - aren’t they safe from unexpected traffic?
How you can mitigate these problems fast
What about robots.txt? Doesn’t that block bots?
Ok, but aren’t spiders and bots getting smarter, so harder to handle?
Why are you proud to use CF?
WWIT to make CF more alive this year?
What are you looking forward to at Into The Box?
The move to the countryside
Charlie says:
For years I've watched people try to tame "server problems" with a focus on their code, their SQL, the jvm, and so on. Yet often it turns out that the root cause is actually unexpected load. And that load may be from things you never expected (automated), at volumes you never expected. I've found folks with as much as 80% of their web traffic to be such unexpected automated traffic! Worse, there are characteristics of such automated visits that may actually have MORE IMPACT than "real users": for instance, did you know they create a new session/client variables, and run session startup code, for each page they visit?!
The good news is there are solutions to better manage (or simply block) such automated requests which may already exist in your environment, and tools you may consider (some free, some commercial) which can be easily implemented. There are even SAAS solutions that could help alleviate such problems with just a single tiny change in your environment! You may also want to consider some admin configuration options related to sessions and/or client variables, as well as reconsider some coding choices in your session startup code.
In this session, veteran ColdFusion server troubleshooter Charlie Arehart will guide a more detailed review of the issues above, including how to identify such traffic, more on these specific impacts, and most important identifying the solutions along with their pros and cons. He has helped shops achieve dramatic reductions in impact from such automated requests, resulting in greater server stability and performance.
Mentioned in this episode
Web spiders
Yandex
Bidu
Yahoo slurp
User agent
robots.txt
Listen to the Audio
Bio
Charlie Arehart
A veteran server troubleshooter who's worked in enterprise IT for more than three decades, Charlie Arehart (@carehart) is a longtime community contributor who as an independent consultant provides short-term, remote, on-demand troubleshooting/tuning assistance for organizations of all sizes and experience levels (carehart.org/consulting).
Links
Website
Twitter
Facebook
LinkedIn
(* WWIT = What Would It Take)
Interview transcript
Michaela Light: I'm here with Charlie Arehart, veteran ColdFusion server troubleshooter. Hey Charlie. We're going to be looking at some very strange stuff, which is why your ColdFusion might be slow, but it's nothing to do with your SQL JVM or even your CF code. And what we're going to be talking about is an unexpected load.
So,
View all episodes
By Michaela Light
4.8
55 ratings
Charlie Arehart talks about “Are spiders eating your servers? The impact of their unexpected load and how to counter it” in this episode of the ColdFusion Alive Podcast, with host Michaela Light.
These days a lot of people are concerned about site security and they'll … Either they or someone in the organization will run some sort of a scam against their server, but when they run that scan, again what are they doing? They're trying to call lots of requests with different forms of potentially bad break-in type code. And those tools, sometimes they're not as careful and they might make a request every second." - Charlie Arehart
Show notes
Why your ColdFusion app being slow might not be your SQL, JVM, or even your CF code
What exactly is unexpected load
What kind of unexpected traffic have you seen
Is it a lot of traffic?
Is it common to find on CF servers?
What can be the crazy problems with unexpected traffic?
But, but, what about intranet-only sites - aren’t they safe from unexpected traffic?
How you can mitigate these problems fast
What about robots.txt? Doesn’t that block bots?
Ok, but aren’t spiders and bots getting smarter, so harder to handle?
Why are you proud to use CF?
WWIT to make CF more alive this year?
What are you looking forward to at Into The Box?
The move to the countryside
Charlie says:
For years I've watched people try to tame "server problems" with a focus on their code, their SQL, the jvm, and so on. Yet often it turns out that the root cause is actually unexpected load. And that load may be from things you never expected (automated), at volumes you never expected. I've found folks with as much as 80% of their web traffic to be such unexpected automated traffic! Worse, there are characteristics of such automated visits that may actually have MORE IMPACT than "real users": for instance, did you know they create a new session/client variables, and run session startup code, for each page they visit?!
The good news is there are solutions to better manage (or simply block) such automated requests which may already exist in your environment, and tools you may consider (some free, some commercial) which can be easily implemented. There are even SAAS solutions that could help alleviate such problems with just a single tiny change in your environment! You may also want to consider some admin configuration options related to sessions and/or client variables, as well as reconsider some coding choices in your session startup code.
In this session, veteran ColdFusion server troubleshooter Charlie Arehart will guide a more detailed review of the issues above, including how to identify such traffic, more on these specific impacts, and most important identifying the solutions along with their pros and cons. He has helped shops achieve dramatic reductions in impact from such automated requests, resulting in greater server stability and performance.
Mentioned in this episode
Web spiders
Yandex
Bidu
Yahoo slurp
User agent
robots.txt
Listen to the Audio
Bio
Charlie Arehart
A veteran server troubleshooter who's worked in enterprise IT for more than three decades, Charlie Arehart (@carehart) is a longtime community contributor who as an independent consultant provides short-term, remote, on-demand troubleshooting/tuning assistance for organizations of all sizes and experience levels (carehart.org/consulting).
Links
Website
Twitter
Facebook
LinkedIn
(* WWIT = What Would It Take)
Interview transcript
Michaela Light: I'm here with Charlie Arehart, veteran ColdFusion server troubleshooter. Hey Charlie. We're going to be looking at some very strange stuff, which is why your ColdFusion might be slow, but it's nothing to do with your SQL JVM or even your CF code. And what we're going to be talking about is an unexpected load.
So,