RTFM

[Read This Fine Material] from Joshua Hoblitt

Can MSNBot behave?

| 0 comments

Since I blocked MSNBot in my robots.txt it seems to have settled down and is only occasionally (about monthly) pinging my robots.txt. Just for fun, I’m going to remove the restriction today to see if it’s still a bandwidth hog … Continue reading