During these days, pratically everything on a Web site dynamic. So, basically, the URLs on the page are subject to change from hit to hit. For example, if you have a page that shows a list of items, those items are subject to change from hours to hours as new items are added. So, basically, this adds new URLs to your page, assuming the list provides a link to the detail page. The same goes with sites which encrypt the URL. In such circumstance, you end up with a new URL everytime for the same link because the encryption routing will generate a new URL each time because of its algorythm.

But, the indexing robots are really smart. Basically, they are going to hit your site a lot as long as those URLs keep changing. So, if they access the home page, assuming such link would be present, they will visit you on a frequent basis because they detect that a URL has changed. So, because they are freaking out on indexing everything, including your refrigerator, this creates simply a loop.

From time to time, we can forget about that. And, sometimes, this is just the way it has to be done. So, if it has to be done like that, the best is to put a javascript link to the link which will call a javascript function from the page which will know where to go. But, the function should be as simple as GoToMyLink() with no parameter as if their is a parameter, they will detect that this is a new link. So, by that, all the links remain as is on the page and occasional indexing will apply.

I just thought I would share that information as I forgot a link like that on my page and I just found out why those robots were so in appetite to vist my page.