What the heck is MonkeyCrawl?
You may have seen a robot called “MonkeyCrawl” running around on your site. This is part of our project to create a BitTorrent search engine, which collects documents from the Web in order to build a large, searchable database of .torrent files (and related metadata) from the web. This project is still under development, but will be available for people to try out soon.
How do I request that the bot not crawl parts or all of my site?
You can use a robots.txt file, which can give specific instructions to MonkeyCrawl and other bots to not visit certain parts of the site. More information is available here.
Why would I want my .torrent files indexed?
Lots of reasons! First, the way we do your search listings ensures that people have a high likelihood of visiting your site if they like your torrent files. All of our titles say “Torrent File from Site” where the site is a link that takes them to your website. We also print out the “comment” in the .torrent file, which you can use to further advertise your site.
Secondly, the heart of the BitTorrent concept is “Give and Ye Shall Receive.” We are trying our best to make a service that gives back to the BitTorrent community, and to help it thrive by making it easier and easier for new people to sign on. There are a couple large sites that very popular, but there are also lots of small sites doing very cool stuff, and we want to help those sites grow.
However, if you would still prefer that your .torrent files don’t appear in our search engine, you can use the robots.txt exclusion protocol. Just read the questions above.
I have a different question that’s not on this page…
Cool. Just e-mail us and we’ll get back to you… click on “Contact” on the left sidebar to give us a holler.