MOSS 2007 has the option to use a dedicated web front end server for crawling content:
Why would you want to do this:
Advantages:
- Search doesn’t compete with the end users – Large environments that need to crawl constantly can cause more traffic than normal user load, you don’t really want your users to experience slow pages just because your doing an index? By using a dedicated web front end that isn’t part of the load balanced cluster, your indexing won’t impact your users as much (I say as much, because you still need to think about the impacts of the database server).
- Easy to move the WFE into the load balanced cluster – It’s a rather crude disaster recovery method, but it’s not that hard to move this box into the load balanced cluster if you really need the extra capacity or if one of your other servers fail. After all it’s just a normal web front end, but one that is reserved for the indexer.
- Perfect place to run a backup central admin – You should always try to have more than one server running central admin (on a large farm anyway), that way if your main central admin server goes down, you still have a way to manage the farm.
Disadvantages:
- More hardware – The obvious disadvantage to having an extra machine is the requirement of more hardware, which also means:
- More cost – New hardware is an additional cost, but now that you have an idea of the advantages it brings, you can make a more informed decision.
Joel has some other tips such as adding a robots.txt to servers that you don’t wish to participate in the indexing process.