MyBB is not very optimized for search engines by default. One of the biggest problems is that it does not come with a robots.txt file.
Search engines will crawl all kind of stuff in your forum that makes no sense. Examples are the calendar, the archive, new reply pages and many more.
They will also find a lot of duplicated content in your forum because the same posts can be displayed in the archive, using a direct post URL and all kinds of thread URLs with lastpost or newpost.html additions. MyBB does not ship with canonical URLs by default so you’ll have issues here too.
My currently recommended robots.txt for MyBB with activated SEO-friendly-URLs looks like this:
User-agent: * Disallow: /board/archive/ Disallow: /board/private.php* Disallow: /board/usercp.php Disallow: /board/usercp2.php Disallow: /board/ratethread.php Disallow: /board/newreply.php Disallow: /board/memberlist.php Disallow: /board/printthread.php Disallow: /board/forumdisplay.php Disallow: /board/showthread.php Disallow: /board/member.php Disallow: /board/calendar.php Disallow: /board/thread-*-lastpost.html Disallow: /board/thread-*-nextnewest.html Disallow: /board/thread-*-nextoldest.html Disallow: /board/thread-*-newpost.html Disallow: /board/thread-*-post-*.html Disallow: /board/post-*.html Disallow: /board/forum-*.html?datecut=9999
Updated 09/06/2015, added one rule.
You might need to adjust the /board path if your forum is in a different subpath of your domain or remove it if your forum is not in a subfolder.
From top to bottom I’m removing the archive, because of duplicated content and to prevent search engine visitors from discovering the website in the ugly archive mode.
I do prevent access to private messages and user cp as google or any other bot don’t need that. I do prevent access to ratethread and newreply for the same reason.
I excluded the memberlist too, you might want to include it. I removed the option to print a thread. I excluded forumdisplay and showthread because we are using SEO-friendly-URLs. I excluded the calendar as mentioned before.
Now we are excluding the different writing options of a thread to minimize duplicated content.
Done – your MyBB does use a proper robots.txt now. Google does not have to crawl and index thousands of meaningless calendar years and focus on the main content of your forum without duplicating it.
MyBB for Business!
5 thoughts on “The best robots.txt for your MyBB”
I just updated this post to modify the robots.txt file.
Thanks for the robots.txt
You might wanna block archive too.. to avoid duplicate content penalty.
Archive is actually already blocked by the first line:
Thank you for your comment!
Is disallowing the archive page the best move? Would it be better to somehow add a canonical tag or 301 redirect on the archive pages to the friendly url? I’m worried about losing the search rankings I have for the archive pages if I’m not telling the search engines what to replace them with.
If you have strong rankings for archive pages I’d redirect them to the default forum threads. They provide not the best user experience and you’ll probably have a high bounce rate at archive pages.