Sometimes, You Just Have to Write Your Own SEO Tools
Posted by drew on Nov. 17, 2015, 11:42 a.m.
The internet is filled with tools to help SEOs, PPC advertisers, and other web marketers do their job. At Fusionbox alone, we use Moz, SpyFu, SEMrush, Google Analytics, and a whole host of other tools to help us manage our analytics, SEO, paid search, social media marketing, and email marketing efforts. Some of these tools are free and others require subscriptions. However, when we can’t find an existing tool that meets our needs (or if we feel it isn’t worth the money to pay for it) we get creative and build our own.
That is the case with eek, our homemade web crawler to review site pages and metadata in a CSV. There are many web crawling tools out there, the most popular and well-known of which is probably Screaming Frog, but we decided to write eek for a few reasons:
-It is written in Python
Python is our favorite language because of its clear and easily readable code. When an entire company of developers could potentially be working on this project, having easy-to-understand code is very important.
-It’s open source
This means it is free, and not just for those of us who created it, but for everyone. This also means that anyone has access to the code. If they see a way to improve it, they can make those changes and submit them. If they want to add an extra feature, it is incredibly easy.
-Eek lets you change the way you crawl the site
Most crawlers simply take a URL as an input and crawl all linked pages on that domain. Eek will do this, but it will also allow you to change your crawl in certain ways. You can set a time delay in between fetches, or determine a certain pattern (using a python regular expression) and only crawl URLs with that pattern. This is useful if you are only concerned with a particular section of the site for the moment (e.g. the blog, or the services pages).