Home Blog Sometimes, You Just Have to Write Your Own SEO Tools


Sometimes, You Just Have to Write Your Own SEO Tools

Posted by drew on Nov. 17, 2015, 11:42 a.m.

The internet is filled with tools to help SEOs, PPC advertisers, and other web marketers do their job. At Fusionbox alone, we use Moz, SpyFu, SEMrush, Google Analytics, and a whole host of other tools to help us manage our analytics, SEO, paid search, social media marketing, and email marketing efforts. Some of these tools are free and others require subscriptions. However, when we can’t find an existing tool that meets our needs (or if we feel it isn’t worth the money to pay for it) we get creative and build our own.

That is the case with eek, our homemade web crawler to review site pages and metadata in a CSV. There are many web crawling tools out there, the most popular and well-known of which is probably Screaming Frog, but we decided to write eek for a few reasons:

-It is written in Python

    Python is our favorite language because of its clear and easily readable code. When an entire company of developers could potentially be working on this project, having easy-to-understand code is very important.

-It’s open source

    This means it is free, and not just for those of us who created it, but for everyone. This also means that anyone has access to the code. If they see a way to improve it, they can make those changes and submit them. If they want to add an extra feature, it is incredibly easy.

-Eek lets you change the way you crawl the site

    Most crawlers simply take a URL as an input and crawl all linked pages on that domain. Eek will do this, but it will also allow you to change your crawl in certain ways. You can set a time delay in between fetches, or determine a certain pattern (using a python regular expression) and only crawl URLs with that pattern. This is useful if you are only concerned with a particular section of the site for the moment (e.g. the blog, or the services pages).

It is our “do it yourself” attitude that led us to create eek and that helps us create custom web software and custom web designs, never relying on templates.