I could hardly tell you what proxy configurations were before this fall. The reason is I started using Screaming Frog’s SEO Spider Tool this fall.

Their tool is great for finding the strengths and weaknesses in a site’s architecture and link profile because it’s so precise and thorough. Site architecture means the stability and ease of a site’s navigation and structure. Link profile means the amount and quality of the sites linking to me, who I’m linking to and how I’m linking within myself. I hope these definitions cleared up the SEO lingo.

Screaming Frog defines their Proxy feature like so:

This feature allows you to use a proxy with the SEO spider by specifying the address and port.

My first few times using Screaming Frog were a combination of beginner’s luck and fool’s paradise. My first crawls either brought more than enough data I was looking for or nearly no data at all. I was able to find some answers on proxy configurations after some research, a few broken computer monitors and teardrops.

*Note* You can successfully crawl with the option of specifying your desired Web address and port. Just make sure you know about proxy configurations in advance. I didn’t, which is why I wrote this tutorial.

Beneath are the steps and screenshots of using correct proxy configurations for a successful crawl.

1. Here’s your starting screen on Screaming Frog.

The standard Screaming Frog interface

2. Select Proxy at the very bottom of the Configuration menu.

Proxy is at the bottom of the Configuration menu

3. This box will pop up on-screen.

A checked selection in the User Proxy Server

4. This is the futile outcome of crawling a site with a checked selection.

The results of a checked User Proxy Server selection

5. This box pops up on-screen when selecting Proxy at the bottom of the Configuration menu.

An unchecked selection in the User Proxy Server

6. This is the fruitful outcome of crawling a site with an unchecked selection.

The results of a checked User Proxy Server selection

Congratulations. You are now on your way to SEO wizardry.

You can now get your crawl on with knowing the proper proxy configurations because it’s good to find out how to improve your site’s architecture and link profile.

Please tell me your success or problems you’re still having crawling below.