102: Crawl Optimization & How To Do Crawl First SEO w/Aysun Akarsu

This is the first of three episodes recorded at TechSEO Boost 2018! I sat down with Aysun Akarsu to chat about:

* How to optimizing your site architecture and crawl paths for better rankings

* Does ‘crawl budget’ actually exist (even though Google says it doesn’t)?

* How to set up a crawl, and what data to pull into your crawl analysis for better insights

* The free common crawl web graph dataset and what she was surprised to learn about backlinks on the web

* What universities can’t teach you about SEO

Listen Now!

BuzzSumo – One of my favorite tools for coming up with content ideas, finding people who share content in an industry, and tons more (like alerts to keep an eye on your competitor’s links). Listen to the show for a special code to get 30% off BuzzSumo for 3 months.

Related Episodes You Might Like

* Optimizing Crawl Budget To Improve Web Traffic w/Vanessa Fox

* The New Technical SEO (In Plain English) w/Mike King

Show Agenda and Timestamps

Episode Introduction [0:22]

Aysun introduction [1:49]

Aysun’s Background and how does she describe what she does? [1:55]

What does analyzing big search data mean on large sites? [2:43]

How to explain crawl budget to clients [4:14]

How does Aysun define “crawl first SEO” and what is the focus? [5:00]

What are some tools Aysun uses to collect the needed data? [5:58]

What tools does Aysun use for storage and calculating metrics  [7:36]

Are there any other data sources that she pulls together for the crawl first analysis?  [9:27]

What metrics does Aysun focus on? [9:55]

How to prepare before the crawl [10:43]

How does Aysun gather the needed information from her clients [12:58]

What is the importance of having the correct IP or user-agent? [16:18]

What is an example of crawl waste? [19:22]

Should you always tell the client when their site will be crawled or does it depend on the size of the website? [20:24]

The test crawl [21:30]

Aysun breaks down crawling into 4 categories- what they are and what they mean [21:53]

What are some things to look for that indicate poor crawl optimization? [23:53]

Is Aysun a fan of large nested XML sitemaps or HTML sitemaps that provide other ways for crawler?  [26:32]

What does Aysun think of crawl budget? [26:56]

Any other common traps when doing a crawl? [27:48]

Does Aysun trust the crawl stats in the google search console? [29:01]

Sponsor Break with discount code!! [30:35]

Common Crawl Web Graph [32:04]

What is the data representing? [35:09]

What is Harmonic Position and Harmonic Value? [35:29]

What did Aysun learn about the distribution of pagerank by visualizing the data? [37:12]

What are some way the data can help SEOs and Data Scientists in their ranking research? [40:04]

Learning SEO [41:58]

Are there challenges that Aysun sees that universities have when teaching SEO. What are they good at? What are their weakness? How to fill in the gaps outside the classroom?  [42:17]

What is a simple way to improve at Data Analysis? [44:46]

Is there a particular language Aysun recommends to begin with? [46:16]

What is a good way to learn the language? [46:23]

TechSEO Boost panel discussion [47:51]

What are some things SEOs can do to work better with Developers [48:26]