This is the podcast for the eleventh class, in which we see traditional and non-traditional methods of collecting data off the web. Traditional way is demonstrated using web crawling using wget, and non-traditional way is instantiated with YouTube harvesting.

Web crawling

Tools for organizing and accessing information have become indispensable. It is critical, therefore, to understand their design and operational foundations. In this course students will have an opportunity to learn about search engines, web crawling, and some Web 2.0 technologies based on hands-on experience and with a focus on techniques that can be used to access, retrieve, organize, and present information. Students will work with practical developmental tools and learn relevant concepts through experimentation. For instance, students will employ an open source search engine and learn about indexing, retrieving, and ranking techniques.

Share Web crawling

Sign up to save your podcasts

Web crawling

Web crawling