Get started with screenscraping using Google Chrome’s Scraper extension

How do you get information from a website to a Excel spreadsheet? The answer is screenscraping. There are a number of softwares and plattforms (such as OutWit Hub, Google Docs and Scraper Wiki) that helps you do this, but none of them are – in my opinion – as easy to use as the Google Chrome extension Scraper, which has become one of my absolutely favourite data tools.

What is a screenscraper?

I like to think of a screenscraper as a small robot that reads websites and extracts pieces of information. When you are able to unleash a scraper on hundreads, thousands or even more pages it can be an incredibly powerful tool.

In its most simple form, the one that we will look at in this blog post, it gathers information from one webpage only.

Google Chrome’s Scraper

Scraper is an Google Chrome extension that can be installed for free at Chrome Web Store.

Image

Now if you installed the extension correctly you should be able to see the option “Scrape similar” if you right-click any element on a webpage.

The Task: Scraping the contact details of all Swedish MPs

Image

This is the site we’ll be working with, a list of all Swedish MPs, including their contact details. Start by right-clicking the name of any person and chose Scrape similar. This should open the following window.

Understanding XPaths

At w3schools you’ll find a broader introduction to XPaths.

Before we move on to the actual scrape, let me briefly introduce XPaths. XPath is a language for finding information in an XML structure, for example an HTML file. It is a way to select tags (or rather “nodes”) of interest. In this case we use XPaths to define what parts of the webpage that we want to collect.

A typical XPath might look something like this:

//div[@id="content"]/table[1]/tr

Which in plain English translates to:

// - Search the whole document...
div[@id="content"] - ...for the div tag with the id "content".
table[1] -  Select the first table.
tr - And in that table, grab all rows.

Over to Scraper then. I’m given the following suggested XPath:

//section[1]/div/div/div/dl/dt/a

The results look pretty good, but it seems we only get names starting with an A. And we would also like to collect to phone numbers and party names. So let’s go back to the webpage and look at the HTML structure.

Right-click one of the MPs and chose Inspect element. We can see that each alphabetical list is contained in a section tag with the class “grid_6 alpha omega searchresult container clist”.

 And if we open the section tag we find the list of MPs in div tags.

We will do this scrape in two steps. Step one is to select the tags containing all information about the MPs with one XPath. Step two is to pick the specific pieces of data that we are interested in (name, e-mail, phone number, party) and place them in columns.

Writing our XPaths

In step one we want to try to get as deep into the HTML structure as possible without losing any of the elements we are interested in. Hover the tags in the Elements window to see what tags correspond to what elements on the page.

In our case this is the last tag that contains all the data we are looking for:

//section[@class="grid_6 alpha omega searchresult container clist"]/div/div/div/dl

Click Scrape to test run the XPath. It should give you a list that looks something like this.

Scroll down the list to make sure it has 349 rows. That is the number of MPs in the Swedish parliament. The second step is to split this data into columns. Go back to the webpage and inspect the HTML code.

I have highlighted the parts that we want to extract. Grab them with the following XPaths:

name: dt/a
party: dd[1]
region: dd[2]/span[1]
seat: dd[2]/span[2]
phone: dd[3]
e-mail: dd[4]/span/a

Insert these paths in the Columns field and click Scrape to run the scraper.

Click Export to Google Docs to get the data into a spreadsheet. Here is my output.

Now that wasn’t so difficult, was it?

Taking it to the next level

Congratulations. You’ve now learnt the basics of screenscraping. Scraper is a very useful tool, but it also has its limitations. Most importantly you are only able to scrape one page at a time. If you need to collect data from several pages Scraper is no longer very efficient.

To take your screenscraping to the next level you need to learn a bit of programming. But fear not. It is not rocket science. If you understand XPaths you’ve come a long way.

Here is how you continue from here:

About these ads

26 Comments on “Get started with screenscraping using Google Chrome’s Scraper extension”

  1. [...] Blogg: Get started with screenscraping using Google Chrome’s Scraper extension [...]

  2. Continued says:

    I think what makes Sims so universally popular is the
    ability to completely manipulate your Sim. This includes the voodoo doll,
    cauldron, crystal ball and new hairstyles and clothing for your magical and fortune telling needs.
    If your Sims have been using the gem spawner all
    this time, they should have found several different types of gem.

  3. ‘s brother Sweet is not too happy with him for ditching his gang for all these years and bullies him into helping take back the neighborhood from their rival gang, the Ballas. The needed driver was automatically found and installed in a few seconds, and I was ready to start getting camcorder clips. I don’t
    believe they have the taxi, EMT, or firefighter missions
    anymore.

  4. […] Jens Finnäs has provided a tutorial on his Dataist blog. He explains how he used the extension to scrape the contact details of all Swedish […]

  5. New research suggests this ‘intestinal’ multiple sclerosis
    journal could even do, before it happened to me. Adults and older children who have had
    an allergic reaction. I believe if you were an only child
    or your siblings were so many years older than you that they already had multiple sclerosis journal.
    But what else would I want free piriton for? Often remains dormant in your
    body for life after someone has been exposed to Multiple Sclerosis Journal, a viral disease also called varicella, hides in the nerve cells that line your spinal cord.

  6. This scaper plugin is really cool. I didn’t know that this kind of plugin really exists on the earth. Thanks for giving such a detailed instructions. I am gonna install it right now.

  7. Anton says:

    This plugin is not so bad, but http://convextra.com can do the same faster in 1 click..

  8. iphone 防水 スピーカー ブーツ ファッション http://www.cnjgov.com/

  9. […] • Scrape screen scraper Chrome extension. Journalist Jens Finnäs wrote a tutorial for it on Dataists. • Time Flow by Martin Wattenberg & Fernanda Viegas • Stately – a symbol font […]

  10. Craig Gooden says:

    I was wondering if anyone can help me – I’m very new to Google data scraping (and loving it!). I need to get data from a website (1000s of pages) – the information I’m after is in the source code – XX12345 01/2013 – what I need for each URL is the ‘XX12345 01/2013. I have a list of the URLs in Excel / CSV format. Is there any automated way of going it? Someting like Google Spreadsheet function – importHTML. Any help appreciated.

  11. Craig Gooden says:

    ah….

    __XX12345 01/2013__

  12. Craig Gooden says:

    I give up!!!!

    XX12345 01/2013

  13. Craig Gooden says:

    Basically it’s the text from within , XX12345 01/2013 , < , / , p

  14. Craig Gooden says:

    Ok, this site basically strips out the code I want to show you – I’m trying to get the text which is after the phrase ” dateCode ” on every webpage. I have a list of URLs in Excel/CSV. Can I use Google spreadsheet function like importHTML?

  15. solarscourge says:

    You should try GrabzIt’s on-line screen scraping tool, which offers more advanced features and is much more customizable: http://grabz.it/scraper

  16. john bole says:

    hello I was trying to scrape the business name, address, phone and URL from this list. The problem is that the scraper first needs to enter the hyperlink for each company and scrape info from second page. Can anybody help?

    Below is the website I need to get the info as you can see there are 37 pages

    http://ces14.mapyourshow.com/5_0/exhibitor_results.cfm?alpha=%40&type=alpha&page=1#GotoResults

    thanks

  17. Khloe says:

    excellent points altogether, you just won a logo new reader. What could you recommend about your submit that you simply made some days ago? Any positive?

  18. john bole says:

    ? what are you talking about ?

  19. iwebscraping says:

    Nice Post, Thanks for explaining screen scraper and how to use Google chrome’s scraper by step by step method.

  20. affordable online marketing says:

    I needed to thank you for this excellent read!!
    I definitely loved every bit of it. I have you book-marked to check out new things you post…

  21. Noelia says:

    I’m gone to inform my little brother, that he should alo visit this webpage on regular basis to get updated from newest news.

  22. Serge says:

    Hi,

    I don’t see the Scrape similar when I right click.

    Any idea why?

    I’m using chrome on win 8.

    Thanks

  23. naehmaschine says:

    It’s truly a nice and usefuil piece of information.
    I am glad that you just shared thiis useful infcormation with us.
    Please keep us up to date like this. Thanks for sharing.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 39 other followers