How to speed up web scraping with Go (Golang) and concurrency ?

I’ve been developing Python web scrapers for years now. Python’s simplicity is great for quick prototyping and so many amazing libraries can help you build a scraper and a result parser (Requests, Beautiful Soup, Scrapy, …). Yet once you start looking into your scraper’s performance, Python can be somewhat limited and Go is a great alternative !

Why Go ?

When you’re trying to speed up information fetching from the Web (for HTML scraping or even for a mere API consumption), 2 ways of optimization are possible:

  • speed up the web resource download (e.g. download http://example.com/hello.html)
  • speed up the parsing of the information you retrieved (e.g. get all urls available in hello.html)

Parsing can be improved either by reworking your code, or using a more efficient parser like lxml, or allocating more resources to your scraper. Still, parsing optimization is often negligible compared to the real bottleneck, namely network access (i.e. web page downloading).

Consequently the solution is about downloading the web resources in parallel. This is where Go is a great help !

Concurrent programming is a very complicated field, and Go makes it pretty easy. Go is a modern language which was created with concurrency in mind. On the other hand, Python is an older language and writing a concurrent web scraper in Python can be tricky, even if Python has improved a lot in this regard recently.

Go has other advantages, but let’s talk about it in another article !

Install Go

I already made a short tuto about how to install Go on Ubuntu.

If you need to install Go on another platform, feel free to read the official docs.

A simple concurrent scraper

Our scraper will basically try to download a list of web pages we’re giving him first, and check it gets a 200 HTTP status code (meaning the server returned an HTML page without an error). We’re not dealing with HTML results parsing here, since the goal is to focus on the critical point: improving network access performance. It’s your turn to write something now !

Final code


/*
Open a series of urls.

Check status code for each url and store urls I could not
open in a dedicated array.
Fetch urls concurrently using goroutines.
*/

package main

import (
    "fmt"
    "net/http"
)

// -------------------------------------

// Custom user agent.
const (
    userAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) " +
        "AppleWebKit/537.36 (KHTML, like Gecko) " +
        "Chrome/53.0.2785.143 " +
        "Safari/537.36"
)

// -------------------------------------

// fetchUrl opens a url with GET method and sets a custom user agent.
// If url cannot be opened, then log it to a dedicated channel.
func fetchUrl(url string, chFailedUrls chan string, chIsFinished chan bool) {

    // Open url.
    // Need to use http.Client in order to set a custom user agent:
    client := &http.Client{}
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", userAgent)
    resp, err := client.Do(req)

    // Inform the channel chIsFinished that url fetching is done (no
    // matter whether successful or not). Defer triggers only once
    // we leave fetchUrl():
    defer func() {
        chIsFinished <- true
    }()

    // If url could not be opened, we inform the channel chFailedUrls:
    if err != nil || resp.StatusCode != 200 {
        chFailedUrls <- url
        return
    }

}

func main() {

    // Create a random urls list just as an example:
    urlsList := [10]string{
        "http://example1.com",
        "http://example2.com",
        "http://example3.com",
        "http://example4.com",
        "http://example5.com",
        "http://example10.com",
        "http://example20.com",
        "http://example30.com",
        "http://example40.com",
        "http://example50.com",
    }

    // Create 2 channels, 1 to track urls we could not open
    // and 1 to inform url fetching is done:
    chFailedUrls := make(chan string)
    chIsFinished := make(chan bool)

    // Open all urls concurrently using the 'go' keyword:
    for _, url := range urlsList {
        go fetchUrl(url, chFailedUrls, chIsFinished)
    }

    // Receive messages from every concurrent goroutine. If
    // an url fails, we log it to failedUrls array:
    failedUrls := make([]string, 0)
    for i := 0; i < len(urlsList); {
        select {
        case url := <-chFailedUrls:
            failedUrls = append(failedUrls, url)
        case <-chIsFinished:
            i++
        }
    }

    // Print all urls we could not open:
    fmt.Println("Could not fetch these urls: ", failedUrls)

}


Explanations

This code is a bit longer than what we could do with a language like Python, but as you can see it is still very reasonable. Go is a statically typed language, so we need a couple of more lines dedicated to variables declaration. But please measure how much time the script is taking, and you’ll understand how rewarding it is !

We chose 10 random urls as an example.

Here, the magical keywords enabling us to use concurrency are go, chan, and select:

  • go creates a new goroutine, which means fetchUrl will be executed within a new concurrent goroutine each time.
  • chan is the type representing a channel. Channels help us communicate among goroutines (main being a goroutine itself as well).
  • select ... case is a switch ... case dedicated to receiving messages sent through channels. Program stays here as long as all goroutines have not sent a message (either to say that url fetching is done, or to say that url fetching failed).

We could have made this scraper without any channel, that’s to say create goroutines and not expect a message from them in return (for instance if every goroutine ends up storing information in database). In such a case, our main goroutine can perfectly end while some goroutines are still working. This is possible because main does not block other goroutines when it stops. But in real life it is almost always necessary to use channels in order to make our goroutines talk to each other.

Don’t forget to limit speed !

Here speed is our goal. This is not a concern because we’re scraping all different urls. However if you need to scrap the same urls multiple times (like in API consumption for example), you’ll probably have to stay under a certain number of requests per second. In this case, you’ll have to set up a counter (maybe we’ll talk about it in another article !).

Have a nice scraping !

Existe aussi en français | También existe en Español
Install Go (Golang) 1.9 on Ubuntu 17.10

Here is a little memento for those who wish to install Go (1.9) on their Ubuntu machine (17.10). As a reminder, Go is a compiled language, meaning no need to install Go on the machine where your application will run.

Update repositories and security patches, just in case:

sudo apt-get update
sudo apt-get -y upgrade

Download and install Go:

sudo curl -O https://storage.googleapis.com/golang/go1.9.linux-amd64.tar.gz  # Download archive. Change the archive's name if you need another version of Go or another system architecture
sudo tar -xvf go1.9.linux-amd64.tar.gz  # Extract archive
sudo mv go /usr/local  # Move binaries to /usr/local
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.profile  # Update our bash profile so Go is in the PATH
source ~/.profile  # Update profile

Go is now installed ! Create your project and initialize environment variables:

mkdir $HOME/my_project
mkdir $HOME/my_project/src
mkdir $HOME/my_project/src/my_app
export GOPATH=$HOME/my_project

Then create your app:

vim $HOME/my_project/src/my_app/my_app.go

And put the following code in it (display a mere Hello World):

package main

import "fmt"

func main() {
    fmt.Printf("hello world\n")
}

Compile app:

go install my_app

If everything went well, an executable has been created inside a new bin folder. Run it:

$HOME/my_project/bin/my_app

Finally you should get:

hello world

In order to understand the differences between go install, go build, and go run read this. And if you cannot or do not want to install Go on your machine, have a look at this Docker image.

Enjoy !

Existe aussi en français | También existe en Español
Why create a blog as a developer ?

Here we are, I’m taking the plunge. I’ve been considering creating a blog for a while. I really wanted it to be multilingual, which made me ask to myself a couple of technical questions about language handling. As a developer I think this blog will be beneficial.

Share

Most developers anticipate that they don’t have enough experience in order to publish on the web. This is very humble but not always correct! IT development is such a big world that we always find people who are more beginner than we are in any field. Those people are relieved that others faced the same issues before them and put it simply.

Promote local languages

I am stunned by how difficult it is to find local content in the IT world. I know English has become the Lingua Franca in this regard but so many people are not fluent enough when it’s about parsing English-speaking blogs efficiently. It’s a shame that skilled and motivated future devs face this barrier (English cannot be learnt overnight…). Defending local languages is important!

OK I must admit that internationalizing a web app is a lot of additional work, and existing localization tools are hard to use IMHO (but let’s talk about that in a future dedicated post), so I understand that everyone wants to avoid it.

Structure ideas

It appears that writing the technical problems you faced in a blog helps structuring your mind and understand concepts better. Let’s see that ! But it is true that this is my main motivation when writing documentation: docs help me remember new concepts much more easily. We must see the blog as another good documentation tool!

Here we go. A quick quote about the above:

Once you get things properly you are able to put it clearly to others.

– Nicolas Boileau

Existe aussi en français | También existe en Español