Plugging a Vue.js SPA frontend to an API backend

Vue.js is a great Javascript frontend framework and its documentation is very clear and straight to the point. You can either choose to integrate Vue into an already existing app (the JQuery way) or build a Webpack based Single Page Application (SPA) in the React.js way. Let’s build a basic SPA here connecting to a REST API backend using Node.js, Webpack, Vue Loader, Vue Router, and Axios. Setting up such a project is not that easy in my opinion, so here is a short memo about how to do it on Ubuntu. For your information, here is why we’re using Webpack.

Set Up a Vue.js Project

Install Node.js and npm.

Then use npm to install vue-cli and use vue-cli to install the vue-loader loader for Webpack that enables you to use Vue components mixing HTML, CSS, and JS in a single .vue file. It also installs the whole ecosystem needed for a SPA like vue-router. Last but not list, install Axios we’ll be using for API fetching.

sudo npm install -g vue-cli
vue init webpack vue_project  # Answer various questions here
cd vue_project
npm install
npm install --save axios

More information about the project’s structure here.

Workflow

Dev

Dev is made dead easy thanks to the local web server and the hot reloading feature (web page is modified on the fly when you change source code). Just run:

npm run dev

and start coding.

Note: I personnaly had to face a bug so that hot reloading was broken because of a permission problem (I’m running Ubuntu 17.10). I fixed it thanks to the following command:

echo 100000 | sudo tee /proc/sys/fs/inotify/max_user_watches

Deployment

Once you need to push your app to production run:

npm run build

and your app is now compiled into the dist directory. It’s up to you to decide how you want to deploy it. Personally I’m putting my app into a Docker container running Nginx and make Nginx point to the folder containing my app thanks to the following block:

location / {
    root /my_app;
}

Of course if you already have another service running on port 80 of the same server you’ll have to think about how you want to organize your services and modify your nginx config accordingly.

Plug to API Backend

Everything is happening inside the src folder from now on.

Set Dev and Prod Server Names Once For All

My API development server is running on http://127.0.0.1 while my API production is running on http://api.example.com so in order to avoid changing my config for every deployment I created the following http-constants.js file at the root of the src folder:

import axios from 'axios'

let baseURL

if (!process.env.NODE_ENV || process.env.NODE_ENV === 'development') {
  baseURL = 'http://127.0.0.1/'
} else {
  baseURL = 'http://api.example.com'
}

export const HTTP = axios.create(
  {
    baseURL: baseURL
  })

and then in every .vue file needing Axios, import HTTP instead of Axios:

import {HTTP} from '../http-constants'

HTTP.get(...).then(...).catch(...)

Note: this proxying feature might be done more easily in the config/index.js using the proxyTable directive but for some reason it did not work for me.

Create The App

Let’s create an app called ShowGreetings that gets an Hello World message from an API. The API endpoint is /greetings and returns the following JSON message when sending a GET request:

{message: "Hello World"}

First create the new Vue component called ShowGreetings.vue in src/components:

<template>
  <div>
    <button @click="getGreetings">Get Greetings</button>
    <h1 v-if="greetings"></h1>
    <p class="error" v-if="errorMessage"></p>
  </div>
</template>

<script>
import {HTTP} from '../http-constants'
export default {
  name: 'ShowGreetings',
  data () {
    return {
      greetings: '',
      errors: ''
    }
  },
  methods: {
    getGreetings: function () {
      HTTP.get('/greetings')
        .then(response => {
          this.greetings = response.data.message
        })
        .catch(e => {
          this.errors = e
        })
    }
  }
}
</script>

<style scoped>
.error {
  color: red;
}
</style>

This component tries to fetch the API backend when you click on a button and displays the returned message. If an error is returned, it displays the error.

Now update the router in order to take this new component into account. Here is our new index.js in src/router:

import Vue from 'vue'
import Router from 'vue-router'
import HelloWorld from '@/components/HelloWorld'
import ShowGreetings from '@/components/ShowGreetings'

Vue.use(Router)

export default new Router({
  routes: [
    {
      path: '/',
      name: 'HelloWorld',
      component: HelloWorld
    },
    {
      path: '/show-greetings',
      name: 'ShowGreetings',
      component: ShowGreetings
    }
  ]
})

We created a named route called “ShowGreetings” so we can now refer to the route by its name rather than its path (much more flexible).

Last of all, edit the App.vue component in src so a link to our new component appears on the home page:

<template>
  <div id="app">
    <img src="./assets/logo.png">
    <router-link :to="{ name: 'ShowGreetings'}">Show Greetings</router-link>
    <router-view/>
  </div>
</template>
<script>
export default {
  name: 'App'
}
</script>
<style>
#app {
  font-family: 'Avenir', Helvetica, Arial, sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
  text-align: center;
  color: #2c3e50;
  margin-top: 60px;
}
</style>

Here we just added a new router-link tag.

Conclusion

Once you understand how all the layers are interacting, it becomes very easy to build an SPA with Vue.js that can scale from a basic project to a complex project in production.

This whole little project is available on my GitHub if needed.

Existe aussi en français
Dockerizing a whole physical Linux server

Docker is usually used in microservice architectures because containers are lightweight (compared to VMs at least), easy to configure, communicate with each other efficiently, and can be deployed very quickly. However Docker can be perfectly used if you want to Dockerize a full physical/VPS server into one single container. Let me show you how and why.

Context

I recently had to work on a project developed by people who had left the company before I arrive. I never had the opportunity to meet those guys and contacting them was not an option either. Unfortunately most of the project lacked a documentation. And in addition to all this, only some parts of the project were managed in a VCS (Git here). Of course there was no dev or staging server: everything was located on a production server… Start seeing the problem now?

It was a web scraping project doing quite a lot of complex things. Technical stack of the prod server was more or less the following:

  • Ubuntu 16.04
  • Nginx
  • Postgresql 9.3
  • Python/Django
  • Python virtualenvs
  • Gunicorn
  • Celery
  • RabbitMQ
  • Scrapy/Scrapyd

First attempt: failed

I tried hard reverse engineering this production server. My ultimate goal was to isolate each application, Dockerize it, and make these containers communicate with each other.

But I failed!

I successfully Dockerized Nginx, the Django app, and the Celery asynchronous tasks management. But I was still struggling with Scrapy and Scrapyd. I think it was mainly due to the fact that changes had been made directly to the Scrapy and Scrapyd source files by the former developers (that’s to say in the python/site-package directory itself!) without any documentation. In addition to that, some of the Python libraries used at the time were very specific Python libs which are not available today anymore, or not in the correct version (you can forget about pip freeze and pip install -r requirements.txt here).

Second attempt: failed

I eventually gave up building a microservice system based on the production server. But still I had to secure the existing production server before it experiences troubles. Database was backed up then but nothing else on the server was backed up.

I thought about making a snapshot of the whole server by using a tool like CloneZilla or a simple rsync command like this one. But a mere backup would not allow me to work easily on new features of the project.

So I thought about converting the physical server to a VMware virtual machine by using their VMware vCenter Converter but the VMware download link was broken and so few people talked about this tool on Internet that I got scared and gave up.

Last of all I tried this Dockerization solution based on Blueprint but could not make it work and Blueprint seemed to be a discontinued project.

Third attempt: successful

Actually the solution was pretty simple: I decided to Dockerize the whole prod server by myself - except the Postgresql data - so I have a backup of the server and I can commit new features to this Docker container whenever I want without being afraid of breaking the server forever. Here is how I did it:

1. Install and setup Docker on the server

  1. Install Docker following this guide.
  2. Login to your Docker Hub account: docker login
  3. Create a docker network (if needed): docker network create --subnet=172.20.0.0/16 my_network

2. Create a Docker image of your server

Go to the root of the server:

cd /

Create the following Dockerfile file based on Ubuntu 16.04 LTS (without the part dedicated to Nginx and Rabbitmq of course):

FROM ubuntu:xenial

# Copy the whole system except what is specified in .dockerignore
COPY / /

# Reinstall nginx and rabbitmq because of permissions issues in Docker
RUN apt remove -y nginx
RUN apt install -y nginx
RUN apt remove -y rabbitmq-server
RUN apt install -y rabbitmq-server

# Launch all services
COPY startup.sh /
RUN chmod 777 /startup.sh
CMD ["bash","/startup.sh"]

Create a .dockerignore file that will mention all the files or folders you want to exclude from the COPY above. This is where you need to use your intuition. Remove as many files as possible so the Docker image’s size is not too big, but do not exclude files that are vital to you application. Here is my example that you should customize based on your own server:

# Remove folders mentioned here:
# https://wiki.archlinux.org/index.php/Rsync#As_a_backup_utility
/dev 
/proc
/sys
/tmp
/run
/mnt
/media
/lost+found

# Remove database's data
/var/lib/postgresql

# Remove useless heavy files like /var/lib/scrapyd/reports.old
**/*.old
**/*.log
**/*.bak

# Remove docker
/var/lib/lxcfs
/var/lib/docker
/etc/docker
/root/.docker
/etc/init/docker.conf

# Remove the current program
/.dockerignore
/Dockerfile

Create a startup.sh script in order to launch all the services and set database connection redirection. This is my script here but yours will be totally different of course:

# Redirect all traffic from 127.0.0.1:5432 to 172.20.0.1:5432
# so any connection to Postgresql keeps working without any other modification.
# Requires the --privileged flag when creating container:
sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -A OUTPUT -p tcp -s 127.0.0.1 --dport 5432 -j DNAT --to-destination 172.20.0.1:5432
iptables -t nat -A POSTROUTING -j MASQUERADE

# Start RabbitMQ.
rabbitmq-server -detached

# Start Nginx.
service nginx start

# Start Scrapyd
/root/.virtualenvs/my_project_2/bin/python /root/.virtualenvs/my_project_2/bin/scrapyd >> /var/log/scrapyd/scrapyd.log 2>&1 &

# Use Python virtualenvwrapper
source /root/.profile

# Start virtualenv and start Django/Gunicorn
workon my_project_1
cd /home/my_project_1
export DJANGO_SETTINGS_MODULE='my_project_1.settings.prod'
gunicorn -c my_project_1/gunicorn.py -p /tmp/gunicorn.pid my_project_1.wsgi &

# Start Celery
export C_FORCE_ROOT=True
celery -A my_project_1 beat &
celery -A my_project_1 worker -l info -Q queue1,queue2 -P gevent -c 1000 &

# Little hack to keep the container running in foreground
tail -f /dev/null

As you can see, I’m using iptables redirections so that all connections to the Postgresql database (port 5432) keep working without any additional change in the configuration files. Indeed my database was initially located on localhost, but it is now located on the Docker host whose ip is 172.20.0.1 (I moved everything to the Docker container except database). Redirections at the kernel level are pretty convenient when you don’t know where all the configuration files are located, and this is necessary when you cannot modify those config files (like in a compiled application you don’t have the source code of).

Now launch the image creation and wait… In my case, the image was about 3Go and was created in about 5 minutes. Make sure you have enough free space on your server before launching this command:

docker build -t your_repo/your_project:your_tag .

If you’ve got no error here, congrats you’ve done the hardest part! Now test you image and see if everything works fine. If not, then you need to adapt one of the 3 files above.

3. Save the newly created Docker image

Just push the new image to Docker Hub:

docker push your_repo/your_project:your_tag

4. Add features to your server image

Now, if you need to work on this server image, you can do the following:

  1. launch a container based on this image with docker run (do not forget to specify network name, ip address, port forwarding, and add the --privileged flag in order for the sysctl command to work in startup.sh)
  2. work in the container
  3. commit changes in the container to a new Docker image with docker commit
  4. push the new image to Docker Hub with docker push and deploy it to staging or production

Conclusion

This solution literally saved my life and is a proof that Docker is great not only for microservice architecture but for whole server containerization as well. Dockerizing a whole server can be a perfect option if you need to secure an existing prod server like mine here with no documentation, no GitHub repo, no initial developers…

The first image creation can be pretty big, but then every commit should not be that heavy thanks to the Docker layer architecture.

Is it a hack? Maybe it is, but it works like a charm!

I would love to here other devs’ opinion on this.

Existe aussi en français
CTOs, developers: how to assess quality of an external API?

Nowadays finding an external API in order to improve your service is getting easier and easier. More and more companies offer APIs. Problem is many developers/CTOs start the API integration right away while it should be the very last step! Before that you need to figure out whether the quality of this API matches some minimum requirements. Let me tell you how I do it. I hope it will help other CTOs and developers.

Quality of data

A lot of APIs expose data in order for you to enrich your system (this is not always the case of course, Stripe is not an enrichment API for example). This is essential that you check the quality of those data. It will take you a long time and I know you do not like testing! Neither do I but you cannot avoid building a serious test scenario here. If you realize data quality was not good enough only 2 weeks after finishing your API integration, trust me you’ll regret it…

Documentation

I recently fell upon an API which exposed great data (much better than his competitors in my opinion), but its documentation was… awful! Actually it almost did not exist. In addition to that it did not always respect the basic REST standards. How can you possibly integrate an external API if error codes are not properly documented ? Well the only solution is for you to test again and again in order to understand how things work behind the hoods. Reverse-engineering might be fun but it takes a lot of time. Remember you have no Github repo to explore here since source code is not available… Bad documentation is a lot of time lost for the devs and certainly bad surprises in the mid term.

Libraries

Can you consume the API with special libraries in your favorite language ? As a Python and Go developer I’m always glad to see APIs offering a Python lib (I know I can forget about Go for the moment). It can save you quite a lot of time, but first make sure the lib is mature enough and covers all the API features (not always the case).

Reputation of the vendor

Reputation can help you find out whether you’ll have bad surprises with your API in the future. By bad surprises I mean service interruption, features regression, or even end of the service… You can partly tackle that by asking yourself the following questions:

  • is this API popular on the internet (in general if you find little information, run away)? Are there a lot of articles/tutorials talking about it? Are those articles positive?
  • are some popular companies using it?
  • if the company developed libs, are they popular on Github? Are the issues on Github solved regularly?
  • were there recent updates of the API or was the last update released a long time ago?

Technical support

Make sure someone answers you quickly by email when you have an issue and the answer is relevant. If you’re based in Europe and the API is run by an American company, check whether time difference is not too much of a problem.

Respect standards

In my humble opinion, you choose only RESTful APIs today. If the API you’re in love with do not respect the REST standard, be suspicious. But keep in mind that it’s not perfectly clear what the REST standard is about, and each API implements its own rules (HTTP codes, POST requests encoding, …). Still, have a close look at the docs, and check that you do not see something original. Originality will slow you down…

Price

Of course price is very important. But be careful, API prices are not always easy to understand. Are you going to be charged per month for an unlimited amount of requests ? Charged per request ? If so are you going to be charged twice for 2 identical requests (in case of an enrichment API) or will the second request be free ? Are you going to be charged for a request returning no result (HTTP 404) ? Make sure you understand all the implications of pricing.

Quality of Service (QoS)

QoS is highly important. Basically you want the API to go fast and have as little downtime as possible. Unfortunately this is not an easy to test point. Indeed QoS may vary a lot over time, and many APIs offer 2 levels of QoS depending on whether you’re using the free version of the API or you paid for it… Sometimes you can also choose between different subscriptions with different levels of response time.

Parallel queries support

Depending on how you’re planning to integrate your API, you might want to speed things up by making multiple parallel queries to the API instead of using it sequentially. Personally I’m using Golang for that most of the time. If so be careful: many vendors do not support parallel queries, and when they do they always set up a limit. In that case make sure to ask them what this limit is (not always told in the docs) and adapt your script based on this.

This post will be a good memo for me. I hope for you too!

Existe aussi en français | También existe en Español
REST API fetching: Go vs Python

APIs are everywhere today. Imagine you want to find business prospect information based on an email. Well there is an API for this. Need to geocode an ugly postal address? There is an API for that. Would you like to make a payment ? There are multiple APIs for that too of course. As a developer I am regularly fetching external APIs using either Python or Go. Both methods are quite different, let’s compare them here on an edge case: JSON data sent through a POST request body.

A real life example

Recently, I’ve used the NameAPI.org API, dedicated to splitting a full name into first name and last name, and determine gender of the person.

In order to use their API you should send JSON data encoded in the request body through POST. Moreover, the request Content-Type should be set to application/json instead of multipart/form-data. This is a pretty tricky case since usually POST data is sent through the request headers, and if we decide to send it through the request body (in case of a complex JSON for example) the usual Content-Type is multipart/form-data.

Here is the JSON data we want to send:

{
  "inputPerson" : {
    "type" : "NaturalInputPerson",
    "personName" : {
      "nameFields" : [ {
        "string" : "Petra",
        "fieldType" : "GIVENNAME"
      }, {
        "string" : "Meyer",
        "fieldType" : "SURNAME"
      } ]
    },
    "gender" : "UNKNOWN"
  }
}

We could do this pretty simply using cURL:

curl -H "Content-Type: application/json" \
-X POST \
-d '{"inputPerson":{"type":"NaturalInputPerson","personName":{"nameFields":[{"string":"Petra Meyer","fieldType":"FULLNAME"}]}}}' \
http://rc50-api.nameapi.org/rest/v5.0/parser/personnameparser?apiKey=<API-KEY>

And here is the NameAPI.org’s response (JSON):

{
"matches" : [ {
  "parsedPerson" : {
    "personType" : "NATURAL",
    "personRole" : "PRIMARY",
    "mailingPersonRoles" : [ "ADDRESSEE" ],
    "gender" : {
      "gender" : "MALE",
      "confidence" : 0.9111111111111111
    },
    "addressingGivenName" : "Petra",
    "addressingSurname" : "Meyer",
    "outputPersonName" : {
      "terms" : [ {
        "string" : "Petra",
        "termType" : "GIVENNAME"
      },{
        "string" : "Meyer",
        "termType" : "SURNAME"
      } ]
    }
  },
  "parserDisputes" : [ ],
  "likeliness" : 0.926699401733102,
  "confidence" : 0.7536487758945387
}

Now let’s see how to do this in Go and Python!

Go implementation

Code

/*
Fetch the NameAPI.org REST API and turn JSON response into a Go struct.

Sent data have to be JSON data encoded into request body.
Send request headers must be set to 'application/json'.
*/

package main

import (
    "encoding/json"
    "io/ioutil"
    "log"
    "net/http"
    "strings"
)

// url of the NameAPI.org endpoint:
const (
    url = "http://rc50-api.nameapi.org/rest/v5.0/parser/personnameparser?" +
        "apiKey=<API-KEY>"
)

func main() {

    // JSON string to be sent to NameAPI.org:
    jsonString := `{
        "inputPerson": {
            "type": "NaturalInputPerson",
            "personName": {
                "nameFields": [
                    {
                        "string": "Petra",
                        "fieldType": "GIVENNAME"
                    }, {
                        "string": "Meyer",
                        "fieldType": "SURNAME"
                    }
                ]
            },
            "gender": "UNKNOWN"
        }
    }`
    // Convert JSON string to NewReader (expected by NewRequest)
    jsonBody := strings.NewReader(jsonString)

    // Need to create a client in order to modify headers
    // and set content-type to 'application/json':
    client := &http.Client{}
    req, err := http.NewRequest("POST", url, jsonBody)
    if err != nil {
        log.Println(err)
    }
    req.Header.Add("Content-Type", "application/json")
    resp, err := client.Do(req)

    // Proceed only if no error:
    switch {
    default:
        // Create a struct dedicated to receiving the fetched
        // JSON content:
        type Level5 struct {
            String   string `json:"string"`
            TermType string `json:"termType"`
        }
        type Level41 struct {
            Gender     string  `json:"gender"`
            Confidence float64 `json:"confidence"`
        }
        type Level42 struct {
            Terms []Level5 `json:"terms"`
        }
        type Level3 struct {
            Gender           Level41 `json:"gender"`
            OutputPersonName Level42 `json:"outputPersonName"`
        }
        type Level2 struct {
            ParsedPerson Level3 `json:"parsedPerson"`
        }
        type RespContent struct {
            Matches []Level2 `json:"matches"`
        }

        // Decode fetched JSON and put it into respContent:
        respContentBytes, err := ioutil.ReadAll(resp.Body)
        if err != nil {
            log.Println(err)
        }
        var respContent RespContent
        err = json.Unmarshal(respContentBytes, &respContent)
        if err != nil {
            log.Println(err)
        }
        log.Println(respContent)
    case err != nil:
        log.Println("Network error:", err)
    case resp.StatusCode != 200:
        log.Println("Bad HTTP status code:", err)
    }

}

Explanations

As you can see we’re facing 2 painful problems with Go:

  • the http lib is quite tricky when it’s about encoding JSON data into the request body and changing the Content-Type header. Go’s documentation is not very clear on this. As a result we cannot use the pretty straightforward http.Post but instead we need to create a http.Client and then use the NewRequest() function and trigger it with client.Do(req). This is the only way to set a custom Content-Type in that case: req.Header.Add("Content-Type", "application/json")
  • decoding the returned JSON into Go data is pretty long and boring (called Unmarshalling in Go). It’s due to the fact that, Go being a statically typed language, we need to know in advance what the final returned JSON will look like. Thus we need to create a dedicated struct that will map the JSON’s structure and receive the data. In case of a nested JSON like the one returned by NameAPI.org, mixing arrays and maps, it is very touchy. Fortunately, our struct does not need to map the whole JSON but only the fields we will need. Another approach, if we have no idea what the final JSON will look like, would be to guess the types of data. Here is a good article on this.

The jsonString input is already a string here. But for a proper comparison with Python, it should have been a struct that we would have turned into a string. I just did not want to make this script too long for the blog.

Python implementation

Code

"""
Fetch the NameAPI.org REST API and turn JSON response into Python dict.

Sent data have to be JSON data encoded into request body.
Send request headers must be set to 'application/json'.
"""

import requests

# url of the NameAPI.org endpoint:
url = (
    "http://rc50-api.nameapi.org/rest/v5.0/parser/personnameparser?"
    "apiKey=<API-KEY>"
)

# Dict of data to be sent to NameAPI.org:
payload = {
    "inputPerson": {
        "type": "NaturalInputPerson",
        "personName": {
            "nameFields": [
                {
                    "string": "Petra",
                    "fieldType": "GIVENNAME"
                }, {
                    "string": "Meyer",
                    "fieldType": "SURNAME"
                }
            ]
        },
        "gender": "UNKNOWN"
    }
}

# Proceed, only if no error:
try:
    # Send request to NameAPI.org by doing the following:
    # - make a POST HTTP request
    # - encode the Python payload dict to JSON
    # - pass the JSON to request body
    # - set header's 'Content-Type' to 'application/json' instead of
    #   default 'multipart/form-data'
    resp = requests.post(url, json=payload)
    resp.raise_for_status()
    # Decode JSON response into a Python dict:
    resp_dict = resp.json()
    print(resp_dict)
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

Explanations

The Python Request library is an amazing library saves us a lot of time here compared to Go! In one line, resp = requests.post(url, json=payload), almost everything is done under the hood:

  • build a POST HTTP request
  • encode the Python payload dictionary to JSON
  • pass the JSON to the request body
  • set header’s 'Content-Type' to 'application/json' instead of the default 'multipart/form-data' thanks to the json keyword argument
  • send the request

Decoding of returned JSON is also a one-liner: resp_dict = resp.json(). No need to create a complicated data structure in advance here!

Conclusion

Python is clearly the winner. Python’s simplicity combined with its huge set of libraries saves us a lot of time of development!

We’re not dealing with performance here of course. If you’re looking for a high-performance API fetcher using concurrency, Go could be a great choice. But simplicity and performance are not good friends as you can see…

Feel free to comment, I would be glad to here your opinion on this!

Existe aussi en français | También existe en Español
How to speed up web scraping with Go (Golang) and concurrency ?

I’ve been developing Python web scrapers for years now. Python’s simplicity is great for quick prototyping and so many amazing libraries can help you build a scraper and a result parser (Requests, Beautiful Soup, Scrapy, …). Yet once you start looking into your scraper’s performance, Python can be somewhat limited and Go is a great alternative !

Why Go ?

When you’re trying to speed up information fetching from the Web (for HTML scraping or even for a mere API consumption), 2 ways of optimization are possible:

  • speed up the web resource download (e.g. download http://example.com/hello.html)
  • speed up the parsing of the information you retrieved (e.g. get all urls available in hello.html)

Parsing can be improved either by reworking your code, or using a more efficient parser like lxml, or allocating more resources to your scraper. Still, parsing optimization is often negligible compared to the real bottleneck, namely network access (i.e. web page downloading).

Consequently the solution is about downloading the web resources in parallel. This is where Go is a great help !

Concurrent programming is a very complicated field, and Go makes it pretty easy. Go is a modern language which was created with concurrency in mind. On the other hand, Python is an older language and writing a concurrent web scraper in Python can be tricky, even if Python has improved a lot in this regard recently.

Go has other advantages, but let’s talk about it in another article !

Install Go

I already made a short tuto about how to install Go on Ubuntu.

If you need to install Go on another platform, feel free to read the official docs.

A simple concurrent scraper

Our scraper will basically try to download a list of web pages we’re giving him first, and check it gets a 200 HTTP status code (meaning the server returned an HTML page without an error). We’re not dealing with HTML results parsing here, since the goal is to focus on the critical point: improving network access performance. It’s your turn to write something now !

Final code


/*
Open a series of urls.

Check status code for each url and store urls I could not
open in a dedicated array.
Fetch urls concurrently using goroutines.
*/

package main

import (
    "fmt"
    "net/http"
)

// -------------------------------------

// Custom user agent.
const (
    userAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) " +
        "AppleWebKit/537.36 (KHTML, like Gecko) " +
        "Chrome/53.0.2785.143 " +
        "Safari/537.36"
)

// -------------------------------------

// fetchUrl opens a url with GET method and sets a custom user agent.
// If url cannot be opened, then log it to a dedicated channel.
func fetchUrl(url string, chFailedUrls chan string, chIsFinished chan bool) {

    // Open url.
    // Need to use http.Client in order to set a custom user agent:
    client := &http.Client{}
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", userAgent)
    resp, err := client.Do(req)

    // Inform the channel chIsFinished that url fetching is done (no
    // matter whether successful or not). Defer triggers only once
    // we leave fetchUrl():
    defer func() {
        chIsFinished <- true
    }()

    // If url could not be opened, we inform the channel chFailedUrls:
    if err != nil || resp.StatusCode != 200 {
        chFailedUrls <- url
        return
    }

}

func main() {

    // Create a random urls list just as an example:
    urlsList := [10]string{
        "http://example1.com",
        "http://example2.com",
        "http://example3.com",
        "http://example4.com",
        "http://example5.com",
        "http://example10.com",
        "http://example20.com",
        "http://example30.com",
        "http://example40.com",
        "http://example50.com",
    }

    // Create 2 channels, 1 to track urls we could not open
    // and 1 to inform url fetching is done:
    chFailedUrls := make(chan string)
    chIsFinished := make(chan bool)

    // Open all urls concurrently using the 'go' keyword:
    for _, url := range urlsList {
        go fetchUrl(url, chFailedUrls, chIsFinished)
    }

    // Receive messages from every concurrent goroutine. If
    // an url fails, we log it to failedUrls array:
    failedUrls := make([]string, 0)
    for i := 0; i < len(urlsList); {
        select {
        case url := <-chFailedUrls:
            failedUrls = append(failedUrls, url)
        case <-chIsFinished:
            i++
        }
    }

    // Print all urls we could not open:
    fmt.Println("Could not fetch these urls: ", failedUrls)

}


Explanations

This code is a bit longer than what we could do with a language like Python, but as you can see it is still very reasonable. Go is a statically typed language, so we need a couple of more lines dedicated to variables declaration. But please measure how much time the script is taking, and you’ll understand how rewarding it is !

We chose 10 random urls as an example.

Here, the magical keywords enabling us to use concurrency are go, chan, and select:

  • go creates a new goroutine, which means fetchUrl will be executed within a new concurrent goroutine each time.
  • chan is the type representing a channel. Channels help us communicate among goroutines (main being a goroutine itself as well).
  • select ... case is a switch ... case dedicated to receiving messages sent through channels. Program stays here as long as all goroutines have not sent a message (either to say that url fetching is done, or to say that url fetching failed).

We could have made this scraper without any channel, that’s to say create goroutines and not expect a message from them in return (for instance if every goroutine ends up storing information in database). In such a case, our main goroutine can perfectly end while some goroutines are still working. This is possible because main does not block other goroutines when it stops. But in real life it is almost always necessary to use channels in order to make our goroutines talk to each other.

Don’t forget to limit speed

Here speed is our goal. This is not a concern because we’re scraping all different urls. However if you need to scrap the same urls multiple times (like in API consumption for example), you’ll probably have to stay under a certain number of requests per second. In this case, you’ll have to set up a counter (maybe we’ll talk about it in another article !).

Have a nice scraping !

Existe aussi en français | También existe en Español