REST API fetching: Go vs Python

APIs are everywhere today. Imagine you want to find business prospect information based on an email. Well there is an API for this. Need to geocode an ugly postal address? There is an API for that. Would you like to make a payment ? There are multiple APIs for that too of course. As a developer I am regularly fetching external APIs using either Python or Go. Both methods are quite different, let’s compare them here on an edge case: JSON data sent through a POST request body.

A real life example

Recently, I’ve used the NameAPI.org API, dedicated to splitting a full name into first name and last name, and determine gender of the person.

In order to use their API you should send JSON data encoded in the request body through POST. Moreover, the request Content-Type should be set to application/json instead of multipart/form-data. This is a pretty tricky case since usually POST data is sent through the request headers, and if we decide to send it through the request body (in case of a complex JSON for example) the usual Content-Type is multipart/form-data.

Here is the JSON data we want to send:

{
  "inputPerson" : {
    "type" : "NaturalInputPerson",
    "personName" : {
      "nameFields" : [ {
        "string" : "Petra",
        "fieldType" : "GIVENNAME"
      }, {
        "string" : "Meyer",
        "fieldType" : "SURNAME"
      } ]
    },
    "gender" : "UNKNOWN"
  }
}

We could do this pretty simply using cURL:

curl -H "Content-Type: application/json" \
-X POST \
-d '{"inputPerson":{"type":"NaturalInputPerson","personName":{"nameFields":[{"string":"Petra Meyer","fieldType":"FULLNAME"}]}}}' \
http://rc50-api.nameapi.org/rest/v5.0/parser/personnameparser?apiKey=<API-KEY>

And here is the NameAPI.org’s response (JSON):

{
"matches" : [ {
  "parsedPerson" : {
    "personType" : "NATURAL",
    "personRole" : "PRIMARY",
    "mailingPersonRoles" : [ "ADDRESSEE" ],
    "gender" : {
      "gender" : "MALE",
      "confidence" : 0.9111111111111111
    },
    "addressingGivenName" : "Petra",
    "addressingSurname" : "Meyer",
    "outputPersonName" : {
      "terms" : [ {
        "string" : "Petra",
        "termType" : "GIVENNAME"
      },{
        "string" : "Meyer",
        "termType" : "SURNAME"
      } ]
    }
  },
  "parserDisputes" : [ ],
  "likeliness" : 0.926699401733102,
  "confidence" : 0.7536487758945387
}

Now let’s see how to do this in Go and Python!

Go implementation

Code

/*
Fetch the NameAPI.org REST API and turn JSON response into a Go struct.

Sent data have to be JSON data encoded into request body.
Send request headers must be set to 'application/json'.
*/

package main

import (
    "encoding/json"
    "io/ioutil"
    "log"
    "net/http"
    "strings"
)

// url of the NameAPI.org endpoint:
const (
    url = "http://rc50-api.nameapi.org/rest/v5.0/parser/personnameparser?" +
        "apiKey=<API-KEY>"
)

func main() {

    // JSON string to be sent to NameAPI.org:
    jsonString := `{
        "inputPerson": {
            "type": "NaturalInputPerson",
            "personName": {
                "nameFields": [
                    {
                        "string": "Petra",
                        "fieldType": "GIVENNAME"
                    }, {
                        "string": "Meyer",
                        "fieldType": "SURNAME"
                    }
                ]
            },
            "gender": "UNKNOWN"
        }
    }`
    // Convert JSON string to NewReader (expected by NewRequest)
    jsonBody := strings.NewReader(jsonString)

    // Need to create a client in order to modify headers
    // and set content-type to 'application/json':
    client := &http.Client{}
    req, err := http.NewRequest("POST", url, jsonBody)
    if err != nil {
        log.Println(err)
    }
    req.Header.Add("Content-Type", "application/json")
    resp, err := client.Do(req)

    // Proceed only if no error:
    switch {
    default:
        // Create a struct dedicated to receiving the fetched
        // JSON content:
        type Level5 struct {
            String   string `json:"string"`
            TermType string `json:"termType"`
        }
        type Level41 struct {
            Gender     string  `json:"gender"`
            Confidence float64 `json:"confidence"`
        }
        type Level42 struct {
            Terms []Level5 `json:"terms"`
        }
        type Level3 struct {
            Gender           Level41 `json:"gender"`
            OutputPersonName Level42 `json:"outputPersonName"`
        }
        type Level2 struct {
            ParsedPerson Level3 `json:"parsedPerson"`
        }
        type RespContent struct {
            Matches []Level2 `json:"matches"`
        }

        // Decode fetched JSON and put it into respContent:
        respContentBytes, err := ioutil.ReadAll(resp.Body)
        if err != nil {
            log.Println(err)
        }
        var respContent RespContent
        err = json.Unmarshal(respContentBytes, &respContent)
        if err != nil {
            log.Println(err)
        }
        log.Println(respContent)
    case err != nil:
        log.Println("Network error:", err)
    case resp.StatusCode != 200:
        log.Println("Bad HTTP status code:", err)
    }

}

Explanations

As you can see we’re facing 2 painful problems with Go:

• the http lib is quite tricky when it’s about encoding JSON data into the request body and changing the Content-Type header. Go’s documentation is not very clear on this. As a result we cannot use the pretty straightforward http.Post but instead we need to create a http.Client and then use the NewRequest() function and trigger it with client.Do(req). This is the only way to set a custom Content-Type in that case: req.Header.Add("Content-Type", "application/json")
• decoding the returned JSON into Go data is pretty long and boring (called Unmarshalling in Go). It’s due to the fact that, Go being a statically typed language, we need to know in advance what the final returned JSON will look like. Thus we need to create a dedicated struct that will map the JSON’s structure and receive the data. In case of a nested JSON like the one returned by NameAPI.org, mixing arrays and maps, it is very touchy. Fortunately, our struct does not need to map the whole JSON but only the fields we will need. Another approach, if we have no idea what the final JSON will look like, would be to guess the types of data. Here is a good article on this.

The jsonString input is already a string here. But for a proper comparison with Python, it should have been a struct that we would have turned into a string. I just did not want to make this script too long for the blog.

Python implementation

Code

"""
Fetch the NameAPI.org REST API and turn JSON response into Python dict.

Sent data have to be JSON data encoded into request body.
Send request headers must be set to 'application/json'.
"""

import requests

# url of the NameAPI.org endpoint:
url = (
    "http://rc50-api.nameapi.org/rest/v5.0/parser/personnameparser?"
    "apiKey=<API-KEY>"
)

# Dict of data to be sent to NameAPI.org:
payload = {
    "inputPerson": {
        "type": "NaturalInputPerson",
        "personName": {
            "nameFields": [
                {
                    "string": "Petra",
                    "fieldType": "GIVENNAME"
                }, {
                    "string": "Meyer",
                    "fieldType": "SURNAME"
                }
            ]
        },
        "gender": "UNKNOWN"
    }
}

# Proceed, only if no error:
try:
    # Send request to NameAPI.org by doing the following:
    # - make a POST HTTP request
    # - encode the Python payload dict to JSON
    # - pass the JSON to request body
    # - set header's 'Content-Type' to 'application/json' instead of
    #   default 'multipart/form-data'
    resp = requests.post(url, json=payload)
    resp.raise_for_status()
    # Decode JSON response into a Python dict:
    resp_dict = resp.json()
    print(resp_dict)
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

Explanations

The Python Request library is an amazing library saves us a lot of time here compared to Go! In one line, resp = requests.post(url, json=payload), almost everything is done under the hood:

• build a POST HTTP request
• encode the Python payload dictionary to JSON
• pass the JSON to the request body
• set header’s 'Content-Type' to 'application/json' instead of the default 'multipart/form-data' thanks to the json keyword argument
• send the request

Decoding of returned JSON is also a one-liner: resp_dict = resp.json(). No need to create a complicated data structure in advance here!

Conclusion

Python is clearly the winner. Python’s simplicity combined with its huge set of libraries saves us a lot of time of development!

We’re not dealing with performance here of course. If you’re looking for a high-performance API fetcher using concurrency, Go could be a great choice. But simplicity and performance are not good friends as you can see…

Feel free to comment, I would be glad to here your opinion on this!

Existe aussi en français | También existe en Español