Developing and deploying a whole website in Go (Golang)

In my opinion Go (Golang) is a great choice for web development:

  • it makes non-blocking requests and concurrency easy
  • it makes code testing and deployment easy as it does not require any special runtime environment or dependencies (making containerization pretty useless here)
  • it does not require any additional frontend HTTP server like Apache or Nginx as it already ships with a very good one in its standard library
  • it does not force you to use a web framework as everything needed for web development is ready to use in the std lib

A couple of years back the lack of libraries and tutorials around Go could have been a problem, but today it is not anymore. Let me show you the steps to build a website in Go and deploy it to your Linux server from A to Z.

The Basics

Let’s say you are developing a basic HTML page called love-mountains. As you might already know, the rendering of love-mountains is done in a function, and you should launch a web server with a route pointing to that function. This is good practice to use HTML templates in web development so let’s render the page through a basic template here. This is also good practice to load parameters like the path to templates directory from environment variables for better flexibility.

Here is your Go code:

package main

import (
    "html/template"
    "net/http"
)

// Get path to template directory from env variable
var templatesDir = os.Getenv("TEMPLATES_DIR")

// loveMountains renders the love-mountains page after passing some data to the HTML template
func loveMountains(w http.ResponseWriter, r *http.Request) {
    // Build path to template
    tmplPath := filepath.Join(templatesDir, "love-mountains.html")
    // Load template from disk
    tmpl := template.Must(template.ParseFiles(tmplPath))
    // Inject data into template
    data := "La Chartreuse"
    tmpl.Execute(w, data)
}

func main() {
    // Create route to love-mountains web page
    http.HandleFunc("/love-mountains", loveMountains)
    // Launch web server on port 80
    http.ListenAndServe(":80", nil)
}

Retrieving dynamic data in a template is easily achieved with {{.}} here. Here is your love-mountains.html template:

<h1>I Love Mountains<h1>
<p>The mountain I prefer is {{.}}</p>

HTTPS

Nowadays, implementing HTTPS on your website has become almost compulsory. How can you switch your Go website to HTTPS?

Linking TLS Certificates

Firstly, issue your certificate and private key in .pem format. You can issue them by yourself with openssl (but you will end up with a self-signed certificate that triggers a warning in the browser) or you can order your cert from a trusted third-party like Let’s Encrypt. Personally, I am using Let’s Encrypt and Certbot to issue certificates and renew them automatically on my servers. More info about how to use Certbot here.

Then you should tell Go where your cert and private keys are located. I am loading the paths to these files from environment variables.

We are now using the ListenAndServeTLS function instead of the mere ListenAndServe:

[...]

// Load TLS cert info from env variables
var tlsCertPath = os.Getenv("TLS_CERT_PATH")
var tlsKeyPath = os.Getenv("TLS_KEY_PATH")

[...]

func main() {
    [...]
    // Serve HTTPS on port 443
    http.ListenAndServeTLS(":443", tlsCertPath, tlsKeyPath, nil)
}

Forcing HTTPS Redirection

For the moment we have a website listening on both ports 443 and 80. It would be nice to automatically redirect users from port 80 to 443 with a 301 redirection. We need to spawn a new goroutine dedicated to redirecting from http:// to https:// (principle is similar as what you would do in a frontend server like Nginx). Here is how to do it:

[...]

// httpsRedirect redirects HTTP requests to HTTPS
func httpsRedirect(w http.ResponseWriter, r *http.Request) {
    http.Redirect(
        w, r,
        "https://"+r.Host+r.URL.String(),
        http.StatusMovedPermanently,
    )
}

func main() {
    [...]
    // Catch potential HTTP requests and redirect them to HTTPS
    go http.ListenAndServe(":80", http.HandlerFunc(httpsRedirect))
    // Serve HTTPS on port 443
    http.ListenAndServeTLS(":443", tlsCertPath, tlsKeyPath, nil)
}

Static Assets

Serving static assets (like images, videos, Javascript files, CSS files, …) stored on disk is fairly easy but disabling directory listing is a bit hacky.

Serving Files from Disk

In Go, the most secured way to serve files from disk is to use http.FileServer. For example, let’s say we are storing static files in a static folder on disk, and we want to serve them at https://my-website/static, here is how to do it:

[...]
http.Handle("/", http.FileServer(http.Dir("static")))
[...]

Preventing Directory Listing

By default, http.FileServer performs a full directory listing, meaning that https://my-website/static will display all your static assets. We don’t want that for security and intellectual property reasons.

Disabling directory listing requires the creation of a custom FileSystem. Let’s create a struct that implements the http.FileSystem interface. This struct should have an Open() method in order to satisfy the interface. This Open() method first checks if the path to the file or directory exists, and if so checks if it is a file or a directory. If the path is a directory then let’s return a file does not exist error which will be converted to a 404 HTTP error for the user in the end. This way the user cannot know if he reached an existing directory or not.

Once again, let’s retrieve the path to static assets directory from an environment variable.

[...]

// Get path to static assets directory from env variable
var staticAssetsDir = os.Getenv("STATIC_ASSETS_DIR")

// neuteredFileSystem is used to prevent directory listing of static assets
type neuteredFileSystem struct {
    fs http.FileSystem
}

func (nfs neuteredFileSystem) Open(path string) (http.File, error) {
    // Check if path exists
    f, err := nfs.fs.Open(path)
    if err != nil {
        return nil, err
    }

    // If path exists, check if is a file or a directory.
    // If is a directory, stop here with an error saying that file
    // does not exist. So user will get a 404 error code for a file or directory
    // that does not exist, and for directories that exist.
    s, err := f.Stat()
    if err != nil {
        return nil, err
    }
    if s.IsDir() {
        return nil, os.ErrNotExist
    }

    // If file exists and the path is not a directory, let's return the file
    return f, nil
}

func main() {
    [...]
    // Serve static files while preventing directory listing
    mux := http.NewServeMux()
    fs := http.FileServer(neuteredFileSystem{http.Dir(staticAssetsDir)})
    mux.Handle("/", fs)
    [...]
}

Full Example

Eventually, your whole website would look like the following:

package main

import (
    "html/template"
    "net/http"
    "os"
    "path/filepath"
)

var staticAssetsDir = os.Getenv("STATIC_ASSETS_DIR")
var templatesDir = os.Getenv("TEMPLATES_DIR")
var tlsCertPath = os.Getenv("TLS_CERT_PATH")
var tlsKeyPath = os.Getenv("TLS_KEY_PATH")

// neuteredFileSystem is used to prevent directory listing of static assets
type neuteredFileSystem struct {
    fs http.FileSystem
}

func (nfs neuteredFileSystem) Open(path string) (http.File, error) {
    // Check if path exists
    f, err := nfs.fs.Open(path)
    if err != nil {
        return nil, err
    }

    // If path exists, check if is a file or a directory.
    // If is a directory, stop here with an error saying that file
    // does not exist. So user will get a 404 error code for a file/directory
    // that does not exist, and for directories that exist.
    s, err := f.Stat()
    if err != nil {
        return nil, err
    }
    if s.IsDir() {
        return nil, os.ErrNotExist
    }

    // If file exists and the path is not a directory, let's return the file
    return f, nil
}

// loveMountains renders love-mountains page after passing some data to the HTML template
func loveMountains(w http.ResponseWriter, r *http.Request) {
    // Load template from disk
    tmpl := template.Must(template.ParseFiles("love-mountains.html"))
    // Inject data into template
    data := "Any dynamic data"
    tmpl.Execute(w, data)
}

// httpsRedirect redirects http requests to https
func httpsRedirect(w http.ResponseWriter, r *http.Request) {
    http.Redirect(
        w, r,
        "https://"+r.Host+r.URL.String(),
        http.StatusMovedPermanently,
    )
}

func main() {
    // http to https redirection
    go http.ListenAndServe(":80", http.HandlerFunc(httpsRedirect))

    // Serve static files while preventing directory listing
    mux := http.NewServeMux()
    fs := http.FileServer(neuteredFileSystem{http.Dir(staticAssetsDir)})
    mux.Handle("/", fs)

    // Serve one page site dynamic pages
    mux.HandleFunc("/love-mountains", loveMountains)

    // Launch TLS server
    log.Fatal(http.ListenAndServeTLS(":443", tlsCertPath, tlsKeyPath, mux))
}

Plus your love-mountains.html template:

<h1>I Love Mountains<h1>
<p>The mountain I prefer is {{.}}</p>

Testing, Deploying and Daemonizing with Systemd

Having a solid and easy test/deploy process is very important from an efficiency standpoint and Go really helps in this regard. Go is compiling everything within a single executable, including all dependencies (except templates but the latter are not real dependencies and this is better to keep them apart for flexibility reasons anyway). Go also ships with its own frontend HTTP server, so no need to install Nginx or Apache. Thus this is fairly easy to test your application locally and make sure it is equivalent to your production website on the server (not talking about data persistence here of course…). No need to add a container system like Docker to your build/deploy workflow then!

Testing

To test your application locally, compile your Go binary and launch it with the proper environment variables like this:

TEMPLATES_DIR=/local/path/to/templates/dir \
STATIC_ASSETS_DIR=/local/path/to/static/dir \
TLS_CERT_PATH=/local/path/to/cert.pem \
TLS_KEY_PATH=/local/path/to/privkey.pem \
./my_go_website

That’s it! Your website is now running in your browser at https://127.0.0.1.

Deploying

Deployment is just about copying your Go binary to the server (plus your templates, static assets, and certs, if needed). A simple tool like scp is perfect for that. You could also use rsync for more advanced needs.

Daemonizing your App with Systemd

You could launch your website on the server by just issuing the above command, but it is much better to launch your website as a service (daemon) so your Linux system automatically launches it on startup (in case of a server restart) and also tries to restart it in case your app is crashing. On modern Linux distribs, the best way to do so is by using systemd, which is the default tool dedicated to management of system services. Nothing to install then!

Let’s assume you put your Go binary in /var/www on your server. Create a new file describing your service in the systemd directory: /etc/systemd/system/my_go_website.service. Now put the following content inside:

[Unit]
Description=my_go_website
After=network.target auditd.service

[Service]
EnvironmentFile=/var/www/env
ExecStart=/var/www/my_go_website
ExecReload=/var/www/my_go_website
KillMode=process
Restart=always
RestartPreventExitStatus=255
Type=simple

[Install]
WantedBy=multi-user.target

The EnvironmentFile directive points to an env file where you can put all your environment variables. systemd takes care of loading it and passing env vars to your program. I put it in /var/www but feel free to put it somewhere else. Here is what your env file would look like:

TEMPLATES_DIR=/remote/path/to/templates/dir
STATIC_ASSETS_DIR=/remote/path/to/static/dir
TLS_CERT_PATH=/remote/path/to/cert.pem
TLS_KEY_PATH=/remote/path/to/privkey.pem

Feel free to read more about systemd for more details about the config above.

Now:

  • launch the following to link your app to systemd: systemctl enable my_go_website
  • launch the following to start your website right now: systemctl start my_go_website
  • restart with: systemctl restart my_go_website
  • stop with: systemctl stop my_go_website

Replacing Javascript with WebAssembly (Wasm)

Here is a bonus section in case you are feeling adventurous!

As of Go version 1.11, you can now compile Go to Web Assembly (Wasm). More details here. This is very cool as Wasm can work as a substitute for Javascript. In other words you can theoretically replace Javascript with Go through Wasm.

Wasm is supported in modern browsers but this is still pretty experimental. Personally I would only do this as a proof of concept for the moment, but in the mid term it might become a great way to develop your whole stack in Go. Let’s wait and see!

Conclusion

Now you know how to develop a whole website in Go and deploy it on a Linux server. No frontend server to install, no dependency hell, and great performances… Pretty straightforward isn’t it?

If you want to learn how to build a Single Page App (SPA) with Go and Vue.js, have a look at my other post here.

Existe aussi en français
Torrengo: a concurrent torrent searcher CLI in Go

I’ve used the Torrench program for quite a while as this is a great Python tool for easy torrent searching in console (CLI). I figured I could try to develop a similar tool in Go (Golang) as it would be a good opportunity for me to learn new stuffs plus I could make this program concurrent and thus dramatically improve speed (Torrench is a bit slow in this regards).

Let me tell you about this new program’s features here, show you how I handled concurrency in it, and then tell you how I organized this Go library for easy reuse by third-parties.

Torrengo Features

Torrengo is a command line (CLI) program that concurrently searches torrents (torrent files and magnet links) from various torrent websites. Looking for torrents has always been a nightmare to me as it means launching the same search on multiple websites, facing broken websites that are not responding, and being flooded by awful ads… so this little tool is here to solve these problems. Here is an insight of the core features of Torrengo as of this writing.

Simple CLI Program

I realized I did not have much experience writing CLI tools so I figured this little project would be a good way to improve my skills in writing command line programs in Go.

CLI programs for very specific use cases like this one and with very few user interactions needed can prove to be much clearer and lighter than a GUI. They can also easily be plugged to other programs.

You can launch a new search like this:

./torrengo Alexandre Dumas

It will return all results found on all websites sorted by seeders, and it’s up to you to decide which one to download then:

Torrengo output

You can also decide to search only specific websites (let’s say The Pirate Bay and Archive.org):

./torrengo -s tpb,arc Alexandre Dumas

If you want to see more logs (verbose mode), just add the -v flag.

Fast

Thanks to Go being a very efficient language and easy to code concurrent programs with, Torrengo is scraping results very fast. Instead of fetching every website sequentially one by one, it fetches everything in parallel.

Torrengo is as fast as the slowest HTTP response then. Sometimes the slowest HTTP response can take more time than you want (for example Archive.org is pretty slow from France as of this writing). In that case, as Torrengo supports timeouts just set a timeout like this:

./torrengo -t 2000 Alexandre Dumas

The above stops every HTTP requests that take more than 2 seconds and returns the other results found.

Use The Pirate Bay Proxies

The Pirate Bay urls are changing quite often and most of them work erratically. To tackle this, the program concurrently launches a search on all The Pirate Bay urls found on a proxy list website (proxybay.bz as of this writing) and retrieves torrents from the fastest response.

The returned url’s HTML is also checked in-depth because some proxies sometimes return a page with no HTTP error but the page actually does not contain any result…

Bypass Cloudflare’s Protection on Torrent Downloads

Downloading torrent files from the Torrent Downloads’ website is difficult because the latter implements a Cloudflare protection that consists of a Javascript challenge to answer. Torrengo tries to bypass this protection by answering the Javascript challenge, which is working 90% of the times as of this writing.

Authentication

Some torrent websites like Ygg Torrent (previously t411) need users to be authenticated to download the torrent files. Torrengo handles this by asking you for credentials (but of course you need to have an account in the first place).

Word of caution: never trust programs that ask for your credentials without having a look into the source code first.

Open in Torrent Client

Instead of copy pasting magnet links found by Torrengo to your torrent client or looking for the downloaded torrent file and opening it, you can just say that you want to open torrent in local client right away. Only supported for Deluge as of this writing.

Concurrency

Thanks to Go’s simplicity, the way concurrency is used in this program is not hard to understand. This is especially true here because concurrency is perfect for networking programs like this one. Basically, the main goroutine spawns one new goroutine for each scraper and retrieves results in structs through dedicated channels. tpb is a special case as the library spawns n goroutines, n being the number of urls found on the proxy website.

I take the opportunity to say that I learned 2 interesting little things regarding channels best practices:

  • every scraping goroutines return both an error and a result. I decided to create a dedicated channel for the result (a struct) and another one for the error, but I know that some prefer to return both inside a single struct. It’s up to you, both are fine.
  • goroutines spawned by tpb don’t need to return an error but just say when they are done. This is something call an “event”. I did it by sending an empty struct (struct{}{}) through the channel which seems the best to me from a memory footprint standpoint as an empty struct weighs nothing. Instead you could also pass an ok boolean variable which is perfectly fine too.

Library Structure

This lib is made up of multiple sub-libraries: one for each scraper. The goal is that every scraping library is loosely coupled to the Torrengo program so it can be reused easily in other third-party projects. Every scraper has its own documentation.

Here is the current project’s structure:

torrengo
├── arc
│   ├── download.go
│   ├── README.md
│   ├── search.go
├── core
│   ├── core.go
├── otts
│   ├── download.go
│   ├── README.md
│   ├── search.go
├── td
│   ├── download.go
│   ├── README.md
│   ├── search.go
├── tpb
│   ├── proxy.go
│   ├── README.md
│   ├── search.go
├── ygg
│   ├── download.go
│   ├── login.go
│   ├── README.md
│   ├── search.go
├── README.md
├── torrengo.go

Let me give you more details about how I organized this Go project (I wish I had more advice like this regarding Go project structures’ best practice before).

One Go Package Per Folder

Like I said earlier, I created one lib for each scraper. All files contained in the same folder belong to the same Go package. In the chart above there are 5 scraping libs:

  • arc (Archive.org)
  • otts (1337x)
  • td (Torrent Downloads)
  • tpb (The Pirate Bay)
  • ygg (Ygg Torrent)

Each lib can be used in another Go project by importing it like this (example for Archive.org):

import "github.com/juliensalinas/torrengo/arc"

The core lib contains things common to all scraping libs and thus will be imported by all of them.

Each scraping lib has its own documentation (README) in order for it to be really decoupled.

Multiple Files Per Folder

Every folder contain multiple files for easier readability. In Go, you can create as many files as you want for the same package as long as you mention the name of the package at the top of the file. For example here every files contained in the arc folder have the following line at the top:

package arc

For example in arc I organized things so that every structs, methods, functions, variables, related to the download of the final torrent file go into the download.go file.

Command

Some Go developers put everything related to commands, that’s to say the part of your program dedicated to interacting with the user, in a special cmd folder.

Here I preferred to put this in a torrengo.go file at the root of the project which seemed more appropriate for a small program like this.

Conclusion

I hope you’ll like Torrengo and I would love to receive feed-backs about it! Feed-backs can be about the features of Torrengo of course, but if you’re a Go programmer and you think some parts of my code can be improved I’d love to hear your ideas as well.

I am planning to write more articles about what I learned when writing this program!

Existe aussi en français
Nice Readings About Go, Python, and Technical Management

I sometimes find interesting information in books that I do not find on the internet and that’s why I’m reading a lot. I read quite a lot of books recently and some were not worth it, but some were great, so let me share it with you here!

Go In-Depth

If you already know the basics about Go (Golang) and want to go deeper, The Go Programming Language (Donovan & Kernighan) is a great book.

Personally, before reading this book I had developed several projects in Go but I was still missing important points about the Go language, especially about the spirit of the language and how to be idiomatic. This book really helped me. It starts with the basics but goes very deep by talking about reflection, testing, low level programming,… Examples are very concise (I hate reading too much code in books) and related to useful real life scenarios.

Go and Concurrency

If you feel the book above does not go far enough about concurrency, you can read Concurrency In Go (Cox-Buday).

This book is very short and to the point. It really emphasizes best practices about concurrency (in general, and more particularly with the Go language). This book gives you patterns that you can apply in your own concurrent programs.

Tips and Tooling in Python

Python is quite an old language so many great tools exist around this language (libraries, frameworks, etc.). It’s not easy to have a clear view about all the possible options though. The Hitchhiker’s Guide to Python! (Kenneth Reitz) summarizes everything you should know about Python’s ecosystem. It can save you a lot of time. Careful: this is not a coding book!

Technical Management

If you are a in charge of a team of developers you already know that this is a very tough task. If you’re not yet, be prepared! In both cases it’s important to work on it. Managing and leading people is a very difficult job, especially in the programming field (lead developer, technical manager, CTO, …). This book was a great help to me: Managing the Unmanageable (Mantle & Lichty). It does not overwhelm you with theoretical concepts about management but instead gives you lots of very useful tips easy to apply.

Conclusion

If you read one of them please let me know your thoughts!

Existe aussi en français
Python Flask vs Django

My experience of Flask is not as extensive as my experience of Django, but still recently I’ve developed some of my projects with Flask and I could not help comparing those 2 Python web frameworks. This will be a quick comparison which will not focus on code but rather on “philosophical” considerations. Of course my opinion might change in the future!

Both Are Still Python Frameworks…

When I switched from Django to Flask, what first occurred to me was that a lot of things are very similar. You won’t be lost. This is mainly due to the fact that both are using Python. Even if Flask is supposed to be more Pythonic (using decorators extensively for example), I did not notice many differences. In addition to that, both are web frameworks and most of web frameworks propose pretty much the same features:

  • an ORM
  • routing features
  • a user request object used extensively
  • templates
  • forms handling
  • static files handling

Even if you switch from Python to another language, using a web framework will help you a lot because this MVC structure will be there (ok this is not always real MVC but whatever) and you will be at home in a second. For example see how easy it is to switch from Python/Django to PHP/Laravel.

Framework VS Micro Framework

Flask is supposed to be a micro framework which is less “opinionated” and with less “batteries included”. In a way you’re supposed to be more free than with Django. And this is true! Django proposes a file/folder structure that everyone follows. Django also ships with its own ORM, templating system, forms handling, registration/login system,… which are used by 99% of people. Problem is that, in my opinion, this freedom is more of a problem than an help…

This lack of structure implies that you know in advance exactly what will be the best structure for your project, which is very difficult if you have never developed a full project with Flask yet and never had the opportunity to test how this structure will fit a growing project over months/years. I saw a lot people ending up asking for advice regarding their project structure on StackOverflow and eventually most people use… Django’s structure (one central configuration file and one folder per application containing a views.py + models.py + forms.py + templates)! I also struggled, like many people, with Python circular imports, which never happens in Django since smart people designed the structure for you in advance especially in order to avoid this kind of problems.

Project’s structure is a very important part of a project’s documentation. A standard structure will considerably help newcomers on your project. Every time I start working on an existing Django project, I am fully operational in a minute because I know in advance how things are organized. It is not the case with Flask so developers on a Flask project need to write additional documentation to explain their project’s structure. Same problem with tooling: every Flask project might ship with different core libraries (for forms, templates, registration/login, etc.) so new developers arriving on a project might have to learn new tools before starting working.

Documentation

Flask’s documentation is good but it is clearly too hard for beginner developers compared to Django’s docs which matches every levels (from beginner to advanced). In addition to that, Flask being a micro framework, most of your work relies on external dependencies like SQLAlchemy for ORM, Flask login for login, Flask WTF for forms, etc. and some of these libs do not have the same documentation standard as Flask does. Let me put it this way: I find SQLAlchemy’s docs awful, and tools like Flask login really need to grow their docs for example (solving my problems by browsing the lib’s source code should never happen in my opinion).

Community

Flask’s community is much smaller than Django’s (but very nice). Consequence is that you won’t find all your answers on StackOverflow or GitHub. However you end up realizing that most issues are actually Python issues (so no problem here) or issues met by Django’s users as well (so in this case Django’s community helps).

Conclusion

Flask is a micro framework that does not recommend standards or conventions. In my opinion this is more of a problem than a solution if you want your project to be easily maintained by other people. However if freedom attracts you, then give Flask a try! If you’re already a Django developer the switch will be super easy.

Existe aussi en français
Building a modern application with a Golang API backend + a Vue.js SPA frontend using Docker

The goal of this article is to show a real application I made recently using Go, Vue.js and Docker which is in production today. Tutorials are sometimes disappointing because they do not talk about real life situations so I tried to put things differently here. I won’t comment the whole code as it would take ages but explain the overall project structure, which important choices I made, and why. I’ll also try to emphasize interesting parts of code worth commenting.

The code of the whole app is here on my GitHub, maybe you should open it in parallel of this article.

Purpose of the Application

This application is dedicated to presenting data from various databases in a user friendly way. The main features are the following:

  • user has to enter credentials in order to use the Single Page Application (SPA) frontend
  • user can select various interfaces in a left panel in order to retrieve data from various db tables
  • user can decide either to only count results returned by db or get the full results
  • if results returned by db are lightweight enough then they are returned by the API and displayed within the SPA app inside a nice data table. User can also decide to export it as CSV.
  • if results are too heavyweight, then results are sent asynchronously to the user by email within a .zip archive
  • as input criteria, the user can enter text or CSV files listing a big amount of criteria
  • some user inputs are select lists whose values are loaded dynamically from db

Project’s Structure and Tooling

This project is made up of 2 Docker containers:

  • a container for a backend API written in Go. No need of an HTTP server here since Go already has a very efficient built-in HTTP server (net/http). This application exposes a RESTful API in order to get requests from frontend and return results retrieved from several databases.
  • a container for a frontend interface using a Vue.js SPA. Here an Nginx server is needed in order to serve static files.

Here is the Dockerfile of my Go application:

FROM golang
VOLUME /var/log/backend
COPY src /go/src
RUN go install go_project
CMD /go/bin/go_project
EXPOSE 8000

Dead simple as you can see. I’m using a pre-built Docker Golang image based on Debian.

My frontend Dockerfile is slightly bigger because I need to install Nginx, but still very simple:

FROM ubuntu:xenial

RUN apt-get update && apt-get install -y \
    nginx \
    && rm -rf /var/lib/apt/lists/*

COPY site.conf /etc/nginx/sites-available
RUN ln -s /etc/nginx/sites-available/site.conf /etc/nginx/sites-enabled
COPY .htpasswd /etc/nginx

COPY startup.sh /home/
RUN chmod 777 /home/startup.sh
CMD ["bash","/home/startup.sh"]

EXPOSE 9000

COPY vue_project/dist /home/html/

The startup.sh simply starts the Nginx server. Here is my Nginx configuration (site.conf):

server {

    listen 9000;

    server_name api.example.com;

    # In order to avoid favicon errors on some navigators like IE
    # which would pollute Nginx logs (do use the "=")
    location = /favicon.ico { access_log off; log_not_found off; }

    # Static folder that Nginx must serve
    location / {
        root /home/html;
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }

    # robots.txt file generated on the fly
    location /robots.txt {
        return 200 "User-agent: *\nDisallow: /";
    }

}

As you can see, authentication is needed in order to use the frontend app. I implemented this within a .htpasswd file.

Actually using Docker for the Go application is not really a big advantage since Go needs no external dependency once compiled making deployment very easy. Sometimes shipping a Go app inside Docker can be useful if you have external files needed in addition to your Go binary (like HTML templates or config files for example). This is not the case here but I still used Docker for consistency reasons: all my services are deployed through Docker so I do not want to have special cases to deal with.

The Go application is made up of multiple files. This is just for readability reasons and everything could have been put into one single file. You must keep in mind that when splitting the application like this, you need to export things (variables, structs, functions, …) you want to use across multiple files (using a capitalized first letter). During development you also need to use go run with a wildcard like this:

go run src/go_project/*.go

I’m using a couple of Go external libraries (so few thanks to the already very comprehensive Go’s standard library!):

  • gorilla/mux for the routing of REST API requests, especially for endpoints expecting positional arguments
  • rs/cors for easier handling of CORS (which can be a nightmare)
  • gopkg.in/gomail.v2 for email handling, especially for easy addition of attachments

Structure and tooling are much more complex regarding the frontend part. Here is an article dedicated to this. Actually this complexity only affects the development part because in the end, once everything is compiled, you only get regular HTML/CSS/JS files that you simply copy paste into your Docker container.

Dev vs Prod

Configuration is different in development and production. During development I’m working on a locally replicated database, I’m logging errors to console instead of file, I’m using local servers, … How to manage this seamlessly?

In the Vue.js app I need to either connect to a local development API (127.0.0.1) or a production API (api.example.com). So I created a dedicated http-constants.js which returns either a local address or a production address depending on whether we launched the npm run dev command or npm run build command. See this article for more details.

In the Go app, multiple parameters change depending on whether I’m in development mode or production mode. In order to manage this, I’m using environment variables passed to the Go app by Docker. Setting configuration through environment variables is supposed to be best practice according to the 12 factor app. First we need to set environment variables during container creation thanks to the -e option:

docker run --net my_network \
--ip 172.50.0.10 \
-p 8000:8000 \
-e "CORS_ALLOWED_ORIGIN=http://api.example.com:9000" \
-e "REMOTE_DB_HOST=10.10.10.10" \
-e "LOCAL_DB_HOST=172.50.0.1" \
-e "LOG_FILE_PATH=/var/log/backend/errors.log" \
-e "USER_EMAIL=me@example.com" \
-v /var/log/backend:/var/log/backend \
-d --name backend_v1_container myaccount/myrepo:backend_v1

Then those variables are retrieved within the Go program thanks to the os.getenv() function. Here is how I managed it in main.go:

// Initialize db parameters
var localHost string = getLocalHost()
var remoteHost string = getRemoteHost()
const (
	// Local DB:
	localPort     = 5432
	localUser     = "my_local_user"
	localPassword = "my_local_pass"
	localDbname   = "my_local_db"

	// Remote DB:
	remotePort     = 5432
	remoteUser     = "my_remote_user"
	remotePassword = "my_remote_pass"
	remoteDbname   = "my_remote_db"
)

// getLogFilePath gets log file path from env var set by Docker run
func getLogFilePath() string {
	envContent := os.Getenv("LOG_FILE_PATH")
	return envContent
}

// getLocalHost gets local db host from env var set by Docker run.
// If no env var set, set it to localhost.
func getLocalHost() string {
	envContent := os.Getenv("LOCAL_DB_HOST")
	if envContent == "" {
		envContent = "127.0.0.1"
	}
	return envContent
}

// getRemoteHost gets remote db host from env var set by Docker run.
// If no env var set, set it to localhost.
func getRemoteHost() string {
	envContent := os.Getenv("REMOTE_DB_HOST")
	if envContent == "" {
		envContent = "127.0.0.1"
	}
	return envContent
}

// getRemoteHost gets remote db host from env var set by Docker run.
// If no env var set, set it to localhost.
func getCorsAllowedOrigin() string {
	envContent := os.Getenv("CORS_ALLOWED_ORIGIN")
	if envContent == "" {
		envContent = "http://localhost:8080"
	}
	return envContent
}

// getUserEmail gets user email of the person who will receive the results
// from env var set by Docker run.
// If no env var set, set it to admin.
func getUserEmail() string {
	envContent := os.Getenv("USER_EMAIL")
	if envContent == "" {
		envContent = "admin@example.com"
	}
	return envContent
}

As you can see, if the production env variable is not set, we set a default value for local development. Then we can use those dedicated functions anywhere in the program. For example, here is how I’m handling the logging feature (log to console in development mode, and log to file in production):

log.SetFlags(log.LstdFlags | log.Lshortfile)            // add line number to logger
if logFilePath := getLogFilePath(); logFilePath != "" { // write to log file only if logFilePath is set
	f, err := os.OpenFile(logFilePath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()
	log.SetOutput(f)
}

Note that logging also involves the use of a shared volume. Indeed I want my log files to be accessed from the Docker host easily. That’s why I added -v /var/log/backend:/var/log/backend to the docker run command above and put a specific VOLUME directive in the Dockerfile.

Design of the Frontend App with Vuetify.js

I have never been fond of spending days working on design, especially for small apps like this one. That’s why I’m using Vuetify.js which is a great framework to be used on top of Vue.js providing you with ready to use beautiful components. Vuetify uses Google’s material design, which looks very good to me.

Memory Usage

I’ve faced quite a lot of memory issues while building this app due to the fact that some SQL queries can possibly return a huge amount of data.

In the Go Backend

Rows returned from db are put in an array of structs. When millions of rows are returned, manipulating this array becomes very costly in terms of memory. The solution is to put as much logic as you can in your SQL request instead of your Go program. PostgreSQL is excellent at optimizing performances and in my case databases are running on PostgreSQL 10 which increased performance considerably thanks to parallel computing on some operations. Plus my databases have dedicated resources so I should use it as much as possible.

Regarding the CSV generation, you also need to consider whether you should store the CSV in memory or write it to disk. Personally I’m writing it to disk in order to reduce memory usage.

Still, I also had to increase the RAM of my server.

In the Vue.js Frontend

Clearly, a browser cannot handle too much content. If too many rows are to be displayed in the browser, rendering will fail. First solution is what I did: above a certain amount of rows returned by db, send results by email in a .zip archive. Another solution could be that results in browser are paginated and each new page actually triggers a new request to server (behind the hood, you would need to use LIMIT in your SQL request).

Touchy Parts of Code

Here some special parts of code that are worth commenting in my opinion because they can be pretty original or tricky.

Multiple Asynchronous Calls with Axios

My frontend contains multiple HTML selects and I want the values of these lists to be loaded dynamically from the API. For this I need to use axios.all() and axios.spread() in order to make multiple parallel API calls with Axios. Axios’ documentation is not that good in my opinion. It is important to understand that you have 2 choices here:

  • catching error for each request in axios.all: HTTP.get('/get-countries-list').catch(...)
  • catching error globally after axios.spread: .then(axios.spread(...)).catch(...)

The first option allows you to display precise error messages depending on which request raised an error, but this is non blocking so we still enter axios.spread() despite the error and some of the parameters will be undefined in axios.spread() so you need to handle it. In the second option, a global error is raised as soon as one of the requests fails at least, and we do not enter axios.spread().

I chose the 2nd option: if at least one of the API calls fails, then all the calls fail:

created () {
    axios.all([
      HTTP.get('/get-countries-list'),
      HTTP.get('/get-companies-industries-list'),
      HTTP.get('/get-companies-sizes-list'),
      HTTP.get('/get-companies-types-list'),
      HTTP.get('/get-contacts-industries-list'),
      HTTP.get('/get-contacts-functions-list'),
      HTTP.get('/get-contacts-levels-list')
    ])
    // If all requests succeed
    .then(axios.spread(function (
      // Each response comes from the get query above
      countriesResp,
      companyIndustriesResp,
      companySizesResp,
      companyTypesResp,
      contactIndustriesResp,
      contactFunctionsResp,
      contactLevelsResp
    ) {
      // Put countries retrieved from API into an array available to Vue.js
      this.countriesAreLoading = false
      this.countries = []
      for (let i = countriesResp.data.length - 1; i >= 0; i--) {
        this.countries.push(countriesResp.data[i].countryName)
      }
      // Remove France and put it at the top for convenience
      let indexOfFrance = this.countries.indexOf('France')
      this.countries.splice(indexOfFrance, 1)
      // Sort the data alphabetically for convenience
      this.countries.sort()
      this.countries.unshift('France')

      // Put company industries retrieved from API into an array available to Vue.js
      this.companyIndustriesAreLoading = false
      this.companyIndustries = []
      for (let i = companyIndustriesResp.data.length - 1; i >= 0; i--) {
        this.companyIndustries.push(companyIndustriesResp.data[i].industryName)
      }
      this.companyIndustries.sort()

    [...]

    }
    // bind(this) is needed in order to inject this of Vue.js (otherwise
    // this would be the axios instance)
    .bind(this)))
    // In case one of the get request failed, stop everything and tell the user
    .catch(e => {
      alert('Could not load the full input lists in form.')
      this.countriesAreLoading = false
      this.companyIndustriesAreLoading = false
      this.companySizesAreLoading = false
      this.companyTypesAreLoading = false
      this.contactIndustriesAreLoading = false
      this.contactFunctionsAreLoading = false
      this.contactLevelsAreLoading = false
    })
},

Generate CSV in Javascript

I wish there was a straightforward solution in order to create a CSV in javascript and serve it to the user as a download, but it seems there isn’t, so here is my solution:

generateCSV: function () {
      let csvArray = [
        'data:text/csv;charset=utf-8,' +
        'Company Id;' +
        'Company Name;' +
        'Company Domain;' +
        'Company Website;' +
        [...]
        'Contact Update Date'
      ]
      this.resultsRows.forEach(function (row) {
        let csvRow = row['compId'] + ';' +
          row['compName'] + ';' +
          row['compDomain'] + ';' +
          row['compWebsite'] + ';' +
          [...]
          row['contUpdatedOn']
        csvArray.push(csvRow)
      })
      let csvContent = csvArray.join('\r\n')
      let encodedUri = encodeURI(csvContent)
      let link = document.createElement('a')
      link.setAttribute('href', encodedUri)
      link.setAttribute('download', 'companies_and_contacts_extracted.csv')
      document.body.appendChild(link)
      link.click()
    }
}

Get Data Sent by Axios in Go

Axios’ POST data are necessarily sent as JSON. Unfortunately currently there is no way to change this. Go has a useful PostFormValue function that easily retrieves POST data encoded as form data but unfortunately it does not handle JSON encoded data, so I had to unmarshal JSON to a struct in order to retrieve POST data:

body, err := ioutil.ReadAll(r.Body)
if err != nil {
	err = CustErr(err, "Cannot read request body.\nStopping here.")
	log.Println(err)
	http.Error(w, "Internal server error", http.StatusInternalServerError)
	return
}

// Store JSON data in a userInput struct
var userInput UserInput
err = json.Unmarshal(body, &userInput)
if err != nil {
	err = CustErr(err, "Cannot unmarshall json.\nStopping here.")
	log.Println(err)
	http.Error(w, "Internal server error", http.StatusInternalServerError)
	return
}

Variadic Functions in Go

The user can enter a variable number of criteria that will be used within a single SQL query. Basically, each new criteria is a new SQL WHERE clause. As we do not know in advance how many parameters will be passed to the database/sql query() function, we need to use the variadic property of the query() function here. A variadic function is a function that accepts a variable number of parameters. In Python you would use *args or *kwargs. Here we’re using the ... notation. The first argument of query() is a string SQL query, and the second argument is an array of empty interfaces that contains all the parameters:

rows, err := db.Query(sqlStmtStr, sqlArgs...)
if err != nil {
	err = CustErr(err, "SQL query failed.\nStopping here.")
	log.Println(err)
	http.Error(w, "Internal server error", http.StatusInternalServerError)
	return compAndContRows, err
}
defer rows.Close()

Managing CORS

Basically, CORS is a security measure that prevents frontend from retrieving data from a backend that is not located at the same URL. Here is a nice explanation of why CORS is important. In order to comply with this behaviour you should handle CORS properly on the API server side. The most important CORS property to be set is the Allowed Origins property. It’s not that easy to handle it in Go since it implies first a “preflight” request (using HTTP OPTION) and then setting the proper HTTP headers.

The best solution in Go in my opinion seems to be the rs/cors library that allows us to handle CORS like this:

router := mux.NewRouter()

c := cors.New(cors.Options{
	AllowedOrigins: []string{"http://localhost:8080"},
})
handler := c.Handler(router)

NULL Values in Go

When making SQL requests to db, you’ll probably get some NULL values. Those NULL values must be handled explicitly in Go, especially if you want to marshal those results to JSON. You have 2 solutions:

  • use pointers for nullable values in your struct that will receive values. It works but NULL values are not detected by the 'omitempty' keyword during JSON marshaling so an empty string will still be displayed in the JSON result.
  • use the sql lib nullable types: replace string with sql.NullString, int with sql.NullInt64, bool with sql.NullBool, and time with sql.NullTime but then you obtain something like {"Valid":true,"String":"Smith"} which is not directly ok in JSON. So it requires extra steps before marshaling to JSON.

I implemented the 2nd option and created a custom type + method that implements the json.Marshaler. Note that, by using this method, I could have turned NULL into an empty string so that it is not included in the final JSON, but here I wanted the NULL values to be kept and sent to frontend in JSON as null:

type JsonNullString struct {
	sql.NullString
}

func (v JsonNullString) MarshalJSON() ([]byte, error) {
	if v.Valid {
		return json.Marshal(v.String)
	} else {
		return json.Marshal(nil)
	}
}

type CompAndContRow struct {
	CompId                       string         `json:"compId"`
	CompName                     JsonNullString `json:"compName"`
	CompDomain                   JsonNullString `json:"compDomain"`
	CompWebsite                  JsonNullString `json:"compWebsite"`
	[...]
}

Concatenation of Multiple Rows in SQL

SQL is a very old but still very powerful language. In addition to that, PostgreSQL provides us with very useful functions that allow us to do a lot of things within SQL instead of applying scripts to the results (which is not memory/CPU efficient). Here I have quite a lot of SQL LEFT JOIN that return a lot of very similar rows. Problem is I want some of these rows to be concatenated within one single row. For example, a company can have multiple emails and I want all these emails to appear in the same row separated by this symbol: ¤. Doing this in Go would mean parsing the array of SQL results a huge number of time. In case of millions of rows it would be very long and even crash if the server does not have enough memory. Fortunately, doing it with PostgreSQL is very easy using the string_agg() function combined with GROUP BY and DISTINCT:

SELECT comp.id, string_agg(DISTINCT companyemail.email,'¤')
FROM company AS comp
LEFT JOIN companyemail ON companyemail.company_id = comp.id
WHERE comp.id = $1
GROUP BY comp.id

Conclusion

I’m covering a wide range of topics inside a single article here: Go, Vue.js, Javascript, SQL, Docker, Nginx… I hope you found useful tips that you’ll be able to reuse in you own application.

If you have questions about the app feel free to ask. If you think I could have optimized better some parts of this code, I would love to hear it. This article is also for me a way to get critical feedbacks and question my own work!

Existe aussi en français