A senior software engineer who relocates with a family does not face the same bill as a single engineer renting a one-bedroom flat. The partner stays home, two children need school and healthcare, and one car is part of daily life. I rebuilt my earlier purchasing-power ranking for that household, across the same 64 tech hubs, and asked a sharper question: what is left each month after tax, benefits, rent, everyday living, school fees that foreigners actually pay, healthcare, and the car.

The answer changes once you stop thinking like a long-settled local and start thinking like a family that has just arrived on a work permit.

Who this ranking is for

The persona is the same everywhere: a senior software engineer of about 35, a partner who is not in paid work, two children aged 5 and 8, and one car. The salary is the Levels.fyi senior base median for each hub. Everyday living costs come from Numbeo’s family-of-four basket excluding rent. Housing is a three-bedroom apartment outside the city centre, which is more realistic for a family with a car than a centre flat.

I framed every tax, benefit, school and healthcare choice for a family relocating from abroad, not for someone who has lived in the country for years. That framing is the whole point of this study.

Why the relocating lens changes the numbers

Special tax regimes that newcomers often receive matter a lot. In the Netherlands I applied the 30 percent ruling. In Spain I used the Beckham law flat rate of 24 percent on employment income. In France I applied the impatriate regime that exempts 30 percent of taxable pay from income tax. In Denmark I used the 27 percent key-employee scheme, and in Portugal the IFICI flat rate of 20 percent for qualifying technology specialists. Ireland’s special assignee relief and Sweden’s expert tax were left out, because a typical direct local hire would not usually qualify.

Child benefits are another trap. Canada’s child benefit needs 18 months of continuous residency before a temporary resident can claim it, so I excluded it for a newly arrived family. Singapore’s baby bonus is for citizens only. Typical United States work-visa families keep the child tax credit when the children have Social Security numbers. Most European child allowances remain available to resident work-permit holders, and those stay in the model.

School is not free for foreigners everywhere. In 48 of the 64 hubs, public primary school is open to resident children of work-permit holders at no tuition, so I used zero. In Singapore, non-ASEAN pupils at government or government-aided primary schools pay Ministry of Education fees that add up to about 1,639 dollars a month for two children. In 15 hubs, including Dubai, Riyadh, the Chinese cities, India, much of Southeast Asia, Lagos and Nairobi, relocating tech families typically use private or international schools, so I used a mid-range private or Numbeo international primary cost for two children.

Healthcare follows the same relocating test. Where a foreign family must buy cover, I included it: the United Kingdom Immigration Health Surcharge of 1,035 pounds a year per adult and 776 pounds a year per child, Australian overseas visitor cover for work-visa holders, Singapore private plans for Employment Pass holders, and mid-range private cover where public insurance does not usefully cover foreign dependents. In Dubai and Riyadh the employer must insure the family, so the employee premium is zero. In the United States, Switzerland and the Netherlands the mandatory employee share was already in the model. I also set aside a monthly out-of-pocket allowance for a bad year, such as a child who breaks a leg and goes to hospital, using official deductibles and co-pay caps where they are published and labelling the rest as approximations.

What the ranking shows

United States cities still dominate the top of the list, even after family healthcare. Austin leaves about 4,473 dollars a month. Silicon Valley follows at about 3,366, then Seattle at 3,093, Chicago at 2,544 and Philadelphia at 1,850. Boston, Los Angeles and San Diego stay in the top ten as well. That lead survives the average employee share of a family health premium and a bad-year deductible of about 163 dollars a month.

Copenhagen appears high in the full ranking, around 1,461 dollars a month, helped by Denmark’s 27 percent scheme for highly paid newcomers. That figure rests on a thin salary sample and is marked as a rough estimate, so it should be read with caution. Among the 39 hubs with solid or partial data, Manchester is the strongest European city in the top ten, at about 914 dollars a month after the United Kingdom health surcharge.

Zurich, often treated as a high earner’s paradise, breaks even at zero once Swiss mandatory health premiums and a modest out-of-pocket allowance are included. Toronto leaves about 207 dollars a month, and Vancouver about 336, both without Canada’s child benefit in the first year. Montreal slips slightly negative.

London is a clear warning for a relocating family. After tax, the health surcharge of about 407 dollars a month for this household, and a three-bedroom rent outside the centre of about 3,782 dollars, the city leaves a shortfall of about 1,813 dollars a month. New York City is also negative, at about 663 dollars short, because rent and healthcare overwhelm a high gross salary.

Singapore and many Asian and African hubs turn sharply negative once school and private health enter the picture. Singapore leaves a shortfall of about 4,089 dollars a month: the car price includes the Certificate of Entitlement, government school fees for non-ASEAN pupils add about 1,639 dollars, and private health for an Employment Pass family adds hundreds more. Dubai is negative mainly because of international school costs, even though the employer pays mandatory health cover. Lagos sits at the bottom with a shortfall of about 6,424 dollars a month. Across the full list, 18 hubs leave a surplus, Zurich breaks even, and 45 leave a shortfall.

The stacked view makes the same point city by city. In Austin and Seattle, net pay still covers rent, living costs, the car and healthcare, with thousands left over. In Zurich the pieces fill the net-pay tick almost exactly. In London and Singapore the cost bar runs past net pay, and school or healthcare are visible reasons why.

What this means if you are thinking of moving

A high gross salary is not enough for a family move. The places that look generous for a single engineer can shrink once a non-working partner, two children, school access and real healthcare rules are included. United States hubs keep a structural advantage for this persona, but only if the visa and the family health plan are realistic. European cities with strong expat tax regimes improve for newcomers, yet rent and health surcharges can erase that gain. Hubs that push foreign children into private school, or foreign workers into private health insurance, need a much higher salary than the headline Levels.fyi number suggests.

I would treat gaps of a few hundred dollars between neighbouring cities as noise, especially where the salary sample is thin. The useful signal is the large gap between the leading United States cities, the break-even European cases, and the deep shortfalls where school and private health dominate.

How I built it

Salaries are Levels.fyi Software Engineer senior base medians, accessed on 5 November 2026. Where a city has no senior page of its own, I used the national Levels.fyi senior median for that country: Munich, Vilnius, Riyadh, Shenzhen, Hangzhou, Bangkok, Lagos, Nairobi, Bogotá, Santiago, Manchester and Bucharest. Miami uses a Bureau of Labor Statistics estimate scaled to senior level, and Hyderabad is scaled from Bengaluru.

Taxes, social contributions and child benefits follow each country’s 2026 rules for a single-earner couple with two children, with the relocating adjustments described above. Rent is Numbeo’s three-bedroom apartment outside the centre. Everyday living is Numbeo’s family-of-four estimate excluding rent. The car is a Toyota Corolla paid off over five years, minus the small share of car costs already inside Numbeo’s basket, plus extra fuel and a simple insurance estimate; in Singapore the Numbeo car price includes the Certificate of Entitlement.

School classifications and fees are documented city by city. Healthcare premiums and the bad-year out-of-pocket allowance are documented with sources; some private premiums and some out-of-pocket figures are labelled approximations. All amounts are in United States dollars at 5 November 2026 spot rates.

Of the 64 hubs, 17 have solid data, 22 have partial data, and 25 are rough estimates that rely on small samples or simplified models. Read the ranking for large differences, not for fine rankings among rough cities.

Full per-country data is available on request: explore@juliensalinas.com.

If you have relocated with children and your experience differs from this picture, tell me which city and what I should adjust.

When EU citizens emigrate, a little over half of them settle in another EU country, which free movement makes easy. Yet millions do leave the EU altogether, and I wanted to know where they go. The answer is less obvious than it seems: the top 25 emigration destinations outside the EU include Canada and Australia, as you would expect, but also Türkiye, Serbia, Brazil and Russia, and the people behind each figure turn out to be very different.

What the numbers measure

Every figure in the main ranking is the total number of EU-born emigrants living in a given country in mid-2024, that is, people who were born in one of the 27 EU countries and now reside in that destination. It is not a yearly flow: it adds up everyone who emigrated there, whether they moved last year or fifty years ago. That distinction matters, because it explains why some destinations rank high for reasons that have more to do with history than with relocation today.

The UK and the US lead by far

The United Kingdom is the first emigration destination outside the EU, with 3.41 million EU-born residents, and the United States comes second with 2.89 million. Together they account for almost half of the 13 million EU-born emigrants counted across the top 25. Switzerland (1.44 million) and Canada (1.12 million) are the only other countries above one million, followed by Türkiye (956,000) and Australia (835,000).

Beyond the top six, the figures drop quickly. Norway hosts 381,000 EU-born emigrants, Israel 340,000 and Serbia 288,000, and the last places on the list, Japan, Uruguay and Andorra, each host a little over 40,000.

Who goes where

The country of birth changes completely from one destination to the next. Poles and Romanians make up almost half of the EU-born emigrants in the UK (26% and 21%), which likely reflects the large movement that followed their countries’ accession to the EU. Germans are the largest group in the United States (39%) and in Switzerland (26%), while Italians remain the first group in Canada (16%) and Australia (18%).

Shared languages and old historical ties likely play a role too. Portuguese-born emigrants make up 55% of the EU-born population in Brazil, Spanish-born emigrants make up 59% in Uruguay and 42% in Argentina, and people born in France account for 80% in Morocco and 99% in New Caledonia, which is a French territory outside the EU. Thailand, for its part, splits almost evenly between Germans and French.

Retirees in some places, younger arrivals in others

Age is where the contrast is sharpest. In Canada and Australia, 39% of the European-born residents are aged 65 or over, and the share reaches 37% in Israel, which likely reflects waves of emigration that took place decades ago. In the United Kingdom, by contrast, only 10% are 65 or over, and the share falls to 8% in Japan, 7% in Norway and 4% in Iceland, likely because most people arrived more recently, many of them to work.

These age shares have an important limit, which I explain in the methodology below: they cover OECD countries only, they date from 2015/16, and they count everyone born in Europe rather than only in the EU.

When history explains the ranking

A few entries reflect history more than recent emigration, and I chose to keep them in the ranking while explaining them rather than dropping them:

• Serbia: 77% of its EU-born residents were born in Croatia before Yugoslavia broke up, which likely explains most of this figure.
• Türkiye: the EU-born population mixes people born in Bulgaria (36%), likely including many ethnic Turks who left in past decades, and people born in Germany (38%), likely including many children of Turkish migrants who returned home.
• Russia and Ukraine: most of their EU-born residents were born in Germany or the Baltic states (Lithuania and Latvia make up 73% in Ukraine), which likely reflects moves within the Soviet bloc.
• Israel: the largest groups were born in Romania (34%) and Poland (18%), and they likely include many older Jewish immigrants.
• Latin America: the Spanish, Portuguese and Italian communities in Brazil, Venezuela, Argentina, Mexico, Chile and Uruguay likely reflect the great waves of 20th-century emigration.

How I built it

• Number of EU-born emigrants: UN DESA international migrant estimates, 2024 edition, table by destination and origin, mid-2024. I added up the 27 EU countries of origin for each destination. Most countries count people by country of birth, but Thailand, Morocco, Japan and Andorra count them by citizenship.
• Main countries of birth: the same UN DESA table, showing the two EU countries that make up the largest shares of each destination’s EU-born population.
• Share aged 65 or over: the OECD Database on Immigrants in OECD Countries (DIOC) 2015/16, based on national censuses and registers. Its public files only break people down by region of birth, so I used everyone born in Europe as an approximation of the EU-born population.
• Scope: I kept every country outside the EU27, including the EFTA countries Switzerland, Norway and Iceland. I left out Réunion, Guadeloupe and Martinique, because these French overseas departments are legally part of the EU.

Limits

The ranking counts all EU-born emigrants, whatever their age, so it includes retirees, children and people who emigrated decades ago. The age data is about ten years older than the main figures, covers only 13 of the 25 destinations, and includes people born in non-EU European countries such as the UK, Russia or Ukraine, so it should be read as an indication rather than an exact profile. I also looked for comparable data on self-employment and household type by EU country of birth, but I did not find any source that covers these destinations consistently, so I left them out rather than guess.

The full per-country data, with the sources and definitions for each destination, is available on request: email explore@juliensalinas.com.

If you spot an error, or know a better source for one of these countries, tell me and I will update the chart.

A software engineer on a $200,000 salary in Silicon Valley keeps less each month than one on $160,000 in Austin. Once you subtract income tax and rent, $7,912 a month in the Valley versus $8,128 in Austin. That is why I stopped comparing salaries and started comparing what is left at the end of the month, for 64 tech hubs.

Why salary alone is misleading

Two things sit between a gross salary and your bank account: what the state takes (income tax and social contributions) and what your landlord takes. Both vary enormously from one hub to the next. In the data below, the share of gross pay going to tax and social contributions ranges from 0% (Dubai, Riyadh) to 41.5% (Bucharest). The rent for a one-bedroom apartment in the city centre ranges from $230 a month in Hyderabad to $4,411 in New York.

This is the second half of a pair. In my previous post I asked “can you actually get a work visa there?”. This one asks “is it worth going?”.

How I measured it

I used one persona everywhere: a mid-level software engineer, single, renting a one-bedroom apartment near the city centre.

• Gross salary: the median base salary for Software Engineers on Levels.fyi, accessed on 16 October 2026.
• Net salary: gross salary minus income tax and employee social contributions for a single person, computed with each country’s official 2026 rules.
• Rent: the average one-bedroom city-centre rent on Numbeo.
• Currency: everything converted to USD at the 16 October 2026 spot rate.
• Result: monthly disposable income = net salary / 12 minus rent.

I left out groceries, transport and everything else. Rent is the one cost I could compare consistently across all 64 hubs. This is not a full cost-of-living index.

What I found

Seattle, Austin, Zurich and Silicon Valley lead.

• Seattle: $8,529 a month
• Austin: $8,128
• Zurich: $8,107
• Silicon Valley: $7,912

Twelve hubs leave more than $5,000 a month. Ten are in the US, plus Zurich and Tel Aviv ($5,641). All 11 US hubs sit in the top 13.

Silicon Valley has the highest salary, but is only fourth. The median base pay is $200,000, the highest of the 64 hubs. But 34% of it goes to tax and social contributions, against 26.8% in Seattle and 24.6% in Austin. Rent in San Jose is $3,086, against $2,454 in Seattle and $1,924 in Austin. Silicon Valley’s net monthly salary ($10,998) is almost the same as Seattle’s ($10,983). The rent difference is what pushes it down.

New York is a rent story. A $170,000 salary leaves $9,312 a month after tax, but rent ($4,411) is the highest on the chart. You keep $4,902, 13th.

Dubai vs. London. Dubai charges no income tax, so an $81,699 salary is all net. After rent you keep $4,401 a month. London pays $121,799 gross, but you keep $4,202. Close enough that I would call it a tie, and Dubai’s figure is graded “partial” (see limitations).

Paris vs. London. Paris leaves $2,336 a month, London $4,202, about 1.8 times more. The difference comes from pay, not rent: the London median is $121,799 gross against $68,256 in Paris, while London rent is $2,851 against $1,556.

China vs. Berlin. Beijing ($3,792), Hangzhou ($3,663) and Shanghai ($3,587) all beat Berlin ($3,544). Shenzhen ($3,494) is just below. Berlin pays more gross ($101,005 against $80,787 in Beijing), but 40.3% goes to tax and social contributions and rent is higher ($1,481 against $937 in Beijing).

Singapore, with a note. Singapore’s gross salary is $78,747, but you keep only $2,109 a month, roughly level with Madrid ($2,087). Rent is $2,924, and I deducted the 20% employee CPF contribution, which applies to citizens and permanent residents. An expat on an Employment Pass pays no CPF, so for them the figure would be higher.

The earlier “sweet spot” hubs. Amsterdam ($3,175), Seoul ($3,162), Paris ($2,336) and Tokyo ($1,909) offered the best jobs-versus-visa balance in my previous post. Here they land between $1.9k and $3.2k.

The bottom of the list. Ho Chi Minh City leaves $353 a month, Jakarta $571 and Mumbai $736. Seattle leaves about 24 times more than Ho Chi Minh City, and about 2.8 times the median hub (about $3.1k). One caution: this is in dollars, so it shows how far a salary goes if you spend in dollars, not how well someone lives locally.

Jobs vs. leftover money. Only six hubs combine a tech jobs score of 7+ with more than $5,000 a month: Seattle, Silicon Valley, Austin, Los Angeles, Boston and Tel Aviv. Apart from Zurich (visa difficulty 6) and Tel Aviv (4), every hub above $5,000 is a US hub with a visa difficulty score of 8. The money is where the visa is hardest.

Limitations

• Levels.fyi medians cover all levels. They are self-reported and lean toward big tech and senior roles. I could not isolate “mid-level”. In emerging markets they probably overstate what a typical local engineer earns.
• Health premiums and occupational pensions are excluded everywhere. I included mandatory government social contributions and statutory provident funds, but not private health insurance (mandatory in the Netherlands and Switzerland, employer-provided in the US) or employer and occupational pensions. This probably flatters Zurich and the US most.
• Data quality varies. Of the 64 hubs, 25 are solid, 29 partial and 10 rough. “Solid” means a large Levels.fyi sample, enough Numbeo contributors and a complete tax model. “Partial” means one of those is weaker, such as a country-level salary used for a city or a simplified tax model. The 10 rough estimates are Miami, Copenhagen, Riyadh, Vilnius, Santiago, Bogotá, Cape Town, Bangkok, Nairobi and Lagos. Treat their rank as indicative only. Lagos comes out negative (-$439 a month) on a very small salary sample.
• Miami uses a different source. Levels.fyi has no Miami page, so I used the US Bureau of Labor Statistics median for software developers in the Miami metro area ($132,650). It covers all employers and is not directly comparable.
• Spot rates. I converted at one day’s exchange rate, so a currency move of a few percent can reorder close hubs. Hangzhou and Shenzhen use the China-wide Levels.fyi median.
• Rent is crowd-sourced. Numbeo’s number of contributors differs by city.

The ranking is therefore reliable for large gaps (Seattle vs. Paris, or London vs. Mumbai) and not for small ones (Dubai vs. London, Berlin vs. Shenzhen).

Sources

• Salaries: Levels.fyi, Software Engineer base salary, median, accessed 16 October 2026. Miami: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics, May 2025, software developers, Miami metro area.
• Rent: Numbeo, one-bedroom apartment in city centre, data updated mid-October 2026.
• Exchange rates: ExchangeRate-API (open.er-api.com), 16 October 2026.
• Taxes and social contributions (2026 rules): official authorities, for example the IRS and SSA (US), CRA (Canada), GOV.UK (UK), §32a EStG (Germany), ESTV and the canton of Zurich (Switzerland), Belastingdienst (Netherlands), IRAS and the CPF Board (Singapore), and the national authority of each other country.

The full per-country data, with the exact sources, assumptions and tax parameters for each hub, is available on request: email explore@juliensalinas.com.

If you spot an error, or a rule that changed, tell me and I will update the chart.

Julien Salinas · @juliensalinas · explore@juliensalinas.com · linkedin.com/in/julien-salinas-673b3811

More tech workers are open to moving abroad for work than a few years ago. But “where is the job market good?” and “where can I actually get a visa?” are two different questions. I put both on one chart for 64 tech hubs.

Why this matters

A few data points I checked:

• In the 2025 CERC/Ipsos Global Mobility Survey of 10,574 workers across 19 countries, “almost half (49%) of employees say they would consider relocating internationally for work; a notable increase compared to 2017, when 46% would consider the same” (Ipsos). The sponsor report frames this as relocation “for up to two years” and lists information technology (58%) as the second most open sector, behind utilities (59%) (CERC/WFPMA sponsor report).
• The most famous tech visa is also a lottery. For the US H-1B cap in fiscal year 2026, USCIS counted about 339,000 eligible unique beneficiaries and selected 118,660 of them in the initial selection. (USCIS)
• Rules keep moving. In 2026 alone, Germany raised its EU Blue Card salary thresholds (Make it in Germany), Sweden moved to a 90%-of-median salary requirement (Migrationsverket), Malaysia roughly doubled its Employment Pass salary minimums (ESD Malaysia), and Singapore announced higher Employment Pass salaries from 2027 (MOM). In the US, the $100,000 H-1B payment introduced in 2025 was blocked by the courts in 2026 while the government appeals (USCIS).

So I wanted a single view: where are the jobs, and where is the door actually open?

The findings

The sweet spot (many jobs, easier visa): Amsterdam, Paris, Tokyo, Seoul. All four combine strong tech ecosystems with a clear, employer-sponsored route.

• Amsterdam: the Netherlands’ highly skilled migrant permit, through an IND-recognised employer.
• Paris: France’s “Talent” residence card.
• Tokyo: Japan’s engineer visa and its points-based highly skilled professional track.
• Seoul: Korea’s E-7 professional visa.

None of them uses a lottery.

Close behind: Singapore, Tel Aviv, Dublin, Hong Kong.

• Singapore and Tel Aviv score higher on jobs than any city in the top group, but their visas are a bit harder. Singapore has a salary bar plus a points test. Israel has a high minimum salary for foreign experts.
• Dublin and Hong Kong are among the easiest visas on the chart for a solid job market. Ireland has its Critical Skills permit. Hong Kong’s Top Talent Pass doesn’t even require a job offer.
• Berlin, Toronto-Waterloo and Bengaluru sit just behind.

The US: the most jobs, the hardest visa. Silicon Valley, New York, Boston and Seattle top the jobs axis. They also sit at the bottom of the visa axis. The H-1B is capped and allocated by lottery, now weighted by wage level. On top of that, policy has been unstable: a $100,000 fee was announced, then blocked by courts, and a separate fee has been proposed.

Other patterns:

• London has a deep job market. But the UK Skilled Worker route needs a licensed sponsor and a £41,700 minimum salary, which puts it just outside the sweet spot.
• Zurich and China’s hubs (Beijing, Shanghai, Shenzhen, Hangzhou) are hard. Switzerland has a fixed annual quota for non-EU workers. China uses a points-based permit system with rising salary thresholds.
• Many smaller hubs are easy to enter but have fewer jobs. Examples are Tallinn, Vilnius, Helsinki, Dubai, Kuala Lumpur and Auckland. They’re worth a look if you already have an offer or work remotely.

Methodology

Tech jobs score (1-10). This is based mainly on the Startup Genome Global Startup Ecosystem Report 2025 ranking. I adjusted it with my judgment of each hub’s size and pay level. Hubs outside that ranking were scored on general market size only.

Visa difficulty score (1-10). Each hub starts at 1 and adds points:

• +3 for a capped lottery (US H-1B)
• +2 for a hard national quota, or +1 for a soft or very large quota or an overall cap on temporary workers
• +1 if employer sponsorship is required
• +1 for a labour-market test or local-hire/understudy requirement
• +1 for a salary or points threshold that is high relative to local tech pay
• +1 if typical processing takes over 3 months (+2 if over 6 months or tied to an annual cycle)
• +1 each for: a language barrier in most tech jobs, recent tightening, policy instability or high fees or security concerns, heavy bureaucracy

Scores max out at 10. In practice they range from 2 to 8. The chart’s quadrant line sits at 4.5 on visa difficulty and 5.5 on jobs.

Verification. On 13 October 2026, I checked the main skilled-worker route in each of the 43 countries behind the 64 hubs against official immigration pages.

• 37 countries: I opened and read official pages, fully for 28 and partially for 9.
• Vietnam: checked against the full text of the relevant decree on a legal database.
• 5 countries: the official sites could not be fetched (blocked, bot checks or dead links). These are India, Israel, Australia, Finland and Colombia, and their scores rely on secondary sources.

A few inputs are also flagged as secondary-only. Examples include Korea’s E-7 salary floor, Japan’s processing times and Poland’s Blue Card salary. The full per-country data (exact URLs read and inputs that remain unverified) is available on request: DM me or email explore@juliensalinas.com.

Assumptions:

• The person is a skilled tech worker from outside the EU, with a job offer.
• Nationality matters a lot. EU citizens move freely within the EU, and bilateral deals exist, so your personal result may differ.
• Points were nudged slightly apart on the chart when scores tie.

Caveats

• These are editorial scores, not measurements. The rubric makes my judgment explicit and consistent, but it is still judgment.
• Rules change fast. Several changed during 2026, and the US fee situation could flip again after appeal. Check the official page before you act.
• A visa route existing doesn’t mean you’ll get it. Approval depends on your profile, your employer and the specific role.
• The jobs score leans on startup ecosystems. It under-weights places with large corporate or outsourcing tech employment.
• Cost of living, salaries after tax, language and quality of life aren’t on this chart. They matter as much as the two axes.

If you spot an error or a rule that changed, tell me and I’ll update the chart.

Julien Salinas · @juliensalinas · explore@juliensalinas.com

Limiting API usage based on advanced rate limiting rule is not so easy. In order to achieve this behind the NLP Cloud API, we’re using a combination of Traefik (as a reverse proxy) and local caching within a Go script. When done correctly, you can considerably improve the performance of your rate limiting and properly throttle API requests without sacrificing speed of the requests.

In this example we’re showing how to delegate the rate limiting of every API request to a dedicated microservice thanks to Traefik and Docker. Then in this dedicated microservice, we will count the number of requests recently made in order to authorize or not the new request.

Traefik As A Reverse Proxy

In order to set up an API gateway, Traefik and Docker are a very good combination.

The idea is that all your API requests should be first routed to a Docker container containing a Traefik instance. This Traefik instance acts as a reverse proxy so it will do things like authentication, filtering, retrying, … and eventually routing the user request to the right container.

For example, if you are making a text summarization request on NLP Cloud, you will first go through the API gateway that will take care of authenticating your request and, if successfully authenticated, your request will be routed to a text summarization machine learning model contained in a dedicated Docker container hosted on a specific server.

Both Traefik and Docker are easy to use, and they make your program quite easy to maintain.

Why Use Go?

A rate limiting script will necessarily have to handle a huge volume of concurrent requests.

Go is a good candidate for this type of application as it processes your requests very quickly, and without consuming too much CPU and RAM.

Traefik and Docker were both written in Go, which must not be a coincidence…

A naive implementation would be to use the database to store API usage, count past user requests, and rate limit requests based on that. It will quickly raise performance issues as making a DB request every single time you want to check a request will overwhelm the DB and create tons of unnecessary network accesses. The best solution is to manage that locally in memory. The flip side, of course, it that in-memory counters are not persistent: if you restart your rate limiting application, you will lose all your ongoing counters. It should not be a big deal in theory for a rate limiting application.

Delegating API Rate Limiting To a Dedicated Microservice Thanks To Traefik And Docker

Traefik has many interesting features. One of them is the ability to forward authentication to a dedicated service.

Basically, each incoming API request will first be forwarded to a dedicated service. If this service returns a code 2XX code, then the request is routed to the proper service, otherwise it is rejected.

In the following example, we will use a Docker Compose file for a Docker Swarm cluster. If you’re using another container orchestrator like Kubernetes, Traefik will work very well too.

First, create a Docker Compose file for your API endpoint and enable Traefik:

version: "3.8"

services:
  traefik:
    image: "traefik"
    command:
      - --providers.docker.swarmmode
  api_endpoint:
    image: path_to_api_endpoint_image
    deploy:
      labels:
        - traefik.http.routers.api_endpoint.entrypoints=http
        - traefik.http.services.api_endpoint.loadbalancer.server.port=80
        - traefik.http.routers.api_endpoint.rule=Host(`example.com`) && PathPrefix(`/api-endpoint`)

Then add a new service dedicated to rate limiting and ask Traefik to forward all requests to it (we will code this Go rate limiting service a bit later):

version: "3.8"

services:
  traefik:
    image: traefik
    command:
      - --providers.docker.swarmmode
  api_endpoint:
    image: path_to_your_api_endpoint_image
    deploy:
      labels:
        - traefik.http.routers.api_endpoint.entrypoints=http
        - traefik.http.services.api_endpoint.loadbalancer.server.port=80
        - traefik.http.routers.api_endpoint.rule=Host(`example.com`) && PathPrefix(`/api-endpoint`)
        - traefik.http.middlewares.forward_auth_api_endpoint.forwardauth.address=http://rate_limiting:8080
        - traefik.http.routers.api_endpoint.middlewares=forward_auth_api_endpoint
  rate_limiting:
    image: path_to_your_rate_limiting_image
    deploy:
      labels:
        - traefik.http.routers.rate_limiting.entrypoints=http
        - traefik.http.services.rate_limiting.loadbalancer.server.port=8080

We now have a full Docker Swarm + Traefik configuration that first forwards requests to a rate limiting service before eventually routing the request to the final API endpoint. You can put the above in a production.yml file and start the application with the following command:

docker stack deploy --with-registry-auth -c production.yml application_name

Note that only the headers of the requests are forwarded, not the content of the requests. This is for performance reasons. So if you want to authenticate a request based on the body of this request, you will need to come up with another strategy.

Handling Rate Limiting With Go And Caching

The Traefik and Docker configurations are ready. We now need to code the Go microservice that will take care of rate limiting the requests: users only have the right to 10 requests per minute. Above 10 requests per minute, every request will be rejected with a 429 HTTP code.

package main

import (
  "fmt"
  "time"
  "log"
  "net/http"

  "github.com/gorilla/mux"
  "github.com/patrickmn/go-cache"
)

var c *cache.Cache

// updateUsage increments the API calls in local cache.
func updateUsage(token) {
  // We first try to increment the counter for this user.
  // If there is no existing counter, an error is returned, and in that
  // case we create a new counter with a 3 minute expiry (we don't want
  // old counters to stay in memory forever).
  _, err := c.IncrementInt(fmt.Sprintf("%v/%v", token, time.Now().Minute()), 1)
  if err != nil {
  c.Set(fmt.Sprintf("%v/%v", token, time.Now().Minute()), 1, 3*time.Minute)
  }
}

func RateLimitingHandler(w http.ResponseWriter, r *http.Request) {
  // Retrieve user API token from request headers.
  // Not implemented here for the sake of simplicity.
  apiToken := retrieveAPIToken(r)
  
  var count int

  if x, found := c.Get(fmt.Sprintf("%v/%v", apiToken, time.Now().Minute())); found {
    count = x.(int)
  }

  if count >= 10 {
    w.WriteHeader(http.StatusTooManyRequests)
    return
  }

  updateUsage(apiToken)

  w.WriteHeader(http.StatusOK)
}

func main() {
 r := mux.NewRouter()
 r.HandleFunc("/", RateLimitingHandler)

 log.Println("API is ready and listening on 8080.")

 log.Fatal(http.ListenAndServe(":8080", r))
}

As you can see, we’re using the Gorilla toolkit in order to create a small API, listening on port 8080, that will receive the request forwarded by Traefik.

Once the request is received, we extract the user API token from the request (not implemented here for the sake of simplicity), and check the number of requests made by the user associated with this API token during the last minute.

The request counter is stored in memory thanks to the go-cache library. Go-cache is a minimalist caching library for Go that is very similar to Redis. It automatically handles important things like cache expiry. Storing the API counters in memory is crucial as it is the fastest solution, and we want this code to be as fast as possible in order not to slow down API requests too much.

If the user has made more than 10 requests during the current minute, the request is rejected with a 429 HTTP error code. Traefik will see that this 429 error is not a 2XX code, so it won’t allow the user request to reach the API endpoint, and it will propagate the 429 error to the user.

If the request is not rate limited, we automatically increment the counter for this user.

I recommend that you deploy this Go application within a simple “scratch” container (FROM scratch): it is the lightest way to deploy Go binaries with Docker.

Conclusion

As you can see, implementing a rate limiting gateway for your API is not that hard, thanks to Traefik, Docker and Go.

Of course, rate limiting based on a number of requests per minute is only a first step. You might want to do more advanced things like:

• Rate limiting per minute, per hour, per day, and per month
• Rate limiting per API endpoint
• Have a variable rate limit per user depending on the plan he subscribed to
• Check concurrency

So many interesting things we can’t mention in this article!

If you have questions please don’t hesitate to reach out to me.

Existe aussi en français | También existe en Español