ESP8266 Tricks: The Royal Road to IoT Hacking

The ESP8266 is the Ford model T of the electronics hobbyist: bringing wifi capabilities to the masses at a ridiculously low price. When I first heard about it, it just blew my mind. A wifi-enabled microcontroller for a couple of bucks?! No way!

However, until today, getting started with that chip could get confusing. The web is flooded with contradictory information and low quality tutorials. There are dozens of ways to program the chip, a seemingly infinite range of devices leveraging it, and little guidance to navigate this ocean.

That’s the problem this guide is addressing. It shows what I believe is the royal road to ESP8266 development. I will show you the fastest way to get the most value out of this nifty module, based on my experience and on what worked best for me.

If you already know a bit of programming and just want a comfortable environment to write robust IoT applications for cheap, then my friend, you are golden.

Which module should I buy?

Let’s start with the hardware. The ESP8266 chip itself is pretty useless to the hobbyist: It has timy SMD terminals, no antenna, very little memory, no USB interface, and requires a few external resistors even just to boot.
Therefore, when people talk about the EPS8266, they generally mean a board which harnesses the ESP8266 chip. There is a vast ecosystem of such boards and I will describe a handful of those here, including the one I recommend.

ESP-XX modules

These are the most basic and widespread modules for the ESP8266. Orignially built by a company called Ai-Thinker, they feature the bare minimum hardware you’ll need.

Modules of the ESP-XX family by ai-Thinker. From left to right: ESP-07, ESP-01 and ESP-12F

The ESP-01 is the oldest module in this familly. It features 4x2 male pins for power, boot modes, GPIO and UART (note that despite using 2.5mm pins, you cannot plug it in a breadboard since this would short pairs of pins together). You’ll find plenty of tutorials about this module on the web, but they apply to its successors as well. Therefore, there is no significant reason to prefer this module over its successors.

The ESP-12x series is an evolution of the ESP-01. These module fullfill exactly the same role as the ESP-01, but they have a dramatically different interface. The dip male pins are supplanted by 2mm solder pads, which means you still can’t plug that in your trusty 2.5mm breadboard. They feature an onboard antenna, just like the ESP-01, but are also shielded, which presumably enhances their immunity to RF interference. The most popular nowadays is the ESP-12f which boasts some “improved antenna performance”, whatever that means.

The ESP-07 comes in the same form factor and pinout as the ESP-12f. The major difference we’ll notice is that it doesn’t have an antenna. Instead it features a U.FL socket in which you are supposed to plug an external antenna. This design has a significantly better range while presumably consuming more power (if someone happens to measure that difference, please let me know the results).

Both the ESP-12f and ESP-07 require an additional breakout board to convert the 2mm pins to 2.5mm.

There are more modules in this familly and you can take a look at the list if you are interested. I will not detail how to use them in this article to keep things short. Just know that I advise beginners against using these modules directly. I made the mistake of buying these and lost a few days fixing silly issues that I wouldn’t have faced with a more complete board.

The NodeMCU devkits

The NodeMCU project provides an open source software and hardware solution for working with the ESP8266. Despite being built specifically for the NodeMCU software, the hardware can also be used as a general purpose ESP8266 development kit. We will talk more about the software later.

The original design incorporates an ESP-12E module, a USB-to-UART converter, 5v to 3.3v voltage regulation, a couple push buttons for reset and flash mode, and exports a micro-usb port and 2 rows of 15 DIP male pins. Needless to say it saves a lot of effort working with the chip. Cost-wise it’s between $0.5 and $1 more expensive than a bare ESP-12f. You need all most of the features on the board anyway so don’t try to cut corners and just buy the board already.

At a price tag of just over $3, this is the only board you’ll ever need for ESP8266-based projects.

By virtue of open source hardware, you will find a lot of variations of the design online and will quickly get lost when shopping. While all boards have similar caracteristics, be aware that some models are impractically large and cover all 10 rows of small breadboards. Therefore, I recommend sticking with the one commonly found under the name “v2” (sometimes also “v1”). See the pictures below to know what to look for. You might also want to take a look at this comparison published by the main contributor to the NodeMCU project.

Two common NodeMCU boards found on AliExpress: the one on the left is branded as "v2", and the right "v3". Notice the pin labels on the v3 are horizontal and take up more space, making the board uncomfortably large.

Note: If you are wondering what NodeMCU has to do with Node.js, well… nothing.

Which software can I use to program the ESP8266?

You should now have a better understanding of the hardware landscape surrounding the ESP8266. Well, that was just the beginning. The software side is even more diverse and confusing than that.
I will teach you the basic principles and show a couple of popular solutions. I will then help you run a proof of concept using the solution that I recommend.

No software

As surprising as it may seem, the ESP8266 is actually first meant to be used as a wifi peripheral, and not to be programmed directly!
The default firmware is a simple serial command interpreter, which means that you can do wifi communication just by sending commands from a terminal (or another microcontroller). For instance, this is how you connect to an access point, and make a GET request to http://httpbin.org/ip:

1
2
3
4
5
6
7
8
AT+CIPMUX=0
AT+CWMODE=1
AT+CIPSTART="TCP","httpbin.org",80
AT+CIPSEND=20
GET /ip HTTP/1.0


// Notice that the two line returns above are part of the HTTP request and should be in \r\n form

There is plenty of documentation about this interface on the web; therefore, we are going to keep this section short. In this post we are interested in programming the chip rather than using it as a mere peripheral. Why bother adding a $20 arduino in the setup when the ESP8266 alone can handle everything for under $3?

Low-level C interfaces

This paragraph is key to understanding the diversity of the software ecosystem for the ESP8266.
Espressif systems (the manufacturer of the ESP8266) initially released a software development kit in C that people could use to program the chip. This SDK only had chinese documentation and was rather complex, so for some time the chip remained obscure and unknown to the hacking community. Nowadays, the documentation has been translated and improved, and the community was able to build powerful frameworks on top of the SDK.

There are two components to this low level SDK: the “RTOS” version and the “SDK” version. A detailed explaination about the differences is way beyond the scope of this article, but just know that developers must chose one and stick with it. One allows you to program inside a real-time operating system (RTOS), the other one hands over complete control to the programmer. Anyways, these details are not super relevant to us. Just know that the tools you’ll use are based on one of these frameworks.

It is very unlikely you’ll ever need to use the low-level C libraries. The frameworks built on top of it are much preferred.

Arduino

Of course it’s a thing! Using the C SDK discussed above, a compatibility layer for arduino was developped. It allows one to program applications on the ESP8266 just like on a regular arduino. No matter how attractive this idea is to you, let me warn you: I do not think it is a good idea to use the Arduino framework to program the ESP8266.

The ESP8266 + arduino combo seems very popular (google around and you’ll see). I think this is mainly due to the authority of Arduino and the fact that many in the Arduino community aren’t programmers, rather than the pure technical characteristics of the solution. Gee, some people love arduino so much they’ll even hook one of those to the ESP8266’s UART for applications that could easily fit on the ESP alone!

The Arduino programming model does not work well with the ESP8266’s architecture.
In arduino, you are usually the sole user of the CPU and you do not care much about power consumption. This allows you to write procedural code in a busy loop, which is easy to understand and is what made arduino so successful to beginners.
On the ESP8266 on the contrary, you share CPU with the networking stack. Monopolize the CPU for too long and the network stack will crash. What’s more, by virtue of being wireless capable, ESP8266 applications are more likely to run off the grid, where battery life is critical; exit the busyloop model!

I salute the effort that has been put in making the ESP8266 as compatible as possible with existing arduino APIs but I think that it is the wrong approach to solving the problem. As we will see next, it is much easier for Arduino enthusiasts to learn a framework tailored for the ESP8266, than it is for the ESP8266 to fit in the Arduino mold.

I am not saying you can’t use the Arduino stack. You can. But in my opinion you will be more productive and save time going directly for the NodeMCU stack.

NodeMCU

This is the way I recommend. NodeMCU is an ambitious idea perfectly put in practice by a group of talented programmers. It solves the problems the arduino-based solution faces while providing an extremely user-friendly environment.

NodeMCU is an open source lua-based operating system for the ESP8266. It contains a lua runtime as well as binary modules to interface with the different hardware capabilities of the EPS8266. It is the best tradeoff between user-friendliness and resource efficiency I am aware of. Its programming model is event-based (the API looks very much like that of Node.js), which makes it particularly efficient and suited for this chip. Despite being based on a scripting language (lua), I found it to run more stable than arduino applications written in C!

Some people might be put off by having to learn lua. The fact is you do not need to know lua in order to use NodeMCU. You will mostly be copy-pasting snippets from the doc, and add a couple functions and if statements. The language is so easy to read and use that you can do basic modifications without any prior knowlege of it. Of course biggger modifications will require a bit of googling, but the time you spend learning lua is time you would spend working around limitations of the other solutions! Trust me, this is the way to go.

It’s lab time!

At this point you should be equipped with a NodeMCU board and be convinced that the NodeMCU software is the way to go. If you have a more guerilla hardware setup, make sure you can talk to the board’s uart at the specified baud rate (115200), and that you drive the boot control pins appropriately (see appendix A).

NodeMCU development is a two step process: First you load a firmware into the chip, then you write your application. Everything happens through the UART port of the chip.

As usual, I provide a complete zip archive containing most of the stuff you’ll need for this tutorial, so you can follow along without interruption. Grab it with this link.

Find your COM port

After you’ve plugged your board to your computer, you will want to find the COM port connected to your ESP8266 chip. One easy way is to go into the arduino IDE, and look at the available options in Tools > Port. Otherwise on OSX, running ls /dev/cu.* will show the list of potential candidates. Just try each of them until you find the right one.
Let me know if you have a method that does not involve the arduino IDE on other platforms so I can add it here.

Finding the COM port using the arduino IDE.

Select one at random, open the serial monitor (the magnifying glass icon on the top right). Make sure you select “115200 baud” and “both NL & CR” in the dropdown at the bottom of the screen. Then depending on what is currently flashed on the board, try either of these two commands: print(node.info()) or else just AT. If the first one returns some numbers, then it means that the chip has already been flashed with a NodeMCU firmware (we are still going to replace it with our own). If the second command works, it means that the chip contains the default firmware by espressif (as discussed in the first section).

If none of them works, try another serial port. It could also be that the chip currently contains garbage and is not able to run anything, or that you booted in “flash mode”. If you are running a bare ESP-XX module, make sure to drive the pins in “boot mode” as shown in appendix A.

Running a command to check that we are talking to an ESP module.

Now that you know which com port talks to the ESP8266, close the serial console, go back to your terminal and save that value to a variable called PORT. For instance, my port is /dev/cu.usbmodem12341 so I type:

1
export PORT="/dev/cu.usbmodem12341"

Initial setup

There is one major drawback with NodeMCU: The complete image with all existing modules is too big to fit on the chip! Therefore, you need to select which modules you need for a given application and create a custom image with only these modules.
Fortunately, there are shortcuts:

  • First, a cloud build service lets you generate custom images online. Just check the boxes you need, hit “Start your build” and within a few minutes you’ll get your image sent in your mail.
  • Alternatively, you can grab a copy of the firmware containing a collection of the most essential modules curated by yours truly. You can download it here.

For this demo, just use the image I provide (second bullet above).

We will flash the image using a handy python script named esptool.py.

All flashing operations happen in “flash” mode. The commands will hang if the board is in “boot” mode. Refer to appendix A to learn more about boot modes.

First, erase the flash memory of the device by running:

1
python esptool.py --port $PORT erase_flash

Now, reset your ESP8266 in flash mode again, and proceed to flash the firmware:
(Note that at this point the chip will tend to reboot automatically in “bootloader mode”. It might take some patience to get it to reboot in flash mode. I found that I sometimes need to disconnect the chip from the power source, and hold the “flash” button down while I connect it back to power).

1
esptool.py --port $PORT write_flash -fm dio 0x00000 nodemcu-essentials-2018-05-02-float.bin

Tadaa! You can now reboot your board one more time, in “boot” mode (the normal mode of operation). It is ready to use!

Application programming

Flashing the firmware was the hardest step involved in the process. The rest is where the fun begins.

The way we interact with a NodeMCU installation is through a lua terminal running on its serial port. Think of it like a remote shell for linux, except the shell is in lua.
A tool called nodemcu-uploader will help us upload files to the board among other things. Install it or just clone the repository somewhere. Check that you can run it with nodemcu-uploader --version (note: if you cloned the repo, you might need to run python /path/to/nodemcu-uploader.py --version instead).

Exploring the interactive prompt

Let’s open an interactive terminal with the chip to get a sense of how it works. Make sure you still have the PORT variable around, then run nodemcu-uploader --port $PORT terminal.

Try the following command:

1
print('Hello world')

The device should respond by printing the string ‘Hello world’. Awesome! Now how about we do a little bit of wifi?

Put your chip in station mode with:

1
wifi.setmode(wifi.STATION)

Then enter this to get a list of available wifi networks (it takes a few seconds to load).

1
2
3
4
5
wifi.sta.getap(1, function (t)
for k,v in pairs(t) do
print(k.." : "..v)
end
end)

Can you see your network? Good, then configure the chip with the following command:

1
wifi.sta.config({ssid = 'YOUR_WIFI_NETWORK_NAME', pwd = 'YOUR_WIFI_PASSWORD'})

Wait for a dozen seconds to give it time to connect, and then confirm that you are online using the following command:

1
print(wifi.sta.getip())

If this shows nil, it means that the NodeMCU did not succesfully connect. Check your wifi password or give it some more time to complete the connection.

Now that we are connected to the internet, let’s make some HTTP requests!

1
2
3
4
5
6
7
http.get("https://istheinternetdown.com/", nil, function(code, data)
if (code < 0) then
print("HTTP request failed")
else
print(code, data)
end
end)

You should get a response telling you that the internet is not down (hopefully) and return your public IP address.

You can now quit the interactive session. (Note that this terminal’s retarded exit sequence is “CTRL+]”. If you are using a non-US keyboard try CTRL + whatever key is next to the return key)

Uploading a real program

Now that you have a sense of how the NodeMCU terminal interface works, let’s write a real program and upload it to the board.

First, create a file called env.lua. This file will contain your wifi credentials (this is a common trick used to keep credentials out of source control).

1
2
3
-- env.lua
WIFI_SSID="YOUR_NETWORK_NAME"
WIFI_PASSWORD="YOUR_NETWORK_PASSWORD"

Then copy the following program into a file called init.lua

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
-- init.lua
-- Loads the WIFI_SSID and WIFI_PASSWORD secret variables
dofile('env.lua')

-- Connect to the wifi network
wifi.setmode(wifi.STATION)
wifi.sta.config({ssid = WIFI_SSID, pwd = WIFI_PASSWORD})


-- index of the gpio connected to the onboard LED
LED_PIN=4

gpio.mode(LED_PIN, gpio.OUTPUT)
gpio.write(LED_PIN, 1)

-- This will print IP informations upon successful connection
wifi.eventmon.register(wifi.eventmon.STA_GOT_IP, function(T)
print('\n\tSTA - GOT IP'..'\n\tStation IP: '..T.IP..'\n\tSubnet mask: '..
T.netmask..'\n\tGateway IP: '..T.gateway)
end)

-- Create a TCP server
sv = net.createServer(net.TCP, 30)

headers = 'Content-Type: text/html; charset=UTF-8\r\n'

-- Configure the server to print the data it gets and respond
-- with a greeting message
if sv then
sv:listen(80, function(conn)
conn:on('receive', receiver)
end)
end

function receiver(sck, data)
print(data)
firstLine = string.sub(data, 0, string.find(data, '\n'))
message = ''
-- note: in practice you'd never perform an action in response to a 'GET' request,
-- this could cause all sorts of troubles. We do it here to keep the demo code brief.
if string.match(firstLine, 'GET /on ') then
message = '<pre>turned on!</pre>'
-- Note that the led shows the inverse of the logic state of the gpio
gpio.write(LED_PIN, 0)
elseif string.match(firstLine, 'GET /off ') then
message = '<pre>turned off!</pre>'
gpio.write(LED_PIN, 1)
end

if string.match(firstLine, 'GET /') or message ~= '' then
-- Serve the homepage
sck:send('HTTP/1.1 200 OK\r\n'.. headers .. '\r\n'..
'<h1>Hello from ESP8266!</h1>' ..
message ..
'<a href="/on">Turn light on</a> - <a href="/off">Turn light off</a>')
else
sck:send('HTTP/1.1 404 NOT FOUND\r\n'.. headers .. '\r\n'..
'Not found.<br/><a href="/">Go back home</a>')
end
sck:on('sent', function () sck:close() end)
end

Finally, upload both files to the board using nodemcu-uploader:

1
nodemcu-uploader --port $PORT upload env.lua init.lua

When you reboot your board, it will look for the file called “init.lua” and run that program. It is useful for debugging purposes to connect to the terminal with nodemcu-uploader --port $PORT terminal while your board is booting. This shows a bunch of info as well as any print statement you put in your code.

Here is what I got:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
--- Miniterm on /dev/cu.usbmodem12341  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
[some redacted garbage characters. This is a normal behavior of the ESP8266]

NodeMCU custom build by frightanic.com
branch: master
commit: 8181c3be7aed9f0a0ceb73ac8137c1a519e8a8e9
SSL: true
modules: cron,enduser_setup,file,gpio,http,i2c,mdns,mqtt,net,node,pwm,rtcfifo,rtcmem,rtctime,sjson,sntp,tmr,uart,wifi,tls
build created on 2018-07-21 18:02
powered by Lua 5.1.4 on SDK 2.2.1(cfd48f3)
>
STA - GOT IP
Station IP: 192.168.0.22
Subnet mask: 255.255.255.0
Gateway IP: 192.168.0.1

Take note of the “Station IP” you get and navigate there with your web browser. In my case for instance, I have to navigate to http://192.168.0.22.

Excuse messy breadboard. None of it would have been necessary had I simply bought the NodeMCU devkit. Hopefully I made this mistake so you won’t have to.

This test program lets you control the onboard LED through an HTML interface. Note that I tested this with the ESP-07. It should also work with the ESP-12 as well as any ESP-XX based devkit, as long as the onboard LED is connected to GPIO2. Other hardware may have the LED connected to a different pin or not have an LED at all.

Let’s get hacking!

That’s it for this tutorial. You now know the basics of the ESP8266 and have a complete development setup that allows you to write any IoT application you can imagine! Here are some additional resources and tips before you leave:

  • The nodemcu documentation is your ultimate resource to learn all of the NodeMCU APIs and more. NodeMCU ships a lot of useful modules and I recommend you spend some time in the doc to see what’s available.
  • HTTP support on the chip is elementary. I advise to use MQTT instead wherever possible. Although it is possible to run an HTTP application server on the ESP8266, that is a complete abuse of the chip and should never be attempted in an application you intend to deploy.
  • The archive provided with this tutorial also contains a Makefile with helpful tasks for NodeMCU development.

I still can not believe the amount of features that come with the ESP8266 at such a low price tag. The NodeMCU stack is icing on the cake and makes the programming of such projects a complete blast! I feel inspired to build lots of IoT stuff and I hope you do too.
I invite you to check out the feeder I built for my betta fish. The ESP8266 was a profitable replacement of the arduino, it enhanced power outage tolerance and logging capabilities while also making it possible to manually trigger the device from anywhere in the world with the push of a button!

And you, what do you want to build with your ESP8266? I’d love to hear your ideas! Please get in touch or share your thoughts in the comment section.


Appendix

Appendix A: Boot mode selection

The following table shows how to select the mode into which the ESP8266 boots. Note that in all cases, GPIO15 must be pulled down and GPIO2 must be pulled high. Some boards, such as the NodeMCU, take care of this for you. Otherwise, use 10k resistors to drive the pins.
More info on boot modes here.

Mode NodeMCU devkit ESP-XX or others
boot press rst Pull GPIO0 down with a resistor
flash hold flash and press rst Pull GPIO0 up with a resistor

How not to suck at GitHub

So you learned a bit of programming and you know how to use git. You are ready to materialize your thoughts through the mighty power of code, and make your art available to the whole world using the most popular open source social network on earth.

But you fear that you will make stupid mistakes and ruin your chance to become the next tech influencer. Rest assured, after reading this post you will know everything you need to know to make the most of Octocat’s home.

I will give you 10 tips to make it to the top faster than Pemba Dorje on oxygen!

1. Don’t look for alternatives

If you think of a novel utility, library or a plug-in, you need to start coding your own solution straight away.

You will only lose time looking for alternatives. You will find that prior art does not quite fit your requirements. Of course, you can not bend your problem definition, even sligthly, so that existing solutions apply. Moreover, during this process you may taint your original idea by looking at too many similar concepts, which will invariably worsen your design.

It would also be a huge waste of time trying to add a feature to an existing project. The legacy codebase of established projects is massive and messy. Just learning the relevant bits would take as much time as coding the whole solution from scratch anyway. And I’m not even talking about the near-infinite number of hours of work lost in feature requests which end up being rejected by the project owner after 3 months of silence.

2. Google and StackOverflow won’t help you

Don’t waste your precious time googling for a solution. Most of the answers on StackOverflow are garbage anyway, and the library you are working with (and which is the root cause of your current issue of course) is not that popular. Hence, chances of finding the exact solution to your problem online are minimal.

Instead, you better bother that project maintainer whose lack of QA considerations is causing your troubles. The project maintainer has the mental map of his project engraved in his memory at all times and can therefore figure out the solution to any problem near instantly. Plus, I suspect projet maintainers keep a secret list of known issues which they do not fix out of pure laziness, and that they actually wait until someone notices it to fix it.

3. Don’t try to fix it yourself

This tip is similar to number 2. If a project you are using doesn’t work, it’s probably the maintainer’s fault. Don’t even bother looking for an answer in the project wiki or investigating the bug yourself. Just go ahead and nag the project maintainer. He or she will probably be able to fix their buggy project just for you.
Don’t forget to check back each day and notify the maintainer if he forgot to respond to your last message.

4. Be as vague as possible

When you open an issue for a bug you’ve encountered (which should be about 2 minutes after you’ve noticed said bug if you follow the advice above), please do yourself a favor and describe your issue in as vague terms as possible. Don’t try to assess the cause of your problem. This way you avoid making a bad diagnosis and mislead the maintainer. Also, don’t post a snippet of your own code, system stats or installed libraries as it may leak sensitive information out to the public (remember hackers are constantly scraping GitHub for sensitive information!).

5. Don’t bother with the markup

You’ve heard about markdown but for your own sake, you couldn’t remember how to format code. Is it a <code> tag? Or maybe [code]? Or is it a back tick? Damn it!

Don’t worry, no one does. Just forget about formatting your issue altogether. Paste large blobs of code inline with your prose like you’re writing lyrics for the next trap banger. After all, if the maintainer is not happy with your format, he can copy and paste it to a text editor.
As a bonus, you don’t even need to indent your code when you proceed this way!

6. Refrain from using reactions

It is outrageous that the GitHub team spent so much effort mimicking facebook with its stupid “reactions”. Ugh… Anyway, when you feel like you agree (or even disagree) with a comment, be respectful and post an additional comment to voice your opinion while adding absolutely no useful piece of information to the debate. And please, please, please, leave that stupid “thumbs up” button for social-network addicted kids.

7. DO NOT, ever, submit a pull request

Everyone knows project maintainers hate it when someone tries to touch their code. After all, why are they constantly finding excuses to reject pull requests?
Also, in order to preserve your ever so precious time, and avoid a painful public shaming, do not attempt to contribute code to an open source project.

8. If you ever should submit code, don’t ask!

Let’s say there is this extremely useful feature you need, and the project seems a bit inactive lately. You really crave that feature and you can’t wait for Mr. slowpoke to code it himself. Despite point 1, you do not have time to write new software from scratch. You realize that it will be a substantial change which will require quite a bit of code, but nothing can discourage you. You need that feature.

Then make sure to secretly develop the feature without letting the maintainer know; otherwise, he or she might try to cut the grass under your feet. Don’t let anyone know you are working on this awesome addition until the very last moment when you submit your huge pull request as a single, solid, code change. Wow factor guaranteed!

9. Do not answer other people’s questions

As you contribute to repositories, you will notice other people try to drown your hard work under a deluge of irrelevant issues and comments. Don’t think helping them will make them go away; they will invariably come back with more.

Instead, keep a cool head and refrain from helping. If needed, “up” your own issue to make sure it stays on top.

10. Use issues as a general purpose Q&A place

As we’ve covered in point number 2, you will not find answers on StackOverflow. What’s more, its elevator system tends to reorder comments in an annoying way which breaks chronological order.

Therefore, if you have anything to say which is even remotely connected to a project, then go ahead and express yourself using the issues section. This will notify everyone who watches the repository and guarantee maximum exposure of your comment.


Hope you liked these tips. Please feel free to contribute your own in the comments!

How to manage and publish a multi-package TypeScript project

In the previous part, we’ve seen a few tricks which can help you split your TypeScript project into small packages. We were still missing a proper build system as well as a way to publish or deploy the packages. Which is what we will cover in this chapter.

To follow along

As in the previous part, I have created a simple repository which you can use to follow along this tutorial. This time however, you must first run a script to customize the project before you can start using the repository. The reason is that at the end of this tutorial, you will be able to publish your project to npm. In order to make this work for everyone, we have to rename the packages with a prefix which is unique to each reader of this tutorial. Don’t worry, it only takes a second:

  1. Download the archive for part 2, and unzip it somewhere.
  2. If you haven’t already, sign up to npm and log in using the command line. Or, if you don’t care about publishing to npm, go straight to 4.
  3. In the project, run ./customize.sh. If it worked, you can skip step 4.
  4. Otherwise, run the script like so: ./customize.sh username, where username is any username you picked. Beware that you will not be able to publish the packages.

Alternatively, if you do not wish to download the files, you can read the diff on github.

Build system

You may have noticed a new file called Makefile at the top of the project. Indeed, we use GNU Make to keep track of dependencies between packages and to define build tasks, and this file is where we tell Make what to do. If you do happen to know a better alternative than Make, please express yourself in the comments.

Make is installed by default on any decent operating system. If you are using windows, you can easily find instructions online to install it.

The top-level Makefile defines three things:

  • global tasks
  • dependencies
  • utility tasks

Global tasks

Global tasks are tasks which must run in all packages. They are defined in this line:

1
TASKS :=build clean test

For instance, the build task will build all packages at once. To run it, type:

1
make build

You will notice that this command also installs npm dependencies, and that it builds the packages in the correct order! Make did not magically figure this out, we instructed it to do so. Keep reading to find out how.

Dependencies

Dependency declaration is the easiest and most useful thing with Make. Simply write the path to the dependent, followed by a colon, followed by a space-separated list of dependencies.
For instance, the dependencies between our packages are defined like so:

1
2
3
packages/tstuto-api:
packages/tstuto-server: packages/tstuto-api
packages/tstuto-web-client: packages/tstuto-api

This is how Make knows in which order to build our packages. But how does it know to install npm dependencies first?

Open the Makefile in tstuto-web-client, which looks something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
BIN=public/dist/main.js
SRC =$(shell find src/ -type f -name '*.ts')
NODE_MODULES=node_modules/.makets
NPM_TASKS=test clean

.PHONY: build
build: $(BIN)

$(NODE_MODULES): package.json package-lock.json
npm install
touch $(NODE_MODULES)

$(BIN): $(NODE_MODULES) $(SRC)
npm run build

.PHONY: $(NPM_TASKS)
$(NPM_TASKS): $(NODE_MODULES)
npm run $@

The BIN variable contains the path to the output file of our package (in this case, it is the client-side javascript bundle).
We create a special file inside the node_modules directory, called .makets (which stands for make TimeStamp). This file records the last time that make ran npm install and helps it figure out if it should run the command again (that is, when either package.json or package-lock.json has changed).

Some tasks are defined in the script property of the package.json and do not need any additional logic in the Makefile. We define those in the NPM_TASKS variable which is used to run the npm script of the same name.

Utility tasks

Let’s go back to the top-level Makefile quickly to talk about utility tasks.

You might encounter repetitive tasks which are specific to one package and cannot be generalized to all packages. For instance, we want a task to start our development server. Such tasks can be defined individually in the top-level Makefile like so:

1
2
3
4
.PHONY: serve
serve:
$(MAKE) build
$(MAKE) -C packages/tstuto-server serve

Here we simply create a proxy-task which delegates to the makefile located in packages/tstuto-server.


That is about it for the build system. If you are familiar with Make, nothing should have surprised you in the above (except maybe the way we declare the dependencies for npm install). Otherwise, the syntax might look daunting at first, but you’ll quickly get used to it.

Without further ado, let’s see how we solve the last remaining problem: publishing.

Publishing/Deployment

The setup we have right now is great for development, but you may be wondering: “How do I deploy this to production?” or “How do I publish this as npm packages?”. Fear not, the answer lies right below!

You could of course clone your whole repository to your production environment and run make serve. That would work but it would also be quite unprofessional to proceed this way.

A more idiomatic way to proceed is to publish your packages to an artifact repository. Open source projects usually rely on npm’s public repository while proprietary software editors have their own private artifact servers. Luckily for us, this means that no matter what you are currently trying to do, whether it’s an open or closed source, whether it’s a library, a microservice or a CLI tool, the process to publish it is exactly the same!

Let’s recap what we want to do before digging into the details: We want to take all of our packages, give them appropriate version numbers and publish them using npm.
Oh, and one more detail: When they get published, our packages need to declare their dependencies to sibling packages in our project (because within the artifact repository, each package stands alone).

All the magic happens inside tools/publish.ts. This script does the following:

  • Determines the next version number
  • Changes all package.json files of all packages to set the correct version number
  • Adds the missing dependencies to each package.json
  • Performs a npm publish
  • Rolls back the changes made in the package.json files

I will not dive into the details of this script and I would not advise you to reuse it as-is for your own projects. Instead, you should try it out and understand how it works so you can apply this knowledge to your specific use-case.

We call this script from the Makefile. Run it with the following command:

1
make publish

If you are currently logged into npm, this will effectively publish the demo package to @username/tstuto-xxx.

You can now test your newly-deployed package by running:

1
npm install -g @username/tstuto-server

(Make sure to replace username with your actual npm username. You may need to run this command with sudo).

And then, start your server by running:

1
my-awesome-app

This should start the application.


Well done! If you made it so far, you now know the key elements to manage a multi-package TypeScript project, develop it and publish it (or ship it for production).

Bonus

Make can run multiple tasks in parallel in a way that is consistent with the dependencies declared in the makefile. For instance, to run up to 4 tasks in parallel, run make -j4 build. This can help you speed up builds when you have many packages and a flat dependency tree.