Tips and tricks to structure multi-package TypeScript projects

I’ve been experimenting with ways to structure a complex TypeScript project lately while working on Vaultage and I have finally found a solution that provides proper isolation and consistency across packages.

Java developers should be familiar with having dozens of sub-projects open simultaneously in their IDE, each with its own build target and dependencies.

In comparison, JavaScript projects tend to be a mess because of the lack of an idiomatic way to split code across packages. Tools like lerna help you create a proper JavaScript monorepo and rationalize the development of a complex project. However these tools pack a lot of features which take some time to properly master. The added complexity may result in misunderstanding and in the end the time gained by deploying the tool may be lost in obscure debugging sessions. Additionally, TypeScript is a different beast and requires more work to integrate properly.

In this tutorial, you will learn the basic principles behind a multi-package TypeScript project so you can apply them in your own work.

I created a minimalist project skeleton so you can follow along this tutorial. You can download it here.

The setup

Download and unpack the tutorial files into your workspace. You should end up with a project containing a packages sub-folder with three packages inside.
Each package is an independent unit of code, with its own build target and test suite:

  • The tstuto-server package contains the NodeJS server files
  • The tstuto-web-client package contains the web application which talks to the server
  • The tstuto-api package contains shared definitions which the client and server will use to communicate in a type-safe way.

As you may have guessed, our example application consists of a simple web server and a web application which talk over HTTP.

Go ahead and open the top-level folder in your favorite TypeScript IDE (it should be VSCode, if it’s not, then go ahead and download it now, I’ll wait…).

First, you’ll want to check that everything works as intended. Navigate to packages/tstuto-web-client and run:

1
2
npm install
npm run build

Then repeat this step for the tstuto-server package.

When you are done, you should have successfully built the demo application. Navigate to the tstuto-server package and run npm start to launch the server. Then, point your web browser at http://localhost:3000. You should see an ugly web page with a button.

Using a shared package

Take a look at the files packages/tstuto-server/src/controllers/MoodController.ts, and packages/tstuto-web-client/src/main-client.ts.

The client uses the axios library to fetch a mood from the server over HTTP. If you are a type safety freak, something should tickle your senses here: the communication is not safe. Indeed, look at the type returned by the axios call in main-client.ts: you’ll find that it is of the any type. You can use this object however you want and the TypeScript compiler will never complain, even though your code might crash at run-time!

An untyped response lets you type anything and provides no intellisense

Enter the tstuto-api package. Take a look at packages/tstuto-api/src/index.ts, we define a type and two factory functions there. Compile them by navigating to packages/tstuto-api and running npm install && npm run build.

Now, going back to MoodController.ts, replace the function by the following, which uses the factory methods instead of the inline object definitions:

1
2
3
4
5
6
7
8
9
10
11
12
13
import { happyMood, sadMood } from '../../../tstuto-api/src/index';
import * as express from 'express';

const HAPPY_THRESHOLD = 0.3;

export function MoodController(_req: express.Request, res: express.Response) {
const indicator = Math.random();
if (indicator > HAPPY_THRESHOLD) {
res.json(happyMood(indicator));
} else {
res.json(sadMood(indicator));
}
}

In main-client.ts, use the interface to type the object returned by axios:

1
2
3
4
5
6
import { IMoodAPIResponse } from '../../tstuto-api/src/index';

/* ... */

// line 23:
const mood = await axios.get<IMoodAPIResponse>('/api/mood');

That is way better, our API is now type-safe. The TypeScript compiler catches typos, and we have proper auto-completion and code refactoring!

A typed response provides completion and catches typos

However, written as is, our import statement actually instructs the TypeScript compiler to compile the target module along with ours. This has multiple nasty side effects and defeats the point of having separate modules altogether.

It would be much cleaner if we could just write:

1
import { IMoodAPIResponse } from 'tstuto-api';

We will see how we can achieve this in the next section.

Proper code sharing

So far, we’ve split our code into three modules and used import statements to borrow code from one module into another. However, we would like to isolate the modules further and import the built artifact rather than importing the raw source code. This means that we want the TypeScript compiler to use the type definitions emitted during compilation and we want node (or webpack for the client) to import the generated JavaScript file rather than the source TypeScript. This helps us avoid bugs spreading across modules and prevents careless developers from producing spaghetti code (to some extent…). It will also speed up your builds because tsc won’t have to compile the same source files over and over.

Go ahead and replace the two imports looking like import xxx from '../../api/src/index' in main-client.ts and MoodController.ts with just import xxx from 'api':

1
2
3
4
5
// main-client.ts:
import { IMoodAPIResponse } from 'tstuto-api';

// MoodController.ts:
import { happyMood, sadMood } from 'tstuto-api';

Don’t worry about the compiler error. All we need to do to make it disappear is instruct the TypeScript compiler to look for our custom packages in the packages folder. Fortunately, there is an option called baseUrl which allows us to do just that.

Edit tsconfig.json at the root of the project and uncomment the line "baseUrl": "packages". This way the TypeScript compiler also looks at the packages directory to resolve package names.

Note 1: Your packages may conflict with packages installed in node_modules; this is why we prefixed all our packages with tstuto-: to make sure that we don’t accidentally shadow an actual npm package.

Note 2: You may need to reload your editor after you changed tsconfig.json. In VSCode, open the command palette (CTRL+SHIFT+P) and chose “reload window”.

There is one more thing we need to do in order for this to work. TypeScript will not recognize your custom module unless you specified the type property in its package.json. Open packages/tstuto-api/package.json. You will see a line with the text “TODO”. Replace it with the following:

1
"types": "dist/index.d.ts"

You want to make extra sure that you got this setting right. If there is an error here, nobody will let you know, your imports might resolve to any and you won’t notice your mistake until it’s too late. The types property in package.json must point to the type definition of your entry point!

Now if you go back to packages/tstuto-server and run npm run build, you should get a successful build. However, if you try to start the server with npm run start, it will fail. Why? Although the TS compiler has figured out your project structure, Node.js is still oblivious to it: it doesn’t know where to find your custom modules at run time!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ npm start

> tstuto-server@0.0.0 start /workspace/ts-project-seed/packages/tstuto-server
> node bin/server.js

module.js:540
throw err;
^

Error: Cannot find module 'tstuto-api'
at Function.Module._resolveFilename (module.js:538:15)
at Function.Module._load (module.js:468:25)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/workspace/ts-project-seed/packages/tstuto-server/dist/src/controllers/MoodController.js:3:20)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)

The next trick we will use is called NODE_PATH.

NODE_PATH is an environment variable node uses for pretty much the same purpose TypeScript uses “baseUrl”: it looks for additional node_modules inside the directory specified by NODE_PATH. The problem is that hacks based on environment variables tend to work poorly cross-platform. That’s why we will use cross-env, a nifty node module that lets you define environment variables in a portable way.

In packages/tstuto-server/package.json, replace the line "start": "node bin/server.js", with "start": "cross-env NODE_PATH=.. node bin/server.js",.

Now, when you run npm start, node will also look at the parent directory when resolving modules. This is how it will know that tstuto-api refers to your custom module in packages/tstuto-api.

Your application should now work properly!

Next steps

You have learned the fundamental tricks which will allow you to structure a multi-package typescript project. There are things left to fix though.

  1. It is tedious to go into each sub-project and manually run npm run build each time we change something. In addition, if we add more modules, manually tracking dependencies can quickly become a nightmare. That’s why we need a build and dependency tracking system to handle all of that for us.
  2. We would like to export our project, either to be distributed as an npm module or to be deployed somewhere. We can not ship our packages as separate npm packages just like that because they now depend on the directory structure of the repository.

See you in the next part of this tutorial, where we discuss those issues.


Appendix: How did you serve the client files again?

You may have noticed that our server also takes care to serve the client (as static files). While there are scenarios where you will want to ship the client separately, serving it from the API server is quite handy for development and suits a broad range of practical use-cases.

The trick fits into these three lines of code:

1
2
3
4
// Bind static content to server
const pathToWebUI = path.dirname(require.resolve('../../../tstuto-web-client'));
const staticDirToServer = path.join(pathToWebUI, 'public');
server.use(express.static(staticDirToServer));

We get the absolute path to the tstuto-web-client module and concatenate the public directory to it; we then instruct express to serve this folder as static content. Doing it this way allows us to keep the server and client completely separated and avoid any copy which would make our build system much more complex.

You should use the module name 'tstuto-web-client' instead of the relative path '../../../tstuto-web-client' now that you have learned the NODE_PATH trick. The reason the tutorial files ship with the relative path is to make it work as-is (even if you don’t set NODE_PATH), but this will break when you’ll try to deploy the app in the next part.


Thanks Ludovic for proofreading this tutorial.

Deal with nullables like they're not even here - Good coding practices in TypeScript.

Today we will take a look at a couple ways to improve your TypeScript code. By applying those techniques we will get code that is more readable, contains less bugs and is objectively higher-level without any downside. How cool is that ?

An example to get started

I like to use examples to illustrate what I’m talking about, so let’s start with the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

class User {
constructor(
public name: string) {
}

/** A user may want to fetch her introduction sentence from an async source so
* we use a callback to handle the asynchronous execution flow.
*/
public introduce(cb: (msg: string) => void): void {
setImmediate(() => {
cb(`Hello, my name is ${this.name}`);
});
}
}

class App {
user?: User;

public login(user: User): void {
this.user = user;
}

public renameUser(newName: string): void {
this.user.name = newName;
}

public introduceUser(cb: (err: string | null, msg?: string) => void): void {
this.user.introduce((msg) => {
cb(null, msg);
});
}
}

const app = new App();
app.introduceUser((err, res) => {
if (err) return console.error(err);
console.log(res);
});
app.login(new User('John'));
app.introduceUser((err, res) => {
if (err) return console.error(err);
console.log(res);
});

This useless piece of code simply illustrates the case where we have a class (Here App containing a member which may be null (user).

I already hear functional programming geeks yelling at me, arguing that mutable values are bad and nullable ones are even worse.
The thing is, in practice there will always be a piece of stateful code with a nullable in there, and although you can avoid it with FP trickery, I want to show you that we can achieve the same amount of protection without sacrificing the ease and comfort we get out of nullables.

OK, let’s get back to business, the above piece of code is bad for an obvious reason. Since user may be undefined, calling introduceUser may result in an error.

strictNullChecks

The first step is to turn on the strictNullChecks compiler option in the project’s tsconfig.json.
What this flag does is that it considers the null and undefined types as completely independent types and thus prevents accidental casting and/or calling methods on variables with those types.
It makes sense that this flag is off by default because TypeScript needs to be as close to JavaScript as possible to ease the learning curve. However, using TypeScript in production without –strictNullChecks is insane. Anytime a nullable value is used without any check is an error.

Now, with the –strictNullChecks flag enabled, our code above won’t compile anymore, so we know we need to handle the case where user is null. The obvious way to do it is to add simple null checks.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class App {

// ...

public renameUser(newName: string): void {
if (this.user == null) {
throw new Error('No user is logged in!');
}
this.user.name = newName;
}

public introduceUser(cb: (err: string | null, msg?: string) => void): void {
if (this.user == null) {
return cb('No user is logged in!');
}
this.user.introduce((msg) => {
cb(null, msg);
});
}
}

Note that since the method introduceUser is using a nodeJS-style asynchronous control flow, we need to call the callback with the error as the first parameter.

The TypeScript compiler analyses the control flow and sees that, because of the newly added null check, the references to the user member are safe.

This works in practice but has two downsides:
1. As your class grows, the number of null checks grows linearly. Each additional branch makes each method more complex and less readable.
2. We need to duplicate the error message which goes against the DRY principle. (even if we factor out the string itself, the string identifier will have to be duplicated anyway.)

Using getters

Who said getters were only good for public interfaces? In this scenario we can actually use a getter to reduce the number of branches in our class and make the code more readable.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class App {

// ...

private getUser(): User {
if (!this.user) {
throw new Error('No user is logged in!');
}
return this.user;
}

public renameUser(newName: string): void {
this.getUser().name = newName;
}

public introduceUser(cb: (err: string | null, msg?: string) => void): void {
this.getUser().introduce((msg) => {
cb(null, msg);
});
}

The getter allows us to factor-out the null-check and our public methods are readable again. If calling getUser() each time looks bad to your taste and you target ES5 or higher, you can actually rewrite the App class using an ES5 getter like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
class App {
_user?: User;

public login(user: User): void {
this._user = user;
}

private get user(): User {
if (!this._user) {
throw new Error('No user is logged in!');
}
return this._user;
}

public renameUser(newName: string): void {
this.user.name = newName;
}

public introduceUser(cb: (err: string | null, msg?: string) => void): void {
this.user.introduce((msg) => {
cb(null, msg);
});
}
}

The most attentive reader might notice that we are back to square one: If the API consumer calls app.renameUser() or app.introduceUser() before logging in, they will get an exception! Even worse, our asynchronous control flow is broken because the callback will never be invoked with an error.

We’ll talk about the asynchronous case in a minute, but first I need to mention one huge advantage this solution has over the initial version: The “unhappy” path is managed.

Before, we relied on the JS engine to crash when the user variable was undefined. The API consumer would get something like “undefined has no member ‘name’”. Any JS dev knows how annoying it is to dig through dozens of stack trace frames to find out what was undefined and why. This kind of error is a legit software bug.
On the other hand, what we got now is an error that we are aware of. We know it may happen, we know why and we know what to do if it does happen. We can even show the error to the end user in the UI because “You are not authenticated” sounds better to end users than “Undefined is not a function”!!!

This is the reason why –strictNullChecks is so important, it helps you, the programmer, avoid bugs. Your users might still get an error but you expect it and you can document it.

Making it airtight

There is one last issue with the current solution: the asynchronous workflow. Indeed, while languages like Java have compile-time checks to prevent you from throwing exceptions where you shouldn’t, TypeScript doesn’t, and even if it did, you could still easily make a mess of your interface declarations.
TypeScript can only protect unchecked access to nullables in a synchronous workflow and that is why we have no other solution but to throw an Error in our getter.
By now you should know that whenever you have a problem with your asynchronous workflow, it’s because you are not using enough promises. Let’s see what promises can do for us in this case.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class App {

// ...

public renameUser(newName: string): void {
this.user.name = newName;
}

public introduceUser(): Promise {
return new Promise(resolve => {
this.user.introduce((s) => resolve(s));
})
}
}

const app = new App();
app.introduceUser().then(console.log, console.error);
app.login(new User('John'));
app.introduceUser().then(console.log, console.error);

note: The promise API is available in node and all major browsers, except IE

When you wrap code in a promise like so, any exception emitted within the execution context of the promise is caught and used to reject the promise itself. Therefore, you get a consistent error handling mechanism where no exception may leak out.

We now have:

  • synchronous methods that return the result or throw an exception
  • asynchronous methods that fullfill a promise or reject it with an error
  • Readable code and managed errors

As advertised in the introduction, we made our code better with no downside at all!

TL;DR / To remember

  • Always use the –strictNullChecks compiler option
  • Dont be ashamed of nullable values, but if you do need one:

    • Always access a nullable through a getter, even within the private context of a class
    • Find the high level meaning of such a null value and throw a user-friendly error in the getter
    • Wrap all async code in a promise and always write promise-based async APIs (no more node-style callbacks)
      Check-out the gist of this guide if you prefer reading code.

You should worry about Vary

We’ve all heard of XSS, SQLi and CSRF. And although they keep occurring all the time, any decent web framework nowadays has some mechanisms to avoid those. Now did you know CSRF had a little sister? It is so poorly known that it seems like it doesn’t even have its own name! Before explaining how it works, let’s see what it can do.

Some background

Meet Bob, a web dev who is in charge of writing a public facing API. Bob sets the Access-Control-Allow-Origin header to allow all origin domains. Indeed, Bob wants any website to be able to talk to his API. Now, being a thoughtful developer, Bob knows about CSRF and he builds an authentication mechanism that delivers access tokens. Without a proper access token, his API won’t talk!

Still following? Good, now you may be wondering what this API does. In fact this API allows different services to store and share credentials in the cloud. The way a service gets the credentials is by issuing a request like this one:

1
2
GET /vault/{account}/{service}
Authorization: Basic QWxhZGRpbjpPcGVuU2VzYW1l

Bob heard one day that Auth basic is unbreakable, therefore he is using this method to pass the API key and username.
But the backend is doing some crypto stuff and hence requesting the credentials is an expensive operation. Bob adds some Cache-Control information to tell the browser to cache the response for a little bit. This way Bob reduces the load on his server and improves the speed of apps consuming his API.
If the credentials check out, the API returns something like this:

1
2
3
4
5
6
7
8
200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Authorization
Cache-Control: no-transform, max-age=600
Content-Length: 42
Content-Type: application/json

{"username":"Alice","password":"I4mab055"}

Everything works fine until one day, a client complains that her credentials got stolen…
Damn it Bob, again?!

You just got hit

Bob can check his server log for a while, he will never find anything. The attacker left no evidence behind because he never even had to talk to Bob’s API; Everything happened locally, in Alice’s browser. Alice received this stupid email a few days ago and couldn’t resist opening a link to a picture of a cat wearing a ninja-turtle mask.
While she was watching this picture, a piece of javascript issued an HTTP request in the background to Bob’s API and stole her credentials. But how did the malicious website get an API token in the first place you’ll ask? Well it didn’t. Because it never had to. A simple request without the Authorization header was enough to get the information.

Introducing: Vary

So there’s this thing in the HTTP spec that you may or may not have heard about. It’s called the Vary header. What it does is inform any cache about which request headers can cause the response to change. If the Vary header is omitted, then caches will disregard the request headers when deciding whether to serve the response from cache. In other words, no matter which headers were sent by the client the first time, the response will be cached and served back to every subsequent request until the response expires, regardless of the next request’s headers. Remember how Bob made the response cacheable for 10 minutes? This means that the attack is successful if it happens within 10 minutes of the original request.

To prevent this issue, the API should send the Vary header in its responses to inform caches that the response will be different for different values of the Authorization header. In our example we get:

1
2
3
4
5
6
7
8
9
200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Authorization
Cache-Control: no-transform, max-age=600
Content-Length: 42
Content-Type: application/json
**Vary: Authorization**

{"username":"Alice","password":"I4mab055"}

Going beyond

So stealing information is fun but let’s see if we can use this flaw to do something better. Imagine a web page that allows a user to set a message of the day, stores it in the cookies and displays it whenever the user visits the page for the rest of the day. The page is cached and can be fetched cross-domain just like in the previous example and it doesn’t have Vary: Cookie. If a user visits my website, I could issue a request to the greetings page with a custom Cookie. If I manage to perform this request when the page is not in cache yet, then the browser will fetch the page with my custom cookie and store it. Yes, we just found ourselves a cache-poisoning vulnerability. Now guess what happens next if this page allows the greeting message to contain javascript?

Depending on the cache headers, it may even be possible that the response gets cached on a server sitting between the victim and the origin. In this situation the attacker can read the victim’s data without even running code in her browser! All that is required is a hit on the cache server.

Wrapping up

I wish this article will at least allow a few people to get awareness about the dangers associated with Cache-Control and the Vary header. This vulnerability is similar to CSRF in that it allows cross domain requests to do some damage. It is harder to exploit and does not allow to reach the server. This however makes it impossible to detect attacks while still allowing some information leakage and defacing. In the case of cache poisoning, it opens up the attacks surface to find other vulnerabilities such as XSS or even SQL injections.

The fact that this vulnerability lies in the shadow of the big names like XSS makes it more likely to be looked over. In fact it has occurred before and I bet there are exploitable scenarios in the wild like those discussed above. So be careful next time you build an API and take some time to review your response headers. It can sometimes be trickier than you’d think figuring out which headers make the response change!

You may want to try out this quick and dirty proof of concept to see this flaw in action.