GitHub's "merge pull request" is wrong

This is the story of Bob. Bob is working on some cool project on github. He’s written tests and runs them continuously on travis. Bob asks his contributors to write tests for new features they introduce. Using GitHub’s pull requests, he can make sure no bugs are introduced by looking for the green tick next to the commit number. If you look at the commit tree in Bob’s repository, you’ll probably find something like this.

Bob states in his contribution guide that people can fork master to work on new features. He guarantees that master’s tests always pass and therefore, individual contributors don’t need to worry that much about errors made by other developers.

If you’re familiar with github then you are probably familiar with this workflow and you’ll think that it’s perfectly reasonable.

But you’ll be wrong. Bob got overly enthusiastic about github’s UI and will soon discover that he’s been wrong all this time.

A catastrophic scenario

Say two contributors, Alice and Charlie, are writing code for Bob’s project in their respective branches. In this project, we find the following code for operating Bob’s car:

1
2
3
4
5
6
7
8

private void doSomething(String action) {
switch (action) {
case "OPEN_DOOR":
open_door();
break;
}
}

Now Alice thinks it would be better if Bob used an enum instead of a string. She changes the function to the following and opens a pull request on Bob’s project.

1
2
3
4
5
6
7
8

private void doSomething(Action action) {
switch (action) {
case Action.OPEN_DOOR:
open_door();
break;
}
}

Bob opens his project page and sees a pull request. He thinks Alice’s modification makes sense and, since all tests passed, he clicks on the green button.

[caption id=”attachment_221” align=”alignnone” width=”776”]The button of doom The button of doom[/caption]

Meanwhile, Charlie thinks Bob will need to close his car’s door at some point. Charlie adds a new case to the switch:

1
2
3
4
5
6
7
8
9
10
11

private void doSomething(String action) {
switch (action) {
case "OPEN_DOOR":
open_door();
break;
case "CLOSE_DOOR":
close_door();
break;
}
}

The next day, Bob opens his project on GitHub and sees Charlie’s pull request. All tests passed, no merge conflict, that’s a clear LGTM and Bob merges the code.

A few days later, Bob receives complaints from multiple developers, they forked master and it doesn’t build!

What could go wrong?

The problem lies in a common misunderstanding of the following assumptions.

  • Different PRs must be completely independent
  • No merge conflicts ≠ No conflicts

If you accept one of these to be violated, then expect your main branch to fail tests!

If we speak in terms of diff, then Alice’s diff is the following:

1
2
3
4
5
6
7
8

- private void doSomething(String action) {
+ private void doSomething(Action action) {
switch (action) {
- case "OPEN_DOOR":
+ case Action.OPEN_DOOR:
open_door();
...

While Bob’s diff looks like this:

1
2
3
4
5
6
7
8

...
break;
+ case "CLOSE_DOOR":
+ close_door();
+ break;
}
...

Put together we get

1
2
3
4
5
6
7
8
9
10
11
12

- private void doSomething(String action) {
+ private void doSomething(Action action) {
switch (action) {
- case "OPEN_DOOR":
+ case Action.OPEN_DOOR:
open_door();
break;
+ case "CLOSE_DOOR":
+ close_door();
+ break;
...

This diff does not cause a merge conflict but it is actually in conflict!

–no-ff and the button of doom

The flaw resides in that big old “merge pull request” button. While the green checks above the button and its fat style make pressing it a rewarding thing, it hides the real danger behind it: a blind merge!

The above example looks like this in terms of commit history:

1
2
3
4
5
6
7
8

+-PR_Charlie-+
/ \
+--PR_Alice--+ \
/ \ \
---+----------------x---x
|
master

All changes introduced by Alice and Charlie on their respective PRs have been tested, but the merge commits (marked ‘x’ above) were not! This is called a “non fast-forward” merge (–no-ff in git) and is basically like committing directly on master.

The right way to do it

Assuming you are like Bob and want your main branch to always pass tests, then here is the right way to do this: Merge master into the PR fisrt, and merge only when fast forward. It would look like that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


Initial state: both PRs are unmerged:

+-PR_Charlie-+
/
+--PR_Alice--+
/
---+
|
master

on master: git merge --ff-only PR_Alice

+-PR_Charlie-+
/
---+--PR_Alice--+
|
master

on PR_Charlie: git merge master

merge branch master into PR_Charlie
|
+-PR_Charlie-+-+
/ /
---+--PR_Alice----+
|
master

Then on master: git merge --ff-only PR_Charlie

+-PR_Charlie-+
/ \
---+--PR_Alice----+-+
|
master

The end result is exactly the same as before EXCEPT that master was merged into the PR first, and this (if properly pushed to github) allows the automated tools to test the last commit. This ensures that master builds because master will always be on a commit that has been tested before.

Many beginner (and even advanced) programmers get fooled into thinking that merging a green pull request cannot harm. In an ideal world, clicking the green button would: merge master into the branch, wait for the tests to pass, and on success, fast-forward master to the merge commit. But unfortunately I don’t think this is coming anytime soon. In the meantime, think about it twice before you press the button of doom. And when in doubt, run the merge sequence manually in the right order. This may save you a few hours of debugging.

Hacking FunRun2 — how to reverse engineer a Corona app

Disclaimer: This article is a bit old and does not apply to recent versions of the corona sdk. It is put here for educational purpose, showing how one may try to break into this kind of applications. It is not a magic formula for hacking all corona apps across all versions of the sdk.

funrun2 hack

If you haven’t played funrun yet, you probably will soon. Funrun2 is one of the best multiplayer mobile games out there. It’s one of these rare games where you can have all the fun from the very first game you play, but still stays enjoyable after month of playing it. It’s my favorite game on mobile and since I had some spare time, I decided to tear it open (yeah, that’s what I do to things I like). It was a bit harder than expected and involves some not so trivial hacking as well as a bit of luck. By the way, yes, all these buttons work.

The easy part

First I need to get my hands on the .apk file. There’s plenty of tools out there to do that, some of them are even open source.

An apk file is an archive, just like a jar or a zip so I can unpack it with regular unzipping tools. This gives me access to a well documented directory structure. The two things I want are the classes.dex file, containing java bytecode, and the assets directory. Reversing the first one allows me to get access to java source code so it’s pretty obvious why I want to look at it.
To understand why the second one is interesting though, you must consider that this game was built with a cross platform framework, namely corona. These kind of framework generally allow you to code your app with some scripting language that gets executed on every platform the same way. Therefore, unless the script is cross compiled to native language (which nobody ever does), it must be stored somewhere. That’s when the assets folder comes to play. It contains only a few images and this resource.car file which extension conveniently matches Corona ARchive. How cool is that ?
I open the file in a text editor and mostly get binary-unreadable-shit but there’s also strings like “lua.whatever.that.is.lu”. There are the scripts.
In case you don’t know: lua is one of the main scripting languages used in games with it’s open-source libraries and tools.

Now I still haven’t figured out where the images and sounds are (the corona archive weights a mere 2MB so it doesn’t contain any asset), but this will come later.

Things getting interesting

Trying to open the corona archive, I quickly realize it’s not a common archive format (even 7-zip can’t open it !). So there are two ways to go: finding the loading procedure in the java code or analyzing the file structure by hand. The first seems more reasonable.
But first, let’s decompile the dalvik bytecode into human readable java. These tools are helpful: dex2jar to covert the .dex to .class (dalvik to jvm) and jd to go back to plain java source code.

Now I have a load of java source code to look into, great! Let’s look for the string “resource.car”: no results. Looking manually, I quickly find corona’s source code in com/ansca/corona, that’s where the magic must happen. However, nothing is nearly dealing with a “resource” nor a “.car” file. I find however references to images. They were actually located on the device’s external storage (sdcard/Android/obb) in a .obb file. This is something common on android devices (that I wasn’t aware of before doing this hack), my guess is that moving resources allows the apk to be lighter and makes the internal memory footprint smaller. A .obb file is just a zip with a fancy extension. I can grab this file on a computer, edit the images/sounds/whatever and put it back in place. The game still runs with my custom images.

[caption id=”attachment_177” align=”aligncenter” width=”700”]funrun_image Besides adding childish drawings to the game, there’s not much that can be done this way.[/caption]

I’m in a dead-end with the java code. Let’s give a shot at manually reversing the resource.car file:

I open it with a hex editor and quickly realize the format used is trivial:

First, a header to rule them all, then comes an index made of each file name + position (address) of it’s contents in the archive, then comes the data block with each file contents.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[header] 16 bytes

[index]
[entry]
[header] (4 bytes) 1
[block addr] (4 bytes)
[nameS] (4 bytes)
[name] (nameS+1 bytes) 0-terminated
[padding] (0s to complete 4x addr)

[files]
[entry]
[header] (4 bytes) 2
[??] (4 bytes)
[size] (4 bytes)
[file] (no 0 termination)
[padding] (to next 4x addr)

The resource.car file format

Now I didn’t have to code a packer for this format myself because luckily, someone already did that. So I grabbed this corona archive packer/unpacker which you won’t find a link to on my blog.

It turns out the archive contains uncompressed precompiled lua scripts, using luadec and then luac again you can decompile and then recompile these scripts which is very convenient. These lua scripts define absolutely every aspect of the game. From where a button might be to how fast a player can move and what a specific powerup does.
Injecting our custom resource.car in the app is a big deal because I could modify anything in the game.

First I try modifying something inside the precompiled lua scripts, such as a string since we can easily recognize them. If this works, I can later try to recompile one file from the (modified) source.
I repack the resource and put it back in the apk file, sign it with dex2jar (as explained here) and install it on my device. I launch the game and, with no real surprise, it crashes.

I figured there’s probably some kind of check performed against the archive to guarantee integrity. However, the loader is nowhere to be found in the java source. This battle is lost but not the war.

Going down

a88

Yeah, maybe Di Caprio is right, we need to go low-level and find this resource loading procedure.
Inside the .apk, there’s another directory called “lib”. It contains architecture-specific libraries. You’d usually build a native library either when you have no choice but use C (for instance, lua is written in C) or when you want higher-than-java performances (for a game framework, it makes sense).

In this lib folder, there’s some lua libraries, but also random libs for sound, video, analytics and whatever. Most importantly I find here a lib called libcorona.so. Let’s quickly open it with a text editor to search for the string “resource.car”, bingo!

I’ll have to disassemble the lib in order to understand what’s happening in there. It turns out, there’s an amazing tool that does this and it’s called IDA. It’s more than a cross-architecture disassembler, it does lot’s of stuff but for my purpose I’ll just use it as a disassembler. It handles libcorona very well and kindly gives us the assembly code.

FYI: I picked the armeabi-v7a version so what follows would be different different with an apk targetting another platform

I look again for the string “resource.car” and find it referenced at offset 0x1081D4.

[caption id=”attachment_180” align=”aligncenter” width=”762”]res_car The only piece of code referring to “resource.car”[/caption]

 

It took me a while to figure out how things worked, but at some point i had a theory:
From the occurence of “resource.car”, I clearly see two execution paths: one that leads to a portion of code referencing the string “could not verify” and exiting, and one that leads to a subroutine that I deduced was meant to initialize the lua runtime or something like that. What stands between them is a fragile Branch No Equal (BNE) that we would like to turn into a rock solid Branch.

[caption id=”attachment_181” align=”aligncenter” width=”761”]jump_annot Take the yellow jump to get to the blue subroutine and avoid the nasty red branches[/caption]

 

Looking at this document and some other branch instructions around in the code, I deduced that I needed to change byte x108247 from 1A to EA to turn my BNE into a B. Going back to my hex editor (I should really learn how to use IDA properly some day) I change the aforementioned value.

[caption id=”attachment_182” align=”aligncenter” width=”715”]patched It’s a branch![/caption]

 

Now let’s pack our patched lib back into the apk, sign it, install it, run it… It runs! Let’s see if our custom value is used.

[caption id=”attachment_185” align=”aligncenter” width=”700”]modded First time ever a guessed binary patch works upon first try.[/caption]

Let’s try the full pipeline: modify the lua source code, compile it, repack the .car and place it back in the .apk. It works provided we use the proper luac version (a quick search for the version string in liblua.so saves you the guessing). That’s it, the path is paved for hours of fun modding the game!

Conclusion

I had seen many game cheats and always wondered how they actually work, now I know at least for this one. To illustrate this article, I managed to strip all ads from the game, create a speed hack, fly mode, and unlimited powerups. Hacking the in-game money is harder because only the server issues coins, so even if you trick your phone into thinking you are rich, the server knows how much you really have.
I’d like to emphasis that I personally think buying or downloading a cheat for a game is lame and those who do that just deserve a trojan. However, hacking a game is really something fun that one should try. It requires patience, and a capability to keep a clear mind even after hours of work leading nowhere.

SailsJS tutorial | expenses tracking app (Part 4/4)

In the previous parts,
we’ve set up a sails application, added some styles and views and implemented a solid backend. It’s now time to finish this application. We will implement the front-end, adjust a few settings and give some tracks to continue the development.

The app page

To render our app page, let’s open the MainController (in api/controllers) and replace the contents of the app action with

1
2
3
4

app: function (req, res) {
res.view();
}

What this does is it renders the view views/main/app.ejs because we are in the app action of the MainController. You should have already added this view in part 2.

Next we only want logged-in users to access our app so let’s add the following lines in config/policies.js

1
2
3
4

MainController: {
'app': ['passport', 'sessionAuth']
},

Also we want some more information about our users than we currently get, like their profile picture and full name. To do this, we’ll need to hack a bit into what sails-generate-auth generated us. Open api/services/passport.js, line 82 there’s the code fetching user attributes


// If the profile object contains a list of emails, grab the first one and
// add it to the user.
if (profile.hasOwnProperty(‘emails’)) {
user.email = profile.emails[0].value;
}
// If the profile object contains a username, add it to the user.
if (profile.hasOwnProperty(‘username’)) {
user.username = profile.username;
}

// If neither an email or a username was available in the profile, we don’t
// have a way of identifying the user in the future. Throw an error and let
// whoever’s next in the line take care of it.
if (!user.username && !user.email) {
return next(new Error(‘Neither a username nor email was available’));
}

1
2
3
4
5
6
7
8
9
10
11
12
**Remove all this** and insert this instead:
<pre lang="javascript" line="82">
if (!profile.hasOwnProperty('id')
|| !profile.hasOwnProperty('displayName')
|| !profile.hasOwnProperty('photos')
|| profile.photos.length == 0 ) {
sails.log.error('not enough info');
next(new Error('Your login provider did not provide enough information.'));
}

user.username = profile.displayName;
user.picture = profile.photos[0].value;

Scripting

This tutorial aims to be neutral regarding what client technology you use, therefore we we’ll do DOM stuff “manually” with jquery and only use backbone and underscore to have a tidy collection of expenditures. You’ll be able to follow even if you’ve never heard of backbone.

First, download this archive. It contains templates, js models and libs as well as a skeleton for our app script. Copy the assets into your assets folder.
You can lift your server and log-into the app. You should see a blank table. Now open your web console and it should display some errors. This is due to the fact that the script you just copied are linked to the layout in no particular order. You can see this by looking at views/layout.ejs.

1
2
3
4
5
6
7
8
9
10
11

<!--SCRIPTS-->
<script src="/js/dependencies/sails.io.js"></script>
<script src="/js/dependencies/backbone.js"></script>
<script src="/js/dependencies/bootstrap.js"></script>
<script src="/js/dependencies/jquery-1.11.1.js"></script>
<script src="/js/dependencies/perfect-scrollbar.js"></script>
<script src="/js/dependencies/underscore.js"></script>
<script src="/js/ExpenditureModel.js"></script>
<script src="/js/app.js"></script>
<!--SCRIPTS END-->

Here, backbone and bootstrap are included after jquery and underscore. To correct this, let’s take a look at tasks/pipeline.js. We will change jsFilesToInject such that our js files are included in the right order:


var jsFilesToInject = [
// Underscore & jquery before backbone
‘js/dependencies/jquery-1.11.1.js’,
‘js/dependencies/perfect-scrollbar.js’,
‘js/dependencies/underscore.js’,

// Dependencies like jQuery, or Angular are brought in here
‘js/dependencies/*/.js’,

// The app classes before the app index
‘js/ExpenditureModel.js’,

// All of the rest of your client-side js files
// will be injected here in no particular order.
‘js/app.js’
];

1
2
3
4
5
While we are here, change _templateFilesToInject_ such that it looks for **.ejs** files instead of **.html**.
<pre lang="javascript" line="51">
var templateFilesToInject = [
'templates/**/*.ejs'
];

Save this file and if sails was running in background, the build tasks should run and now our js files should be included in the right order and our templates are compiled.

Now, let’s write real code. Open assets/js/app.js. This file contains already everything we need to handle the DOM of the page. What we’ll focus on is how to get data from the server and keep it in sync using Sails’ real-time capabilities. For simplicity we’ll do everything over the socket connection so that we don’t mix protocols.

We have a backbone expenditure collection that we’ll use to store our expenditures locally and propagate events.

First thing we’ll do is fetch a list of expenditures from the server. This is what io.socket.get is for.
Let’s add those models to our collection as soon as we get them.

1
2
3
4

io.socket.get('/expenditure', function(models) {
expenditures.set(models, {parse: true});
});

We pass the parse: true option to turn the date string into an actual date object.
Add some expenditures with the shortcut routes (for instance /create?amount=32&description=resto) and now you should see them displayed in the table like this:

Yay, we got some data !
If you can’t see the picture, that’s probably because you created a user before we told passport to get profile pictures of new users. Stop the server, delete .tmp/localDiskDb.db, then start the server again. Next time you log in, it will create a user from scratch and now your profile picture should be included with it.

Now it would be better if we could add expenditures directly from the app. We can insert the code that handles the create form submit action on line 58.


createForm.submit(formAction(createForm, function(formData) {
io.socket.post(‘/expenditure’, formData, function(data, res) {
if (res.statusCode === 201) {
expenditures.add(new expenditures.model(data, {parse: true}));
} else {
console.log(data);
}
});
}));

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Here we use the _post_ method of our socket to create the model, then we check the status code and create a new model if everything went fine. If there was an error, we just log the response body. It would be better to have a proper error handling mechanism of course but we won't do that in this tutorial.

You can now see that our models are added to the collection and if you refresh they are still there. Good, now if you open a second tab it won't automatically update one tab while you add a model in the other. To do this, we need to fill the blank on line 23.
This callback is invoked every time something changes in our expenditures collection. The passed-in _evt_ object has a _verb_ property which can take one of "created", "destroyed", "updated" values.
With a simple switch, we can handle each of these cases:
```javascript

io.socket.on('expenditure', function(evt) {
switch (evt.verb) {
case 'created':
expenditures.add(evt.data, {parse: true});
break;
case 'destroyed':
expenditures.remove(evt.id);
break;
case 'updated':
var data = _.extend({id: evt.id}, evt.data)
expenditures.set(data, {
parse: true,
remove: false
});
break;
default:
throw new Error("Unknown verb: "+evt.verb);
}
});

That’s all we need to make our app real-time !

Now how about deleting items. We have an event handler for that on what should now be line 48 and I think you’ve guessed what we’ll write here.


io.socket.delete(‘/expenditure/‘+id, function(data, res) {
if (res.statusCode === 200) {
expenditures.remove(id);
} else {
console.log(data);
}
});

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

And last but not least, we want to edit our records. This is done by clicking the edit icon which shows up a dialog. We need to write what happens when this dialog is submitted.

```javascript

$('#editModal-accept-btn').click(itemAction(function(evt) {
// Retrieve form data
var formData = getFormData(editForm);
var id = editedItem.id;

// Save to the server
io.socket.put('/expenditure/'+id, formData, function(data, res) {
// If successful
if (res.statusCode === 200) {
// update the collection locally
var data = _.defaults({id: id}, data);
expenditures.set(
data,
{parse: true, remove: false}
);
} else {
console.log(data);
}
});

Congratulations, you now have a fully functionnal app !

Now if you log-in as another user, the edit and delete actions are shown on expenditures you did not create. Of course, the server forbids this but it would be better if they were hidden. To do this, we’ll need to know which client is currently connected from the client-side javascript. The easiest way to do it is by adding a script in layout.ejs which injects a user object for the logged-in user in the client’s javascript scope. We need to be careful though as layout can be displayed even when the user is not logged in. Add this in views/layout.ejs right before the other scripts.

1
2
3
4

<script type="text/javascript">
var user = <%- JSON.stringify(req.user) || 'null' %>;
</script>

Now we can use this client object in our client-side scripts. Let’s edit assets/templates/expenditureItem.ejs, we will wrap the actions in an if statement like this:


<% // person is available as a template variable. user is taken from the global scope
if (person.id === user.id) { %>
[“>

](#)
[“>

](#)
<% } %>

`

Conclusion

From there, if you want to use mongodb as your data store, the only thing you need to do is put your configuration in config/connections.js and set that connection in config/model.js. I think it’s a really great framework with powerful features despite the fact it currently lacks contributors (and documentation).

I hope this tutorial gave you a good insight at how Sails works and that you’ll have fun building awesome app with it! Feel free to ask any question on twitter or add a comment below.