Analyse an app's web traffic (Reverse Engineer)

In this post I will show how to analyse / sniff most internet requests made by your phone and the apps on it.

We will then reverse engineer these requests and replicate them in Python.
This is a good start to writing your own software to interact with particular services without published API documentation.

I will be using Windows 10 (to view the requests) and an Android phone.
It should work on all other devices, but the steps to set the proxy on that device will differ slightly.
I have suggested google search terms under each step for other devices.

The software we will install on our PC is Betwixt (not Brexit)
This will create a proxy on the PC which our phone will connect to.
This will then show us all the requests the phone makes as they happen.

There is other similar proxy software but I prefer Betwixt because:
  1. It's open-source & free
  2. Chrome Developer Tools UI
  3. Portable (doesn't require installation)
  4. It works

Install Betwixt

Visit their Github release page and download the latest release for your OS.
Once it's downloaded, extract the archive to a location where you want it to live.
Remember, it's portable - so there is no setup / installer.
Once it's extracted, open the folder and run the Betwixt.exe file.
The first time it runs, Windows firewall will ask to allow it access.
Select "Allow Access".
Betwixt is now ready to receive requests.

Install SSL Cert

To allow the phone to still make HTTPS calls, when need to install a SSL cert on the phone.

In Betwixt, select Tools -> Root Certificate.
This will open a directory with the required certificate ca.pem selected.
Copy this file to somewhere handy (eg. Desktop).

We need to copy this file onto the phone.
There are a few ways to do this (email, cloud storage etc)
I however will just do it via USB transfer.

Plug the phone into the PCs USB port.
On the phone, swipe down from the top and select "USB charging this device".
Now select "Transfer Files"
The phone is now accessible on the PC.
Navigate into the phones "Internal shared storage" directory.
Copy the ca.pem file from the computer to this directory on the phone.

Once copied, navigate to Settings -> Security on the phone.
Scroll down to the credential storage group and select "Install from SD card"

Find and select the ca.pem file.
It may now ask you to authorize yourself.

Name the certificate "Betwixt".
Set Credential use to VPN and apps,
and then select OK.
You should see a small message saying the cert has been installed.

Not using Android?
Try googling "How to install certificate on [device name]"

Find IP Address

We will need our PCs IP address to allow us to set the proxy on the phone.

Right click on the network / Wi-Fi icon in the bottom right of your task bar.
Then select "Open Network and Sharing Centre"
Now click on the network name (Ethernet or WiFi).
Then select "Details"
Your local IP address is listed next to IPv4 Address.

Make a note of this IP address as it is required below.

Not using Windows?
Try googling "How to find my IP address on [OS name]"

Setup Proxy

We now need to tell our phones WiFi to use our newly created proxy.

Navigate to Settings -> Wi-Fi.
Long press on your connected Wi-Fi network.
Select "Modify network"

Scroll down and set Proxy to "Manual".
Set Proxy hostname to your PCs IP address
Set Proxy port to 8008.
Select SAVE.
A side affect of the proxy is that Android will think your Wi-Fi does not have internet access.
It then reverts to sending the data via the mobile network.
Therefore, we need to disable Mobile data to force Android to use the Wi-Fi.

Navigate to Settings -> Data usage.
Disable "Mobile data" for your network.
Your phone will now be sending all it's requests to our proxy (Betwixt) on the PC.

Not using Android phone?
Try googling "How to setup a proxy on [device name]" & 

              "How to turn off mobile data on [device name]

Recording the Requests

If all has gone well, you should now start seeing requests from your phone in Betwixt.

Open your phones browser and try visiting a view pages.
You should see the corresponding requests show in Betwixt,
We are now ready to reverse engineer an app.

For this example, I will be reverse engineering the RugbyPass Android app.
This is what I did to allow me to build my RugbyPass KODI Add-on.

Their API will only return data to IP's located in their allowed regions.
Therefore, you most likely won't be able to follow along with this particular example,
but the process should be the same for any other apps.

I want to retrieve the rugby games information shown when I open the app.
First, click the crossed circle icon next to the record icon in Betwixt (this clears the log).
Make sure it's recording (red record icon) and now open the app on your phone.

Once the app has loaded the content you want, click the record icon to stop it recording new requests.
We can now work with the requests without more coming in and changing the list.

I quickly notice all the requests of interest to me are from ''.
So, I enter this in the filter to only show requests to this domain.
When a request is selected, a new pane is shown with more details.
We are interest in the response tab which shows us the returned data from the request.

I go through each request until I find the response with the data I am looking for (rugby games).
This is the request I want to replicate.
Navigate to the "Headers" tab.
Here we can view all the data sent to the server which we can replicate.
Of interest to us is the Request URL, Request Method, Request Headers & Query String Parameters.
As this is a GET request, the query string parameters are also appended to the request URL.

Python Code

Let's replicate this request in Python.

We will first need to install the Python Requests library.
This can be installed using pip

pip install requests

And here is the Python code showing the values I have copied from Betwixt.
For GET queries, you can either have them appended to the URL, or pass a dictionary as a params argument like I have above.

The only request headers I have replicated is the user-agent string.
This is a header that tells the server what device you are.
Some servers may ignore requests from other devices.

As the data being returned is json, I can use the requests .json() function to get the JSON dictionary. If plain text is returned, just use response.text instead.

There are also shorter methods in requests eg. requests.get() to do a GET request.
Refer to the quick start guide to learn more.

You can grab my simple Python Requests template code HERE.
Fill it in with your own request details and start experimenting.