Reverse engineering of navigators' API to compare means of transport (Bachelor Thesis)

2021-02-25

TL;DR

For my bachelor thesis I proposed to compare means of transport using various public API from navigation services able to estimate duration of any route with real time traffic data. Since many of them weren't working, I did the reverse engineering of their GUI to scrape the data directly. The scraper was written in Go, and then the JSON results have been analized with Jupyter Notebook.

1. The goal

The question was if using the car inside one of the most trafficated city in the world (Milan, 100th on TomTom report) was a good idea compared to using alternative means of transport like public transport and bicycle for small routes (2 to 5 km). I chose to answer this by using navigation services like Moovit, GMaps, Waze and others that offer a public API to give accurate ETA (Estimated Time of Arrival) based on real time traffic data for any given route. Here is the high level steps of the work to be done:

for every 2 minute {
  from, to := generateRandomRoute()
  for t in transportMeans {
    travelTime, travelDistance := t.API.estimateRoute(from, to)
    save(date(), t.type, travelTime, travelDistance)
  }
}

2. Choosing the ingredients

After doing my own research I found the API needed for every means of transport:

• Google Direction APIs (for almost everything)
  • API too expensive
• Waze API (for cars)
  • public API not working, reverse engineering
• Moovit API (for public transport)
  • public API not working, reverse engineering (failed, too many random errors)
• HERE WeGo (for public transport)
  • unfortunately no real time data for Milan
• OpenStreetMap API (bicycle and pedestrian)
  • public API

3. Writing the software in Go

3.1 Moovit API (failed)

Since the public API of Moovit was not working at all, I decided to open the Firefox console and analyze the network traffic of the web application trying to understand the underlying communication protocol. Here's a code snippet that gives the idea:

func GetRoutes(fromLat, fromLon, toLat, toLon string) Result {
  var empty Result

  fromName := getLocationName(fromLat, fromLon)
  toName := getLocationName(toLat, toLon)
  headerParams := getParamsNeededForHeader(fromName, toName)
  cookie := getMagicCookie(headerParams)
  fromMetadata := getLocationInfo(fromName, headerParams, cookie)
  toMetadata := getLocationInfo(toName, headerParams, cookie)
  token := getMagicToken(fromMetadata, toMetadata, headerParams, cookie)
  routes := getSuggestedRoutes(fromMetadata, toMetadata, token, cookie)

  return routes
}

After testing it, I found that 7 out of 10 results were always wrong (negative or zero time travel). A total waste of energies... But I moved on.

3.2 Waze API

With the Waze API I had the same problem: public API wasn't working (docs gave me HTTP 502 error). I decided to follow the same way of Moovit, and fortunately it was way more easier than that:

High level code:

func GetRoutes(fromLat, fromLon, toLat, toLon string) Result {
  var empty Result

  cookies, err := getCookies()
  err = setCookieConsent()
  routes := getSuggestedRoutes(fromLat, fromLon, toLat, toLon, cookies)

  return routes
}

HTTP request (one of many):

func getSuggestedRoutes(fromLat, fromLon, toLat, toLon string, cookies []*http.Cookie) (Result, error) {
  var tripPlans Result

  urlWaze := "https://www.waze.com/row-RoutingManager/routingRequest?"

  header := http.Header{}
  header.Add("Host", "www.waze.com")
  header.Add("User-Agent", "Mozilla/5.0 (Linux x64) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0")
  ...

  params := url.Values{}
  params.Add("from", "x:" + fromLon + " y:" + fromLat)
  params.Add("options", "AVOID_TRAILS:t,ALLOW_UTURNS:t")
  params.Add("to", "x:" + toLon + " y:" + toLat)
  ...

  response := httpwrap.Get(urlWaze, header, params, cookies)
  bodyBytes := ioutil.ReadAll(response.Body)
  json.Unmarshal(bodyBytes, &tripPlans)

  return tripPlans, nil
}

4. A bit of statistics

5. Conclusions

Even though the car always outperformed every alternative means of transport in terms of time, the results showed that, for small routes (2 to 4 km) during the rush hours (8:00-10:00 and 17:00-19:00), most of the time the outperform was less than 6 minutes, that means that most of the times you could have taken the bicycle or public transport and you would have arrived at destination only 6 minutes later than car in the worst case.