Intel machine learning makes GTA 5 look even more realistic

GTA 5 is a great-looking game, but Intel's machine learning experiment takes it up a notch.

Even if GTA 5 is not that impressive graphically (it is a game from 2013, after all), that doesn't mean the game looks terrible. However, attempts to make it look more photorealistic have been made by several modders in the past, yet none of them compare to the Intel machine-learning experiment that Stephan R. Richter, Hassan Abu AlHaija, and Vladlen Koltun did recently.

Recently, a YouTube channel known as Intel ISL posted a video where they use machine learning to make GTA 5 look stunning. It's neither a mod that players can download per se, nor is it a pre-existing mod that they downloaded from somewhere random. Instead, it is a legitimate approach to make GTA 5 look more photorealistic via machine learning.

---

Intel machine learning in GTA 5

The above YouTube video was posted by Intel ISL to showcase their use of machine learning in GTA 5. If players are interested in looking at their code, they should click here to check their Github. Likewise, players can click here to check out the paper these fine gentlemen wrote on the matter.

Otherwise, the remainder of this article will cover the above links, alongside what goes on in the video listed above.

What is machine learning?

A few examples of where machine learning is used (Image via R-bloggers)

Not every GTA 5 player understands what machine learning is. As impressive as it is to have improved GTA 5 graphics, it's vital for players to have a basic understanding of what machine learning is on a fundamental level.

At its core, machine learning is essentially the idea that AI can "learn" how to do certain actions without being programmed to do so. Given how difficult it is to properly program machine learning into something, it's very impressive for Intel to pull it off in a competent way.

In some ways, it's almost like the machine is sentient. Machine learning, in essence, is allowing machines to make use of what they're capable of learning without forcing them to automatically know it.

There are advantages for choosing to incorporate machine learning into various projects, but this article will focus more on the GTA 5 examples as shown above.

How Intel is using machine learning in the above GTA 5 example

A split-screen example of how it works (Image via Intel ISL (YouTube))

What some players don't understand about this machine learning example is that it's always being updated frame by frame. Hence, every passing moment continues to look stunning and not out of place. There aren't random stutters or visual bugs that happen, simply because it was coded phenomenally.

More specifically, the machine learning aspect comes from the fact that it's drawing data from the GTA 5 model and is translating it into a cityscape dataset that Intel programmed into it. Hence, they're allowed to go anywhere on the GTA 5 map with any camera angle, and the game will automatically update its graphics to look photorealistic.

The YouTube video is 720p, so it does seem less impressive than what it actually is. However, players can also compare what GTA 5 looks like in 720p within the same video, and the differences are astonishing.

Note: this machine learning example isn't limited to just 720p, that's just what the cap is on this YouTube video.

The dataset

An example of the dataset used (Image via Intel ISL (YouTube))

Interestingly, Intel used German cities as the data within the dataset in the video. Intel even compared their machine-learning technique to a popular method known as CUT. However, it's evident that Intel's method is clearly superior in terms of stability and visual aesthetics.

Likewise, Intel also compared their GTA 5 example to other methods, such as TSIT. However, it's still widely apparent that Intel's method of machine learning is the most stable and produces the least amount of visual irregularities by far.

Given the sheer technical aspects of machine learning, it's bound to go over a lot of people's heads. For example, most people won't understand what a G-buffer Encoder is. If people are interested in the more technical aspects of how it works, it's advisable for them to check out the full video.

More important details on the technical side of things

Some technical details of how it works (Image via Intel ISL (YouTube))

Intel ISL went into great detail about how certain strategies for perfecting this photorealistic approach were suboptimal in the context of GTA 5. For example, GTA 5 isn't 100% based on a real location, so one cannot just use real-world examples for every possible image frame by frame.

So, Intel chose to decrease the field of view that the machine interprets and allowed the machine to learn how to differentiate the individual aspects of a scene between the real world and what is shown on GTA 5.

All of these details go into an object's color, texture, shading, and other visual details. It's also important to note that they wanted the machine to not change too much about GTA 5 to the point it would become unrecognizable.

The end result is certainly remarkable and shows how far graphical improvements have come since 2013.