Google shows off what's possible using the JavaScript programming language for machine learning

• Updated on: 2019-05-14
• Read original article here

From head-controlled Pacman to browser-based upload filters — here's what's possible using TensorFlow.js.

Building and training machine-learning models using a web-scripting language might seem ambitious, but in 2019 it's perfectly feasible.

Helping make machine learning possible in the browser is TensorFlow.js, Google's open-source library for carrying out machine learning using JavaScript. The possibilities opened up by the library were showcased recently with a Google Doodle that generated a fresh Bach-style melody on demand.

In a talk at the Google I/O conference last week Sandeep Gupta, product manager at Google, said TensorFlow.js could be used by developers to create new machine-learning models, as well as to run or retrain pre-trained models.

"It's built specifically to make it easier for JavaScript developers to build and use machine-learning models within their JavaScript applications," he said, adding that machine-learning models built using the broader TensorFlow framework can be converted to run using TensorFlow.js.

SEE: Tips for building a successful career as a software engineer (free PDF) (TechRepublic)

TensorFlow.js can be used in JavaScript applications running in the browser, on servers inside a Node.js environment, on the desktop using Electron and on mobile browsers on Android and iOS devices. However, it is within the browser that Gupta sees the most possibilities.

"We see a ton of use cases in the browser and it has a lot of advantages because the browser is super interactive, you have easy access to sensors, such as webcams and microphones, which you can then bring into your machine-learning models," he said.

"Also we use WebGL-based acceleration, so if you have a GPU in your system you can take advantage of that and get really good performance."

That said, not every developer is won over by TensorFlow.js, with some arguing the library still has significant limitations.

So what exactly is possible using TensorFlow.js? While the framework is still relatively new, only hitting 1.0 this year, Gupta said there has been "really good adoption and usage by the community", and used his talk to demo some of the most interesting uses of TensorFlow.js.

You might not be crying out for a new way to play Pacman, but TensorFlow.js has made a novel spin on the classic arcade game possible.

Gupta showed a head-controlled version of the game, a JavaScript application running on the WeChat social media messaging platform on a smartphone.

After a quick calibration step, Gupta was able to control Pacman using head gestures, tracked by his phone's camera, looking left to move left, right to move right, and so on.

"It's a really fun way of interacting with the device, and the nice thing is that you can do a variety of things using web cams, using text, using speech, and have a very convenient way of sharing these applications without having to install anything," he said.

The taxi and delivery company Uber uses machine learning to tackle a wide variety of problems at a very large scale.

Helping it achieve that is Manifold, a browser-based application that Uber uses to visualize and debug their machine-learning models and data pipelines.

"This application runs in the browser and they're using TensorFlow.js for a lot of numerical computations, so for example distance calculations and visualization, as well as clustering of data," said Gupta, adding that "because of the WebGL acceleration, they could accelerate these computations more than 100x compared to just natively using JavaScript".

Online property rental service AirBnB uses machine learning in the browser to stop people from inadvertently uploading sensitive information when adding a picture to their profile.

"When a user is trying to upload a profile picture to the AirBnB website, sometimes people accidentally use a driver's license picture or a passport picture, which may end up containing personal sensitive information," said Gupta.

"So AirBnB runs a machine-learning model client-side in the browser or on device, so that if you were to choose a picture which may have such sensitive information it will alert you before you upload that picture."

Clinic.js provides a tool for sys admins and software engineers to profile server-side performance in a Node.js environment.

"This is a node.js-based application which is used for profiling node jobs or node processes and they're using TensorFlow.js to look for anomalies, or spikes in CPU usage or memory consumption of these node applications," said Gupta.

One of the main destinations for showing what is possible using TensorFlow.js is Creatability, Google's creative labs team's showcase for experiments using machine learning.

Gupta showed a machine-learning powered demo that allows a person to play a piano keyboard using head gestures.

Other examples of using TensorFlow with JavaScript online include Google's Gallery page for TensorFlow.jsand Magenta.js plug-ins offering machine-learning models for music generation.

If you're interested in finding out more about TensorFlow then check out TechRepublic's cheat sheet or to find out more about the popular new JavaScript spin-off language TypeScript, read TechRepublic's round-up of the best free resources for learning the language online.

More from Google I/O on ZDNet