Don't fall for the AI hype: Here are the ingredients you need to build an actual useful thing

• Updated on: 2018-10-13
• Read original article here

Artificial intelligence these days is sold as if it were a magic trick. Data is fed into a neural net – or black box – as a stream of jumbled numbers, and voilà! It comes out the other side completely transformed, like a rabbit pulled from a hat.

That's possible in a lab, or even on a personal dev machine, with carefully cleaned and tuned data. However, it is takes a lot, an awful lot, of effort to scale machine-learning algorithms up to something resembling a multiuser service – something useful, in other words.

Interest in AI is soaring. There's a lot of hype, and the reality is that the technology is still very raw and difficult to implement in production. Making the jump from prototype to product introduces new challenges. Where does the training data come from? How is it stored, organized, sanitized, and prepared for teaching a system? How do people query the system? Who can query it? What about security: how is sensitive information managed and protected? How fast does my hardware need to be to deliver results? Where are the performance bottlenecks and concurrency hurdles?

It goes on and on. Suddenly, your AI code, your supposed crown jewels, is just a small cog in a very large and complex and buggy machine.

Adam Gibson, cofounder and CTO of Skymind, a startup geared towards supporting AI for enterprise, says: “It’s a challenge to get AI into production. Infrastructure is one of the biggest problems to solve.”

Scaling AI is a complicated process with several steps – think of it like water flowing through a set of pipes, Chris Nicholson, cofounder and CEO of Skymind, explained to The Register.

Raw data is like liquid. It becomes digitized and is gathered before it enters the production pipeline. Next, the streams of data need to be processed by tools such as Apache Kafka or Apache Storm before they can be stored using something like Hadoop. The data – whether it’s an image, text, or sound file – then needs to be extracted, transformed so that it’s formatted in a way that it can be used for vector calculations, and loaded into a neural network for training.

After training, the model's inference code is tested with more data to check its performance and accuracy. In other words, you take the freshly trained AI, give it some queries and check the responses against what you'd expect it to output.

At this point, a little tuning and fudging takes place, as well as extra training, to nudge the software in the right direction. When the system is up to scratch – its accuracy is sufficient and it's working fast enough to give impatient humans what they want – it is plugged into a larger framework so it can be securely and safely accessed by users directly or through APIs to yet more software. An example of this is using machine learning to generate the best recommendations on Netflix or Amazon based on user profiles.

It doesn’t stop there; the final stage is to scale up. The system described thus far is packaged into a microservice so that the AI model can be spun up thousands, if not hundreds of thousands, of times over on many servers to cope with demand. Imagine a system checking credit card transactions for fraud detection: it needs to cope with millions of purchases coming in.

“Researchers don’t know or care about this. They are just interested in developing better algorithms, not serving customers. That’s too boring,” says Gibson.

Nicholson agrees. “They’re not interested in infrastructure and scaling up. Here’s a good analogy: Imagine a customer call service with one person – that’s how research works. They just have one person. But imagine if someone like General Electric is getting thousands of calls – they can’t just have one person. You’d need to clone that person an infinite amount of times to handle as many calls as needed.”

It’s likely that small businesses like startups will hype the inference stage of their AI system and “hand-wave” the rest. “They won’t be dealing with enough data to bother with all the infrastructure and scaling up, so ideally they’ll just outsource it to the cloud,” Gibson told The Register. “But as they grow, they will need to worry about those problems.”

Founded in 2014, Skymind develops AI models using Deeplearning4j, a deep learning framework written in Java and integrated with Hadoop and Spark.

Cloud platforms such as Google Cloud, Microsoft Azure and Amazon Web Services are all competing to offer AI as a service. They provide users with pre-trained models and a way to generate more tailored models by hooking together different tools, such as image recognition or natural language processing.

Using AI models through the cloud is restrictive and expensive. Neural nets learn iteratively and require intense training over several GPUs. Doing this through the cloud can cost anywhere between $15,000 and $30,000 per model, says Nicholson.

“Startups who can’t afford this will use pre-trained models downloaded from the internet and then they customize them – that’s typically what they do – then upload the model to the cloud.”

The advantage is that small teams don’t have to go around looking for computer scientists who understand machine learning to kickstart their idea. But they will be at a disadvantage, since they are constrained by the prepackaged models.

In fact, it provides a way to verify how serious a company is about using AI. The AI hype doesn’t seem to be slowing down, and boasting about using deep learning or machine learning increases the chances of a snazzy headline in a publication, or increasing investor funding.

“There are four things a company needs to build AI: talent, tools, data and infrastructure,” says Nicholson.

“They need to have people with machine learning backgrounds, computer scientists, and hackers. They need to be able to explain what types of algorithms they’re using, and if they are using the cloud then they probably don’t have as much data and don’t need to think about infrastructure.

“If they can’t talk about their algorithms and technology stack, then it’s a warning sign. It’s hard to take them seriously, because they could be doing anything. They could be using if-then statements in logistic regression and calling it AI.”

From the pharmaceutical trade to government agencies, there is no shortage of businesses interested in applying AI to their data. But it’s not for everyone, Nicholson and Gibson say. Deep learning requires huge swathes of training data to develop models with high performance.

“The reality is most startups don’t have enough data for it to be useful. Companies also have to be realistic. People have to get past the magical thinking stage to reach a real, solvable problem.” ®