The AI Hierarchy of Needs

And what it means to me!

Have you ever encountered an idea that changed how you look at everything? You may not know it when you get introduced to it. You may even completely dismiss it. But over time, you come to appreciate it more and more. That is what happened to me with the AI Hierarchy of Needs. I was introduced to it by a co-worker. For me, it started as an excellent conversation piece, but it matured into a whole philosophy. This article will walk you through the AI Hierarchy of Needs and how it grew on me.

When I got introduced to this concept, the most popular article concerning it was this one on Hacker Noon, it is certainly worth a read. I’ve simplified the Hierarchy a bit into these five layers:

• Collect
• Move
• Explore
• Analyze
• Learn

If AI (Machine learning) is the goal, we must first possess the ability and perform some analysis. Analysis begets exploration; an exploration often requires some munging of the data, and to munge, we must first have the data in our possession. And that is the premise of the AI Hierarchy of Needs. If we want AI, our data must be ready for AI.

Before you leave, because I am just listing the obvious, I want to tell you a little about the context of my introduction to this hierarchy. The company I was working for had hired a Ph.D.-level mathematician to bring about the AI revolution. But, unfortunately, as he launched into AI-oriented endeavors, he came to a shocking conclusion: our data wasn’t nearly ready for AI; worse, we didn’t have the data we needed!

Our deficit was evident in two ways when talking with the pyramid sitting in front of us. First, this mathematician was hired to do AI but found himself building each layer of the pyramid. There are many implications: cost, time, proficiency, and best practices. If the right person isn’t doing the right job, it’s more complicated, takes longer, and often isn’t done the best possible way.

Second, the most devastating realization was that we didn’t have the needed data. Hiring, re-allocating, or contracting out the different pyramid layers is doable, maybe even in short order. But coming up with data you don’t have can be much more challenging and take longer if historical data is needed.

Understanding that we weren’t where we needed to be, we launched a Data Science Readiness initiative with the AI Hierarchy of Needs at its core. We knew we needed to address each layer.

Collect

More stating of the obvious, but you can’t get AI out of data you don’t have; you can’t even build a dashboard for it. Even if you have some data, you don’t have enough for the AI projects that would be transformative for your company. Storage costs are cheap, and data doesn’t have to be processed before it’s stored. We adopted a mantra of saving everything now and determining how to use it later.

Move

If you attempt to gather each piece of information possible for your company, it will likely be stored in many formats. If you’re a worldwide company, it’s also possible that your data is spread around the world as well. So the second layer of the pyramid is all about taking the raw information you’ve collected and moving it to the right location in the correct format.

Explore

The beginning step of using your data is just knowing what you have. Once the data is where it needs to be, please put it in the hands of someone who understands it or can figure it out. When we think of exploration, we often think of visual data exploration tools like Tableau, PowerBI, etc. There’s a specialty here for people who can touch a dataset and see the possibilities of what it could enable.

Analyze

Once you’ve got the data you need and someone has the desired use for it, it’s time to take everything up a notch. Apply some statistics, monitor for data quality, and anomalies harden your solution. We’ve found that many projects, even ones that ask for AI, end in the exploration or analysis phase. Satisfying needs with the simplest solution possible is an overall win.

Learn

It took a lot of work to get here, but now your data is ready to apply some artificial intelligence. All of the steps below were necessary so that the data coming to this stage an established, understood, and monitored pipeline feeding it. AI doesn’t have to be challenging, but algorithms often embody the garbage in garbage out rule. And when it does get tough, if your pyramid is reliable, you won’t have to question the data coming in.

The layers of the Hierarchy can inform many things about your data strategy. It can be the outline for a project, a team, or a process. So keep it in the back of your mind and try to see if it applies the next time you have a data conversation.

Want to know more and go deeper with the AI Hierarchy of Needs? I’ve turned a lot of my experience applying it into a simple course. Find out more here.