The chasm between Business and IT is well documented and has existed since the first punch-card mainframe dimmed the lights of MIT to solve the ballistic trajectory of WWII munitions. Analytics and now Data Science are trapped in the middle. Everyone hopes they'll deliver the productivity gains, but the jury is still out.
Some studies suggest that analytics projects have an 80% failure rate. A recent HBR article put it at 100% for data science projects. That’s abysmal. And there are dozens of reasons why it's so poor. In this article, we'll look at the team.
A helpful starting point is to imagine your dream team. Who would you hire, and what would their roles be? I suggest that there are five distinct job descriptions:
So who's in the analytics dream team.
Data Steward – this skillset is alive and well in most organizations. Almost everyone has a data warehouse, talks about the ETL process, and has had discussions around the business rules of cleaning up and storing their data. What they should be talking about is how to get the data out more quickly and cleanly. A typical project is 80% data wrangling, so don't skimp on number or quality here. The data steward will use tools such as MongoDB, MySQL, Oracle, and if she’s a superstar, she’ll dabble in Python and web scraping and know the difference between JSON and XML. Maybe you'll give her a raise can call her a data engineer.
Analytic Explorer – this skillset is a tough one to find. It requires math, statistics, and modeling along with a healthy dose of creativity and skepticism. This is a person who can spin straw into gold or write tomorrow’s news today. His job is to ask the right questions, explore your data, and distill it down to insights that will support your most critical decisions. He’ll use tools such as TensorFlow, R, MATLAB, ArcInfo, SAS, Tableau, and SPSS. If he’s a superstar, he’ll know all about Reinforcement Learning, Bayes, Optimization, and the difference between precision, accuracy, and skill.
Information Artist – This is the role of a creative analytical. Her goal is to sell the results to the decision-maker. And the lack of emphasis on this skillset is one of the reasons analytics is such a failure (and why Apple is such a success). Edward Tufte – the godfather of data visualization - speculates that the lack of good data design contributed to both the Columbia and Challenger space shuttle tragedies. Think of this person as being as crucial as your sales force. In fact, that is her job – to sell the right answer. But she's a whole lot more than a graphic designer. She gets aggregation, normalization and signal versus noise. And she also get mood, white space, and font kerning. Excel and PowerPoint may be her go to tools, although she's more likely to use Photoshop, Moqups, and D3. If she’s a superstar, she’ll be as comfortable talking about the math behind the visuals as she is talking about the psychology behind her design.
Automator – If the Explorer finds the path through the dark forest to the fountain of youth, and the visualizer designs a beautiful bottle for the elixir, then the Automator turns that path into an eight-lane highway and builds a factory to bottle that stuff as soon as it comes out of the ground. His job is to operationalize the work of the Explorer and Visualizer. He makes sure that results are timely and fast. He adds scale. He might use traditional coding methods like C# or Java or he might fiddle with JavaScript and D3. Or he might even be the guru of Vue.JS or React.
The Champion – The champion stands with one foot in the land of “gut feel”, and the other planted firmly in the side of “evidence”. She can speak the language of the geeks, and translate it to that of the battle-hardened general. She believes strongly in data-driven decision making, but also recognizes the value of deep domain experience. She’s tireless in her efforts to sculpt the processes of the organization to support analytics. She aims to harvest the brightest insights from the sharp young analysts and the cleverest hacks from the wily old veterans. Her focus is adoption and impact. If she’s a superstar, she’ll make you believe that this analytics thing was your idea in the first place.
So that’s your dream team: a steward, an explorer, an artist, an automator and a champion.
The dream team in the real world.
But there’s a problem. This team rarely exists in the wild. Most companies hire the Data Steward, and then try to do the rest through a major software implementation. Unfortunately, the software is not meant to explore and discover. And it was designed by engineers who don’t understand the psychology of data visuals. It’s like expecting your bookkeeper to be your CFO. Sure they can both do accounting, but you won’t be happy with the results.
In other instances, organizations will try to shoehorn engineers into the roles “in their spare time”. Again, with neither the training nor the time to explore the data or design the results, they’re doomed to fail. These skillsets are distinct, and they shouldn’t be ignored.
So what’s the canny company to do? If you’re extremely lucky, you’ll find the unicorn of the 21st century known as the Data Scientist, pay her a quarter million, and watch the magic happen. (A data scientist can do all five roles.) Or you can try to develop these skills in-house. Or you can hire contractors – perhaps engaging a consulting firm to take on the Explorer or Visualizer roles for a time. Or you can outsource the whole thing.
What’s important is that you recognize that each of these roles is necessary. Neither software nor “Dave in Engineering” can replace them. Happy hunting.