Business Intelligence

In my last post, I kicked off our new Executive Insights Series and shared my perspective on the struggles marketing organizations are facing as they try to measure, report and analyze their performance, so they can become more agile and effective while increasing their business impact. Most marketing executives I talk to are still not getting the insights they want. So one of the big initiatives many of them are undertaking is the development of what we, at Origami Logic, call a Marketing Intelligence solution. This is a system that takes all of the diverse data in marketing and brings it together to create a centralized hub of marketing data. This hub is complemented with reporting, visualization and exploration tools that make it easy for marketers to drive decisions.

In the last post, I introduced the different approaches I see organizations taking to develop a marketing intelligence solution:

  1. Use spreadsheets
  2. Use generic business intelligence tools
  3. Use an enterprise data hub

In this post, I am going to focus on the generic business intelligence (BI) tools approach. After typically starting their Marketing Intelligence efforts by consolidating data into spreadsheets, large brand marketing organizations soon realize that it doesn’t make sense to rely on such manual (and error prone) measurement and reporting efforts. So in order to automate and evolve their efforts, it is becoming increasingly common for organizations to try to build a Marketing Intelligence solution based on generic BI and visualization tools and a data warehouse/data cubes.

When I talk to organizations who are taking this approach, without fail, I hear something similar to the following…

  • From marketers who are in the middle of a project: ”We started this project a couple years ago. The project is going way over, in terms of time and budget (in the millions of dollars). We got something initially delivered to us but it turned out to be quite different from what we really need now since the underlying data and our requirements have significantly changed since the project was originally conceived.”
  • From marketers who are using a solution in production: “It’s way too difficult for most of us to use and the system is extremely rigid. Every small change requires significant IT cycles. Ultimately, most marketers don’t use the system very much and instead, they keep asking for ad-hoc reports from IT so we don’t have the ability to work independently and make agile decisions like we hoped.”

Does any of this sound familiar?

Some marketing organizations are trying to use modern, cloud-based BI products for their Marketing Intelligence solution. These products claim to have “out of the box” capabilities specifically for marketing that address the shortcomings of legacy BI offerings. Based on feedback I have received from large brands who have gone down this route, these “modern” products are no different than their older counterparts. While packaged as cloud-based platforms, they still suffer from the same fundamental shortcomings. Projects still take a long time and are still dependent on technical resources (in this case, it is often a vendor, rather than an IT organization).

So, what’s the problem here? Well, if you take a step back and look at what BI tools and data warehouses were originally designed for and then compare that to the current marketing business and data environment, it would be obvious that the two are grossly mismatched. Here are three of the major mismatches:

Generic BI was designed for a very different environment than modern marketing

The BI-centric technology stack — BI and visualization tools, data warehouses (and data marts/cubes), ETL tools (extraction, transformation and load) — was initially designed to deal with highly structured, slowly changing data (like in finance), coming from a small number of internal data sources. The attributes of today’s marketing data environment, however, are vastly different:

  • There is so much volume and diversity of data from the different marketing channels and systems, where each system represents data in a different manner.
  • New data is being added at a fast pace and metrics related to the data need to be updated often. The best example of this is social media messages and the metrics associated with them.
  • The form and nature of the underlying data continuously changes as the marketing platforms keep evolving at a rapid pace. For example, the nature of ads data on Facebook, Twitter or DoubleClick continuously changes as new ad types are introduced.
  • Marketers need to make decisions that span both the left (quantitative) and right (creative) sides of the brain and as such, they need both highly structured performance data and highly unstructured marketing asset/creative data at their fingertips.
  • The questions marketers ask of their data and the way the want to organize it, so it aligns with their business environment, constantly change.

Generic BI relies on an extremely rigid data structure

Generic BI solutions typically access data stored in a data warehouse (or data marts) that is based on relational database technology. Such technology requires a top-down design process that starts with understanding the detailed reporting and visualization needs of end users and then designing a structured data schema that is tightly tied to those needs (and no other needs!). A structured data schema is necessary for two key reasons:

  1. To optimize the performance and scalability of a relational database as it performs queries.
  2. To provide a structured interface between the various building blocks in the BI technology stack. For example, front-end visualization tools need to know the organizational principles (the schema) of the marketing data in a data warehouse in order to read, slice and dice, and visualize the data.

The downside of structured data schemas is that they are very inflexible. Whenever the end user requirements change or new data elements are introduced, the schema has to be redesigned. This takes a while to implement and requires a never-ending investment in technical resources. Unfortunately, in today’s marketing world, end user requirements change often and new data sources and elements are introduced regularly by the marketing platform providers. So, every time Facebook release a new ad unit, or Google upgrades the API for Google Analytics with revised metric definitions, the schema needs to be changed. Or, every time you launch a new marketing campaign or create a new AdWords account, the schema needs to be adjusted. Or, every time there’s a change in your company’s regional, product or organizational taxonomy…well, you get the drift.

Generic BI requires a lot of effort to optimize it for marketing

Possibly the most important factor is that the generic BI technology stack is well, generic. In order to build a solution that meets the needs of marketers, IT resources have to get involved. They need to design the data structure; define how data gets extracted, transformed, and loaded from data sources into a data warehouse; and develop reports and visualizations for the end users. Projects like this are known to take years and millions of dollars to develop.

In addition to the effort required to develop the solution, IT resources also have to maintain the solution. For this type of project, it is always a challenge to keep up with the changing needs of end users but the maintenance requirement for marketing is more significant than other functional areas. This is because the vast majority of marketing data comes from third-party data sources and as described above, the third parties change their APIs and add new data elements regularly. These type of modifications cause a ripple effect on any solution that is built internally.

So, what does all of this mean? The lesson here is that trying to implement a Marketing Intelligence solution using the generic BI technology stack ends up missing the mark on a few fronts:

  1. Timeliness: It’s pretty darn difficult to take generic products, modify them, and then integrate them to build a Marketing Intelligence solution. So brace yourself for a long wait.
  2. Initial development cost: Expect to spend millions of dollars, mostly in personnel costs, to implement the solution.
  3. Ongoing maintenance cost: When the project is delivered, the investment can significantly go down, right? WRONG. You’ve just spent millions of dollars to earn the right to spend millions more. The rapid changes in the modern marketing environment requires continuous development and changes in your Marketing Intelligence implementation. You’re on a treadmill that keeps going faster. So you need to keep running.
  4. Rigidity: The outcome, bound by the fundamental limitations of generic BI tools, is likely to be extremely rigid. This means that every change will take a long time to implement. The rate of change required by the modern marketing environment vastly outpaces the rate of change generic BI-based solutions can absorb, even with the most talented IT organization in charge.
  5. Usability: Marketers need significant analytical flexibility, for both quantitative and creative analytics. For broad adoption and usability across your marketing organization, any solution needs to be super easy and intuitive to use. Marketers CANNOT be bothered to learn SQL, understand data models and schemas, or become Excel and data cube pivot experts. If you took a quick poll in your marketing organization, how many folks do you think will say they just LOVE their BI or visualization tool for its simplicity and utility?

Needless to say, the generic BI technology stack is not a good match when an organization needs a Marketing Intelligence solution. From my perspective, this is a classic case of using old tools to solve a new problem.

In my next post for this series, I am going to address another approach I am starting to see organizations take to implement a Marketing Intelligence solution – the enterprise data hub. I will then conclude the series with an overview of a new type of solution that is now available – vertically-integrated solutions that deliver much of the capabilities marketers need right out of the box.

Next-Gen Marketing Measurement