Co-authored by Tali Dolev and Siamack Sanaie
These days, every marketing team is looking for ways to collect and analyze the vast amounts of data generated by dozens of different channels and sources. When a marketing team has grown to the point where their performance measurement needs can no longer be satisfied by basic analytics tools, they have two choices: find and buy an advanced marketing intelligence platform or build their own. Many companies currently face the buy vs build dilemma, and even though we would tell you that the buy option is optimal, some companies have decided to build. For those companies, no matter how skilled their IT team is, they’ll run into some roadblocks when building connectors.
As an engineer, I understand the challenges many IT teams go through during the build process. Our team at Origami Logic has built hundreds of connectors that pull marketing metrics from channels and sources like Facebook, Twitter, Double Click, AdWords and Google Analytics just to name a few. This was no easy task, but our team of engineers overcame a number of obstacles that IT teams will face while building their own connectors. We feel your pain and we want to share our insights with you.
Here are 8 common roadblocks faced when building connectors.
1. Finding the API can be a challenge
When building your connectors, the first step is to find the appropriate API and build that into your system. Unfortunately, this first step isn’t always the easiest. Even though many platforms like Facebook and Twitter make it easy to find and implement their native APIs, not all data channels do the same.
The challenge is to find the API. Normally, a quick Google search should reveal any documentation or helpful support pages where you can either find the API or be told that it doesn’t exist. Again, not all data channels and sources adhere to the same level of organization when listing API documentation. Occasionally, you will need to have a direct contact at the channel in order to get access to the API documentation.
- Search developer forums associated with that platform and ask a fellow developer where you might find the API
- Reach out to the developer support team
- You might want to get familiar with Excel. If the channel does not have its own API, you’ll be importing that data in the form of CSV files
Congratulations! You’ve found the API and you’re ready to collect some data. Now that you’ve found the API, the challenges don’t stop there. You’re going to be looking at a few roadblocks that can slow down the process of making it work.
2. Accessing an account is not a simple task
For most platforms, an account is the entry point to the API. Creating an account can be troublesome and costly. It may require working with account managers and paying for the account and even the API access. You can either create an account of your own (which can be costly) or reach out to marketing for their credentials. Either way, you will not get far without an account.
- Enlist the help of the developer support team
- Reach out to marketing for account information if available
3. API Calls do not work without data
Even after you’ve accessed the platform and have set up your API, you can’t get started unless there is data to work with. Without data in the account, your APIs have nothing to request. This leads to a number of errors and issues that will keep you from testing API calls successfully.
- Use a Sandbox account to test your APIs (unfortunately, this is not always available)
- Populate the account with marketing data before starting API calls
4. Documentation isn’t always up-to-date
When developing code for the APIs, we usually rely on the documentation provided by the platform. The problem is that not all of these channels provide up-to-date or even correct documentation. Any discrepancies in the documentation can affect your code and create error messages.
- Enlist the help of the developer support team
- Create an error message catch-all to help in the debugging process
5. The API is always changing
Like any software, these native APIs are always being updated and changed. The problem is that many of these sources do not provide timely notice to engineering teams. When changes occur in the APIs that the engineering teams aren’t aware of, error messages occur and things come to a full stop. This is a challenge that engineering teams face regularly.
- Sign up for all developer email alerts. These alerts will help keep your team ahead of updates and your code fresh
- Maintain a friendly relationship with platform support. Having a contact on the support team can be beneficial when you need to know about any major changes to the API
6. Rolling API changes require extra programming
Another challenge with API updates and changes is the gradual roll out of changes over an extended period of time. These gradual changes require engineering teams to start preparations for the new versions of the API while supporting the older version of the API that is still in use. Essentially, engineers need to develop code for the latest API and make it backwards compatible.
- Stay up to date on API related news
7. API quota limits
One of the biggest challenges engineering teams face are API data request limits. Every API has them and they’re often times different from platform to platform. These API request limits can usually be found within the API documentation but should be double checked for accuracy. Some APIs create limits based on number of days, hours, 15 minute intervals or rolling windows. Your team should get very familiar with these limits and build safeguards to prevent reaching them.
- Keep a detailed list of API quota limits and the time frames that they operate in
- Schedule your API calls based on quota limits
- A simple retry with a delay can do the trick.
- Build a mechanism which takes into account the rate limit for each connector and schedules the API calls according to the rate limit.
8. You can’t always retrieve historical data
One API challenge that should be noted is that some sources will not allow the retrieval of historical data. This means that if you’ve missed your window to collect the marketing data from that source today, you will not have an opportunity to retrieve that data again.
- Build a tool to detect gaps in data, so that you can fetch missing data as soon as it’s discovered.
- For data that is missing and cannot be retrieved: interpolate new data points from the data points which have been correctly fetched.
- Use statistical methods to fill in the gaps. The simplest method is linear, but there are other statistical methods to estimate missing values from a set of data points.
There are a lot of challenges that lie ahead for engineering teams looking to build their own marketing analytics platforms. These were just a few of the roadblocks teams will face, but hopefully these insights will help companies sidestep some of the more common challenges of developing connectors.