
Survey Platform Integration – Notes From the Coalface
By codeit
- article
- Integrated Data
- Survey Software
- APIs
- Data Connectivity
- Survey Data Centralisation
- Survey Research
Introduction
Do you wish your survey tools could work more seamlessly together? Do you waste too much time exporting and importing data between systems? Wouldn’t it be great if you could connect mainstream tools, like Tableau or PowerBI directly to your survey platform?
Recently, Matt Gibbs wrote an article explaining that connecting systems together like this requires a lot of custom development. This development takes too much time and money and is therefore a barrier to innovation in the Market Research industry.
He wrote about the need for standardization in the way that survey software products talk to each other. In particular, how TSAPI sets out to be that standard.
I’ve worked on many integrations with survey systems during my time developing codeit, so I know first hand the pain involved. In this article, I’d like to dig a bit deeper into this to examine the reasons and barriers that make this kind of development so frustratingly difficult.
But first, I’d like to take you on a trip back in time…
A History Lesson
Cast your mind back, if you’re old enough, to 1994. The Channel Tunnel has just started operation, Sony have just released the first Playstation and a new sitcom featuring 6 friends has just aired for the first time.
Meanwhile, the Market Research industry has a problem. Transferring survey data between different survey platforms is painful, error prone and time consuming. Each platform exports to a different or proprietary file format which can’t be easily imported into other systems. Huge amounts of effort and cost are spent either manually cajoling files into systems or writing converters to get the data into shape. Back then, a group of people developed triple-s, a file format standard designed to solve this problem and ensure that data can flow seamlessly between systems. To date, over 100+ products have implemented the standard and this has saved the industry from unnecessarily wasting huge amounts of time and money.
Fast forward a few years, and Market Research is moving progressively towards web-based interviewing, software moves more and more online and vendors start offering access to data via APIs rather than just file exports. Since this has happened organically, there’s no standardization and, you guessed it, every API is implemented in a different way. We’re literally back to square one – the same problem from 1994 has returned. Those who cannot remember history are doomed to repeat it, as the saying goes.
Except now the problem is much, much worse. Back in 1994, if you were given a file to import it would be a pain, but as it’s just a stand-alone file you’d probably figure it out eventually. In the world of APIs there are a whole new set of extra barriers in the way. I’d like to elaborate on some of these to help illustrate how the TSAPI standard solves them.
Barriers
1. The Expert Trap
Put simply, in order to develop an integration with a survey platform, I need to become an expert user of it. I need to learn every question type to make sure I’ve got everything covered. I need to learn how to script and deploy a survey, so I can test everything thoroughly. Does it support loops? How do grids work? What about multiple response questions? How are completes defined? and so on…
With hundreds of survey platforms out there, I will be long dead before I manage to become an expert in all of them. Probably killed under an avalanche of training manuals! Of course, I could take a short-cut and become a partial-expert: learn just enough to get by. A little knowledge is a dangerous thing – sooner or later, something unexpected will come out of the woodwork and my integration will break.
The TSAPI standard solves this problem. I don’t need to become an expert in any survey platform. I don’t need to know anything really about any platform. I can just develop a single connector against the standard and know that it will work against all platforms that implement the standard, in one fell swoop.
2. Cooperation and Access
Currently, in order to create an API connector you need access to that API for development and testing. A lot of software companies are very unwilling to cooperate and provide access for third parties. Often this is because they consider API access an “Enterprise” feature (a laughable attitude in my opinion, in this day and age) or because they just don’t want the inevitable support hassle (see point 3 below) that will come with it.
The TSAPI standard solves this problem. I don’t need access to. I can develop a compliant connector and know that it will work with all other compliant systems.
3. Support
APIs seem to be like fingerprints or snowflakes – each one is different in its own unique way. As a developer this means you’re starting from scratch every time, and you’re going to get stuck. This means I need to start hassling the survey platform support team. “Why isn’t this working?”, “What am I doing wrong?”, “What does this setting do?”.
Sometimes the API does such weird and wonderful things that the developer needs a 1:1 tutorial in order to understand how it all works. Clearly, this is not scalable and creates hassle and work for everyone involved.
The TSAPI standard solves this problem. I don’t need support from platform vendors. Once I’ve implemented a TSAPI compliant connector I know it will automatically work on all systems without the need to bother support and take up precious resources.
4. Bad Design
In writing this, my goal is not to API-shame any companies out there. This is an industry-wide problem and so nobody is blameless.
Seemingly, all APIs have their good points, but in my experience, there’s always some kind of horror lurking in there which makes it a real pain to develop against. As a developer, you end up forced into writing hacky code, to work your way around the quirks, inconsistencies, dodgy design and bugs(!) in the API you’re targeting.
This is an inevitable consequence of the way software products evolve. An API gets bolted on, it grows organically, systems become dependent on it and then it’s legacy so you’re stuck with it and you can’t change it.
The TSAPI standard solves this problem. It provides good, peer reviewed design out of the box. Software companies do not need to change their existing APIs. The TSAPI standard only requires that they add three simple new methods to their existing API. Most software companies could add these in a matter of days (it took us about 5 days to implement it in codeit).
Conclusion
As an industry, we’re experiencing a disappointing déjà vu – systems can’t talk to each other which is a serious dampener on innovation. The solution is for us to adopt a standard so that systems can start speaking a common language.
As an example of this in action, consider the Open Banking initiative. This has allowed financial data to be shared between banks and third parties using APIs in a way that was never possible before. This initiative has then led to a new wave of FinTech innovation, competition and customer empowerment. Of course, change like this requires pressure from the top, and so the onus is on everyone in our industry to apply that pressure and demand that TSAPI be adopted industry-wide.
So, if you’re a user of survey software, help spread the word and tell your vendor you’re interested in TSAPI and the benefits it can bring to the industry. Ask them whether they’ve got plans to implement it and if not, why not? If you’re a software vendor, take a look at the TSAPI website, learn about the standard and start spreading the word internally in your organization.