Why Survey Data Is Still an Outlier in the Modern Data Stack

If you work with survey data, this will probably sound familiar.

The fieldwork closes. The data arrives. Then somebody exports an SPSS file or CSV, opens up an old script and starts working through the same processing steps they used last time.

Perhaps the script runs perfectly. More often, something has changed. A question has been added, an answer code has moved or the questionnaire has been structured slightly differently. Before anyone can get to the analysis, they first have to work out what the data means and how this particular version needs to be handled.

We have made huge advances in how surveys are designed and delivered. But once the responses have been collected, much of the underlying workflow would still be recognisable to a researcher from 20 years ago.

Why has survey data been left behind?

The survey experience has changed. The data workflow has not.

Survey platforms have become much better at collecting respondent input. We can build mobile-first questionnaires, use complex routing and create far more engaging experiences than the static forms of the past.

That is good progress. The problem is that most innovation has focused on the front of the process.

At the other end, survey data is still commonly delivered as an SPSS file or CSV. Both formats have their place, but they usually feed a project-based workflow:

1. Collect the responses
2. Export the data
3. Clean and process it
4. Produce the tables, dashboard or presentation
5. Archive the project

Then the next wave arrives and we do much of it again.

This is not because research teams are resistant to change. It is because survey data genuinely is difficult to represent. Unfortunately, the result is that a valuable source of organisational information is often treated as a temporary project file rather than a reusable data asset.

A survey file is never just a table of responses

At first glance, survey data can look straightforward: one respondent per row and one variable per column.

But a response rarely makes sense on its own. We may also need to know:

• The exact question the respondent was shown
• Which answer options were available
• Whether they could select one answer or several
• Whether routing determined if they saw the question
• How the variable relates to a loop or repeated question block
• Which weight should be used
• Whether the question changed between waves

That context is not supplementary documentation. It is part of the data.

This helps explain why SPSS remains widely used in market research. An SPSS .sav file can retain a dictionary containing variable names, labels, value labels, missing-value rules and formats alongside the response values. IBM’s documentation explains what is held in that file dictionary.

A CSV is easier for almost any tool to open, but much of this context may be lost. If nobody selected a particular option, the response data alone might not tell us that the option existed. It may not tell us whether a blank means “not selected”, “not asked” or “missing”.

So we end up reconstructing the survey from the questionnaire, codebook, processing syntax and whatever notes happen to have survived.

I have spent enough time working with survey exports to know that the difficult part is rarely reading the values. It is preserving their meaning.

APIs help, but they do not create consistency

APIs have made it possible to pull data directly from survey platforms rather than waiting for somebody to export a file. That is an important step towards automation.

However, every platform has its own way of representing surveys, questions, options, responses and metadata. Connecting to one API does not necessarily give you a reusable method for connecting to another.

Even two projects from the same platform may need different downstream treatment because their questionnaires were built using different conventions.

The Transfer Survey API project, usually known as TSAPI, is trying to address this by defining an open standard for transferring survey data. It is a valuable initiative, and the Market Research Society has endorsed TSAPI as an industry standard. Adoption, however, remains rare.

For most research teams, the practical reality is still a collection of platform-specific APIs, exports and quirks.

Meanwhile, the rest of the data world moved on

Over the past decade, organisations have invested heavily in cloud data platforms, automated pipelines and self-service analytics.

Sales, marketing, finance and product data can be collected continuously, transformed through governed processes and made available across tools such as Tableau and Power BI. Analysts expect to work from shared data rather than asking somebody to rebuild an extract for every new question.

Survey data often sits outside that environment. It goes through its own specialist processing route and ends up in a separate dashboard, report or archive.

There are good historical reasons for this separation, but that does not mean we should preserve it forever.

The wider modern data stack was not designed around the complexities of survey research. As we will explore in the next article in this series, general-purpose ETL tools can move and transform data, but they do not automatically understand questions, codebooks, multi-response structures, universes or weighting.

We need to bridge those two worlds.

What would first-class survey data look like?

The answer is not to throw away SPSS, abandon specialist research techniques or insist that every analyst becomes a data engineer.

It is to take the useful engineering practices applied to other business data and adapt them to the realities of surveys.

That means:

• Preserving the original response data and metadata
• Converting different survey sources into a consistent structure
• Recording transformations as visible, reusable steps
• Separating raw inputs from analysis-ready outputs
• Making the finished data available to statistical, BI and warehouse tools
• Keeping the structured result available after the immediate project ends

This changes the destination of the workflow.

Instead of ending with a set of archived files, it ends with a governed data asset that can support the next wave, the next dashboard and questions we have not thought to ask yet.

That is the idea behind Purify Intelligence: survey data should work like the rest of the modern data stack without losing the context and rigour that make it valuable in the first place.

In the next post, we will look more closely at why generic ETL tools struggle with survey data and what a survey-specific processing layer needs to do differently.