· Andrew Le Breuilly · 5 min read · Rethinking Survey Data — Part 4
Bronze Silver and Gold for Survey Data
A medallion architecture separates raw survey inputs, governed transformations and analysis-ready outputs into clear, auditable layers.
When something looks wrong in a set of survey results, three questions tend to follow quite quickly:
- What did we originally receive?
- What did we change?
- Which version did we actually use?
In a traditional project folder, the answers may be spread across an SPSS file, several syntax scripts, an Excel workbook and a collection of filenames containing words such as “final”, “final2” and “final_revised”.
Most of us have encountered some version of that folder.
A Bronze, Silver and Gold architecture gives us a cleaner way to organise the journey from source data to approved output.
What do the colours mean?
The terminology comes from the medallion architecture used in modern data engineering. Databricks describes it as a series of progressively refined data layers.
Applied to survey data, the model is straightforward:
- Bronze preserves what arrived
- Silver contains the governed processing
- Gold provides data ready for analysis or delivery
The names matter less than the separation. We want a clear boundary between the source, the work we perform and the outputs people consume.
Bronze answers “What did we receive?”
The Bronze layer contains the survey data as it arrived, before cleaning, recoding or reshaping.
That could be an SPSS file, an API response or another supported input. The key principle is that the original data and metadata are preserved.
Why keep an unchanged source when we already know we need to transform it?
Because transformation rules change. Mistakes happen. New analytical questions emerge. If we retain the source, we can rerun the pipeline without trying to reverse-engineer an already processed file.
Bronze also helps us distinguish between a source-data problem and something introduced during processing. If a label is missing or a value looks odd, we can check what the collection platform actually supplied.
This layer is not designed to be friendly for everyday analysis. Its job is to be faithful.
Silver answers “What have we done to it?”
Silver is where the real survey processing takes place.
Typical steps might include:
- Cleaning text and labels
- Fixing encoding issues
- Normalising multi-response questions
- Restructuring loops
- Creating respondent attributes
- Applying recodes and derived variables
- Linking code frames and reference data
- Creating and validating weights
The important thing is that these are explicit steps in a pipeline rather than a collection of changes hidden inside files.
If we create a derived audience, the definition should be visible. If we clean a text field, the rule should be repeatable. If a new wave arrives, we should be able to apply the agreed logic again.
Silver is also where we can inspect intermediate results. When a value changes unexpectedly, we can identify which transformation produced it instead of staring at the final output and guessing.
Gold answers “What is ready to use?”
Gold contains approved, analysis-ready data.
For one team, that might be an SPSS file. For another, it could be Parquet tables feeding a dashboard or a dimensional model inside a corporate warehouse. The same pipeline can produce several Gold outputs for different audiences.
This is worth emphasising: Gold does not mean one universal file that everybody must use.
It means that each output has been produced from the same controlled source and processing logic. Researchers, BI developers and data teams can receive structures suited to their work without separately rebuilding the survey.
The boundaries make governance practical
Without these layers, source data, temporary calculations and final outputs easily become entangled.
A manual correction can end up inside the only remaining copy of a file. A calculation created for one dashboard can quietly become the accepted definition. Two analysts can build different versions of the same net without realising it.
The layered approach gives everyone a shared language:
- Check Bronze to see what arrived
- Check Silver to understand the transformation
- Check Gold to see what was approved for use
That makes conversations about quality much more concrete. It also helps teams decide where checks, approvals and documentation belong.
It is especially useful for trackers
Repeat studies are rarely as repeatable as their name suggests.
Questions change, markets are added and client requirements evolve. A layered pipeline allows each wave to retain its own source while passing through a consistent processing structure.
Changes between waves become visible rather than disappearing into the latest export. If the definition of a measure changes, the transformation can be versioned and the affected outputs regenerated.
Every new wave can benefit from the work completed on the previous one.
Automation needs this kind of clarity
People sometimes think of automation as taking an existing script and scheduling it. That can save time, but it does not necessarily create a reliable process.
Once ingestion, transformation, checking and delivery are separated, we can automate them with much more confidence. New data can enter Bronze, pass through the approved Silver steps and generate Gold outputs on a schedule.
Human review still matters, particularly when questionnaire changes or unusual responses require judgement. The difference is that people can focus on those exceptions rather than repeating routine handling.
The previous article explains the survey data model sitting underneath these layers.
Next, we will look at how reusable processing improves delivery and reduces dependence on individual team members.
