Making a recurring survey actually useful
How I improved the sample, made the monthly analysis repeatable, and gave teams a better way to work with user sentiment.
Method: Survey design, oversampling, stratified weighting, significance testing, factor analysis, Python, Amplitude, Streamlit, and executive writing.
What we needed to figure out
We needed a reliable way to track the health of the product from the user’s perspective. That sounds simple until the survey has to represent very different kinds of users, run on the same schedule every month, and answer questions from teams across the company.
What I did
I own the recurring study end to end. I gather questions from teams, write and field the survey, run the analysis, improve the methodology, and write the companywide report each month.
What changed after the first wave
The first wave gave us plenty of responses from veteran users and almost no signal from newer users. That sent me back to the drawing board. I increased the fielding volume, oversampled new users, and used stratified weighting to bring the results closer to the actual user population. As the study matured, I added significance testing and factor analysis to the descriptive reporting.
Doing all of that by hand every month was not practical, so I worked with AI-assisted coding tools to build a reproducible Python workflow. I built dashboards in Amplitude for product-insights reporting, too. I also created a Streamlit dashboard because people across the company kept asking for different cuts of the data. It shows trends over time, compares cohorts, creates downloadable views, and includes a simple question-and-answer layer.
Why it mattered
The study became more representative, the monthly work became repeatable, and teams no longer had to wait for me to answer every routine question. They could see how users felt, look at the cohort that mattered to them, and come back to me for the questions that actually needed deeper research.