Verbatim Analysis: What to Actually Do With 40,000 Open-Ended Responses

TL;DR#

Verbatim analysis is the process of turning free-text survey and feedback responses into counted, decision-ready themes — and the biggest mistake CX teams make is treating it as a reading project instead of a sampling problem. You do not need to read 40,000 open-ended responses; a properly stratified random sample of 400 gives you a ±4.9-point margin of error on any theme's prevalence, versus ±0.5 points for the full 40,000, at one-hundredth of the labor. The workflow that actually ships is five steps: name the decision, build a coding frame with explicit inclusion and exclusion rules, sample instead of exhausting, machine-code the bulk after a human-validated codebook exists, then gate the output on inter-rater reliability. Krippendorff's alpha of 0.80 is the accepted threshold for publishable content analysis, with 0.667 as the floor for tentative conclusions. Large language models are now statistically non-inferior to human coders at deductive coding — applying a codebook you already wrote — but a 2026 blinded comparison in PLOS Digital Health found a 12.4% comprehensive error rate on inductive theme generation, which is exactly the step teams are most tempted to hand over. The structural fix is upstream: most verbatim piles are unreadable because nobody asked a follow-up question at capture time.

What Is Verbatim Analysis?#

Verbatim analysis is the systematic coding of open-ended, free-text customer responses into a defined set of themes so those themes can be counted, tracked over time, and tied to a decision. It differs from sentiment analysis, which assigns polarity (positive/negative/neutral) without explaining what the customer was talking about, and from generic text analytics, which surfaces frequent terms and phrases without committing to a stable category scheme. Verbatim coding is the labeling step inside verbatim analysis: one coder, one response, one or more codes from an agreed frame.

Why Open-Ended Response Analysis Stalls at Scale#

Open-ended response analysis stalls because the effort scales linearly with volume while the insight scales logarithmically — and nobody budgets for the linear part. Careful coding of a short verbatim against a 30-code frame takes roughly 20 to 40 seconds once a coder is trained. At 30 seconds per response, 40,000 verbatims is about 333 person-hours, or more than eight work-weeks of uninterrupted labeling for a single analyst. Double-code it for reliability and you are at four months.

So it does not get done. The verbatims sit in an export, the quarterly deck reports the score, and the free-text column becomes a quote mine — someone searches for a memorable line, pastes it onto slide 14, and the other 39,997 responses never influence anything. That pattern is the single most common failure we see in customer experience data sources, quality, and the gaps that break analysis, and it is why so many dashboards report movement without explanation. The metric layer is fine; the why behind the numbers is stranded in a text field.

There is a quieter problem too. Item nonresponse is consistently higher for open-ended questions than for closed-ended ones, and the people who do write something are not a random subset of your customers — they skew toward the extremes. A 40,000-row verbatim pile is not a census of customer opinion. It is a self-selected sample with unknown bias, which is another argument for treating it statistically rather than exhaustively.

Step 1: Decide the Decision Before You Read Anything#

The first step in verbatim analysis is writing down the decision the analysis will inform, in one sentence, before a single response is read. Not "understand customer sentiment." Something falsifiable: "Decide whether the Q3 onboarding rebuild should prioritize data import or permissions setup." Or "Decide whether the 6-point NPS drop in the enterprise segment is about pricing or support responsiveness."

This matters mechanically, not philosophically. The decision determines the unit of analysis (response? customer? account?), the population you sample from (all responses, or only detractors in the affected segment?), and the granularity of the coding frame. A frame built to answer "what should we fix first in onboarding" needs 8 to 15 codes at feature-level granularity. A frame built to feed a CX scorecard for the board needs 5 or 6 codes at theme level, stable across quarters. Building one and using it for the other is where most projects go sideways.

Write down, alongside the decision, what result would change the answer. If a theme appearing in 12% of responses would trigger a roadmap change but 4% would not, you have just specified your precision requirement — which sets your sample size in Step 3. Teams that skip this end up doing driver analysis on categories nobody can act on.

Step 2: Build a Verbatim Coding Frame That Survives Contact With the Data#

A coding frame survives contact with data when every code carries a definition, an inclusion rule, an exclusion rule, and two real example verbatims — and when it was built iteratively on samples rather than invented in a meeting.

The best public example of the process comes from the Pew Research Center, whose coding methodology for a 2026 global open-ended study documents the whole pipeline: six researchers open-coded random samples to map concepts, drafted a preliminary codebook, then tested it across four iterations on random samples of at least 120 responses each before declaring it stable. The final frame held 30 codes and cleared a Krippendorff's alpha of at least 0.70 for every one of them. Only then did coding begin on the full set of 29,810 substantive responses across 36 languages.

Copy that structure. A workable frame for a CX verbatim pile looks like this:

Two rules of thumb. First, watch the residual bucket: if other_uncategorized exceeds roughly 15% of coded responses, the frame is under-built and you should run another iteration rather than pushing through. Second, freeze the frame before production coding starts. Mid-stream edits are the primary cause of theme drift — the same code meaning something different in March than it did in January, which quietly destroys any trend line you were hoping to build. If the frame genuinely must change, version it, note the effective date, and re-code a sample of the earlier batch so the series stays comparable. This is the same discipline that keeps customer experience benchmarking honest instead of self-flattering.

Step 3: Sample Properly — Why 400 Verbatims Beat 40,000#

Sampling beats exhaustive coding because precision improves with the square root of sample size, so the last 39,600 responses buy you almost nothing. For a theme with prevalence near 50%, a random sample of 400 yields a margin of error of roughly ±4.9 points at 95% confidence. A sample of 1,000 gets you to ±3.1 points. Coding all 40,000 gets you to ±0.5 points — a hundred times the labor for 4.4 points of precision you will never use, since no roadmap decision hinges on whether a theme is at 31% or 33%. The Pew Research Center's own explainer on the variability of survey estimates is a good primer on why that curve flattens so fast, and on why margin of error only captures random sampling error in the first place.

Rare-theme detection is the one legitimate objection, and it also resolves in favor of sampling. A theme present in 1% of responses has about a 98% chance of appearing at least once in a random sample of 400, and would show up roughly four times. If your decision depends on catching a 0.2% signal — a safety issue, a compliance complaint, a specific security objection — do not solve that with a bigger coding sample. Solve it with a targeted keyword sweep across the full pile for that specific risk, run separately from the thematic analysis.

Practical sampling checklist:

1. Stratify before you randomize. Draw proportionally within segment, channel, and month. An unstratified sample of a pile dominated by one high-volume touchpoint tells you about that touchpoint, not your customer base.
2. Set the size from Step 1's precision requirement, not from what feels thorough. 300–500 is the right order of magnitude for most CX decisions.
3. Draw a fresh reserve sample of 100 for the validation step, and never let a coder see it during frame development.
4. Record the draw. Seed, date, filters. Someone will ask in six months whether the trend is real or a sampling artifact.
5. Re-sample rather than top-up when you need more precision. Topping up a sample after seeing early results imports your expectations into the data.

Qualitative research reached the same conclusion from the other direction decades ago. Guest, Bunce, and Johnson's landmark study on how many interviews are enough found basic thematic saturation arriving within the first 12 interviews, with the most prevalent themes evident within six. Later work pushed the number up for harder cases — Hagaman and Wutich (2017) found 20 to 40 interviews were needed for metathemes across multiple sites, and Hennink, Kaiser, and Marconi (2017) put meaning saturation at 16 to 24 — but the shape of the finding is consistent. New themes stop arriving long before your data does.

Step 4: Machine Coding — Where LLMs Help and Where They Drift#

Large language models are reliable at applying a coding frame you already built and unreliable at inventing one, and that distinction should determine exactly where you deploy them. A 2026 blinded mixed-methods comparison published in PLOS Digital Health tested three models against human analysts on thematic analysis in healthcare research. On deductive coding — applying a pre-defined codebook — the models hit 93.5% agreement (95% CI 92.5–94.5) against a human benchmark of 92.7%, achieved statistical non-inferiority at p < 0.0001, and posted a specificity of 0.98 on descriptive codes. On inductive analysis, where the model had to generate the themes, only one of the three models cleared non-inferiority, and the comprehensive error rate ran to 12.4% (SD 5.1%) even though strict hallucination stayed low at 1.2%.

The qualitative failure pattern in that study is the operationally important part. Models handled explicit, well-documented themes and struggled with latent meaning, interpersonal dynamics, and fragmentation across a customer's account of events — and they coded facilitator speech despite clear labeling. Translated to CX verbatims: an LLM will reliably tell you that a response mentions billing. It will less reliably tell you that a response about billing is actually about a broken expectation set during the sales cycle.

So the division of labor is:

• Humans build the frame (Step 2), on a sample, with the residual bucket monitored.
• Humans code the calibration set — 150 responses is a reasonable benchmark size, matching Pew's coder-training standard.
• The model codes the bulk against the frozen frame, with the definition, inclusion rule, exclusion rule, and both examples in the prompt for every code. Frames compressed to bare labels are the single biggest cause of machine drift.
• Humans adjudicate the residual bucket and any response the model flags as low-confidence or multi-code.

Two further guards. Run the model on the same 150-response calibration set twice, on different days, and check that it produces the same labels — non-determinism is a real drift source, not a theoretical one. And re-run the calibration check every time the underlying model version changes, because a silent provider upgrade will move your trend line without touching your codebook. If you are evaluating tooling for this, the practical differences are covered in our comparison of text analytics for customer feedback and in the roundups of Qualtrics Text iQ alternatives and Qualtrics XM Discover alternatives, where coding-frame control varies enormously between platforms.

Step 5: Validate With Inter-Rater Reliability, Not Vibes#

Validation means computing an agreement statistic on double-coded responses and refusing to publish results that fall below a pre-declared threshold. Percentage agreement is not enough, because two coders assigning the dominant code to everything will agree 85% of the time while carrying no information. Use a chance-corrected statistic: Cohen's kappa for two coders and nominal codes, Krippendorff's alpha when you have more than two coders, missing data, or multi-label responses — which describes most verbatim work.

The thresholds are settled. Published guidance summarized in the K-Alpha methodological literature puts α ≥ 0.80 as the standard for drawing firm conclusions from coded content, 0.667 to 0.79 as acceptable only for tentative conclusions that should be flagged as such, and anything below 0.667 as unreliable. Pew's production standard of α ≥ 0.70 per code is a reasonable working target for a 30-code commercial frame — note that it is per code, not averaged across the frame, which is a much harder test and the right one. An average of 0.80 can easily hide two codes at 0.35 that are functionally interchangeable.

A validation routine that fits inside a normal sprint:

• Double-code the reserve sample of 100 from Step 3. Compute alpha per code.
• Any code below 0.667 gets fixed or merged, not excused. Usually the exclusion rule is missing and two neighboring codes are competing.
• Check reliability every 100 units during production coding rather than only at the end. Coder drift and machine drift both show up as a slow decay, and catching it at unit 300 costs you 200 responses instead of 40,000.
• Report the alpha alongside the percentages. A theme at 22% with α = 0.81 is a finding. The same 22% with α = 0.44 is noise wearing a number, and it will get repeated in a board deck for a year.

This is also what separates a defensible verbatim program from an impressive-looking one. When you present customer experience analytics examples that changed a decision, the reliability figure is what lets a skeptical exec trust the theme counts — and what keeps the reporting cadence from degrading into anecdote circulation.

The Five-Step Verbatim Analysis Workflow at a Glance#

Total elapsed time for a 40,000-response pile using this workflow is typically two to three weeks, most of it in Step 2 — versus the four months that exhaustive double-coding would take. Teams running this as a standing quarterly process usually fold it into a broader voice of customer program blueprint so the frame and the reporting calendar stay in sync.

The Structural Fix: Ask the Follow-Up at Capture Time#

The reason verbatim analysis is so expensive is that the data arrives ambiguous, and the fix is to resolve ambiguity at capture rather than at coding. Consider what you are actually doing in Steps 2 through 5: building elaborate machinery to guess what a customer meant, months after they typed 14 words into a box that could not ask them a single question back.

A response like "support was slow" forces a coder to choose between three or four plausible codes — first-response time, resolution time, channel availability, agent competence — with no way to know which one the customer meant. Multiply that across 40,000 responses and a large share of your coding disagreement is not coder error at all. It is irreducible ambiguity baked into the collection instrument. No coding frame, no reliability statistic, and no model can recover information the form never collected.

This is the case for conversational capture. Perspective AI runs AI interviewers that ask the follow-up in the moment — "slow in what way? How long did you wait, and what were you trying to get done?" — so the response arrives with the disambiguation already attached. When the "why" is captured at source, three things change in your verbatim workflow: the residual bucket shrinks because responses self-classify, inter-rater reliability rises because there is less to interpret, and the analysis moves from guessing intent to counting stated intent. The same mechanic is what makes AI-first platforms structurally different from the survey suites they replace, as covered in what a customer experience platform is and why AI is replacing the survey suite and in the 12 capabilities that separate a CXP from a survey tool.

The practical migration path does not require abandoning your existing program. Keep the score-based instrument for trending, and replace the single open-ended box with a short conversational follow-up on a sample of respondents — the same 300 to 500 you were going to code anyway. Our voice-of-customer survey template and customer journey interview template are both built for that swap, and Perspective AI for CX teams and for research teams shows how the coded output feeds an existing reporting stack. Teams running this pattern also find closing the voice-of-customer loop gets dramatically easier, because a follow-up-rich transcript already contains the specific fix the customer wanted.

Frequently Asked Questions#

What is verbatim coding?#

Verbatim coding is the act of assigning one or more predefined theme labels to an individual open-ended response. It is the labeling step inside the broader verbatim analysis workflow, and it depends entirely on the quality of the coding frame — each code needs a definition, an inclusion rule, an exclusion rule, and example responses. Without exclusion rules, neighboring codes blur together and reliability statistics collapse.

How many open-ended responses do you need to analyze?#

For most CX decisions, a stratified random sample of 300 to 500 responses is sufficient. At 400 responses, the margin of error on a theme's prevalence is roughly ±4.9 points at 95% confidence, and a theme present in just 1% of the population has about a 98% chance of appearing in the sample. Coding all 40,000 improves precision to about ±0.5 points — a gain no roadmap decision requires.

Can AI do verbatim analysis on its own?#

No — AI is reliable at applying a coding frame but not at creating one. A 2026 blinded comparison in PLOS Digital Health found large language models statistically non-inferior to human analysts on deductive coding, at 93.5% agreement, but showed a 12.4% comprehensive error rate on inductive theme generation, with specific weakness on latent meaning. The reliable pattern is humans build and validate the frame, the model codes the bulk, humans adjudicate the residual.

What inter-rater reliability score is good enough for verbatim coding?#

Krippendorff's alpha of 0.80 or above is the accepted standard for firm conclusions, 0.667 to 0.79 supports only tentative conclusions, and below 0.667 is unreliable. Apply the threshold per code rather than averaged across the frame, because an acceptable average routinely conceals two or three interchangeable codes scoring near 0.35. Report the alpha alongside every theme percentage you publish.

What is the difference between verbatim analysis and sentiment analysis?#

Verbatim analysis identifies what customers are talking about; sentiment analysis identifies how they feel about it. Sentiment assigns polarity without explaining the subject, which is why sentiment-only dashboards show a score moving without indicating a cause. The two are complementary: code the theme first, then layer sentiment within each theme to see which topics are driving negative feeling.

How do you stop themes from drifting between quarters?#

Freeze and version the coding frame before production coding, and re-code a sample of the prior period whenever the frame changes. Theme drift happens when a code's working definition shifts silently while the label stays the same, which invalidates any trend line built on it. Running reliability checks every 100 coded units catches both human coder drift and model drift before they contaminate the full dataset.

Conclusion: Verbatim Analysis Is a Decision Process, Not a Reading Project#

The 40,000-response backlog is not a labor problem to be brute-forced or a text-analytics feature to be purchased. Verbatim analysis works when you name the decision first, build a coding frame with real exclusion rules, sample 400 instead of exhausting 40,000, use machine coding for application rather than invention, and gate every published theme on a per-code reliability threshold of 0.70 or better. That is a two-to-three-week process, and it produces numbers you can defend in a board meeting rather than quotes you can decorate a slide with.

Then fix the upstream cause. Most of the ambiguity you are paying coders and models to resolve was created the moment a form accepted 14 words and asked nothing back. Start an interview with Perspective AI to see what a verbatim looks like when the AI interviewer probed the vague answer in real time — or browse live studies to see how the coded output reads when the "why" was captured at the source instead of reconstructed six months later.