Analysis & synthesisOpen-response analysisCo-occurrence networksKH CoderGenerative AILarge text datasets
Text Mining and AI-Assisted Analysis
Text Mining and AI-Assisted Analysis
Also known as: Quantitative text analysis、KH Coder、Natural language processing、NLP
Processes text as words or documents to examine frequency and co-occurrence patterns. Record units and preprocessing and interpret findings against source texts. [1]
- Difficulty
- Intermediate
- Cost
- Low
- Duration
- Planning example, not a universal standard duration: 1–4 weeks for preprocessing, analysis, interpretation and validation
Content updated: 2026-09-13Source checked: 2026-09-13 (includes AI cross-checking)No documented human reviewIncludes fictional examples
How it works at a glance
Extract words and define treatment of spelling variants, stop words and compounds. Co-occurrence means appearing in the same sentence or document; units and word selection alter diagrams. [1]
Frequent words or networks alone cannot establish meaning. This site's operational recommendation requires source comparison, confidentiality handling and revision records even when generative AI assists classification or summaries. [1]
- Prepare and preprocess text
- Tokenize and adjust word units
- Review frequent words and stop words
- Analyze co-occurrence networks and clusters
- Integrate coding where needed
- Interpret and verify manually
When to use it and when it is unsuitable
When to use it
- When several hundred or more open responses make full manual thematic or content analysis difficult within available time
- When comparing vocabulary or theme changes across years
- When examining themes or keyword distributions in large interview or meeting datasets
- When using generative AI for coding assistance or draft summaries with evaluator verification and correction
When it is unsuitable
- When volume or vocabulary variation is inadequate for the purpose. Do not apply uniform document thresholds; consider qualitative analysis where originals remain manageable.
- When deep interpretation of context, nuance or contradictions matters more than word frequencies. Surface pattern extraction cannot replace interpretive qualitative analysis.
- When data cannot be handled with appropriate permissions and protection. Sending data to external AI is not required for text mining.
Participants, roles, and deliverables
Participants and roles
Staff handling preprocessing, aggregation and source checks, understanding software units and settings and data-use conditions
Main deliverables
- Frequent words and vocabulary statistics
- Co-occurrence network with clusters or subgraphs
- Text-mining-assisted codebook linking words and themes
- Methods and settings: preprocessing, software versions, etc.
Implementation steps
- Prepare and preprocess textFormat open responses, transcripts and other text as CSV or plain text. Use UTF-8 in KH Coder. Remove irrelevant URLs, symbols or boilerplate.
- Tokenize and adjust word unitsRun KH Coder morphological analysis, such as MeCab, to segment words. Register proper names or compounds, such as “program evaluation” as 1 word, as forced extraction terms. Set part-of-speech filters, such as nouns, verbs and adjectives.
- Review frequent words and stop wordsCheck frequencies and exclude analytically irrelevant frequent function words, such as “think,” “exist” or “do.” Remaining words provide an overview of what is discussed.
- Analyze co-occurrence networks and clustersUse measures such as Jaccard coefficients to analyze co-occurrence and generate word networks. Examine clusters, meaning groupings, to identify themes relevant to questions.
- Integrate coding where neededLink candidate themes to originals through KH Coder coding, or use frequent words and patterns to support thematic or content analysis codebook development.
- Interpret and verify manuallyText mining is a starting point, not a conclusion. Return to originals to interpret and verify patterns. Critical evaluator interpretation of tool-generated themes is essential.
Strengths and limitations
Strengths
- Aggregates words and patterns across documents to guide original-text examination.
- Visualization reveals overlooked themes and unexpected patterns.
- KH Coder is free, Japanese-compatible and established academically, and practitioners can learn it.
- Supports preliminary exploration for thematic or content analysis and improves efficiency.
Limitations
- Word frequencies and co-occurrence cannot fully capture context, irony or implied meaning; original-text checks are necessary.
- Interpretation needs qualitative analysis knowledge; tool-generated networks cannot simply be reported unchanged as findings.
- LLM coding assistance has unresolved reproducibility, verifiability and privacy issues.
- Morphological settings, stop words and minimum frequency alter results, requiring transparent records.
Common pitfalls
- Presenting a network without interpretation. Diagrams begin analysis; explain clusters' meanings and relevance to questions.
- Entering personal or confidential data into generative AI. Direct cloud transmission of beneficiary responses risks privacy or confidentiality violations. Anonymization and organizational rule checks are mandatory.
- Using unverified AI outputs as findings. LLM coding can hallucinate false content; source comparison and human verification are essential.
Conditions of use and evidence
- Networks depend on co-occurrence units and word or edge selection, so check settings and originals. [1]
Check different difficulties expressed by “time”
A fictional illustrative extraction of “time” from invented open responses.
Scroll horizontally to view the table
| Fictional original | Unknown from counts alone | Contextual interpretation |
|---|---|---|
| Not enough time, so stopped entering data | Time for what? | Application burden |
| Had more time available | Problem or outcome? | Change after support |
| It did not take time. | Negation handling | Experience of low burden |
Common errors in judgment
Common mistake
Reading co-occurring words as causation
How to revise
Treat them as occurring within the same unit.
ReasonCo-occurrence does not establish causal direction.
Common mistake
Accepting AI theme names without review
How to revise
Check source evidence and counterexamples for each theme.
ReasonSummaries may omit negation or minority voices.
Organize your thinking by writing
Text mining implementation and validation worksheet
A practical template is available to fill in and print on A4 paper.
Open worksheetRelated methods
References and checked sections
- KH Coder 3 Reference ManualKoichi Higuchi
- Section consulted
- Part A: Co-Occurrence Network of Words
- How it relates to the text
- Networks depend on co-occurrence units and word or edge selection, so check settings and originals.
Editorial and checking information
How to cite this page
EvalCompass Editorial Team. “Text Mining and AI-Assisted Analysis”. EvalCompass, updated 2026-09-13. https://evalcompass.jun-nakatani.com/en/methods/text-mining/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.
Change history
- — Checked sources for definitions and applicability conditions, and added fictional examples, reasons for judgments and examples for every worksheet field. Revised unverified claims about history and adoption. Also reviewed existing procedures and cautions, and adjusted fixed participant counts and thresholds and generalizations about institutions.