Exploring vocabulary diversity

Analysis snapshot

Analysis date: October 7, 2026.

Findings

Among the 12 writers with at least 1,000 token occurrences in our exported data, Baba Sheikh Farid had the highest average number of distinct tokens in a random 1,000-token sample: 700.47. Guru Nanak Dev (649.31) and Bhagat Kabir (647.02) followed. Among the six Gurus represented in this comparison, Guru Nanak Dev had the highest average.

This answers a specific question: how many different token strings should we expect when randomly drawing 1,000 occurrences from each writer's recorded text? It does not establish who had the greatest overall vocabulary or literary merit.

The earlier whole-text counts gave Guru Arjan Dev the largest recorded vocabulary (17,745 distinct tokens), followed by Guru Nanak Dev (11,906). Those counts measure the complete exported vocabulary rather than diversity at an equal sample size. The two rankings therefore answer different questions.

Data and token definition

The source is the local SQLite cache, cache/guru-granth-sahib.sqlite, with Punjabi text stored in lines.punjabi and author attribution from the authors table. The sampling analysis reads the previously exported per-author frequency files in outputs, using outputs/author-unique-words.tsv to identify writers. It does not re-query SQLite or independently verify the exports against it.

A token is a nonempty whitespace-separated string, as produced by Go's strings.Fields. A unique token is an exact distinct string. Punctuation, verse-number tokens, and punctuation attached to words are retained intentionally. There is no spelling standardization, Unicode normalization, stemming, or merging of grammatical forms. Thus “words” in the output means tokens under this definition.

For each writer, total tokens are the sum of the frequencies in their file; unique tokens are the number of frequency rows. Sampling is weighted by occurrence frequency, not uniformly by unique word: a token occurring 100 times represents 100 selectable occurrences.

Sampling design

• Sample size: 1,000 token occurrences per sample.
• Number of samples: 1,000 per included writer.
• Included: 12 writers, including all six Gurus in the exported data.
• Excluded: 25 writers with fewer than 1,000 occurrences.
• Each sample draws occurrences uniformly without replacement from the writer's complete exported text. Different occurrences of the same token can appear in one sample, but count as only one distinct token.
• The pool resets between samples. Samples can overlap and reuse occurrences.
• There are 1,000,000 selections per included writer, and 12,000,000 overall; these are repeated selections, not that many distinct words or original occurrences.
• Sampling uses Floyd's algorithm over occurrence indices and cumulative frequencies to identify the selected token strings.
• The pseudorandom seed is 20261007, reset for each writer. This makes the current procedure repeatable with the same inputs and implementation; it is not a set of independently seeded streams across writers.

The choice of 1,000 tokens is a practical compromise, not a universally optimal sample size. Including every writer would cap a shared sample at 44 occurrences, the size of the smallest export. Larger samples would exclude more writers.

Mathematics

For writer a, let f(a,w) be the frequency of token w. Then:

N_a = sum_w f(a,w)                    total token occurrences
V_a = number of tokens with f(a,w)>0  whole-text unique tokens
TTR_a = V_a / N_a                    whole-text type-token ratio

Whole-text TTR generally declines as text length increases: repeated common tokens add to the denominator without adding to the numerator. Equal sample sizes reduce this length dependence.

For sample t of n=1,000 occurrences, let U(a,t) be the number of distinct tokens selected. With R=1,000 samples, the reported statistics are:

Mean unique tokens:     mean_a = (1/R) * sum_t U(a,t)
Mean sampled ratio:     mean_a / n
Sample variance:        s_a^2 = sum_t (U(a,t) - mean_a)^2 / (R - 1)
Sample standard deviation: s_a = sqrt(s_a^2)

Because n is the same for every writer, ranking by mean unique tokens and by the mean sampled ratio gives the same order. The implementation calculates the mean and variance incrementally using Welford's method.

The exact expected distinct-token count for this sampling design can also be expressed using combinations:

E[U_a(n)] = sum_w [1 - C(N_a - f(a,w), n) / C(N_a, n)]

Here C(x,n) is the number of ways to choose n occurrences from x, with C(x,n)=0 when x<n. The fraction is the probability that a sample contains no occurrence of token w. Subtracting from one gives its probability of appearing; adding these probabilities gives the expected number of distinct tokens. Our reported values are Monte Carlo estimates of this expectation, rather than evaluations of the exact formula.

Included writers and results

Sorted by average distinct tokens per 1,000-occurrence sample. Standard deviation describes variation between samples, not uncertainty about historical authorship.

Excluded writers

Every writer below was excluded solely because their export had fewer than 1,000 occurrences. Exclusion is not evidence of low diversity. Names follow the source metadata, including its spacing and spelling.

Caveats and interpretation

1. This measures token variety, not literary or spiritual quality. It does not measure meaning, conceptual depth, creativity, fluency, or a writer's complete personal vocabulary. Findings concern this cached corpus only.
2. Punctuation affects the result. Punctuation and verse numbers count as types, and attached punctuation can split one linguistic word into multiple token strings. Retaining these was an explicit choice for this exercise.
3. Equal sample sizes do not eliminate every corpus-size effect. A sample of 1,000 covers about 87% of Bhatt Gayand's export, but only about 0.58% of Guru Arjan Dev's. Without-replacement sampling behaves differently at these sampling fractions. The scores describe each finite corpus at this sample size.
4. Samples are dispersed, not passages. Frequency files discard order, so samples mix occurrences from across each writer's text. Results do not describe a consecutive 1,000-word passage. Refrains, genre, subject matter, and compositional conventions can affect token frequencies and rankings.
5. More samples add precision, not new source material. Reusing a small corpus cannot supply missing works or make it representative of all writing by an author. The 25 exclusions limit any claim about all writers.
6. Close scores should not be treated as decisive. The TSV reports sample SD, not a confidence interval. Under independent random draws, an approximate Monte Carlo standard error of a mean is SD/sqrt(1,000). For example, the Bhatt Kalh versus Bhatt Gayand difference is only 0.244 distinct tokens. Shared seeds can couple simulations across writers, so an independent-stream difference calculation is not justified without further work. No formal significance tests or adjustments for multiple comparisons were performed.
7. Input and attribution quality matter. Incorrect author labels, missing lines, repeated source content, spelling variants, or stale exports can change results. Counts are not an independent scholarly validation of the source. Similar author names are not merged automatically.
8. Rankings depend on the analysis choices. Other sample sizes, punctuation handling, linguistic normalization, or passage-based sampling could change rankings. Four decimal places document computational output, not substantive certainty. Checking several shared sample sizes would help assess stability.

Additional analysis: words appearing once in the entire text

This separate analysis asks how many globally single-occurrence tokens each writer uses. It uses the same exact whitespace-token definition, retaining punctuation and verse-number tokens. There is no sampling or normalization.

A token qualifies only when its frequency in outputs/fullMap.txt is exactly one. For each of the 37 authors, we count qualifying tokens present in their exported frequency file:

Global frequency(w) = count in fullMap.txt
Singleton count(a) = number of tokens w in author's file
                    for which Global frequency(w) = 1

This differs from words exclusive to one author: an author-exclusive word used twice would not qualify. It also differs from a word used once by an author but used elsewhere in the text; that word would not qualify either.

Guru Arjan Dev has the highest count in these exports. This is an absolute count and still depends on how much recorded text each author has; it does not measure diversity independent of text length or establish literary quality.

There are 16,030 qualifying tokens in the full-text export. Of these, 15,870 appear in the 37 author files and 160 do not appear in any of those files. The latter are left unassigned rather than attributed by assumption. Missing author metadata or inconsistent exports could explain the unmatched tokens; their cause has not been established by this file-based analysis. Every assigned token was checked to have frequency one in its author file and to appear under only one author.

• All 37 author counts
• Per-author qualifying token lists
• Unassigned qualifying tokens

Punctuation, spelling variants, attached markers, and source errors can generate single-occurrence token strings without representing distinct linguistic words. The results describe the existing exports, which must be regenerated together if the cache changes.

Sampling files and reproduction

• Whole-text unique counts
• Whole-text unique/total ratios
• Sampled results
• Excluded writers
• Sampling implementation

From the repository root, with the existing per-author exports available:

go run ./scripts/vocabulary-samples

The command uses the constants in the sampling script and replaces the sampled results and exclusions TSVs. If the cache changes, regenerate the per-author exports and whole-text summaries before rerunning this analysis. This findings document is a snapshot and must be updated separately when results change.