Go / Concurrent data collection

A map of words
to counts.

A simple question—how often does each word appear?—became the foundation for our other text explorations.

Gathering the pages with goroutines

We used Go goroutines to fetch pages concurrently, with at most ten active HTTP requests. A single database writer saves the results to a local SQLite cache. Later runs fetch missing pages, so interrupted imports can resume without starting over.

From cached text to a frequency map

Once all 1,430 pages are cached, we read the Punjabi lines, split them on whitespace using strings.Fields, and increment a Go map from token to count. The fullMap export lists each token and its frequency, separated by a tab, sorted by descending count and then token for ties.

for each line in the cached text:
    for each whitespace-separated token:
        counts[token] += 1

Download fullMap.txt ↓ View the frequency map

A simple definition, with limits

Tokens retain punctuation, verse numbers, and spelling variants. No Unicode normalization, stemming, or linguistic word merging is applied. Counts describe exact strings in the cached corpus. The map supports our Raagmala exclusivity comparison and writer-vocabulary analysis.

Reproduce the export

go run .
go run . --frequency --output=outputs/fullMap.txt

The first command populates missing cache pages. The frequency command reads the complete cache without fetching from the network and writes UTF-8 directly to the output file.

Continue to vocabulary diversity →