WISPRnet
Tutorials

Case Study: Building an N-Gram Language Model with WisprNet API for Mark Rutte's NATO Speeches

Trevor French •
#python#tensorflow

This case study highlights how we used WisprNet’s API to analyze public statements made by Mark Rutte, the newly appointed Secretary General of NATO, to build a basic trigram language model. The goal was to use recent speeches from Rutte to train the model and generate plausible extensions of seed sentences based on his speech patterns.

Our app, WisprNet, serves as a comprehensive database of public statements, empowering users with tools to filter, analyze, and process speech data for various use cases like sentiment analysis, trend detection, and predictive modeling. This project demonstrates how our API facilitates data retrieval and processing to enable innovative applications.

Step 1: Setting Up the Environment

In this step, we import the necessary libraries to fetch and process public statements data. We use standard Python libraries for HTTP requests and text processing, along with NLTK for tokenization and n-gram generation.

Imports:

import requests
import re
from nltk.tokenize import word_tokenize
from nltk.util import ngrams
from collections import defaultdict, Counter
import nltk

We also download the NLTK tokenizer models:

nltk.download('punkt')

This step sets up the tools needed for data processing and NLP tasks in the following stages.

Step 2: Fetching Public Statements

The first step involved retrieving the last week of Mark Rutte’s public speeches. Using the WisprNet API, we filtered data by person_id, specifying Rutte as the individual of interest. Here’s a snippet of the API query:

url = "http://api.wisprnet.com/statement/"
params = {
    "filter_value": "1",
    "filter_field": "person_id",
    "per_page": "20",
    "page": "1"
}
headers = {
    'accept': 'application/json',
    'X-API-Key': '[YOUR FREE API KEY]'
}

response = requests.get(url, headers=headers, params=params)
statements = [item['statement'] for item in response.json()['data']] if response.status_code == 200 else []

This API query retrieved the most recent statements by Mark Rutte, providing a textual dataset to train the language model.

Step 3: Preprocessing Text

To prepare the text for modeling, we performed preprocessing steps to remove special characters, normalize the case, and tokenize the text into individual words. Each statement was processed using a custom function:

def preprocess_text(text):
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    text = text.lower()
    tokens = word_tokenize(text)
    return tokens

tokenized_statements = [preprocess_text(statement) for statement in statements]

This step ensured a clean and consistent dataset for subsequent n-gram modeling.

Step 4: Training an N-Gram Model

Using the preprocessed data, we trained a trigram model to capture the relationships between words in the dataset. The model was built using defaultdict to store the counts of words following specific contexts:

def train_ngram_model(tokenized_texts, n=3):
    ngram_model = defaultdict(Counter)
    for tokens in tokenized_texts:
        for ngram in ngrams(tokens, n):
            context = ngram[:-1]
            target = ngram[-1]
            ngram_model[context][target] += 1
    return ngram_model

n = 3
ngram_model = train_ngram_model(tokenized_statements, n=n)

This allowed us to predict the next most probable word given a sequence of preceding words.

Step 5: Predicting and Completing Sentences

To test the model, we designed a function to generate extensions of a given seed sentence. For example, starting with the seed text “We must,” the model predicted the following completion:

def predict_next_word(context, ngram_model):
    context_tuple = tuple(context)
    if context_tuple in ngram_model:
        next_word = ngram_model[context_tuple].most_common(1)[0][0]
        return next_word
    else:
        return None

def complete_sentence(seed_text, ngram_model, max_words=10):
    tokens = preprocess_text(seed_text)
    for _ in range(max_words):
        context = tokens[-(n-1):]
        next_word = predict_next_word(context, ngram_model)
        if next_word:
            tokens.append(next_word)
        else:
            break
    return ' '.join(tokens)

seed_text = "We must"
completed_sentence = complete_sentence(seed_text, ngram_model)
print("Completed Sentence:", completed_sentence)

Output: “We must go further to decisively change the course of this war.”

This prediction reflects the training data’s themes, likely drawn from Rutte’s recent speeches on NATO’s stance in ongoing global conflicts.

Impact and Applications

The case study demonstrates how WisprNet’s API enabled the seamless collection and analysis of real-world speech data. By providing an easy-to-use platform for accessing public statements, our app empowers researchers, analysts, and policymakers to explore linguistic patterns, sentiment, and trends.

This specific example, focused on Mark Rutte, highlights how our platform supports applications like:

Conclusion

Using the WisprNet API, we successfully built a trigram language model trained on Mark Rutte’s NATO speeches. This demonstrates the power of our app in enabling real-world applications of AI and NLP. Whether for predictive text modeling, speech analysis, or public policy research, WisprNet’s robust data pipeline opens the door to limitless possibilities.

← Back to Blog