{"id":20265,"date":"2025-03-08T00:00:00","date_gmt":"2025-03-08T00:00:00","guid":{"rendered":"https:\/\/thierrymoudiki.github.io\/\/blog\/2025\/03\/08\/r\/python\/llms\/word-online"},"modified":"2025-03-08T00:00:00","modified_gmt":"2025-03-08T00:00:00","slug":"word-online-recreating-karpathys-char-rnn-with-supervised-linear-online-learning-of-word-embeddings-for-text-completion","status":"publish","type":"post","link":"https:\/\/python-bloggers.com\/2025\/03\/word-online-recreating-karpathys-char-rnn-with-supervised-linear-online-learning-of-word-embeddings-for-text-completion\/","title":{"rendered":"Word-Online: recreating Karpathy’s char-RNN (with supervised linear online learning of word embeddings) for text completion"},"content":{"rendered":"
\r\nThis article was first published on \r\n T. Moudiki's Webpage - Python <\/a><\/strong>, and kindly contributed to python-bloggers<\/a>. (You can report issue about the content on this page here<\/a>)\r\nWant to share your content on python-bloggers? click here<\/a>.<\/i>\r\n<\/div>\n

In this post, I implement a simple word completion model, based on Karpathy\u2019s char-RNN<\/a>, but using supervised linear online learning of word embeddings<\/strong>. More precisely, I use the SGDClassifier<\/a> from scikit-learn<\/code>, which is a simple linear classifier that can be updated incrementally<\/strong>.<\/p>\n

Keep in mind that this is an illustrative example, based on a few words and small vocabulary. There are many, many ways to improve the model, and many other configurations could be envisaged. So, feel free to experiment and extend this example<\/a>. Nonetheless, the grammatical structure of the generated text (don\u2019t generalize this result yet) is surprisingly good.<\/p>\n

My 2-cents-non-scientific (?) extrapolation about this is, is that artificial neural<\/em> networks are not intrinsically better than other methods: it takes a model with high capacity, capable of learning and generalize well.<\/p>\n

Here is how to reproduce the example, assuming you named the file word-online.py<\/code> (the repository is named word-online<\/code><\/a>):<\/p>\n

uv venv venv --python=3.11\nsource venv\/bin\/activate\nuv pip install -r requirements.txt\n<\/pre>\n
python word-online.py\n<\/pre>\n
word-online.py<\/code> contains the following code:<\/p>\n
Python version<\/h1>\n
import numpy as np\nimport gensim\nimport time  # Added for the delay parameter\n\nfrom collections import deque\nfrom tqdm import tqdm\nfrom scipy.special import softmax\nfrom sklearn.linear_model import SGDClassifier\n\n\n# Sample text \ntext = \"\"\"Hello world, this is an online learning example with word embeddings.\n          It learns words and generates text incrementally using an SGD classifier.\"\"\"\n\ndef debug_print(x):\n    print(f\"{x}\")\n\n# Tokenization (simple space-based)\nwords = text.lower().split()\nvocab = sorted(set(words))\nvocab.append(\"<UNK>\")  # Add unknown token for OOV words\n\n# Train Word2Vec model (or load pretrained embeddings)\nembedding_dim = 50  # Change to 100\/300 if using a larger model\nword2vec = gensim.models.Word2Vec([words], vector_size=embedding_dim, window=5, min_count=1, sg=0)\n\n# Create word-to-index mapping\nword_to_idx = {word: i for i, word in enumerate(vocab)}\nidx_to_word = {i: word for word, i in word_to_idx.items()}\n\n# Hyperparameters\ncontext_size = 12  # Default 10, Words used for prediction context\nlearning_rate = 0.005\nepochs = 10\n\n# Prepare training data\nX_train, y_train = [], []\n\nfor i in tqdm(range(len(words) - context_size)):\n    context = words[i:i + context_size]\n    target = words[i + context_size]\n    # Convert context words to embeddings\n    context_embedding = np.concatenate([word2vec.wv[word] for word in context])\n    X_train.append(context_embedding)\n    y_train.append(word_to_idx[target])\n\nX_train, y_train = np.array(X_train), np.array(y_train)\n\n# Initialize SGD-based classifier\nclf = SGDClassifier(loss=\"hinge\", max_iter=1, learning_rate=\"constant\", eta0=learning_rate)\n\n# Online training (stochastic updates, multiple passes)\nfor epoch in tqdm(range(epochs)):\n    for i in range(len(X_train)):\n        clf.partial_fit([X_train[i]], [y_train[i]], classes=np.arange(len(vocab)))\n\n# \ud83d\udd25 **Softmax function for probability scaling**\ndef softmax(logits):\n    exp_logits = np.exp(logits - np.max(logits))  # Stability trick\n    return exp_logits \/ np.sum(exp_logits)\n\n\ndef sample_from_logits(logits, k=5, temperature=1.0, random_seed=123):\n    \"\"\" Applies Top-K sampling & Temperature scaling \"\"\"\n    logits = np.array(logits) \/ temperature  # Apply temperature scaling\n    probs = softmax(logits)  # Convert logits to probabilities\n    # Select top-K indices\n    top_k_indices = np.argsort(probs)[-k:]\n    top_k_probs = probs[top_k_indices]\n    top_k_probs \/= top_k_probs.sum()  # Normalize\n    # Sample from Top-K distribution\n    np.random.seed(random_seed)\n    return np.random.choice(top_k_indices, p=top_k_probs)\n\n\ndef generate_text(seed=\"this is\", length=20, k=5, temperature=1.0, random_state=123, delay=3):\n    seed_words = seed.lower().split()\n\n    # Ensure context has `context_size` words (pad with zero vectors if needed)\n    while len(seed_words) < context_size:\n        seed_words.insert(0, \"<PAD>\")\n\n    context = deque(\n        [word_to_idx[word] if word in word_to_idx else -1 for word in seed_words[-context_size:]],\n        maxlen=context_size\n    )\n\n    generated = seed\n    previous_word = seed\n\n    for _ in range(length):\n        # Generate embeddings, use a zero vector if word is missing\n        context_embedding = np.concatenate([\n            word2vec.wv[idx_to_word[idx]] if idx in idx_to_word else np.zeros(embedding_dim)\n            for idx in context\n        ])\n        logits = clf.decision_function([context_embedding])[0]  # Get raw scores\n        # Sample next word using Top-K & Temperature scaling\n        pred_idx = sample_from_logits(logits, k=k, temperature=temperature)\n        next_word = idx_to_word.get(pred_idx, \"<PAD>\")\n        \n        print(f\"Generating next word: {next_word}\")  # Added this line\n        time.sleep(delay)  # Added this line\n        \n        if previous_word[-1] == \".\" and previous_word[-1] != \"\" and previous_word[-1] != seed:\n          generated += \" \" + next_word.capitalize()\n        else: \n          generated += \" \" + next_word\n        previous_word = next_word\n        context.append(pred_idx)\n\n    return generated\n\n# \ud83d\udd25 Generate text\nprint(\"\\n\\n Generated Text:\")\nseed = \"This is a\"\nprint(seed)\nprint(generate_text(seed, length=12, k=1, delay=0)) # delay seconds for next word generation, optimal for delay=0 seconds \n<\/pre>\n
100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 10\/10 [00:00<00:00, 12164.45it\/s]\n100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 10\/10 [00:01<00:00,  8.34it\/s]\n\n\n Generated Text:\nThis is a\nGenerating next word: classifier.\nGenerating next word: an\nGenerating next word: sgd\nGenerating next word: classifier.\nGenerating next word: and\nGenerating next word: generates\nGenerating next word: text\nGenerating next word: incrementally\nGenerating next word: using\nGenerating next word: an\nGenerating next word: sgd\nGenerating next word: classifier.\nThis is a classifier. An sgd classifier. And generates text incrementally using an sgd classifier.\n<\/pre>\n
\"image-title-here\"<\/p>\n
R version<\/h1>\n
%%R\nlibrary(reticulate)\nlibrary(progress)\nlibrary(stats)\n\n# Initialize Python modules through reticulate\nnp <- import(\"numpy\")\ngensim <- import(\"gensim\")\ntime <- import(\"time\")  # Added for the delay parameter\n\n# Sample text \ntext <- \"This is a model used for classification purposes. It applies continuous learning on word vectors, converting words into embeddings, learning from those embeddings, and gradually producing text through the iterative process of an SGD classifier.\"\n\ndebug_print <- function(x) {\n  print(paste0(x))\n}\n\n# Tokenization (simple space-based)\nwords <- strsplit(tolower(text), \"\\\\s+\")[[1L]]\nvocab <- sort(unique(words))\nvocab <- c(vocab, \"<UNK>\")  # Add unknown token for OOV words\n\n# Train Word2Vec model (or load pretrained embeddings)\nembedding_dim <- 50L  # Change to 100\/300 if using a larger model\nword2vec <- gensim$models$Word2Vec(list(words), vector_size=embedding_dim, window=5L, min_count=1L, sg=0L)\n\n# Ensure \"<UNK>\" is in the Word2Vec vocabulary\n# This is the crucial step to fix the KeyError\nif (!(\"<UNK>\" %in% word2vec$wv$index_to_key)) {\n  word2vec$wv$add_vector(\"<UNK>\", rep(0, embedding_dim))  # Add \"<UNK>\" with a zero vector\n}\n\n\n# Create word-to-index mapping\nword_to_idx <- setNames(seq_along(vocab) - 1L, vocab)  # 0-based indexing to match Python\nidx_to_word <- setNames(vocab, as.character(word_to_idx))\n\n# Hyperparameters\ncontext_size <- 12L  # Default 10, Words used for prediction context\nlearning_rate <- 0.005\nepochs <- 10L\n    \n# Prepare training data\nX_train <- list()\ny_train <- list()\n\npb <- progress_bar$new(total = length(words) - context_size)\nfor (i in 1L:(length(words) - context_size)) {\n  context <- words[i:(i + context_size - 1L)]\n  target <- words[i + context_size]\n  # Convert context words to embeddings\n  context_vectors <- lapply(context, function(word) as.array(word2vec$wv[word]))\n  context_embedding <- np$concatenate(context_vectors)\n  X_train[[i]] <- context_embedding\n  y_train[[i]] <- word_to_idx[target]\n  pb$tick()\n}\n\n# Initialize SGD-based classifier\nsklearn <- import(\"sklearn.linear_model\")\nclf <- sklearn$SGDClassifier(loss=\"hinge\", max_iter=1L, learning_rate=\"constant\", eta0=learning_rate)\n\n# Online training (stochastic updates, multiple passes)\npb <- progress_bar$new(total = epochs)\nfor (epoch in 1L:epochs) {\n  for (i in 1L:length(X_train)) {\n    # Use the list version for indexing individual samples\n    clf$partial_fit(\n      np$array(list(X_train[[i]])), \n      np$array(list(y_train[[i]])), \n      classes=np$arange(length(vocab))\n    )\n  }\n  pb$tick()\n}\n\n# Softmax function for probability scaling\nsoftmax_fn <- function(logits) {\n  exp_logits <- exp(logits - max(logits))  # Stability trick\n  return(exp_logits \/ sum(exp_logits))\n}\n\nsample_from_logits <- function(logits, k=5L, temperature=1.0, random_seed=123L) {\n  # Applies Top-K sampling & Temperature scaling\n  logits <- as.numeric(logits) \/ temperature  # Apply temperature scaling\n  probs <- softmax_fn(logits)  # Convert logits to probabilities\n  \n  # Select top-K indices - ensure k doesn't exceed the length of logits\n  k <- min(k, length(logits))\n  sorted_indices <- order(probs)\n  top_k_indices <- sorted_indices[(length(sorted_indices) - k + 1L):length(sorted_indices)]\n  \n  # Handle case where k=1 specially\n  if (k == 1L) {\n    return(top_k_indices)\n  }\n  \n  top_k_probs <- probs[top_k_indices]\n  # Ensure probabilities sum to 1\n  top_k_probs <- top_k_probs \/ sum(top_k_probs)\n  \n  # Check if all probabilities are valid\n  if (any(is.na(top_k_probs)) || length(top_k_probs) != length(top_k_indices)) {\n    # If there are issues with probabilities, just return the highest probability item\n    return(top_k_indices[which.max(probs[top_k_indices])])\n  }\n  \n  # Sample from Top-K distribution\n  set.seed(random_seed)\n  return(sample(top_k_indices, size=1L, prob=top_k_probs))\n}\n\ngenerate_text <- function(seed=\"this is\", length=20L, k=5L, temperature=1.0, random_state=123L, delay=3L) {\n  seed_words <- strsplit(tolower(seed), \"\\\\s+\")[[1L]]\n  \n  # Ensure context has `context_size` words (pad with zero vectors if needed)\n  while (length(seed_words) < context_size) {\n    seed_words <- c(\"<PAD>\", seed_words)\n  }\n  \n  # Use a fixed-size list as a ring buffer\n  context <- vector(\"list\", context_size)\n  for (i in 1L:context_size) {\n    word <- tail(seed_words, context_size)[i]\n    if (word %in% names(word_to_idx)) {\n      context[[i]] <- word_to_idx[word]\n    } else {\n      context[[i]] <- -1L\n    }\n  }\n  \n  # Track position in the ring buffer\n  context_pos <- 1L\n  \n  generated <- seed\n  previous_word <- seed\n  \n  for (i in 1L:length) {\n    # Generate embeddings, use a zero vector if word is missing\n    context_vectors <- list()\n    for (idx in unlist(context)) {\n      if (as.character(idx) %in% names(idx_to_word)) {\n        word <- idx_to_word[as.character(idx)]\n        context_vectors <- c(context_vectors, list(as.array(word2vec$wv[word])))\n      } else {\n        context_vectors <- c(context_vectors, list(np$zeros(embedding_dim)))\n      }\n    }\n    \n    context_embedding <- np$concatenate(context_vectors)\n    logits <- clf$decision_function(np$array(list(context_embedding)))[1L,]\n    \n    # Sample next word using Top-K & Temperature scaling\n    pred_idx <- sample_from_logits(logits, k=k, temperature=temperature, random_seed=random_state+i)\n    next_word <- if (as.character(pred_idx) %in% names(idx_to_word)) {\n      idx_to_word[as.character(pred_idx)]\n    } else {\n      \"<PAD>\"\n    }\n    \n    print(paste0(\"Generating next word: \", next_word))\n    if (delay > 0) {\n      time$sleep(delay)  # Added delay\n    }\n    \n    if (substr(previous_word, nchar(previous_word), nchar(previous_word)) == \".\" && \n        previous_word != \"\" && previous_word != seed) {\n      generated <- paste0(generated, \" \", toupper(substr(next_word, 1, 1)), substr(next_word, 2, nchar(next_word)))\n    } else {\n      generated <- paste0(generated, \" \", next_word)\n    }\n    \n    previous_word <- next_word\n    \n    # Update context (ring buffer style)\n    context[[context_pos]] <- pred_idx\n    context_pos <- (context_pos %% context_size) + 1L\n  }\n  \n  return(generated)\n}\n    \ncat(\"\\n\\n Generated Text:\\n\")\nseed <- \"This classifier is\"\ncat(seed, \"\\n\")\nresult <- generate_text(seed, length=2L, k=3L, delay=0L)  # delay seconds for next word generation\nprint(result)    \n<\/pre>\n
Generated Text:\nThis classifier is \n[1] \"Generating next word: for\"\n[1] \"Generating next word: text\"\n[1] \"This classifier is for text\"\n<\/pre>\n\n
\r\n
To leave a comment<\/strong> for the author, please follow the link and comment on their blog:  T. Moudiki's Webpage - Python <\/a><\/strong>.<\/div>\r\n
\r\nWant to share your content on python-bloggers? click here<\/a>.\r\n<\/div>","protected":false},"excerpt":{"rendered":"
R and Python implementations of word completion<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4],"tags":[],"class_list":["post-20265","post","type-post","status-publish","format-standard","hentry","category-data-science"],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/posts\/20265","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/comments?post=20265"}],"version-history":[{"count":2,"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/posts\/20265\/revisions"}],"predecessor-version":[{"id":20267,"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/posts\/20265\/revisions\/20267"}],"wp:attachment":[{"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/media?parent=20265"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/categories?post=20265"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/python-bloggers.com\/wp-json\/wp\/v2\/tags?post=20265"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}