Merge pull request #13 from clemsciences/old-norse

Old Norse tutorial

Author: kylepjohnson

languages/old-norse/old-norse-tutorial.ipynb

@@ -0,0 +1,358 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Old Norse with CLTK\n",
+    "\n",
+    "Process your Old Norse texts thanks to cltk. Here are presented several tools adapted to Old Norse."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set your own user path\n",
+    "USER_PATH = \"/home/pi\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Import Old Norse corpora\n",
+    "* old_norse_text_perseus contains different Old Norse books\n",
+    "* old_norse_texts_heimskringla contains the Eddas\n",
+    "* old_norse_models_cltk is data for a Part Of Speech tagger \n",
+    "\n",
+    "By default, corpora are imported into ~/cltk_data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from cltk.corpus.utils.importer import CorpusImporter\n",
+    "onc = CorpusImporter(\"old_norse\")\n",
+    "onc.import_corpus(\"old_norse_text_perseus\")\n",
+    "onc.import_corpus(\"old_norse_texts_heimskringla\")\n",
+    "onc.import_corpus(\"old_norse_models_cltk\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Configure IPython\n",
+    "\n",
+    "Configure IPython if you want to use this notebook\n",
+    "```bash\n",
+    "$ ipython profile create\n",
+    "$ ipython locate\n",
+    "$ nano ~/profile_default/ipython_config.py\n",
+    "```\n",
+    "Add it a the end of the file (without '#'):\n",
+    "```python\n",
+    "c.InteractiveShellApp.exec_lines = [\n",
+    "    'import sys; sys.path.append(\"~/cltk_data/old_norse\")'\n",
+    "]\n",
+    "```\n",
+    "And... It's done!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### old_norse_text_perseus"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Ögmundr er maðr nefndr, er kal\n",
+      "Sigurðr hefir átt sér einn fós\n",
+      "Nú halda þeir þangat, ok er þe\n",
+      "Nú líða stundir fram, ok var s\n",
+      "Herruðr hét jarl ríkr ok ágætr\n",
+      "Í þann tíma réð fyrir Danmörku\n",
+      "Nú er þat eitt sumar, at hann \n",
+      "Þetta spyrst til skipa Ragnars\n",
+      "Nú ráða þeir þetta með sér, at\n",
+      "HEIMIR í Hlymdölum spyrr nú þe\n",
+      "Nú halda þeir í brott þaðan, þ\n",
+      "Eptir þetta fara þeir Hvítserk\n",
+      "Nú er sú stund var liðin, er á\n",
+      "Nú er þar til máls at taka, er\n",
+      "Nú ráða þeir þat með sér, at þ\n",
+      "Sá atburðr hefir verit út í lö\n",
+      "Nú segir hann, at honum lízt v\n",
+      "Nú er þat eitthvert sinn, at m\n",
+      "Nú berr svá til, at þeir koma \n",
+      "Eysteinn hefir konungr heitit,\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "\n",
+    "corpus = os.path.join(USER_PATH, \"cltk_data/old_norse/text/old_norse_text_perseus/plain_text/Ragnars_saga_loðbrókar_ok_sona_hans\")\n",
+    "chapters = []\n",
+    "for filename in os.listdir(corpus):\n",
+    "    with open(os.path.join(corpus, filename)) as f:\n",
+    "        chapter_text = f.read()  # json.load(filename)\n",
+    "        print(chapter_text[:30])\n",
+    "        chapters.append(chapter_text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### old_norse_texts_heimskringla"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['Snorra-Edda', '__pycache__', 'Sæmundar-Edda']\n",
+      "\n",
+      "Atlakviða\n",
+      "\n",
+      "Dauði Atla\n",
+      "\n",
+      "Guðrún Gjúkadóttir hefndi bræðra sinna, svá sem frægt er orðit. Hon drap fyr\n"
+     ]
+    }
+   ],
+   "source": [
+    "import sys\n",
+    "from old_norse.text.old_norse_texts_heimskringla.text_manager import *\n",
+    "corpus_path = USER_PATH+\"/cltk_data/old_norse/text/old_norse_texts_heimskringla\"\n",
+    "here = os.getcwd()\n",
+    "os.chdir(corpus_path)\n",
+    "loader = TextLoader(os.path.join(corpus_path, \"Sæmundar-Edda\", \"Atlakviða\"), \"txt\")\n",
+    "print(loader.get_available_names())\n",
+    "complete_text = loader.load()\n",
+    "print(complete_text[:100])\n",
+    "os.chdir(here)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### POS tagging\n",
+    "Unknown tags are marked with 'Unk'."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[('Hlióðs', 'Unk'),\n",
+       " ('bið', 'VBPI'),\n",
+       " ('ek', 'PRO-N'),\n",
+       " ('allar', 'Q-A'),\n",
+       " ('.', '.')]"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from cltk.tag.pos import POSTag\n",
+    "import cltk.tag.pos as cltkonpos\n",
+    "tagger = POSTag('old_norse')\n",
+    "sent = 'Hlióðs bið ek allar.'\n",
+    "tagger.tag_tnt(sent)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Word tokenizing\n",
+    "For now, the word tokenizer is basic, but Old Norse actually does not need a sophisticated one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['Gylfi', 'konungr', 'var', 'maðr', 'vitr', 'ok', 'fjölkunnigr', '.']"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from cltk.tokenize.word import WordTokenizer\n",
+    "word_tokenizer = WordTokenizer('old_norse')\n",
+    "sentence = \"Gylfi konungr var maðr vitr ok fjölkunnigr.\"\n",
+    "word_tokenizer.tokenize(sentence)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Old Norse Stop Words\n",
+    "A list of stop words was elaborated with the most insignificant words of a sentence. Of course, according to your needs, you can change it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['var',\n",
+       " 'einn',\n",
+       " 'morgin',\n",
+       " ',',\n",
+       " 'karlsefni',\n",
+       " 'rjóðrit',\n",
+       " 'flekk',\n",
+       " 'nökkurn',\n",
+       " ',',\n",
+       " 'glitraði']"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from nltk.tokenize.punkt import PunktLanguageVars\n",
+    "from cltk.stop.old_norse.stops import STOPS_LIST\n",
+    "sentence = 'Þat var einn morgin, er þeir Karlsefni sá fyrir ofan rjóðrit flekk nökkurn, sem glitraði við þeim'\n",
+    "p = PunktLanguageVars()\n",
+    "\n",
+    "tokens = p.word_tokenize(sentence.lower())\n",
+    "[w for w in tokens if not w in STOPS_LIST]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Swadesh list for Old Norse\n",
+    "In the following Swadesh list, an item may have several words if they have a similar meaning, and some words lack because I have not found any corresponding Old Norse word."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['ek',\n",
+       " 'þú',\n",
+       " 'hann',\n",
+       " 'vér',\n",
+       " 'þér',\n",
+       " 'þeir',\n",
+       " 'sjá, þessi',\n",
+       " 'sá',\n",
+       " 'hér',\n",
+       " 'þar',\n",
+       " 'hvar',\n",
+       " 'hvat',\n",
+       " 'hvar',\n",
+       " 'hvenær',\n",
+       " 'hvé',\n",
+       " 'eigi',\n",
+       " 'allr',\n",
+       " 'margr',\n",
+       " 'nǫkkurr',\n",
+       " 'fár',\n",
+       " 'annarr',\n",
+       " 'einn',\n",
+       " 'tveir',\n",
+       " 'þrír',\n",
+       " 'fjórir',\n",
+       " 'fimm',\n",
+       " 'stórr',\n",
+       " 'langr',\n",
+       " 'breiðr',\n",
+       " 'þykkr']"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from cltk.corpus.swadesh import Swadesh\n",
+    "swadesh = Swadesh('old_norse')\n",
+    "words = swadesh.words()\n",
+    "words[:30]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By Clément Besnier, email address: clemsciences@aol.com"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.6",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}