cookbook examples

Author: Rangeet Pan

cldk/core.py

@@ -114,7 +114,7 @@ def treesitter_parser(self):
         else:
             raise NotImplementedError(f"Treesitter parser for {self.language} is not implemented yet.")
 
-    def tree_sitter_utils(self, source_code: str):
+    def tree_sitter_utils(self, source_code: str) -> [TreesitterSanitizer| NotImplementedError]:
         """
         Parse the project using treesitter.
 

docs/examples/java/code_summarization.ipynb

@@ -1,5 +1,17 @@
 {
  "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "!pip install ollama"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "eebee2515df69b96"
+  },
   {
    "cell_type": "markdown",
    "source": [

docs/examples/java/generate_unit_tests.ipynb

@@ -1,5 +1,17 @@
 {
  "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "!pip install ollama"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "b8da254b236c8a4b"
+  },
   {
    "cell_type": "markdown",
    "source": [
@@ -80,7 +92,7 @@
    "execution_count": null,
    "outputs": [],
    "source": [
-    "def prompt_ollama(message: str, model_id: str = \"granite-code:8b-instruct\") -> str:\n",
+    "def prompt_ollama(message: str, model_id: str = \"granite-code:20b-instruct\") -> str:\n",
     "    \"\"\"Prompt local model on Ollama\"\"\"\n",
     "    response_object = ollama.generate(model=model_id, prompt=message)\n",
     "    return response_object[\"response\"]"

docs/examples/java/validating_code_translation.ipynb

@@ -1,9 +1,22 @@
 {
  "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from cldk.utils.treesitter.tree_sitter_utils import TreeSitterUtils\n",
+    "!pip install ollama"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "3195a8c0612cb428"
+  },
   {
    "cell_type": "markdown",
    "source": [
-    "Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. In our recent paper [https://dl.acm.org/doi/10.1145/3597503.3639226] published at ICSE'24, we found that LLM-based code translation is very promising. In this example, we will walk through the steps of translating each Java class to Python and checking various properties of translated code, such as the number of methods, number of fields, formal arguments, etc.\n",
+    "Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. In our recent paper [https://dl.acm.org/doi/10.1145/3597503.3639226]  published at ICSE'24, we found that LLM-based code translation is very promising. In this example, we will walk through the steps of translating each Java class to Python and checking various properties of translated code, such as the number of methods, number of fields, formal arguments, etc.\n",
     "\n",
     "(Step 1) First, we will import all the necessary libraries"
    ],
@@ -83,6 +96,57 @@
     "collapsed": false
    },
    "id": "1c86224032a6eb70"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "(Step 4) Translate each class in the application (provide the application path as an environment variable, ```JAVA_APP_PATH```) and check certain properties of the translated code, such as (a) number of translated method, and (b) number of translated fields. "
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "518efea0d8c4d307"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from cldk.analysis.python.treesitter import PythonSitter\n",
+    "from cldk.analysis.java.treesitter import JavaSitter\n",
+    "\n",
+    "# Create a new instance of the CLDK class\n",
+    "cldk = CLDK(language=\"java\")\n",
+    "# Create an analysis object over the java application. Provide the application path using JAVA_APP_PATH\n",
+    "analysis = cldk.analysis(project_path=\"JAVA_APP_PATH\", analysis_level=AnalysisLevel.symbol_table)\n",
+    "# Go through all the classes in the application\n",
+    "for class_name in analysis.get_classes():\n",
+    "    # Get the location of the Java class\n",
+    "    class_path = analysis.get_java_file(qualified_class_name=class_name)\n",
+    "    # Read the file content\n",
+    "    if not class_path:\n",
+    "        class_body = ''\n",
+    "    with open(class_path, 'r', encoding='utf-8', errors='ignore') as f:\n",
+    "        class_body = f.read()\n",
+    "    # Sanitize the file content by removing comments.\n",
+    "    tree_sitter_utils = cldk.tree_sitter_utils(source_code=class_body)\n",
+    "    sanitized_class =  JavaSitter.remove_all_comments(source_code=class_body)\n",
+    "    translated_code = prompt_ollama(\n",
+    "                message=sanitized_class,\n",
+    "                model_id=\"granite-code:20b-instruct\")\n",
+    "    py_cldk = PythonSitter()\n",
+    "    all_methods = py_cldk.get_all_methods(module=translated_code)\n",
+    "    all_functions = py_cldk.get_all_functions(module=translated_code)\n",
+    "    all_fields = py_cldk.get_all_fields(module=translated_code)\n",
+    "    if len(all_methods) + len(all_functions) != len(analysis.get_methods_in_class(qualified_class_name=class_name)):\n",
+    "        print(f'Number of translated method not matching in class {class_name}')\n",
+    "    if len(all_fields) != len(analysis.get_class(qualified_class_name=class_name).field_declarations):\n",
+    "        print(f'Number of translated field not matching in class {class_name}')    "
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "fe3be3de6790f7b3"
   }
  ],
  "metadata": {