copy docs

Author: CarkHT

docs/.DS_Store

@@ -0,0 +1,82 @@
+# Running and Testing Agents Locally
+
+You can run the Python function for your agent itself by writing a `main()` function, or you can call the [`testbench_prompts.py`](https://github.com/lambda-feedback/lambda-chat/blob/main/src/agents/utils/testbench_prompts.py) script that runs a similar pipeline to the `module.py`.
+
+```bash
+python src/agents/utils/testbench_prompts.py
+```
+
+You can also use the `test_prompts.py` script to test the agents with example inputs from Lambda Feedback questions and synthetic conversations.
+```bash
+python src/agents/utils/test_prompts.py
+```
+
+## Testing using the Docker Image [:material-docker:](https://www.docker.com/)
+
+You can also build and run the docker pipeline for the agents. The chatbot agents are deployed onto a AWS Lambda serverless cloud function using the docker image. Hence, for final testing of your chatbots, we recommend completing those steps.
+
+#### Build the Docker Image
+
+To build the Docker image, run the following command in the root folder of the project (where the Dockerfile is located):
+
+```bash
+docker build -t llm_chat .
+```
+
+### Running the Docker Image
+
+To run the Docker image, use the following command:
+
+#### Without .env file:
+
+```bash
+docker run -e OPENAI_API_KEY={your key} -e OPENAI_MODEL={your LLM chosen model name} -p 8080:8080 llm_chat
+```
+
+#### With container name (for interaction, e.g. copying file from inside the docker container):
+
+```bash
+docker run --env-file .env -it --name my-lambda-container -p 8080:8080 llm_chat
+```
+
+This will start the evaluation function and expose it on port `8080` and it will be open to be curl:
+
+```bash
+curl --location 'http://localhost:8080/2015-03-31/functions/function/invocations' --header 'Content-Type: application/json' --data '{"message":"hi","params":{"conversation_id":"12345Test","conversation_history": [{"type":"user","content":"hi"}]}}'
+```
+
+### Call Docker Container From Postman
+
+POST URL:
+
+```bash
+http://localhost:8080/2015-03-31/functions/function/invocations
+```
+
+Body:
+
+```JSON
+{
+    "message":"hi",
+    "params":{
+        "conversation_id":"12345Test",
+        "conversation_history": [{"type":"user","content":"hi"}]
+    }
+}
+```
+
+Body with optional Params:
+```JSON
+{
+    "message":"hi",
+    "params":{
+        "conversation_id":"12345Test",
+        "conversation_history":[{"type":"user","content":"hi"}],
+        "summary":" ",
+        "conversational_style":" ",
+        "question_response_details": "",
+        "include_test_data": true,
+        "agent_type": {agent_name}
+    }
+}
+```

docs/advanced/chat_functions/local.md

@@ -0,0 +1,70 @@
+# Developing Chat Agents: Getting Started
+
+## What is a Chat Agent?
+
+It's a function which calls Large Language Models (LLMs) to respond to the student's messages given contxtual data:
+
+- question data
+- user data such as past responses to the problem
+  Chatbot Agents capture and automate the process of assisting students during their learning process when outside of classroom.
+
+## Getting Setup for Development
+
+1. Get the code on your local machine (Using github desktop or the `git` cli)
+
+	- For new functions: clone the main repo for [lambda-chat](https://github.com/lambda-feedback/lambda-chat) and create a new branch. Then go under `scr/agents` and copy the `base_agent` folder.
+
+	- For existing functions: please make your changes on a new separate branch
+
+2. _If you are creating a new chatbot agent_, you'll need to set it's name as the folder name in `scr/agents` and its corresponding files.
+3. You are now ready to start making changes and implementing features by editing each of the three main function-logic files:
+
+	1. **`scr/agents/{base_agent}/{base}_agent.py`**: This file contains the main LLM pipeline using [LangGraph](https://langchain-ai.github.io/langgraph/) and [LangChain](https://python.langchain.com/docs/introduction/).
+
+   	- the agent expects the following inputs when it being called:
+
+   	Body with necessary Params:
+
+   	```JSON
+   	{
+   		"message":"hi",
+   		"params":{
+   				"conversation_id":"12345Test",
+   				"conversation_history": [{"type":"user","content":"hi"}]
+   		}
+   	}
+   	```
+
+   	Body with optional Params:
+
+   	```JSON
+   	{
+   		"message":"hi",
+   		"params":{
+   				"conversation_id":"12345Test",
+   				"conversation_history":[{"type":"user","content":"hi"}],
+   				"summary":" ",
+   				"conversational_style":" ",
+   				"question_response_details": "",
+   				"include_test_data": true,
+   				"agent_type": {agent_name}
+   		}
+   	}
+   	```
+
+   2. **`scr/agents/{base_agent}/{base}_prompts.py`**: This is where you can write the system prompts that describe how your AI Assistant should behave and respond to the user.
+
+   3. Make sure to add your agent `invoke()` function to the `module.py` file.
+
+   4. Please add a `README.md` file to describe the use and behaviour of your agent.
+
+4. Changes can be tested locally by running the pipeline tests using:
+	```bash
+	pytest src/module_test.py
+	```
+   [Running and Testing Agents Locally](local.md){ .md-button }
+
+
+5. Merge commits into any branch (except main) will trigger the `dev.yml` workflow, which will build the docker image, push it to a shared `dev` ECR repository to make the function available from the `dev` and `localhost` client app.
+
+6. In order to make your new chatbot available on the LambdaFeedback platform, you will have to get in contact with the ADMINS on the platform.

docs/advanced/chat_functions/quickstart.md

@@ -0,0 +1,37 @@
+# Alternate Evaluation Function Languages
+---
+
+## Lambda-Compatible Images
+### Extending a pre-built Lambda image
+- Available for: Node.js, Python, Java, .NET, Go, Ruby
+- [Docs](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-images.html#runtimes-images-lp)
+- [Repo](https://github.com/aws/aws-lambda-base-images)
+- These base images are regularly updated, and the most widely used (more docs)
+- They also come with pre-packaged runtime interface clients - a HTTP interface for runtimes to receive invocation events and respond
+	- Good for local development
+
+### Creating custom base images
+- Using the [lambda/provided](https://gallery.ecr.aws/lambda/provided) image
+	- This "contains all the required components to run functions packaged as container images on Lambda"
+- Building a custom runtime from scratch 
+	- [Custom AWS Lambda runtimes](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html#runtimes-custom-build)
+	- [Runtimes walkthrough tutorial](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html)
+- Emulate execution locally?
+> Lambda provides a runtime interface emulator (RIE) for you to test your function locally. The AWS base images for Lambda and base images for custom runtimes include the RIE. For other base images, you can download the [Runtime interface emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator) from the AWS GitHub repository.
+
+### Misc Notes/Sources
+- [The Lambda Execution Environment](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html)
+- [Create Images from Alternative base images](https://docs.aws.amazon.com/lambda/latest/dg/images-create.html#images-create-from-alt)
+
+## Development Philosophy
+Ultimately we want to call a function made by a user in any language. Two ways to do this:
+
+- We write and provide runtime in all the different languages. This means that all the logic happens in that language. We write the code that actually receives the requests from lambda function events. In this case, the user function can be imported from those handlers.
+	- Writing handlers in each of those languages requires time and extensive knowledge (in order to write robust code)
+	- Handler code needs to:
+		- Have clean and reliable error catching
+  
+- We write a global runtime, which makes a call to their function via a sub-process. We call their script, which must recieve the payload as a commandline argument.
+	- User has to write more code 
+		- For allowing cmdline arguments, and parsing of inputs
+	- Might be slower than in other languages. Since another script has to be executed.

docs/advanced/evaluation_functions/alternate_languages.md

@@ -0,0 +1,40 @@
+# Base Layer Feedback Implementation
+
+Input structure:
+
+```json
+{
+	"response": "user input",
+	"answer": "original answer",
+	"params": {
+		"cases": [
+			{
+				"answer": "same shape as original answer",
+				"feedback": "feedback string",
+				"params": {...} # Any parameters to set or override
+			},
+			...
+		]
+	}
+}
+```
+
+## Execution Logic for the `eval` command
+1. First `evaluation_function` is called using the response, answer and params
+3. If evaluation threw an error, then return the error message
+2. If evaluation was successful, check for matching cases
+	1. If "params" contains a non-empty list of "cases", determine the correct feedback, add it to the result and return the block (Logic for this is described in the next section) 
+	2. If "params" doesn't contain a list of cases, simply return the result
+
+## Determining the correct feedback case
+1. Iterate through each case in the list of `cases`:
+	1. Validate the case has an 'answer' and 'feedback'
+	2. If the case contains 'params', then merge them with the original 'params', overwriting values if they already exist
+	3. Call `evaluation_function` with the student "response", case "answer" and merged "params"
+		1. If the function returns "is_correct: true", we have a match, store case and feedback returned from the evaluation function
+		2. If the function returns an error, catch it and add it to a list of warnings
+2. If no matches were found, don't return any feedback 
+3. If exactly one match was found, check if `override_eval_feedback` is in parameters
+	1. If `override_eval_feedback` is set to true, return the case feedback
+	2. If `override_eval_feedback` is not set or set to false, append the evaluation function feedback to the case feedback, separated by a linebreak and the return the result
+4. If more than one matches were found, return the first one (using the same procedure as if only one match was found) and add a warning explaining which cases matched, and why only the first was selected.

docs/advanced/evaluation_functions/feedback.md

@@ -0,0 +1,5 @@
+# Deployed Evaluation Functions
+
+Documentation for each of the functions registered to the LambdaFeedback platform are pulled in this section automatically. This is done using a custom MkDocs plugin [EvalDocsLoader](https://github.com/lambda-feedback/EvalDocsLoader).
+
+If you can't see any documentation files as subsections here, please contact an admin.

docs/advanced/evaluation_functions/index.md

@@ -0,0 +1,65 @@
+# Running and Testing Functions Locally
+
+## Simple
+
+
+## Using Docker [:material-docker:](https://www.docker.com/)
+This method builds and runs evaluation functions in the same way they are deployed on AWS as Lambda functions. Extending a pre-built and AWS-maintained [base python image](https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-base), the container contains a HTTP client which can be used to locally simulate Lambda execution events. 
+
+Note that this is different from the [simple](#simple) method proposed, in that it gives access to all the functionality provided by the base layer. This means that commands such as `docs` and `healthcheck` can be tested.
+
+1. Install [Docker](https://docs.docker.com/get-docker/) on your machine
+
+2. Navigate to the root directory of your function
+
+3. Build the image. This will pull our base image from Dockerhub, extend it with files specific to your evaluation function and name it `eval-tmp`.
+    ```bash
+    docker image build -t eval-tmp app
+    ```
+
+4. Spin up a container using the image built in the previous step.
+    ```bash 
+    docker run --rm -d --name eval-function -p 9000:8080 eval-tmp 
+    ```
+
+5. You can now simulate requests to the function using any request client (like [Insomnia](https://insomnia.rest/) or [Postman](https://www.postman.com/)). By default, the url you can hit is:
+    ```url 
+    http://localhost:9000/2015-03-31/functions/function/invocations
+    ```
+
+    ???+ warning
+        *When deployed, our Lambda functions are triggered by calls made through an AWS [API Gateway](https://aws.amazon.com/api-gateway/). This means that when testing locally, events sent should follow the structure of events triggered by that resource. That is, if you want to simulate what it would be like to make web requests to the deployed function.*
+
+        Specifically, this means structuring requests in the following way:
+        ```json 
+        {
+          "headers": {
+            "command": "eval"
+          },
+          "body": {
+            "response": "a",
+            "answer": "a",
+            "params": {
+              "garlic": "moreish"
+            }
+          }
+        }
+        ```
+
+        The main difference is that `headers` and `body` are sent as keys in the main body of the local request. When hitting the deployed function through the API Gateway, the `command` field would instead be passed in the actual HTTP headers of the request - and the actual request body would only contain the `response`, `answer` and `params` fields.
+
+6. *(Optional)* The `run` command specifies the **-d** flag, which spins up the container in detached mode. If you want to inspect the logs of the function, you can run:
+    ```bash 
+    docker container logs -f eval-function 
+    ```
+
+??? note "Tip"
+    You will very rarely need this, but you can peek into the running container by opening a shell within it using:
+
+    ```bash 
+    docker exec -it eval-function bash
+    ```
+
+## Useful Links 
+
+- 

docs/advanced/evaluation_functions/local.md

@@ -0,0 +1,60 @@
+# evaluation-function-utils Package
+
+- Error Reporting 
+- Schema validation
+- Local testing
+
+## Errors 
+Submodule containing custom error and exception classes, which can be properly caught by the base evaluation layer, and return more detailed and appropriate errors.
+
+### class `EvaluationException`
+This class extends the usual python `Exception`, with additional functionality. It can be used to package additional fields and values to errors thrown and returned by evaluation functions.
+
+!!! example 
+    If at some point in the execution of the [`evaluation_function`](specification.md#the-evaluationfunction), an error is thrown:
+
+    ```python
+    from evaluation_function_utils.errors import EvaluationException
+
+    if isinstance(input, str):
+        raise EvaluationException(
+            "The input must not be a string", 
+            valid_types=["int", "float", "array"],
+        )
+    ``` 
+
+    Then the output generated by the lambda function will look like:
+
+    ```python
+    {
+      "command": "eval",
+      "error": {
+        "message": "The input must not be a string",
+        "valid_types": [
+          "int", "float", "array"
+        ]
+      }
+    }
+    ```
+
+This class contains an error_dict property, which packages the additional arguments given to the Exception instance into a JSON-serializable object. It does so in an error-safe way, also reporting serialization errors if they occur.
+
+## Client 
+This submodule contains a custom `EvaluationFunctionClient`, which can be used to call other deployed evaluation functions.
+
+### class `EvaluationFunctionClient`
+Client wrapped around the botocore.client.Lambda, for invoking deployed evaluation functions. On initialisation, it fetches credentials from environment variables "INVOKER_KEY", "INVOKER_ID" and "INVOKER_REGION", or from an optional environment file prescrived by `env_path`. 
+
+!!! example 
+
+    ```python 
+    from evaluation_function_utils.client import EvaluationFunctionClient
+    client = EvaluationFunctionClient()
+
+    def evaluation_function(response, answer, params): 
+        return client.invoke('isExactEqual', response, answer, params)
+    ```
+
+    In this example, the evaluation_function completely offloads grading to the deployed 'isExactEqual' function. 
+
+*Note:* The `EvaluationFunctionClient.invoke` method was designed to behave exactly as if the [`evaluation_function`](specification.md#the-evaluationfunction) function defined in the targeted deployed function was called directly. This means that if errors are encountered an `EvaluationException` is raised.