TalkBack is an interactive note taking app that allows you to upload audio recordings of seminars, lectures, personal notes, or any other audio content. The audio is transcribed and uploaded to a Large Language Model, allowing you to interact with the audio notes in a conversational manner. Since the model has access to a word-for-word recording of the lecture or seminar, you can ask questions about the content and get answers in real time. TalkBack is effectively an interactive digital notebook, perfect for students, professionals, and anyone who wants to utilize notes in a more engaging way.
Note
This is a template for your startup application. You must modify this README.md file for each phase of your development. You only need to fill in the section for each deliverable when that deliverable is submitted in Canvas. Without completing the section for a deliverable, the TA will not know what to look for when grading your submission. Feel free to add additional information to each deliverable description, but make sure you at least have the list of rubric items and a description of what you did for each item.
Note
If you are not familiar with Markdown then you should review the documentation before continuing.
Note
Fill in this sections as the submission artifact for this deliverable. You can refer to this example for inspiration.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- Proper use of Markdown
- A concise and compelling elevator pitch
- Description of key features
- Description of how you will use each technology
- One or more rough sketches of your application. Images must be embedded in this file using Markdown image references.
TalkBack is an interactive note taking app that allows you to upload audio recordings of seminars, lectures, personal notes, or any other audio content. The audio is transcribed and uploaded to a Large Language Model, allowing you to interact with the audio notes in a conversational manner. Since the model has access to a word-for-word recording of the lecture or seminar, you can ask questions about the content and get answers in real time. TalkBack is effectively an interactive digital notebook, perfect for students, professionals, and anyone who wants to utilize notes in a more engaging way.
This squence diagram shows the flow of the application. The user uploads an audio recording, the audio is transcribed and uploaded to the LLM, and the user can then interact with the notes in a conversational manner.
sequenceDiagram
actor User
participant Website
participant LLM
User->>Website: Upload audio recording
Website->>Website: Transcribe audio
Website->>LLM: Upload to LLM
LLM->>User: Display Transcript
User->>Website: Ask questions
Website->>User: Answer questions
- Secure user authentication and data storage
- Ability to store multiple audio recordings and notes
- User-friendly interface for uploading and interacting with audio recordings
- Ability to summarize the audio content
- Real-time transcription and LLM interaction
- Ability to ask questions about the audio content
I am going to use the required technologies in the following ways.
- HTML - Uses correct HTML elements and attributes to build the website and display chat interface. Also Includes library dropdown and login page.
- CSS - Asthetic design of the website and chat interface. Compatibility across different screen sizes and types.
- React - Provides login capabilities and allows for the chat interface to be displayed. Also allows for the library dropdown to be displayed and function.
- Service - backend service that handles the audio transcription and LLM interaction.
- Supports login and registration systems.
- Supports storing audio recordings and notes/chat history.
- Supports audio transcription and LLM interaction.
- DB/Login - Stores user data and authentication information. Guest mode available but does not save any data.
- WebSocket - Used to send and receive messages between the LLM and server. Also used to send audio recordings to the transcription service and receive the transcription back.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
<<<<<<< HEAD
- Server deployed and accessible with custom domain name - My server link. =======
- Server deployed and accessible with custom domain name - TalkBack.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- HTML pages - Three HTML pages. One for tha chat/home page, one for log in, and one to view your library.
- Proper HTML element usage - Body, Nav, Header, Main, Footer, Form, Input, Button, and Menu are all used correctly.
- Links - The links are functional and take the user to the correct page.
- Text - The text is functional and displays the correct information. Placeholders are used for the text that will be added by the user or by the LLM.
- 3rd party API placeholder - The api placeholder is the chat history, currently represented by a paragraph tag.
- Images - The logo is displayed on all pages as an image.
- Login placeholder - There is a placeholder for the login page.
- DB data placeholder - The uploaded audio files are displayed via a placeholder in the library page.
- WebSocket placeholder - Realtime communication will be the chat interface and history. The LLM api will provide real time feedback to the user.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- Header, footer, and main content body - Header, footer, and main content body are all present and styled.
- Navigation elements - Navigation elements are present and styled.
- Responsive to window resizing - The website is responsive to window resizing.
- Application elements - Application elements are present and styled.
- Application text content - Application text content is present and styled.
- Application images - Application logo image is present and styled.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- Bundled using Vite - i bundled using vite.
- Components - All the components are present and styled.
- Router - Routing between login and voting components is present and functional.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- All functionality implemented or mocked out - The only thing that doesn't work yet other than API calls is things involving databases.
- Hooks - Lots of hooks. Also a couple of useContext things.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- Node.js/Express HTTP service - Use Node.js to host my backend server.
- Static middleware for frontend - lots and lots and lots of middlewhere.
- Calls to third party endpoints - use openAI api calls for chat and file transcription.
- Backend service endpoints - lots and lots and lots of endpoints
- Frontend calls service endpoints - lots and lots and lots of endpoints.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- User registration - Users can log in and store data to their accounts.
- User login and logout - Users can log in and out using accounts stored to the database.
- Stores data in MongoDB - User accounts and associated data are stored to the database.
- Stores credentials in MongoDB - User credentials are stored in the database.
- Restricts functionality based on authentication - If a user is not logged in, thy will not be able to use some features.
For this deliverable I did the following. I checked the box [x] and added a description for things I completed.
- Backend listens for WebSocket connection - Backend Websocket on PeerProxy.
- Frontend makes WebSocket connection - Connects on notificationsDisplay.jsx
- Data sent over WebSocket connection - Uses Websocket to alert whenever users upload a file
- WebSocket data displayed - Displays alerts on login and library pages
- Application is fully functional - We're 100% ready to rock and roll!