AmI HMAS is a proof of concept for goal-driven interaction in smart environments using Hypermedia Multi-Agent Systems (HMAS), Web of Things Thing Descriptions, LLM-based agents, and signifiers.
The main idea is simple: instead of making a user control every device separately, the system should be able to understand a higher-level request such as "It's too dark in here", reason about the environment, propose a plan, ask for confirmation, and then execute the necessary actions.
This repository accompanies the demo paper AmI HMAS: goal-driven interaction support in Smart Environments, presented at EUMAS 2025, and contains the proof-of-concept implementation of the HMAS architecture described there.
A short demo video of the system is available here: Conference demo video
This project is built around a simulated smart lab called lab308, running on top of the Yggdrasil Hypermedia MAS platform. The environment contains artifacts such as a smart lamp, motorized blinds, and environmental sensors. These artifacts are exposed through Web of Things Thing Descriptions, while a set of specialized agents handle planning, monitoring, semantic discovery, experience reuse, and execution.
The project is meant to show how an HMAS-style environment can support more flexible and context-aware interaction than a setup based only on fixed scripts or direct device commands.
A central goal of this project is to treat the environment as a hypermedia space, not as a hardcoded list of devices.
In practice, this means that artifacts and workspaces are represented as web resources that can be discovered, described, and queried through standard web technologies. Agents do not need to rely only on fixed integrations. Instead, they can inspect what exists in the environment, understand what each artifact exposes, and adapt their behavior based on that information.
This proof of concept focuses on several HMAS ideas:
- web-addressable artifacts and workspaces
- semantic descriptions of capabilities
- runtime discovery of available affordances
- agent-to-agent interaction over web interfaces
- reuse of previous successful interactions through signifiers
Because of this, the project is not just about controlling a lamp or some blinds. It is about exploring how a hypermedia multi-agent environment can support reasoning, adaptation, and coordination between agents.
A key part of the system is the way artifacts are semantically represented and understood.
Each artifact in lab308 is exposed as a resource with a machine-readable description based on Web of Things Thing Descriptions. These descriptions do more than just expose endpoints. They describe what an artifact is, what properties it has, what actions it supports, what input schema is required, and how each affordance can be invoked.
Because of this, the agents can reason over artifact descriptions in a structured way. They can inspect:
- what type of artifact they are dealing with
- what properties are available
- what actions can be invoked
- what parameters are required
- what HTTP method and target are associated with each action
In the current system:
EnvExplorerAgentretrieves artifact descriptions and indexes action affordancesEnvMonitoringAgentkeeps RDF-based snapshots of live artifact state
Together, these two services give the planner both a semantic understanding of what the environment can do and a live view of what the environment is doing at a given moment.
Another important part of the project is the use of signifiers.
In this system, signifiers are used as reusable, context-rich records of successful past interactions. After a plan is confirmed and executed, the system can store that experience in a form that connects:
- the user intent
- the artifact affordance that was useful
- the context in which that affordance made sense
This allows the system to reuse previous experience instead of planning from scratch every time. When a similar request appears again, the InteractionSolverAgent first checks whether the EnvExplorerAgent already has a signifier that matches the current goal and context. If a matching signifier exists, the system can reuse it. If not, it can still fall back to first-principles planning from raw affordances and live state.
Because of this, signifiers act as a bridge between semantic affordances and accumulated experience. They help the system move from "what the environment can do" to "what has already worked before in a similar situation."
- Understands natural-language requests and turns them into actionable intents.
- Discovers semantic artifact capabilities and tracks live environment state.
- Reuses previous successful interactions through signifiers when the context matches.
- Generates, confirms, and executes plans when direct reuse is not possible.
The system is organized around four agent roles and one environment platform:
UserAssistantAgent: the conversational entry point. It receives the user request, summarizes the plan, asks for confirmation, and starts execution.InteractionSolverAgent: the reasoning component. It builds a JSON plan from capabilities, signifiers, and live environment state.EnvExplorerAgent: the capability and signifier service. It indexes artifact affordances and stores reusable experience.EnvMonitoringAgent: the state service. It keeps an in-memory view of artifact state from WebSub updates.yggdrasil-signifiers: the Lab308 simulation and frontend, built on Yggdrasil.
Main ports used by the system:
8080: Yggdrasil / Lab308 backend5173: Yggdrasil frontend8081: EnvMonitoringAgent8082: EnvExplorerAgent8083: InteractionSolverAgent8084: UserAssistantAgent
- A user sends a natural-language request to the
UserAssistantAgent. - The
UserAssistantAgentdecides whether the request is in scope and forwards a structured intent to theInteractionSolverAgent. - The
InteractionSolverAgentasksEnvExplorerAgentfor signifiers and available affordances, andEnvMonitoringAgentfor live state. - If a matching signifier exists, the solver tries to reuse it. Otherwise, it plans from raw capabilities and current context.
- The resulting JSON plan is sent back to the
UserAssistantAgent, which explains it to the user and asks for confirmation. - After confirmation, the plan is stored as a signifier and executed against the Lab308 environment.
yggdrasil-signifiers/- the HMAS environment, Lab308 simulation, and React frontendUserAssistantAgent/- user-facing orchestration and plan executionInteractionSolverAgent/- intent-to-plan reasoning serviceEnvExplorerAgent/- affordance indexing and signifier managementEnvMonitoringAgent/- live artifact state tracking
Each service also has its own local README with more specific details.
- Java 21
- Kotlin + Spring Boot + WebFlux
- Eclipse LMOS / ARC agent tooling
- LangChain4j
- RDF4J and RDF/Turtle models
- Web of Things Thing Descriptions
- WebSub notifications
- Yggdrasil Hypermedia MAS
- CArtAgO-based artifact simulation
- React + Vite frontend
- Yggdrasil - the Hypermedia MAS platform used to model and run the Lab308 environment.
- Eclipse LMOS / ARC - the agent platform and framework used to build the four LLM-based services.
- W3C Web of Things Thing Description 1.1 - the semantic description model used to represent artifacts, properties, actions, and invocation metadata.
- WebSub - the notification mechanism used to propagate environment updates.
- JaCaMo - the broader multi-agent ecosystem that includes CArtAgO, the artifact-oriented environment model behind the simulated setup.
If you want to run the full setup, the easiest path is the following.
- Java 21+
- Node.js + npm
- One configured LLM backend
OPENAI_API_KEYis the simplest default for all agentsUserAssistantAgentalso supportsOLLAMA_URLandOPENROUTER_API_KEY
From yggdrasil-signifiers/:
./gradlew.bat
java -jar build/libs/yggdrasil-0.0.0-SNAPSHOT-all.jar -conf conf/cartago_config_light308.jsonFrom yggdrasil-signifiers/frontend/:
npm install
npm run devIn separate terminals:
cd EnvMonitoringAgent
./gradlew.bat bootRuncd EnvExplorerAgent
./gradlew.bat bootRuncd InteractionSolverAgent
./gradlew.bat bootRuncd UserAssistantAgent
./gradlew.bat bootRun- Open the Lab308 frontend at
http://localhost:5173 - Use the LMOS chat UI with the
UserAssistantAgent:
https://eclipse.dev/lmos/chat/index.html?agentUrl=http://localhost:8084#/chat
Example request:
It's too dark in here.
The current implementation is still a research prototype rather than a fully hardened platform. It already demonstrates:
- multi-agent separation of responsibilities
- runtime discovery of device affordances
- semantic interpretation of artifact descriptions
- context-aware planning over a WoT-described environment
- reuse of prior experience through signifiers
Some parts still need more work:
- stronger plan validation before execution
- better rollback and error recovery for multi-step plans
- broader evaluation scenarios
- richer user-guided teaching of new signifiers