`MDNS` integration and `NodeGraph` structure expansion

[tegefaulkes]

Specification

With the creation of MDNS we need to integrate it's usage into Polykey. With that comes some considerations surrounding
what connection information needs to be stored in the NodeGraph and what information needs to be shared between nodes.

Right now the NodeGraph is based on the Kademlia spec and is pretty simple. It only stores a single Address for each node. A There are only 20 nodes per bucket and only the oldest active ones are stored. This is all based on the Kademlia spec.

The NodeGraph logic needs to be expanded to allow for more information relating to each node to be stored. We need to store IPv4 as well as IPv6 addresses. We need to track information about if the node is publicly contactable or needs to be hole punched. We need sticky nodes such as the seed nodes or user defined static information.

Step 1 - MDNS Sidecar onto NodeConnectionManager and NodeManager

I've decided as the simplest solution, MDNS can be integrated into methods of NodeConnectionManager and NodeManager related to discovery, like findNode, etc. This essentially allows NodeConnectionManager and NodeManager use both MDNS and NodeGraph as discovery mechanisms independent of each other.

Note: MDNS should not be integrated into findNode directly, but as a separate method to be called by the callers of findNode in non-kademllia contexts.

The NodeGraph related data structures will still have to be edited like so to facilitate the separation of MDNS addresses and NodeGraph addresses:

type NodeAddressScope = 'local' | 'external';
type NodeAddress = {
  host: Host | Hostname;
  port: Port;
  scopes: Array<NodeAddressScope>;
};

This will mean that there will be overlapping NodeData that exists on both the NodeGraph and MDNS, this will have to be dealt with.

The methods on NCM: getConnectionWithAddress and getConnection will need to be amended to support multiple NodeAddresses as input. This isn't the ideal way to do this, but it will have to be like this until the NodeGraph itself is made to support storing multiple IP addresses in Step 2. Much of the flow will need to be rewritten to support this change.

Scopes

Currently in Step 1, scopes will only be used for disabling holepunching on local addresses. An address can be either local, external, or both. Local means that the address is locally routable, so holepunching is not required when connecting to that address. External means the that the address is gliobally routable, but will require holepunching to establish a connection.

Scopes aren't just for disabling signaling or not, they are also used for the purpose of classifying IP addresses into different levels. This means that in future, when scopes are incorporated as part of the nodegraph, we can specify for only external addresses to be shared on the public mainnet.

Being able to define scopes as an array also means that i address could have multiple classifications, which would be useful in the future if we ever needed it expanded. Also addresses with both local and external will be understood to be shared on the mainnet, but not require holepunching to be connected to.

Custom Group Address and Port

We're going to need to select a port and ipv4 + ipv6 groups for now, as to not interfere with existing MDNS stacks. However, as native integration of MDNS improves, this may eventually no longer be needed.

I've decided to take inspiration from our slogan fait accompli, to choose these addresses:

	IPv4	IPv6	Port
Value	224.0.0.250	ff02::fa17	64023
Reasoning	250 is decimal representation for fa	fa17 looks like fait in fait accompli	64023 is the decimal representation of fa17

All these Values have been checked against IANA to be not of conflict with any existing reservations.

Step 2 - Full Integration of MDNS to Supply Data to NodeGraph

The NodeGraph is expanded to support multiple IPv4 and IPv6 addresses.

This will allow us to integrate MDNS into the NodeGraph, so that it is used as the central Node address book.

type NodeData = {
  addresses: Array<NodeAddress>;
  lastUpdated: number;
};

This might require alot of API changes. We'll see if we can get there in time.

Additional context

• Related: js-quic integration and Agent migration #525
• Related: Using QUIC/HTTP3 to replace utp-native for the Data Transfer Layer in the networking domain #234
• Related: IPv6 Support #400
• Related: MDNS integration and NodeGraph structure expansion #537
• Related: Fix hole punching to work in both directions #605

Tasks

• 1. Step 1 - Integrate MDNS and it's usage into initial network entry and discovery.
  • Integrate MDNS querying into NodeConnectionManager.findNode
• 2. Step 2 - NodeGraph functionality needs to be expanded to handle connection metadata, support IPv6 and general policy for garbage collection, data updating and data sharing policy.
  • The initial sync node graph proceedure should be extracted out of NodeManager.start and be made a public function. It's usage should be called in PolykeyAgent when starting. We should make it optional for testing.
  • [ ] Integrate MDNS querying into NodeManager.syncNodeGraph
  • [ ] Integrate MDNS querying into NodeManager.getNodeAddress
  • [ ] Integrate MDNS querying into NodeManager.setNode
  • Integrate MDNS into NodeManager.findNode - this was moved out from NCM into NM now.

[tegefaulkes]

@amydevs @CMCDragonkai

[CMCDragonkai]

Also that node addresses can now have both local node addresses and global node addresses. In the mainnet/testnet, local addresses shouldn't be given out, as that leaks local network environment and could be a security risk.

However when connecting to a PKE portal, local network addresses could be shared, just like in tailscale.

In that case, it might be part of the bootstrapping configuration of PK nodes in a private network, where they can be enabled to share those local addresses.

Even when it is shared, receiving nodes need to intelligently decide and pick out which address to use to connect. In some cases, even in a LAN, they still cannot connect to the LAN address because they are not part of the same LAN. So in some cases, there may need to be concurrent/parallel attempts to the addresses, the node graph should then remember which ones worked and which ones didn't and sort addresses by concurrent priority.

[amydevs]

Some context regarding MDNS
The responder I've implemented will not re-query for records that have expired. This behaviour is deemed to be optional by the MDNS RFC. But since Polykey checks for reachability anyway, the MDNS service-removed event should only be treated as a hint and not that it has actually been taken offline.

[CMCDragonkai]

I tried out enabling LAN loopback, and I have an empirical sense of how it works now.

Most routers do not have it enabled, and many routers don't even have this feature at all.

When it is enabled, it may only be enabled on specific port forwarding rules (maybe port triggering rules), and not for the entire network.

The basic idea is that when it is enabled it is possible to use the external IP address while you're in the local network.

For example if you have your router forwarding 55555, and have a nc -l 55555 on a local computer. Then this will succeed:

nc -vz `curl -Ss -4 icanhazip.com` 55555

If you don't have a service listening 55555, you'd get Connection refused.

If loopback isn't enabled, you'd also get a connection refused or a timeout.

Basically this feature is primarily for convenience. It means that while you're in the local network, you can use the external IP to access internal services. It's convenient because if you are sharing scripts between others, you don't have to remember to swap the IP address from internal to external or vice versa when running the scripts from an external location or from a local location.

I asked myself why is this not on by default? It seems like a convenient feature. The only security issue I discovered is mentioned here on SO:

Anyway this means we generally have to assume that this feature does not exist or does not work and not rely on it. It appears most organisations (that need to expose an internal service over NAT) rely on split DNS to deal with this problem, and instead use a hostname that resolves to an external IP or local IP depending on the situation.

What does this mean for the NodeGraph? Basically for P2P to work between agents on the local network, it is necessary for us to expand our NodeGraph to contain both local IPs and external IPs. When contacting the seed node, the seed nodes will tell the local node what its external IP is, but it cannot tell it what its internal IP is. Only the node itself can find that out by inspecting its local interfaces.

This means MDNS is necessary to use multicast to find other nodes on the same local network. Multicast of course is also subject to the local network rules that the router has, if it disables multicast, then multicast becomes broken too. Finally there's also broadcast... but if multicast is disabled, most likely broadcast is disabled too. The idea is that the nodegraph will have multiple IPs that are potential IPs of the same node.

To prevent leaking internal network information, one has to add classification rules (categorisation or tagging) to the node graph entries. So that internal address entries are only shared with other nodes that are on the same internal network. This can be done by inspecting and comparing the source-IP of any RPC requests. This is a fairly minimal security risk... and for private PK networks, it's fine to tell the root nodes what the internal addresses are. We can observe this behaviour in tailscale, where all nodes advertise their internal network to tailscale too. But in our public mainnet and testnet, we would not want to do this.

Another thing... we might want to also expand our address types to include non-network addresses. Let's consider the generic idea of a "beacon" or "lighthouse" as a rendezvous point where nodes can discover each other.

There are multiple layers of rendezvous points, each one increasing its scope of operations.

1. The first rendezvous point can be a filesystem location. This is useful for PK agents running on the same host operating system, and enables easy virtualisation, by simply changing this filesystem location (imagine VMs and docker containers bind-mounts).
2. The second rendezvous point can be MDNS, this is useful for local networks.
3. The third rendezvous point can be a PK private network hosted on PKE.
4. The fourth rendezvous point can be PK public mainnet and testnet thus representing the global network (also hosted on PKE - but presented publically).

Given our desire to create a decentralised signalling and relaying system, all rendezvous point is just a "starting" point, subsequent signalling and relaying should rely on the network itself to help scale it. We can call this a "self-scaling" technology.

[amydevs]

I remember in a conversation we had earlier in the year
, that we will have two hard coded groups of addresses for now, being public (accessible via Holepunching, and is shared with other nodes) and private (accessible via LAN, is not shared with other nodes, and discovered through MDNS)

I also remember one of us saying that private addresses should be shared with other nodes when the node is bootstrap with a private PKE instance by default.

[CMCDragonkai]

It's like classification tiers.

There may actually be more tiers than that, so I was thinking of adding a classification label into the node graph structure some how.

Just make a quick solution for the launch, then new issue to rearchitect it properly. But try not to build too much technical debt.

[CMCDragonkai]

I remember at the very least we have:

1. Local Machine
2. Local Network
3. Global Network

Right now the goal is for MDNS to cover 1 and 2, since kademlia can't discover 1 and 2.

Note that 1 should actually be found via also a filesystem discovery using like a common directory location like /var/run or /run. However we can do that later.