The async fn Future state machine overhead (~1.9us) dominated
MpscTunnelSender::send, while RingSink operations were only ~40ns.
Breakthrough: make send() an async fn that completes synchronously
on the first poll for the direct (ring tunnel) path. Uses
futures::task::noop_waker() to construct a dummy Context, then calls
Sink trait methods (poll_ready, start_send, poll_flush) directly.
RingSink always returns Ready immediately, so the waker is never
invoked and the async fn completes without yielding.
Channel mode (TCP/UDP/WG tunnels) still uses async send_async()
with proper backpressure. Ring tunnels detected via tunnel_info()
type check in PeerConn.
Results (4 threads, 1400B, 15s):
pps: 249K → 474K (+90%)
send_msg_by_ip: 3.53us → 1.67us (-53%)
send_msg_internal: 2.40us → 502ns (-79%)
MpscTunnelSender::send: 1.97us → 144ns (-93%)
All 207 peers:: tests pass. Netns-requiring tests (three_node,
credential) unchanged (require root).
tokio::sync::Mutex and std::sync::Mutex both have !Send guards that
cannot cross await points in multi_thread runtime. Replace with a
custom SpinSink using AtomicBool CAS — the SpinGuard contains only a
&SpinSink reference (SpinSink: Sync via unsafe impl), so it is Send.
Benchmark: pps unchanged (~249K), MpscTunnelSender::send avg 1.97us.
The bottleneck is confirmed to be async fn Future state machine
overhead (~1.9us), not the lock mechanism. RingSink operations are
only ~40ns (poll_ready 15ns + start_send 10ns + poll_flush 15ns).
Further breakthrough requires either:
- Sync send API (bypassing async entirely)
- Concrete type instead of dyn ZCPacketSink (to call RingSink::try_send directly)
Replace 3 await points (lock().await + feed().await + flush().await)
with try_lock() (sync) + single poll_fn (merged poll_ready + start_send
+ poll_flush).
parking_lot::Mutex cannot be used because MutexGuard is !Send (cannot
cross await in multi_thread runtime). tokio::sync::Mutex try_lock()
returns synchronously and MutexGuard is Send.
Benchmark: pps 250K → 251K (+0.4%), MpscTunnelSender::send avg
2.07us → 1.98us (-90ns). Improvement is small because tokio async
machinery overhead (Future state machine + poll) dominates over
RingSink's actual 40ns operation cost.
MpscTunnelSender now supports two modes:
- Channel mode (existing): try_send to tokio mpsc → receiver task → sink
- Direct mode (new): MpscTunnelSender holds Arc<Mutex<sink>> directly,
bypassing the channel + receiver task entirely
PeerConn uses new_direct to skip the channel intermediary.
Benchmark result: pps unchanged (~245K). The async fn overhead of
Mutex::lock().await + SinkExt::feed().await + SinkExt::flush().await
(~2us) is comparable to channel try_send (~2us). The bottleneck is
the Sink trait's async poll machinery, not the channel itself.
However, this change provides:
- RingSink timing now fully visible (start_send 10ns, poll_ready 13ns,
poll_flush 17ns = 40ns/pkt total)
- Reduced architectural complexity (no receiver task for PeerConn)
- Foundation for a sync fast path using RingSink::try_send directly
channel(32) was frequently full under high pps, causing try_send to
fail and fall back to send().await (semaphore wait). Increasing to 1024
reduces fallback frequency.
Benchmark (4 threads, 1400B, 15s):
MpscTunnelSender::send avg: 2.23us → 2.01us (-220ns)
MpscTunnelSender::send P95: 6.39us → 5.74us (-650ns)
send_msg_internal avg: 2.67us → 2.45us (-220ns)
pps: ~250K (unchanged, receiver-bound)
pps unchanged because bottleneck moved to receiver (forward_one_round →
sink.feed/flush). The 220ns/pkt saving is pure CPU efficiency gain.
Adds hotpath::measure to two critical blind spots in the send chain:
1. MpscTunnelSender::send — the tokio mpsc channel send point, which
accounts for 84% of PeerConn::send_msg wall time (2.33us/pkt).
2. PeerManager::send_msg_by_ip — the top-level packet send entry point,
revealing a 1.04us gap between send_msg_by_ip and send_msg_internal
(encryption + routing + ACL + fan-out).
Full send chain timing now visible:
send_msg_by_ip: 3.83us
└─ send_msg_internal: 2.79us (gap: 1.04us = encrypt + route + ACL)
└─ MpscTunnelSender::send: 2.33us (84% of internal)
MpscTunnelSender::send now tries try_send first, falling back to
send().await only when the channel is full. try_send bypasses the
tokio batch_semaphore Acquire::poll + add_permits_locked machinery
(~9.4% of CPU in samply profiling), which is pure overhead when the
channel has capacity.
In the ring-tunnel bench (4 threads, 1400B, 15s) the channel(32) fast
path hits >99%, so the fallback rarely triggers.
Benchmark improvement:
pps: 230K -> 246K (+7.0%)
send_msg_internal avg: 3.25us -> 3.13us (-120ns/pkt)
forward_one_round calls: 2.46M -> 706K (-71%, bigger batches)
All mpsc tests pass.
This patch implements:
1. A dns server that handles .et.net. zone in local and forward all other queries to system dns server.
2. A dns server instance which is a singleton in one machine, using one specific tcp port to be exclusive with each other. this instance is responsible for config system dns and run the dns server to handle dns queries.
3. A dns client instance that all easytier instance will run one, this instance will try to connect to dns server instance, and update the dns record in the dns server instance.
this pr only implements the system config for windows. linux & mac will do later.
* fix peer rpc stop working because of mpsc tunnel close unexpectedly
* fix gui:
1. allow set network prefix for virtual ipv4
2. fix android crash
3. fix subnet proxy cannot be set on android
This patch removes Tarpc & Tonic GRPC and implements a customized rpc framework, which can be used by peer rpc and cli interface.
web config server can also use this rpc framework.
moreover, rewrite the public server logic, use ospf route to implement public server based networking. this make public server mesh possible.
fanyang