eBPF From Rust: The 34ms Ghost We Couldn't See

It sounds like you've uncovered some significant performance bottlenecks in your system using eBPF (Extended Berkeley Packet Filters), which is a powerful tool for tracing and monitoring Linux systems at the kernel level. Here's a summary of what you discovered:

1. Dynamic Library Loading Overhead:

   • You were loading dynamic plugins on every request, causing the dynamic linker to perform extensive file system operations like stat(), open(), and mmap() for each plugin.
   • This resulted in significant kernel time consumption (37ms of kernel time per 2ms reported by userspace profilers).
   • Solution: Cache loaded plugins to avoid redundant loading on every request.

2. TCP Retransmissions:

   • You traced TCP retransmissions and found a high number of retransmits (847 per second) for one specific connection.
   • This was due to incorrect TCP window size tuning, buffer overflow in the high-throughput path, and issues specific to an AWS availability zone.
   • Solution: Adjusted TCP window scaling, which reduced retransmissions by 99.2% and improved latency.

3. Filesystem Operations:

   • You were able

Read the full article at DEV Community

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.