When you are running a telemetry server that maintains 2,000+ concurrent TCP sessions from GPS and RFID devices, memory matters. We discovered our production pipeline was consuming 4.4GB of RAM - for what should have been a lightweight data relay. Here is how we diagnosed the problem and reduced it to 35.9MB.
The Symptom
Our telemetry pipeline follows a straightforward architecture: devices connect via TCP, data flows through a session manager, gets routed through RabbitMQ, and lands in a database handler. During routine monitoring, we noticed the process had ballooned to 4.4GB. For a service that reads bytes from sockets, parses them, and forwards to a message broker, that is orders of magnitude too high.
The Root Cause: Unbounded Deduplication Cache
The culprit was our MessageDeduplicationCache. The original implementation used a ConcurrentDictionary with no eviction policy. Every unique message hash was stored indefinitely. With thousands of devices sending telemetry every few seconds, the dictionary grew without bound.
The Fix: LRU Eviction with TTL
We redesigned the cache with three constraints:
- Entry limit: Maximum 50,000 entries
- TTL: 5-minute expiration per entry
- LRU eviction: Least recently seen hashes evicted first
public class MessageDeduplicationCache
{
private readonly int _maxEntries = 50_000;
private readonly TimeSpan _ttl = TimeSpan.FromMinutes(5);
private readonly ConcurrentDictionary<string, DateTime> _cache = new();
public bool IsDuplicate(string hash)
{
CleanExpired();
if (_cache.TryGetValue(hash, out _)) return true;
if (_cache.Count >= _maxEntries) EvictOldest();
_cache[hash] = DateTime.UtcNow;
return false;
}
}Results
Memory dropped from 4.4GB to 35.9MB. Duplicate detection rate went from about 35% down to about 3.3%, which represented actual retransmissions.
The lesson: any collection that grows with input volume and never shrinks is a memory leak waiting to happen.
What Else We Found
While investigating, we also fixed async void event handlers swallowing exceptions, Thread.Sleep in async contexts blocking the thread pool, and blocking waits on async operations causing thread starvation.
