Linux EDAC & rasdaemon Memory Telemetry: Detecting ECC DRAM Degradation in Dedicated Servers

In mission-critical enterprise environments—whether running high-frequency fintech databases, distributed Redis clusters, or high-throughput virtual machine hypervisors—unplanned downtime is unacceptable.

Modern enterprise servers rely on Error-Correcting Code (ECC) DDR4 and DDR5 memory to detect and correct single-bit memory flips caused by cosmic rays, thermal stress, and microscopic silicon degradation.

However, many sysadmins treat ECC memory as a magic shield: as long as the server doesn’t crash, they assume the memory subsystem is in pristine health.

In reality, silicon degrades progressively. A memory cell rarely fails catastrophically on day one; instead, it begins generating an escalating cascade of Correctable Errors (CE) on a specific memory channel, bank, or row. If left unmonitored, these correctable flips inevitably cascade into an Uncorrectable Error (UE), triggering a fatal Machine Check Exception (MCE) and hard kernel panic.

To prevent sudden outages, modern Linux kernels provide advanced hardware telemetry through EDAC (Error Detection and Correction) and rasdaemon. In this engineering guide, we examine how to deploy and query kernel RAS telemetry, track DRAM degradation down to the physical DIMM socket, and automate proactive memory page retirement.

---

🧠 The Evolution: From Legacy EDAC to rasdaemon

Historically, the Linux kernel handled memory error telemetry through the EDAC driver subsystem (/sys/devices/system/edac/mc/).

Legacy EDAC queried hardware memory controller registers through periodic polling. While effective on older dual-channel systems, modern multi-socket server platforms (such as dual AMD EPYC 9004 series or Intel Xeon Scalable Emerald Rapids) feature complex topologies with up to 12 memory channels per socket, sub-channels, and NUMA nodes. Polling registers at scale introduced kernel jitter and struggled to map errors accurately to physical motherboard silkscreen labels.

+-----------------------------------------------------------+
|               Linux Kernel Tracepoints (tracefs)          |
|    ras:mc_event  |  ras:non_standard_event  |  ras:aer_event  |
+-----------------------------------------------------------+
                            │ (Zero-copy Perf Ring Buffer)
                            ▼
+-----------------------------------------------------------+
|                 Userspace rasdaemon Daemon                |
|       Logs events, timestamps, socket, channel, DIMM      |
+-----------------------------------------------------------+
                            │
                            ▼
+-----------------------------------------------------------+
|            SQLite Database / Journal Telemetry            |
|              /var/lib/rasdaemon/ras-mc_event.db           |
+-----------------------------------------------------------+

Enter rasdaemon: the modern Reliability, Availability, and Serviceability (RAS) logging tool for Linux. Instead of polling, rasdaemon taps directly into Kernel Tracepoints (ras:mc_event). When the CPU’s integrated memory controller (IMC) corrects an ECC error, the hardware issues an interrupt, the kernel logs the event via tracepoint, and rasdaemon captures the exact physical location without CPU polling overhead.

---

📦 Step 1: Installing and Enabling rasdaemon

rasdaemon is packaged in standard enterprise repositories across AlmaLinux, Rocky Linux, Ubuntu Server, and Debian.

On RHEL / AlmaLinux / Rocky Linux 9:

sudo dnf install -y rasdaemon sqlite
sudo systemctl enable --now rasdaemon

On Ubuntu Server 22.04 / 24.04 LTS:

sudo apt update
sudo apt install -y rasdaemon sqlite3
sudo systemctl enable --now rasdaemon

Verify that the daemon is actively listening to kernel RAS tracepoints:

sudo rasdaemon --status

Output:

rasdaemon: ras:mc_event event enabled
rasdaemon: ras:aer_event event enabled
rasdaemon: ras:extlog_event event enabled

---

🔍 Step 2: Querying Live DRAM Health & Error Records

rasdaemon maintains an internal SQLite database storing every hardware telemetry event recorded since deployment:

# Query memory controller error summaries
sudo ras-mc-ctl --error-count

A healthy enterprise server will report clean counters:

Memory controller events summary:
    Corrected errors: 0
    Uncorrected errors: 0

Detailed Event Auditing

If correctable errors have been logged, inspect the exact DIMM location:

sudo ras-mc-ctl --summary
sudo ras-mc-ctl --errors

Sample output indicating a degrading DIMM:

************************************************
Timestamp: 2026-10-04 11:24:18 +0500
Error type: Corrected error
Error count: 42
Label: 'CPU_SrcID#0_MC#1_Chan#0_DIMM#0'
MC: 1, Topo: channel:0, slot:0
Location: csrow:0, channel:0
Grain: 8
Syndrome: 0x00000000
Driver detail: Memory read error at physical address 0x7b4a28000
************************************************

Notice the level of granular forensic detail:

1. Error Count: 42 corrected single-bit flips within a short window.
2. Label: CPU_SrcID#0_MC#1_Chan#0_DIMM#0 identifies the exact physical slot on the motherboard.
3. Physical Address: 0x7b4a28000 pinpoints the precise memory page triggering the fault.

---

🧮 Calculating Error Thresholds for Preventive DIMM Replacement

Not every single correctable error warrants pulling a server offline. Cosmic rays naturally cause random single-bit flips at an expected rate of approximately 1 flip per 16GB of DRAM per month.

However, hardware degradation follows distinct statistical patterns:

Error Signature	Rate of Occurrence	Root Cause Diagnosis	Action Required
Isolated CE	1–2 events per month on random addresses	Cosmic ray soft error	Normal operation; no action needed
Repeat Cell CE	Same physical address (0x7b4a...) repeating	Leaky capacitive cell in DRAM die	Soft-offline page; monitor
Row / Column Flood	Hundreds of CEs across the same row	Failing address line or sense amplifier	Replace DIMM within 48 hours
Multi-Bit Burst	Rapidly accelerating CE count (>50/hour)	Imminent silicon breakdown	Emergency maintenance (UE risk)

---

🛡️ Step 3: Proactive Kernel Mitigation: Soft-Offlining Degraded Pages

When rasdaemon flags a specific physical memory address repeatedly experiencing correctable errors, you do not have to wait for hardware replacement to protect your system from a crash.

The Linux kernel features Soft-Offline Page Retirement (CONFIG_MEMORY_FAILURE). This allows the kernel to migrate any active processes residing in the degraded 4KB memory page to healthy RAM, and permanently decommission the physical page from the buddy allocator:

# Verify kernel support for memory failure isolation
cat /boot/config-$(uname -r) | grep CONFIG_MEMORY_FAILURE

To manually soft-offline a degraded physical memory page (e.g., address 0x7b4a28000 identified in rasdaemon logs):

# Convert physical address to page frame number (PFN = address >> 12)
# 0x7b4a28000 >> 12 = 0x7b4a28 (or decimal 8080040)

# Soft offline the page safely
echo 0x7b4a28 > /sys/devices/system/memory/soft_offline_page

Check the kernel buffer to confirm clean page migration:

dmesg | tail -n 10

[ 1420.892014] Soft offlining pfn 0x7b4a28 at address 0x7b4a28000
[ 1420.892150] Soft offlining pfn 0x7b4a28: page migrated successfully

The faulty memory address is now permanently isolated. Your database or hypervisor continues operating uninterrupted without risking an uncorrectable crash!

---

🏆 Enterprise Hardware Telemetry on Nextgen Bare-Metal Infrastructure

Monitoring hardware down to the silicon layer is a standard best practice across Nextgen Tier-3 datacenter deployments:

• For developers and growing SaaS platforms, our Cloud VPS in Pakistan utilizes enterprise KVM hypervisors with continuous ECC RAM telemetry and automated host failover.
• For financial institutions, high-frequency trading platforms, and mission-critical databases requiring complete bare-metal hardware ownership, deploy on Nextgen Dedicated Servers in Pakistan and international Dedicated Servers with 24/7 out-of-band IPMI hardware telemetry, automated DIMM diagnostics, and 4-hour on-site SLA hardware replacement.

---

📚 Related Server Hardware, Memory & Performance Guides

• ECC Memory Single-Bit vs Multi-Bit Errors in Servers – Deep dive into SECDED algorithms and hardware fault boundaries.
• Liquid Cooling vs High-CFM Air in Servers – Maintain optimal thermal stability for high-density DDR5 DIMMs.
• PCIe AER Advanced Error Reporting in Dedicated Servers – Catch PCIe bus and NVMe drive degradation before catastrophic failures.

---