Debugging & Troubleshooting — Top 150 Interview Q&A


Core Debug Tools: strace

Q1: What is strace and how does it work internally?

Answer: strace uses the ptrace() syscall to intercept every system call a process makes. The kernel pauses the tracee before and after each syscall, allowing the tracer to inspect arguments and return values. Output format: syscall(arg1, arg2, ...) = return_value. The ptrace mechanism has ~2–10× overhead.

Q2: How do you attach strace to a running process?

Q3: How do you filter strace to only specific syscalls?

Q4: How do you use strace to find why a program can't find a file?

Look for openat(...)= -1 ENOENT (No such file or directory) to identify missing files.

Q5: How do you measure syscall time with strace?

Q6: What does strace output tell you about a hung process?

Answer: Attach strace -p <pid>. If the process is stuck in a blocking syscall, strace will show it unfinished (e.g., read(5, <unfinished ...>) or futex(...) or poll(...)) — revealing what it's waiting on (pipe, socket, lock, signal).

Q7: How do you trace a program and all its child processes?

Q8: How do you find which files a process opens?

Or check /proc/<pid>/fd/ for a running process.


Core Debug Tools: ltrace

Q9: What is ltrace and how does it differ from strace?

Answer: ltrace intercepts calls to dynamic library functions (e.g., malloc, printf, strlen) using breakpoints on PLT entries. strace intercepts kernel boundary (system calls). Use ltrace to debug application-level logic; strace for kernel interactions.

Q10: How do you filter ltrace output?


Core Debug Tools: GDB

Q11: How do you debug a segfault with GDB?

Answer:

  1. Compile with -g (debug symbols).

  2. Enable core dumps: ulimit -c unlimited.

  3. Run program → crash → gdb ./prog core.

  4. (gdb) bt — backtrace shows call stack at crash.

  5. (gdb) frame N — switch to frame N; (gdb) info locals — view variables.

Q12: How do you set a conditional breakpoint in GDB?

Q13: How do you inspect memory in GDB?

Q14: What are watchpoints in GDB?

Hardware watchpoints use CPU debug registers for near-zero overhead.

Q15: How do you debug a multi-threaded program in GDB?

Q16: How do you debug a running production process with GDB without stopping it?

Q17: What is GDB's reverse debugging?

Answer: GDB supports recording execution and stepping backwards (record, reverse-next, reverse-continue). Useful for finding the cause of corruption. Overhead is significant; typically used for complex heisenbug scenarios.

Q18: How do you use GDB with Python scripts?

Answer: GDB has a Python API for automation:

Useful for custom pretty-printers and automated analysis.


Core Debug Tools: perf

Q19: What is perf stat and what does it measure?

Reports: task clock, context switches, CPU migrations, page faults, cycles, instructions, IPC (instructions per cycle), cache references/misses, branch misses. Reveals CPU efficiency and memory bottlenecks.

Q20: How do you profile CPU hotspots with perf?

Shows which functions consume the most CPU time.

Q21: What is perf top?

Like top but at the function level. Shows which kernel/user functions are hottest.

Q22: How do you use perf to find cache misses?

High cache miss rate indicates poor data locality.

Q23: What is flame graph and how do you generate one from perf?

Flame graphs visualize the call stack frequency — wider frames = more samples = more CPU time. Brendan Gregg's FlameGraph scripts generate them.

Q24: What is perf trace?

Uses tracepoints and syscall events with significantly lower overhead than ptrace-based strace.


Log Analysis

Q25: How do you parse logs to find the most common errors?

Q26: How do you follow multiple log files simultaneously?

Q27: How do you find when a system last rebooted?

Q28: How do you search for patterns in compressed log files?

Q29: How do you extract lines between timestamps in logs?

Q30: What does /var/log/auth.log or /var/log/secure contain?

Answer: Authentication events: SSH logins (successful and failed), sudo usage, PAM events, su attempts. Critical for security auditing. journalctl _SYSTEMD_UNIT=ssh.service provides the same on systemd systems.

Q31: How do you investigate a kernel oops from logs?

A kernel oops shows: the fault type, register dump, and call trace. The address can be decoded with addr2line or ksymoops.

Q32: What is logrotate and how does it work?

Answer: logrotate compresses and rotates log files on a schedule (daily/weekly/size-based), controlled by /etc/logrotate.conf and /etc/logrotate.d/. After rotation, it can send SIGHUP to the logging daemon or run postrotate scripts to reopen the log file (copytruncate avoids this need).


Process & Resource Issues

Q33: How do you diagnose a full process table?

Answer:

Common causes: fork bomb, runaway process spawning workers, zombie accumulation. Fix: identify the creator process, kill it, fix the parent to call wait().

Q34: How do you handle a fork bomb?

Answer: A fork bomb (:(){ :|:& };:) rapidly exhausts the process table. Prevention: ulimit -u 100 per user; systemd TasksMax in service files. Recovery: need a root shell not subject to limits, kill the fork bomb processes. Reboot may be necessary if the system is unresponsive.

Q35: How do you find which process is consuming the most memory?

Q36: How do you find which process is consuming the most CPU?

Q37: How do you investigate high CPU usage by a specific process?

Q38: How do you diagnose a process stuck in D state (uninterruptible sleep)?

Answer: D-state means the process is waiting for I/O (usually). It cannot be killed. Diagnose:

Common causes: NFS hang, disk error, kernel bug. Fix: unmount stuck NFS, fix disk, or reboot.

Q39: How do you find and kill zombie processes?

Q40: What does high iowait (wa%) in top indicate?

Answer: High iowait means CPUs are idle while waiting for I/O to complete — disk I/O is the bottleneck. Check:

Q41: How do you find what is holding a file descriptor open after a file is deleted?

Q42: How do you investigate high system call rate?

High system time relative to user time suggests too many syscalls — batching I/O, reducing lock contention, or using poll() instead of multiple syscalls.

Q43: What is a "thundering herd" problem?

Answer: When many processes/threads are waiting on the same event (e.g., accept()), and the event fires, all are woken but only one can proceed — the rest go back to sleep. This wastes CPU. Fix: EPOLLEXCLUSIVE flag in Linux 4.5+, SO_REUSEPORT for accept, pthread_cond_signal instead of broadcast.

Q44: How do you debug a deadlock?

Answer:

Look for circular wait: thread A holds lock1, waits for lock2; thread B holds lock2, waits for lock1.

Q45: How do you find open network connections and their state?

High TIME_WAIT: normal after many short connections. High CLOSE_WAIT: application not closing sockets.

Q46: What is a file descriptor leak and how do you detect it?

Answer: FD leak occurs when a process opens files without closing them, eventually hitting EMFILE (too many open files for process) or ENFILE (system-wide limit).

Q47: How do you diagnose memory pressure on a system?

Q48: What does /proc/<pid>/smaps tell you?

Answer: smaps provides per-mapping memory details: start/end address, permissions, size, RSS, PSS, shared/private clean/dirty pages. Useful for understanding exactly what memory a process is using and why RSS is high (e.g., shared library vs. heap).

Q49: How do you investigate a sudden spike in memory usage?

Q50: How do you diagnose high context switch rate?

Causes: too many threads competing, excessive locking, high interrupt rate. Reduce thread count, improve lock granularity, batch work.


Network Debugging

Q51: How do you capture and analyze network traffic?

Q52: How do you check for network packet loss?

Q53: How do you diagnose slow network connections?

Q54: What does netstat -s or ss -s show?

Answer: Aggregate socket statistics: TCP/UDP/ICMP packet counts, errors, retransmits, resets, failed connection attempts. Useful for spotting TCP health issues without per-connection detail.

Q55: How do you check if a firewall is blocking traffic?

Q56: How do you investigate high network latency from a process perspective?

Q57: What does SO_REUSEADDR vs SO_REUSEPORT do?

Answer: SO_REUSEADDR: allows binding to a port in TIME_WAIT state (essential for server restarts). SO_REUSEPORT: allows multiple sockets on the same process/different processes to bind to the same port — kernel load-balances incoming connections. Used for multi-threaded/multi-process servers.

Q58: What is /proc/net/tcp and how do you read it?

Answer: Contains per-socket TCP state in hex. Fields: local address:port, remote address:port, state (01=ESTABLISHED, 0A=LISTEN), tx/rx queue, uid, inode. Decode hex addresses with printf '%d.%d.%d.%d\n' $(echo "0100007F" | sed 's/../0x& /g' | awk '{for(i=NF;i>0;i--) printf $i" "}'). Usually easier to use ss.

Q59: How do you debug DNS resolution issues?

Q60: How do you measure network bandwidth utilization?


Performance Profiling

Q61: How do you profile Python application performance?

Q62: How do you identify I/O bottlenecks?

%util > 90% means device saturated. High await means high I/O latency.

Q63: What is the difference between throughput and latency in I/O?

Answer: Throughput: data transferred per unit time (MB/s). Latency: time to complete a single I/O operation (ms). They trade off: large sequential I/O = high throughput but potentially higher latency per operation. Random small I/O = low throughput, high latency per op.

Q64: How do you use vmstat for performance analysis?

Key columns: r (run queue length), b (blocked on I/O), si/so (swap in/out), us/sy/id/wa (CPU breakdown), cs (context switches/s), in (interrupts/s).

Q65: What does pidstat do?

Q66: How do you trace kernel function calls?

Q67: What is bpftrace and give an example use case?

bpftrace is a high-level language for eBPF tracing — powerful and low overhead.

Q68: How do you profile lock contention?

Q69: How do you identify NUMA issues?

Q70: What is slab memory usage and how do you check it?

High slab usage for inodes/dentries indicates many open files or heavy filesystem operations.


Full Process Table Scenarios

Q71: A system shows "fork: retry: Resource temporarily unavailable". What do you check?

Answer:

Fix: increase pid_max temporarily, find and stop the runaway process/user.

Q72: How do you kill all processes of a specific user?

Q73: How do you prevent a fork bomb from bringing down the system?

Answer: In /etc/security/limits.conf or /etc/security/limits.d/:

In systemd service files: TasksMax=100. This limits the number of processes/threads per user or service before a fork bomb can exhaust system PIDs.

Q74: A service keeps spawning new processes; how do you trace the origin?

Q75: What is the /proc/sys/kernel/pid_max limit and can you change it?

Answer: Default is 32768 (32-bit compat). Max on 64-bit systems is 4194304. Change temporarily:

Persist in /etc/sysctl.conf: kernel.pid_max = 65536.

Q76: How do you identify which process is spawning excessive children?

Q77: What is pstree useful for?

Visualizes the fork hierarchy — useful for understanding process relationships and finding the root of a process storm.


Crash Analysis & Core Dumps

Q78: How do you configure the system to save core dumps?

Q79: What information does a core dump contain?

Answer: The core dump contains: all memory mappings (heap, stack, text, mmap regions), register state, open file descriptors, signal information, and thread stacks. It's a snapshot of the process at crash time. Does NOT include disk files or network state.

Q80: How do you analyze a SIGSEGV (segfault) from a core dump?

Q81: What causes a SIGBUS vs SIGSEGV?

Answer:

Q82: How do you debug a stack overflow?

Answer: Stack overflow causes SIGSEGV when accessing a guard page (unmapped page below the stack). GDB backtrace may be corrupted (infinite recursion). Signs in backtrace: many identical frames or ?? entries. Fix: increase stack size (ulimit -s), reduce recursion depth, or use heap-allocated recursion.

Q83: What is an address sanitizer (ASan) and how do you use it?

Detects: use-after-free, heap/stack/global buffer overflow, use-after-return, use-after-scope. ~2× slowdown. Essential in CI/CD testing.

Q84: What is ThreadSanitizer (TSan)?

Detects data races in multi-threaded programs by tracking memory accesses and synchronization operations at runtime.

Q85: What is UndefinedBehaviorSanitizer (UBSan)?

Detects: signed integer overflow, null pointer dereference, misaligned accesses, invalid enum values, and other C/C++ undefined behavior.

Q86: How do you analyze a kernel panic?

Answer: A kernel panic prints a call trace to the console/serial port and optionally writes a crashdump (via kdump). Analyze with crash utility on the vmcore:

Q87: What is kdump and how does it work?

Answer: kdump uses a crash kernel (second kernel reserved in memory) that boots automatically when the primary kernel panics. It saves the crashed kernel's memory to a vmcore file (usually /var/crash/). The system then reboots normally. Configure in /etc/kdump.conf; enable with systemctl enable kdump.

Q88: How do you debug a memory corruption issue (heap corruption)?

Answer:

  1. Compile with gcc -fsanitize=address — AddressSanitizer gives exact location.

  2. Use valgrind --tool=memcheck for runtime detection.

  3. Check glibc malloc debugging: MALLOC_CHECK_=3 ./prog.

  4. Use electric fence (efence) library to detect buffer overflows immediately.

  5. In GDB: watch *(type*)address to catch the write.

Q89: What does "double free" mean and how do you detect it?

Answer: Calling free() on the same pointer twice — undefined behavior that corrupts the allocator's internal structures (freelist), potentially leading to arbitrary code execution or crashes. Detection: AddressSanitizer reports "double free"; valgrind --tool=memcheck also catches it.

Q90: How do you debug a use-after-free bug?

Answer: Use-after-free: accessing memory after free() — it may have been reallocated. Symptoms: corrupted data, crashes at seemingly unrelated points. Detection: AddressSanitizer (-fsanitize=address) quarantines freed memory; valgrind memcheck poisons it. Both report the exact allocation, free, and use locations.


System-Level Troubleshooting Scenarios

Q91: A server is slow — what is your systematic approach?

Answer:

  1. High level: top / uptime (load average, CPU/memory/IO wait).

  2. CPU: top, mpstat, perf top — is it user, kernel, or iowait?

  3. Memory: free -h, vmstat (swap activity), /proc/meminfo.

  4. I/O: iostat -x 1 (disk saturation), iotop (top I/O process).

  5. Network: ss -s, iftop, netstat -s (errors/retransmits).

  6. Processes: ps, strace -c -p <pid>.

  7. Logs: journalctl, /var/log/syslog.

Q92: A process consumes 100% CPU and you can't kill it. What do you do?

Answer:

Q93: How do you recover from a full disk (/ is 100% full)?

Answer:

Q94: A process is leaking memory. How do you confirm and identify the source?

Answer:

Q95: How do you debug a program that hangs at startup?

Q96: SSH connection drops after some time — how do you diagnose?

Answer:

Q97: A service fails to start. How do you debug it?

Q98: How do you debug a "too many open files" error?

Q99: How do you investigate a "connection refused" error?

Q100: How do you debug intermittent failures that are hard to reproduce?

Answer:

  1. Enable verbose logging in the application.

  2. Set up continuous monitoring: strace/perf trace with output to file.

  3. Use bpftrace probes at key kernel points without stopping the process.

  4. Analyze metrics over time with sar, collectd, or Prometheus.

  5. Enable core dumps (ulimit -c unlimited, systemd-coredump) to capture crash state.

  6. Add timing instrumentation and check for timing-sensitive (race condition) patterns.


Advanced Debugging Techniques

Q101: What is addr2line and when do you use it?

Used to decode hex addresses in crash logs, kernel oops traces, or profiling output into human-readable source locations. Requires debug symbols or compile with -g.

Q102: What is objdump and what can it show?

Q103: What is nm used for?

Q104: What is readelf and how does it differ from objdump?

Answer: Both read ELF files. readelf is ELF-specific and more detailed for ELF structures; objdump also supports other binary formats and has disassembly. Use readelf -d binary for dynamic linking info, -S for section headers, -l for program headers.

Q105: How do you trace system calls across a fork without losing output?

Q106: How do you use gdb's printf to add instrumentation without recompiling?

Insert print statements at runtime without modifying/recompiling source.

Q107: What is SystemTap?

Answer: SystemTap is a scripting language/infrastructure for live kernel and user-space instrumentation. Like bpftrace but older; uses kernel modules. Allows tracing kernel internals, user functions, and system calls with full scripting. Largely superseded by eBPF/bpftrace on modern kernels.

Q108: What is ftrace and how do you use it?

ftrace is the kernel's built-in function tracer. Can trace function call graphs, latency, interrupts. Zero overhead when disabled. trace-cmd provides a user-friendly wrapper.

Q109: How do you debug inter-process communication issues?

Q110: How do you debug a race condition in production?

Answer:

  1. Build with ThreadSanitizer (-fsanitize=thread) in staging — it will detect races with context.

  2. In production: use lockstat/perf lock to identify heavily contended locks.

  3. Add atomic counters or timestamps around suspected regions.

  4. Use rr (Mozilla's record-and-replay debugger) to deterministically replay non-deterministic bugs.

  5. Review code for lock-free data structure misuse.

Q111: What is Mozilla rr and when is it useful?

Answer: rr records a deterministic execution trace of a process (including all hardware inputs, syscalls, signals) and allows replaying it exactly, including backwards. Invaluable for heisenbug: record once, reproduce/debug indefinitely. Supports GDB interface (rr replay).

Q112: How do you find where a crash is coming from in a stripped binary?

Answer:

Q113: What is catchsegv?

Wrapper that catches SIGSEGV/SIGBUS and prints a backtrace using libc's backtrace mechanism. Quick alternative to GDB when debug symbols aren't available.

Q114: How do you debug high CPU in a kernel thread (kworker, ksoftirqd)?

Q115: What tools help with Java application debugging on Linux?

Q116: How do you debug Python memory leaks?

Q117: What is sysdig and how does it differ from strace?

Answer: sysdig is a system-level observability tool using a kernel module (or eBPF). Unlike strace (per-process, ptrace), it captures all system calls system-wide with rich filtering. Can save to file and replay. csysdig is an interactive UI. Good for container environments (aware of container metadata).

Q118: How do you debug a performance regression between two versions of code?

Answer:

  1. Use perf stat to compare hardware counters (cycles, instructions, cache misses) between versions.

  2. Use perf diff to compare two perf.data files.

  3. Use cachegrind (valgrind) for instruction-level cache simulation — deterministic.

  4. Profile both with flame graphs and compare visually.

  5. Use microbenchmarks (benchmark library) to isolate specific functions.

Q119: What is numactl and when do you need it?

On multi-socket systems, remote NUMA accesses are 2–4× slower. Binding workloads to a single NUMA node eliminates remote memory latency.

Q120: How do you investigate TLB thrashing?

High TLB misses indicate working set too large for TLB coverage. Use huge pages or madvise(MADV_HUGEPAGE).


Crash Dump & Post-Mortem

Q121: What is the difference between a core dump and a minidump?

Answer: A core dump (Linux/POSIX) contains all mapped memory regions of the process — can be several GB. A minidump (Windows-originated, also used in some tools) is a smaller, structured dump with thread stacks, registers, and selected memory — easier to transfer and analyze. Tools like Breakpad/Crashpad generate minidumps.

Q122: How do you extract a stack trace from a core dump in a CI system?

Q123: How do you correlate a core dump to a specific build?

Answer: Embed build ID in binaries (-Wl,--build-id). Check readelf -n binary | grep "Build ID" and match against symbol server. systemd-coredump stores metadata (timestamp, UID, command, build ID) with each core.

Q124: What is coredumpctl and how do you use it?

Part of systemd; automatically captures and stores core dumps with metadata.

Q125: How do you debug a problem that only occurs under load?

Answer:

  1. Use wrk/ab/vegeta to apply controlled load.

  2. Attach perf record -g -p <pid> during load test.

  3. Monitor: vmstat 1, iostat -x 1, ss -s.

  4. Set up core dumps for crashes under load.

  5. Use strace -c to summarize syscall distribution under load.

  6. Profile with perf record during high-load phase, analyze afterwards.


System Observation Tools

Q126: What is dstat and what does it show?

Combines vmstat, iostat, netstat in one output. Useful for a quick holistic view.

Q127: What is atop and how is it different from top?

Answer: atop logs system and process activity persistently, allowing historical replay. Unlike top, it shows disk and network per process, indicates process exits between samples, and keeps accounting data in /var/log/atop/. atop -r /var/log/atop/atop_20240115 replays recorded data.

Q128: How do you monitor network connections over time?

Q129: What is collectl and when is it useful?

Answer: collectl is a lightweight system metrics collector that can record all major subsystem metrics (CPU, memory, disk, network, I/O) to files for later analysis. Lower overhead than sar. Useful for continuous baseline collection.

Q130: What is /proc/sys/net/ipv4/tcp_tw_reuse?

Answer: Allows reusing TIME_WAIT sockets for new connections when safe (when TCP timestamps are enabled). Helps when local ports are exhausted due to many short connections. Different from tcp_tw_recycle (deprecated in kernel 4.12 — caused issues with NAT).

Q131: How do you find which process is writing to a specific file?

Q132: How do you debug a process that ignores SIGTERM?

Answer:

Q133: What is /proc/<pid>/maps and what does it show?

Answer: Shows the virtual memory map of a process: each line has start-end address range, permissions (rwxp), offset, device, inode, and path. p=private, s=shared. Used to understand memory layout, find library versions loaded, diagnose large mappings.

Q134: What is execsnoop and opensnoop?

Answer: eBPF/bpftrace-based tools from the bcc package:

Invaluable for watching what a service does at startup or under load without attaching to a specific PID.

Q135: How do you debug NFS mount performance issues?

Q136: How do you profile interrupt handling overhead?

Q137: What is schedd_lat and how do you measure scheduling latency?

Scheduling latency = time between when a task becomes runnable and when it actually gets CPU. High latency indicates a busy system or real-time requirements not met.

Q138: How do you detect CPU frequency throttling?

Q139: What are cgroup memory events and how do you monitor them?

Q140: How do you analyze TCP connection latency?

Q141: How do you debug high system time (%sy in top)?

Answer:

High system time usually means many syscalls, memory allocation, or kernel lock contention.

Q142: What is /proc/buddyinfo and why does it matter?

Answer: Shows the kernel buddy allocator free block list per memory zone. Each column represents blocks of 2^N pages. If large allocations fail despite free memory, fragmentation is the cause — the allocator can't find a contiguous block. echo 1 > /proc/sys/vm/compact_memory triggers memory compaction.

Q143: How do you debug getaddrinfo performance issues?

Q144: What does ss --options --extended --memory --processes --info show?

Answer: The -e/--extended flag shows additional socket options. --memory/-m shows socket memory usage. Combined: for each socket shows send/receive buffer sizes, which is useful when diagnosing buffer bloat or sockets consuming excessive memory.

Q145: How do you find the source of a kernel soft lockup?

Q146: How do you test for memory errors in hardware (not software)?

Q147: What is numastat and what does it reveal?

Shows numa_miss (allocations that fell through to remote node) and numa_foreign (allocations intended for this node that went elsewhere). High miss rates indicate NUMA imbalance.

Q148: How do you analyze CPU cache behavior of a program?

Q149: What is blktrace and how do you use it?

blktrace records every block I/O request from issue through completion, revealing I/O scheduler behavior, queue depth, and per-operation latency.

Q150: How do you systematically approach debugging an unknown binary in production?

Answer:

  1. file binary — type/architecture check.

  2. ldd binary — shared library dependencies.

  3. strings binary | head -50 — readable strings (configuration, paths).

  4. strace ./binary 2>&1 | head -50 — first syscalls (what it opens first).

  5. ltrace ./binary 2>&1 | head -50 — library calls.

  6. objdump -d binary | head -100 — first instructions.

  7. readelf -d binary — dynamic section (imported symbols, RPATH).

  8. /proc/<pid>/maps — memory layout at runtime.

  9. perf top -p <pid> — CPU hotspots.

  10. lsof -p <pid> — files and network connections.