strace and how does it work internally?Answer: strace uses the ptrace() syscall to intercept every system call a process makes. The kernel pauses the tracee before and after each syscall, allowing the tracer to inspect arguments and return values. Output format: syscall(arg1, arg2, ...) = return_value. The ptrace mechanism has ~2–10× overhead.
xxxxxxxxxxstrace -p <pid>strace -p <pid> -f # follow child processes tooxxxxxxxxxxstrace -e trace=openat,read,write ./progstrace -e trace=network ./prog # all network callsstrace -e trace=file ./prog # all file-related callsstrace -e trace=process ./prog # fork, exec, wait, etc.xxxxxxxxxxstrace -e trace=openat,stat,access ./prog 2>&1 | grep -E "ENOENT|EACCES"Look for openat(...)= -1 ENOENT (No such file or directory) to identify missing files.
xxxxxxxxxxstrace -c ./prog # summary: calls, errors, time per syscallstrace -T ./prog # time spent in each syscall callstrace -tt ./prog # absolute timestamps (microseconds)Answer: Attach strace -p <pid>. If the process is stuck in a blocking syscall, strace will show it unfinished (e.g., read(5, <unfinished ...>) or futex(...) or poll(...)) — revealing what it's waiting on (pipe, socket, lock, signal).
xxxxxxxxxxstrace -f -o /tmp/trace.txt ./prog # -f follows forks, -o saves outputstrace -ff -o /tmp/trace ./prog # separate file per PIDxxxxxxxxxxstrace -e trace=openat,open -f ./prog 2>&1 | grep -v ENOENTOr check /proc/<pid>/fd/ for a running process.
ltrace and how does it differ from strace?Answer: ltrace intercepts calls to dynamic library functions (e.g., malloc, printf, strlen) using breakpoints on PLT entries. strace intercepts kernel boundary (system calls). Use ltrace to debug application-level logic; strace for kernel interactions.
xxxxxxxxxxltrace -e malloc+free+realloc ./prog # trace memory functionsltrace -l libssl.so.* ./prog # trace specific libraryAnswer:
Compile with -g (debug symbols).
Enable core dumps: ulimit -c unlimited.
Run program → crash → gdb ./prog core.
(gdb) bt — backtrace shows call stack at crash.
(gdb) frame N — switch to frame N; (gdb) info locals — view variables.
xxxxxxxxxxbreak my_func if x == 5break file.c:42 if ptr == NULLxxxxxxxxxxx/10xw 0xaddress # examine 10 words in hexx/s 0xaddress # examine as stringp *ptr # dereference pointerp/x variable # print in hexxxxxxxxxxxwatch variable # break when variable changesrwatch variable # break when variable is readawatch variable # break on read or writeHardware watchpoints use CPU debug registers for near-zero overhead.
xxxxxxxxxxinfo threads # list threadsthread N # switch to thread Nthread apply all bt # backtrace of all threadsset scheduler-locking on # only run current threadxxxxxxxxxxgdb -batch -ex "thread apply all bt" -p <pid> # non-interactivegcore <pid> # dump core without killing processgdb ./prog -c corefile # analyze the corereverse debugging?Answer: GDB supports recording execution and stepping backwards (record, reverse-next, reverse-continue). Useful for finding the cause of corruption. Overhead is significant; typically used for complex heisenbug scenarios.
Answer: GDB has a Python API for automation:
xxxxxxxxxx# In .gdbinit or gdb --batch -x script.pyimport gdbgdb.execute("bt")frame = gdb.selected_frame()Useful for custom pretty-printers and automated analysis.
perf stat and what does it measure?xxxxxxxxxxperf stat ./progReports: task clock, context switches, CPU migrations, page faults, cycles, instructions, IPC (instructions per cycle), cache references/misses, branch misses. Reveals CPU efficiency and memory bottlenecks.
xxxxxxxxxxperf record -g ./prog # record with call graphsperf report # interactive TUI reportperf report --stdio # text outputShows which functions consume the most CPU time.
perf top?xxxxxxxxxxperf top # live symbol-level CPU usageperf top -p <pid> # focus on one processLike top but at the function level. Shows which kernel/user functions are hottest.
xxxxxxxxxxperf stat -e cache-references,cache-misses,L1-dcache-loads,L1-dcache-load-misses ./progHigh cache miss rate indicates poor data locality.
xxxxxxxxxxperf record -F 99 -g -p <pid> -- sleep 30perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > flame.svgFlame graphs visualize the call stack frequency — wider frames = more samples = more CPU time. Brendan Gregg's FlameGraph scripts generate them.
perf trace?xxxxxxxxxxperf trace ./prog # like strace but eBPF-based, lower overheadperf trace -p <pid>Uses tracepoints and syscall events with significantly lower overhead than ptrace-based strace.
xxxxxxxxxxgrep -i error /var/log/syslog | awk '{print $NF}' | sort | uniq -c | sort -rn | headjournalctl -p err --since "1 hour ago"xxxxxxxxxxtail -f /var/log/syslog /var/log/nginx/error.logmultitail /var/log/syslog /var/log/auth.log # tool for multiple logsxxxxxxxxxxlast reboot # from /var/log/wtmpwho -b # last boot timeuptime # current uptimejournalctl -b -1 -e # end of previous boot's journalxxxxxxxxxxzgrep "error" /var/log/syslog.2.gzzcat /var/log/syslog.*.gz | grep patternxxxxxxxxxxawk '/2024-01-15 10:00/,/2024-01-15 11:00/' /var/log/app.logsed -n '/10:00:00/,/11:00:00/p' /var/log/app.log/var/log/auth.log or /var/log/secure contain?Answer: Authentication events: SSH logins (successful and failed), sudo usage, PAM events, su attempts. Critical for security auditing. journalctl _SYSTEMD_UNIT=ssh.service provides the same on systemd systems.
xxxxxxxxxxdmesg | grep -A 20 "Oops"journalctl -k | grep -A 20 "general protection"A kernel oops shows: the fault type, register dump, and call trace. The address can be decoded with addr2line or ksymoops.
logrotate and how does it work?Answer: logrotate compresses and rotates log files on a schedule (daily/weekly/size-based), controlled by /etc/logrotate.conf and /etc/logrotate.d/. After rotation, it can send SIGHUP to the logging daemon or run postrotate scripts to reopen the log file (copytruncate avoids this need).
Answer:
xxxxxxxxxxps aux | wc -l # current process countcat /proc/sys/kernel/pid_max # max PIDsulimit -u # per-user process limitps aux --sort=-pid | head # highest PIDs (recent)ps aux | awk '{print $1}' | sort | uniq -c | sort -rn # by userCommon causes: fork bomb, runaway process spawning workers, zombie accumulation. Fix: identify the creator process, kill it, fix the parent to call wait().
Answer: A fork bomb (:(){ :|:& };:) rapidly exhausts the process table. Prevention: ulimit -u 100 per user; systemd TasksMax in service files. Recovery: need a root shell not subject to limits, kill the fork bomb processes. Reboot may be necessary if the system is unresponsive.
xxxxxxxxxxps aux --sort=-%mem | head -10smem -r | head # by PSS (proportional)cat /proc/<pid>/status | grep -E "VmRSS|VmSwap"xxxxxxxxxxps aux --sort=-%cpu | head -10top -b -n1 | head -20 # non-interactive toppidstat -u 1 5 # per-process CPU every 1sxxxxxxxxxxstrace -c -p <pid> # see which syscalls dominateperf top -p <pid> # see which functions dominategdb -batch -ex "thread apply all bt" -p <pid> # where is it spending timecat /proc/<pid>/wchan # what kernel function it's waiting inAnswer: D-state means the process is waiting for I/O (usually). It cannot be killed. Diagnose:
xxxxxxxxxxcat /proc/<pid>/wchan # kernel function it's waiting instrace -p <pid> # shows blocked syscallcat /proc/<pid>/stack # kernel stack trace (need root)dmesg | tail # check for I/O errorsCommon causes: NFS hang, disk error, kernel bug. Fix: unmount stuck NFS, fix disk, or reboot.
xxxxxxxxxxps aux | awk '$8 == "Z"' # find zombies# Get the parent:ps -o ppid= -p <zombie_pid># Signal the parent to reap:kill -SIGCHLD <parent_pid># If parent is unresponsive, kill it:kill -9 <parent_pid>top indicate?Answer: High iowait means CPUs are idle while waiting for I/O to complete — disk I/O is the bottleneck. Check:
xxxxxxxxxxiostat -x 1 # look at %util and await for devicesiotop # which processes are causing I/Olsof | grep " REG " | awk '{print $1}' | sort | uniq -c | sort -rn # top file usersxxxxxxxxxxlsof | grep "(deleted)"# The file still uses disk space until all FDs are closed.# Free space: kill the process or wait for it to close the FD.xxxxxxxxxxstrace -c -p <pid> # which syscalls and how manyperf stat -p <pid> # system vs user timeHigh system time relative to user time suggests too many syscalls — batching I/O, reducing lock contention, or using poll() instead of multiple syscalls.
Answer: When many processes/threads are waiting on the same event (e.g., accept()), and the event fires, all are woken but only one can proceed — the rest go back to sleep. This wastes CPU. Fix: EPOLLEXCLUSIVE flag in Linux 4.5+, SO_REUSEPORT for accept, pthread_cond_signal instead of broadcast.
Answer:
xxxxxxxxxxgdb -batch -ex "thread apply all bt" -p <pid> # check all threads' stacksstrace -p <pid> # see if stuck in futexcat /proc/<pid>/fd | xargs ls -la # check held file lockslslocks -p <pid> # list locks heldLook for circular wait: thread A holds lock1, waits for lock2; thread B holds lock2, waits for lock1.
xxxxxxxxxxss -antp # all TCP with process infoss -s # summary of socket statesnetstat -an | awk '{print $6}' | sort | uniq -c | sort -rn # state distributionHigh TIME_WAIT: normal after many short connections. High CLOSE_WAIT: application not closing sockets.
Answer: FD leak occurs when a process opens files without closing them, eventually hitting EMFILE (too many open files for process) or ENFILE (system-wide limit).
xxxxxxxxxxls /proc/<pid>/fd | wc -l # count open FDscat /proc/sys/fs/file-nr # system-wide: used, free, maxlsof -p <pid> | wc -lwatch 'ls /proc/<pid>/fd | wc -l' # watch FD count growxxxxxxxxxxfree -h # overall memory/swapvmstat 1 # si/so (swap in/out per second)cat /proc/vmstat | grep pswp # cumulative swap activitydmesg | grep "Out of memory" # OOM eventssar -r 1 10 # memory stats over time/proc/<pid>/smaps tell you?Answer: smaps provides per-mapping memory details: start/end address, permissions, size, RSS, PSS, shared/private clean/dirty pages. Useful for understanding exactly what memory a process is using and why RSS is high (e.g., shared library vs. heap).
xxxxxxxxxxps aux --sort=-%mem | head # which process spikedcat /proc/<pid>/smaps | awk '/^Size/{sum+=$2} END{print sum}'valgrind --tool=massif ./prog # heap profiling/proc/<pid>/status # VmRSS, VmPeak, VmSwapxxxxxxxxxxvmstat 1 # cs column: context switches/secpidstat -w 1 # per-process context switchesperf stat -e context-switches -p <pid>Causes: too many threads competing, excessive locking, high interrupt rate. Reduce thread count, improve lock granularity, batch work.
xxxxxxxxxxtcpdump -i eth0 -w capture.pcaptcpdump -r capture.pcap -A # read, print ASCIIwireshark capture.pcap # GUI analysisxxxxxxxxxxping -c 100 <host> # packet loss percentagemtr --report <host> # per-hop packet lossnetstat -s | grep retransmit # TCP retransmitsss -ti | grep retrans # per-connection retransmitsxxxxxxxxxxmtr <host> # find high-latency hopstraceroute <host> # path analysisiperf3 -s / iperf3 -c <host> # bandwidth testss -ti | grep rtt # TCP RTT per connectionethtool eth0 # NIC speed/duplexnetstat -s or ss -s show?Answer: Aggregate socket statistics: TCP/UDP/ICMP packet counts, errors, retransmits, resets, failed connection attempts. Useful for spotting TCP health issues without per-connection detail.
xxxxxxxxxxiptables -L -n -v --line-numbers # list rules with packet countsiptables -t nat -L -n -v # NAT rulesnft list ruleset # nftables rulestcpdump -i lo port 8080 # check if packets arrive at allxxxxxxxxxxstrace -e trace=network -T -p <pid> # time on each network syscallss -ti # TCP socket internals (cwnd, rtt, retrans)SO_REUSEADDR vs SO_REUSEPORT do?Answer: SO_REUSEADDR: allows binding to a port in TIME_WAIT state (essential for server restarts). SO_REUSEPORT: allows multiple sockets on the same process/different processes to bind to the same port — kernel load-balances incoming connections. Used for multi-threaded/multi-process servers.
/proc/net/tcp and how do you read it?Answer: Contains per-socket TCP state in hex. Fields: local address:port, remote address:port, state (01=ESTABLISHED, 0A=LISTEN), tx/rx queue, uid, inode. Decode hex addresses with printf '%d.%d.%d.%d\n' $(echo "0100007F" | sed 's/../0x& /g' | awk '{for(i=NF;i>0;i--) printf $i" "}'). Usually easier to use ss.
xxxxxxxxxxdig @8.8.8.8 example.com # query specific DNS servernslookup example.com # basic lookupresolvectl status # systemd-resolved statuscat /etc/resolv.conf # DNS server configstrace -e trace=network -f ./prog 2>&1 | grep socket # DNS callsxxxxxxxxxxiftop -i eth0 # per-host bandwidthnethogs # per-process bandwidthsar -n DEV 1 # per-interface stats per secondip -s link # cumulative byte/packet countsxxxxxxxxxxpython -m cProfile -o prof.out script.pypython -m pstats prof.out # analyze# Or:py-spy top --pid <pid> # live sampling profilerxxxxxxxxxxiostat -x 1 # %util (saturation), await (latency), r/s, w/siotop # which processblktrace /dev/sda # block layer tracingperf record -e block:block_rq_issue -g -- sleep 10%util > 90% means device saturated. High await means high I/O latency.
Answer: Throughput: data transferred per unit time (MB/s). Latency: time to complete a single I/O operation (ms). They trade off: large sequential I/O = high throughput but potentially higher latency per operation. Random small I/O = low throughput, high latency per op.
vmstat for performance analysis?xxxxxxxxxxvmstat 1 10Key columns: r (run queue length), b (blocked on I/O), si/so (swap in/out), us/sy/id/wa (CPU breakdown), cs (context switches/s), in (interrupts/s).
pidstat do?xxxxxxxxxxpidstat -u 1 # per-process CPUpidstat -d 1 # per-process disk I/Opidstat -r 1 # per-process memorypidstat -w 1 # per-process context switchesxxxxxxxxxx# ftrace (in-kernel tracer):echo function_graph > /sys/kernel/debug/tracing/current_tracerecho <pid> > /sys/kernel/debug/tracing/set_ftrace_pidcat /sys/kernel/debug/tracing/trace# bpftrace:bpftrace -e 'kprobe:sys_read { @[comm] = count(); }'bpftrace and give an example use case?xxxxxxxxxx# Trace slow disk I/Os > 1ms:bpftrace -e 'tracepoint:block:block_rq_complete /args->nr_sector > 0 && args->error == 0/ { @us = hist(nsecs/1000); }'# Count syscalls by process:bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[comm, probe] = count(); }'bpftrace is a high-level language for eBPF tracing — powerful and low overhead.
xxxxxxxxxxperf lock record ./progperf lock report# Or with bpftrace:bpftrace -e 'tracepoint:lock:mutex_lock_slowed { @[comm] = count(); }'xxxxxxxxxxnumactl --hardware # NUMA topologynumastat -p <pid> # per-process NUMA allocationperf stat -e node-load-misses,node-store-misses ./prognumactl --membind=0 ./prog # bind to NUMA node 0slab memory usage and how do you check it?xxxxxxxxxxslabtop # live slab cache usagecat /proc/slabinfo # raw datavmstat -m # slab memory summaryHigh slab usage for inodes/dentries indicates many open files or heavy filesystem operations.
Answer:
xxxxxxxxxxcat /proc/sys/kernel/pid_max # max PIDs (default 32768)ulimit -u # per-user limitps aux | wc -l # current countps aux | awk '{print $1}' | sort | uniq -c | sort -rn # by usercat /proc/sys/kernel/threads-max # max threadsFix: increase pid_max temporarily, find and stop the runaway process/user.
xxxxxxxxxxpkill -u usernamekillall -u username# If the user's shell is in a zombie/D state:ps -u username -o pid= | xargs kill -9Answer: In /etc/security/limits.conf or /etc/security/limits.d/:
xxxxxxxxxx* hard nproc 500
In systemd service files: TasksMax=100. This limits the number of processes/threads per user or service before a fork bomb can exhaust system PIDs.
xxxxxxxxxx# Use auditctl to trace fork/exec:auditctl -a always,exit -F arch=b64 -S execve -k exec_traceausearch -k exec_trace | grep -A 5 "service_name"# Or with bpftrace:bpftrace -e 'tracepoint:syscalls:sys_exit_fork /retval > 0/ { printf("parent=%s pid=%d child=%d\n", comm, pid, retval); }'/proc/sys/kernel/pid_max limit and can you change it?Answer: Default is 32768 (32-bit compat). Max on 64-bit systems is 4194304. Change temporarily:
xxxxxxxxxxsysctl -w kernel.pid_max=65536echo 65536 > /proc/sys/kernel/pid_maxPersist in /etc/sysctl.conf: kernel.pid_max = 65536.
xxxxxxxxxxps --ppid <suspected_parent_pid> # children of a specific parentps -eo ppid,pid,cmd | sort | head -30 # sort by PPID to group childrenwatch 'ps -eo ppid --no-headers | sort | uniq -c | sort -rn | head'pstree useful for?xxxxxxxxxxpstree -p # full process tree with PIDspstree -p <pid> # subtree of a processpstree -u # show user changesVisualizes the fork hierarchy — useful for understanding process relationships and finding the root of a process storm.
xxxxxxxxxxulimit -c unlimited # per-shellecho '/tmp/core.%e.%p' > /proc/sys/kernel/core_pattern # name pattern# With systemd-coredump (default on many distros):coredumpctl list # list saved corescoredumpctl gdb <pid> # open in gdbAnswer: The core dump contains: all memory mappings (heap, stack, text, mmap regions), register state, open file descriptors, signal information, and thread stacks. It's a snapshot of the process at crash time. Does NOT include disk files or network state.
xxxxxxxxxxgdb ./binary core(gdb) bt # where did it crash?(gdb) frame 0 # innermost frame(gdb) info locals # local variables(gdb) print offending_ptr # check null/invalid pointer(gdb) x/10i $pc # disassemble around crashAnswer:
SIGSEGV: access to unmapped memory or protection violation (e.g., write to read-only page, NULL dereference).
SIGBUS: bus error — misaligned memory access (e.g., reading a 4-byte int from an odd address on strict alignment architectures), or accessing beyond the end of a memory-mapped file.
Answer: Stack overflow causes SIGSEGV when accessing a guard page (unmapped page below the stack). GDB backtrace may be corrupted (infinite recursion). Signs in backtrace: many identical frames or ?? entries. Fix: increase stack size (ulimit -s), reduce recursion depth, or use heap-allocated recursion.
xxxxxxxxxxgcc -fsanitize=address -g -O1 -fno-omit-frame-pointer prog.c -o prog./prog # ASan will report errors with full contextDetects: use-after-free, heap/stack/global buffer overflow, use-after-return, use-after-scope. ~2× slowdown. Essential in CI/CD testing.
xxxxxxxxxxgcc -fsanitize=thread -g -O1 prog.c -o prog./prog # reports data races with full contextDetects data races in multi-threaded programs by tracking memory accesses and synchronization operations at runtime.
xxxxxxxxxxgcc -fsanitize=undefined -g prog.c -o progDetects: signed integer overflow, null pointer dereference, misaligned accesses, invalid enum values, and other C/C++ undefined behavior.
Answer: A kernel panic prints a call trace to the console/serial port and optionally writes a crashdump (via kdump). Analyze with crash utility on the vmcore:
xxxxxxxxxxcrash /boot/System.map /var/crash/vmcore(crash) bt # kernel backtrace(crash) log # kernel ring buffer(crash) ps # process list at panickdump and how does it work?Answer: kdump uses a crash kernel (second kernel reserved in memory) that boots automatically when the primary kernel panics. It saves the crashed kernel's memory to a vmcore file (usually /var/crash/). The system then reboots normally. Configure in /etc/kdump.conf; enable with systemctl enable kdump.
Answer:
Compile with gcc -fsanitize=address — AddressSanitizer gives exact location.
Use valgrind --tool=memcheck for runtime detection.
Check glibc malloc debugging: MALLOC_CHECK_=3 ./prog.
Use electric fence (efence) library to detect buffer overflows immediately.
In GDB: watch *(type*)address to catch the write.
Answer: Calling free() on the same pointer twice — undefined behavior that corrupts the allocator's internal structures (freelist), potentially leading to arbitrary code execution or crashes. Detection: AddressSanitizer reports "double free"; valgrind --tool=memcheck also catches it.
Answer: Use-after-free: accessing memory after free() — it may have been reallocated. Symptoms: corrupted data, crashes at seemingly unrelated points. Detection: AddressSanitizer (-fsanitize=address) quarantines freed memory; valgrind memcheck poisons it. Both report the exact allocation, free, and use locations.
Answer:
High level: top / uptime (load average, CPU/memory/IO wait).
CPU: top, mpstat, perf top — is it user, kernel, or iowait?
Memory: free -h, vmstat (swap activity), /proc/meminfo.
I/O: iostat -x 1 (disk saturation), iotop (top I/O process).
Network: ss -s, iftop, netstat -s (errors/retransmits).
Processes: ps, strace -c -p <pid>.
Logs: journalctl, /var/log/syslog.
Answer:
xxxxxxxxxxkill -9 <pid> # SIGKILL — should always work unless D-state# If D-state (uninterruptible sleep):cat /proc/<pid>/wchan # what kernel function it's in# Options: wait for I/O to complete, fix underlying I/O issue, reboot# Cannot kill D-state processes without resolving the blocking condition/ is 100% full)?Answer:
xxxxxxxxxxdf -h # confirmdu -sh /* 2>/dev/null | sort -rh | head # find large directorieslsof | grep "(deleted)" # files deleted but held open# Free up:journalctl --vacuum-size=500M # trim journalfind /tmp /var/tmp -mtime +7 -deleteapt-get clean / yum clean all# Extend LVM if available:lvextend -L +10G /dev/vg0/root && resize2fs /dev/vg0/rootAnswer:
xxxxxxxxxx# Confirm: watch RSS growwatch -n 5 "cat /proc/<pid>/status | grep VmRSS"# Identify location:valgrind --leak-check=full --track-origins=yes ./prog# Or compile with -fsanitize=address (best for development)# Production: use heaptrack or jemalloc with profilingxxxxxxxxxxstrace ./prog # see where it gets stuck (last unfinished call)lsof -p $(pidof prog) # what files/sockets are opencat /proc/<pid>/wchan # kernel wait channelgdb ./prog # run under debugger, interrupt with Ctrl+C, btAnswer:
xxxxxxxxxxssh -v user@host # verbose — see where it disconnects# Check:dmesg | grep -i "tcp\|ssh"/var/log/auth.log # server-side auth log# Fixes:# Client: ~/.ssh/config: ServerAliveInterval 60# Server: /etc/ssh/sshd_config: ClientAliveInterval 60# Possible firewall/NAT idle timeout dropping connectionsxxxxxxxxxxsystemctl status service-name # status and recent logsjournalctl -u service-name -n 50 # last 50 log linessystemctl cat service-name # view unit file# Run the binary manually to see errors:$(systemctl cat service-name | grep ExecStart | awk '{print $2}')strace -e trace=file,process binary # trace file/exec operationsxxxxxxxxxxulimit -n # current limit (default: 1024)cat /proc/sys/fs/file-max # system-wide maxlsof -p <pid> | wc -l # FDs used by process# Increase limit:ulimit -n 65535 # in shell# For service: systemd LimitNOFILE=65535 in unit file# System-wide: /etc/security/limits.conf: * soft nofile 65535xxxxxxxxxxss -tlnp | grep <port> # is anything listening?iptables -L -n | grep <port> # firewall blocking?telnet localhost <port> # local testcurl -v http://localhost:<port> # HTTP testjournalctl -u service -n 20 # is service running/crashing?Answer:
Enable verbose logging in the application.
Set up continuous monitoring: strace/perf trace with output to file.
Use bpftrace probes at key kernel points without stopping the process.
Analyze metrics over time with sar, collectd, or Prometheus.
Enable core dumps (ulimit -c unlimited, systemd-coredump) to capture crash state.
Add timing instrumentation and check for timing-sensitive (race condition) patterns.
addr2line and when do you use it?xxxxxxxxxxaddr2line -e ./binary -f 0x401234 # convert address to file:line functionUsed to decode hex addresses in crash logs, kernel oops traces, or profiling output into human-readable source locations. Requires debug symbols or compile with -g.
objdump and what can it show?xxxxxxxxxxobjdump -d ./binary # disassembleobjdump -t ./binary # symbol tableobjdump -R ./binary # dynamic relocationsobjdump -p ./binary # program headersnm used for?xxxxxxxxxxnm ./binary # list symbols (T=text, U=undefined, D=data)nm -D ./library.so # dynamic symbolsnm ./binary | grep "T main" # find specific symbolreadelf and how does it differ from objdump?Answer: Both read ELF files. readelf is ELF-specific and more detailed for ELF structures; objdump also supports other binary formats and has disassembly. Use readelf -d binary for dynamic linking info, -S for section headers, -l for program headers.
xxxxxxxxxxstrace -f -ff -o /tmp/trace ./prog# Creates /tmp/trace.<pid> for each processls /tmp/trace.*grep -l "execve" /tmp/trace.* # find exec callsgdb's printf to add instrumentation without recompiling?xxxxxxxxxxbreak function_namecommands printf "var=%d\n", variable continueendInsert print statements at runtime without modifying/recompiling source.
SystemTap?Answer: SystemTap is a scripting language/infrastructure for live kernel and user-space instrumentation. Like bpftrace but older; uses kernel modules. Allows tracing kernel internals, user functions, and system calls with full scripting. Largely superseded by eBPF/bpftrace on modern kernels.
ftrace and how do you use it?xxxxxxxxxxecho 1 > /sys/kernel/debug/tracing/tracing_onecho function > /sys/kernel/debug/tracing/current_tracerecho my_func > /sys/kernel/debug/tracing/set_ftrace_filtercat /sys/kernel/debug/tracing/traceftrace is the kernel's built-in function tracer. Can trace function call graphs, latency, interrupts. Zero overhead when disabled. trace-cmd provides a user-friendly wrapper.
xxxxxxxxxx# Shared memory:ipcs -m # list shared memory segmentsipcs -s # semaphoresipcs -q # message queues# Pipes/sockets:lsof -p <pid1> -p <pid2> # shared FDsstrace -e trace=ipc ./prog # IPC syscallsAnswer:
Build with ThreadSanitizer (-fsanitize=thread) in staging — it will detect races with context.
In production: use lockstat/perf lock to identify heavily contended locks.
Add atomic counters or timestamps around suspected regions.
Use rr (Mozilla's record-and-replay debugger) to deterministically replay non-deterministic bugs.
Review code for lock-free data structure misuse.
rr and when is it useful?Answer: rr records a deterministic execution trace of a process (including all hardware inputs, syscalls, signals) and allows replaying it exactly, including backwards. Invaluable for heisenbug: record once, reproduce/debug indefinitely. Supports GDB interface (rr replay).
Answer:
xxxxxxxxxx# If DWARF info is in separate file:gdb ./stripped_binary(gdb) add-symbol-file debug_binary 0x400000# Use addr2line with unstripped binary on same build:addr2line -e unstripped_binary -f 0xcrash_addr# Strip only removes .debug_*, not .symtab for internal symbolscatchsegv?xxxxxxxxxxcatchsegv ./program argsWrapper that catches SIGSEGV/SIGBUS and prints a backtrace using libc's backtrace mechanism. Quick alternative to GDB when debug symbols aren't available.
kworker, ksoftirqd)?xxxxxxxxxxperf top # shows kernel symbolsperf record -g -a sleep 10 && perf report # full system# kworker high CPU: check workqueue:cat /proc/sys/kernel/workqueue/max_activebpftrace -e 'kprobe:queue_work_on { @[kstack] = count(); }'# ksoftirqd: check soft IRQswatch -n 1 cat /proc/softirqsxxxxxxxxxxjstack <pid> # thread dumpjmap -heap <pid> # heap infojmap -histo <pid> # object histogramasync-profiler # low-overhead CPU/memory profilerperf record -g -p <pid> # native CPU profiling (with JVM flags)xxxxxxxxxx# Use objgraph:import objgraphobjgraph.show_most_common_types(limit=20)# Use tracemalloc:import tracemalloctracemalloc.start()# ... code ...snapshot = tracemalloc.take_snapshot()# Or with memory_profiler:@profiledef my_func(): ...sysdig and how does it differ from strace?Answer: sysdig is a system-level observability tool using a kernel module (or eBPF). Unlike strace (per-process, ptrace), it captures all system calls system-wide with rich filtering. Can save to file and replay. csysdig is an interactive UI. Good for container environments (aware of container metadata).
Answer:
Use perf stat to compare hardware counters (cycles, instructions, cache misses) between versions.
Use perf diff to compare two perf.data files.
Use cachegrind (valgrind) for instruction-level cache simulation — deterministic.
Profile both with flame graphs and compare visually.
Use microbenchmarks (benchmark library) to isolate specific functions.
numactl and when do you need it?xxxxxxxxxxnumactl --hardware # show NUMA topologynumactl --membind=0 ./prog # bind memory allocations to NUMA node 0numactl --cpunodebind=0 ./prog # bind CPU to NUMA node 0On multi-socket systems, remote NUMA accesses are 2–4× slower. Binding workloads to a single NUMA node eliminates remote memory latency.
xxxxxxxxxxperf stat -e dTLB-load-misses,dTLB-store-misses ./prog# Or:perf stat -e iTLB-load-misses ./prog# Fix: use huge pages, improve data locality, pin to fewer coresHigh TLB misses indicate working set too large for TLB coverage. Use huge pages or madvise(MADV_HUGEPAGE).
Answer: A core dump (Linux/POSIX) contains all mapped memory regions of the process — can be several GB. A minidump (Windows-originated, also used in some tools) is a smaller, structured dump with thread stacks, registers, and selected memory — easier to transfer and analyze. Tools like Breakpad/Crashpad generate minidumps.
xxxxxxxxxx# In CI script:gdb -batch -ex "bt" -ex "quit" ./binary core > stack_trace.txt 2>&1# More thorough:gdb -batch -ex "set print frame-arguments all" \ -ex "thread apply all bt full" -ex "quit" \ ./binary core 2>/dev/nullAnswer: Embed build ID in binaries (-Wl,--build-id). Check readelf -n binary | grep "Build ID" and match against symbol server. systemd-coredump stores metadata (timestamp, UID, command, build ID) with each core.
coredumpctl and how do you use it?xxxxxxxxxxcoredumpctl list # list captured core dumpscoredumpctl info <PID> # details about a specific dumpcoredumpctl gdb <PID> # open in gdbcoredumpctl dump <PID> -o core.out # export core filePart of systemd; automatically captures and stores core dumps with metadata.
Answer:
Use wrk/ab/vegeta to apply controlled load.
Attach perf record -g -p <pid> during load test.
Monitor: vmstat 1, iostat -x 1, ss -s.
Set up core dumps for crashes under load.
Use strace -c to summarize syscall distribution under load.
Profile with perf record during high-load phase, analyze afterwards.
dstat and what does it show?xxxxxxxxxxdstat -cdngy # cpu, disk, net, page, system all-in-onedstat --top-cpu --top-mem # top CPU and memory consumersCombines vmstat, iostat, netstat in one output. Useful for a quick holistic view.
atop and how is it different from top?Answer: atop logs system and process activity persistently, allowing historical replay. Unlike top, it shows disk and network per process, indicates process exits between samples, and keeps accounting data in /var/log/atop/. atop -r /var/log/atop/atop_20240115 replays recorded data.
xxxxxxxxxxwatch -n 1 'ss -s' # socket summarysar -n TCP 1 # TCP connection ratenetstat -s | grep -i "failed\|reset" # error ratesconntrack -L | wc -l # connection tracking table sizecollectl and when is it useful?Answer: collectl is a lightweight system metrics collector that can record all major subsystem metrics (CPU, memory, disk, network, I/O) to files for later analysis. Lower overhead than sar. Useful for continuous baseline collection.
/proc/sys/net/ipv4/tcp_tw_reuse?Answer: Allows reusing TIME_WAIT sockets for new connections when safe (when TCP timestamps are enabled). Helps when local ports are exhausted due to many short connections. Different from tcp_tw_recycle (deprecated in kernel 4.12 — caused issues with NAT).
xxxxxxxxxxinotifywait -m /path/to/file # watch for eventslsof /path/to/file # who has it openauditctl -w /path/to/file -p w # audit writesausearch -f /path/to/file # search audit logAnswer:
xxxxxxxxxx# Check signal handler:cat /proc/<pid>/status | grep SigIgn # hex bitmask of ignored signals# Decode: signal 15 (SIGTERM) = bit 14 set in SigIgn# Alternatives:kill -SIGKILL <pid> # cannot be ignored# If still stuck: check if it's in D-state (kernel bug, I/O issue)/proc/<pid>/maps and what does it show?Answer: Shows the virtual memory map of a process: each line has start-end address range, permissions (rwxp), offset, device, inode, and path. p=private, s=shared. Used to understand memory layout, find library versions loaded, diagnose large mappings.
xxxxxxxxxxcat /proc/<pid>/maps | grep heap # heap locationcat /proc/<pid>/maps | grep libssl # verify SSL libraryexecsnoop and opensnoop?Answer: eBPF/bpftrace-based tools from the bcc package:
xxxxxxxxxxexecsnoop # trace all exec() calls system-wide with argsopensnoop # trace all file opens system-wideInvaluable for watching what a service does at startup or under load without attaching to a specific PID.
xxxxxxxxxxnfsstat -c # client NFS statisticsmountstats # detailed per-mount statistics including latencynfsiostat 1 # I/O stats per NFS mounttcpdump -i eth0 port 2049 # capture NFS trafficxxxxxxxxxxcat /proc/interrupts # interrupt counts per CPUwatch -n 1 cat /proc/interrupts | diff - <(sleep 1; cat /proc/interrupts)perf record -e irq:irq_handler_entry -a sleep 10mpstat -I ALL 1 # per-interrupt CPU timeschedd_lat and how do you measure scheduling latency?xxxxxxxxxxbpftrace -e 'tracepoint:sched:sched_wakeup { @start[tid] = nsecs; } tracepoint:sched:sched_switch /comm == "myapp"/ { @lat = hist(nsecs - @start[tid]); }'Scheduling latency = time between when a task becomes runnable and when it actually gets CPU. High latency indicates a busy system or real-time requirements not met.
xxxxxxxxxxcpupower frequency-infocpupower monitorcat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freqperf stat -e cpu-clock,task-clock ./prog # effective CPU frequency ratiodmesg | grep -i "throttling\|thermal"cgroup memory events and how do you monitor them?xxxxxxxxxxcat /sys/fs/cgroup/memory/<group>/memory.stat | grep -E "pgfault|oom"cat /sys/fs/cgroup/memory/<group>/memory.oom_control# V2:cat /sys/fs/cgroup/<group>/memory.eventsxxxxxxxxxxss -ti # per-connection TCP internals (rtt, variance, retrans)# Fields: rtt, rttvar, retrans, rcv_space, cwnd, lastsnd, lastrvc# High rtt + retrans = network congestion or lossping <host> # ICMP RTT as baseline%sy in top)?Answer:
xxxxxxxxxxperf top --sort comm,dso # which processes/libraries use kernelperf record -e cycles:k # sample kernel CPU cyclesperf stat -e syscalls:sys_enter_* ./prog # count syscallsstrace -c -p <pid> # syscall summary for a processHigh system time usually means many syscalls, memory allocation, or kernel lock contention.
/proc/buddyinfo and why does it matter?Answer: Shows the kernel buddy allocator free block list per memory zone. Each column represents blocks of 2^N pages. If large allocations fail despite free memory, fragmentation is the cause — the allocator can't find a contiguous block. echo 1 > /proc/sys/vm/compact_memory triggers memory compaction.
getaddrinfo performance issues?xxxxxxxxxxstrace -e trace=network ./prog 2>&1 | grep -A3 "socket\|connect"# Slow DNS:tcpdump -i any port 53 # see DNS queriesnscd # name service cache daemon helpscat /etc/nsswitch.conf # resolution orderss --options --extended --memory --processes --info show?Answer: The -e/--extended flag shows additional socket options. --memory/-m shows socket memory usage. Combined: for each socket shows send/receive buffer sizes, which is useful when diagnosing buffer bloat or sockets consuming excessive memory.
xxxxxxxxxxdmesg | grep "soft lockup"# Output shows CPU number and task.# Common causes: long interrupts-off sections, I/O spinningecho 1 > /proc/sys/kernel/softlockup_panic # panic on soft lockup (for kdump)xxxxxxxxxxmemtest86+ # boot-time memory test (bootable ISO)# From running system:stress-ng --vm 1 --vm-bytes 80% --timeout 60s# Check EDAC:edac-util -s 4 # ECC memory errorscat /sys/bus/platform/drivers/i7core_edac/*/mc/mc*/ce_count # correctable errorsnumastat and what does it reveal?xxxxxxxxxxnumastat # per-node memory statisticsnumastat -p <pid> # per-process NUMA allocationShows numa_miss (allocations that fell through to remote node) and numa_foreign (allocations intended for this node that went elsewhere). High miss rates indicate NUMA imbalance.
xxxxxxxxxxvalgrind --tool=cachegrind ./prog # cache simulationcg_annotate cachegrind.out.<pid> # annotated sourceperf stat -e L1-dcache-loads,L1-dcache-load-misses,LLC-loads,LLC-load-misses ./progblktrace and how do you use it?xxxxxxxxxxblktrace -d /dev/sda -o traceblkparse -i trace # parse tracebtt -i trace.blktrace.0 # latency statisticsblktrace records every block I/O request from issue through completion, revealing I/O scheduler behavior, queue depth, and per-operation latency.
Answer:
file binary — type/architecture check.
ldd binary — shared library dependencies.
strings binary | head -50 — readable strings (configuration, paths).
strace ./binary 2>&1 | head -50 — first syscalls (what it opens first).
ltrace ./binary 2>&1 | head -50 — library calls.
objdump -d binary | head -100 — first instructions.
readelf -d binary — dynamic section (imported symbols, RPATH).
/proc/<pid>/maps — memory layout at runtime.
perf top -p <pid> — CPU hotspots.
lsof -p <pid> — files and network connections.