Oracle Releases DTrace for Linux Version 2.0.0-1.14

Presented experimental release of dynamic debugging tools DTrace for Linux 2.0.0-1.14, implemented as a user-space process that uses the eBPF subsystem and the standard tracing mechanisms provided by the Linux kernel. In terms of functionality, the eBPF-based DTrace implementation is close to the first DTrace implementation for Linux, implemented in the form of a kernel module. Project code distributed by licensed under GPLv2.

The toolkit can be used with stock Linux kernels that support BPF. To work, you need the libctf library with implementation of support for the CTF (Compact Type Format) debugging format, which is included in the binutils package starting from release 2.40, or the library libdtrace-ctf, ported from Solaris. Optionally, two patches are offered for the 6.7 kernel, which allow you to use advanced features to obtain additional data about the modules and the kernel.

DTrace technology was developed to solve the problems of dynamically tracing the kernel and end applications in the Solaris operating system. DTrace gives the user the ability to monitor system behavior in detail and diagnose problems in real time. During the debugging process, DTrace does not affect the operation of the applications under study and does not in any way affect their performance, which allows you to organize the analysis of running systems on the fly. One of the strengths of DTrace is the high-level D language, similar to AWK, in which it is much easier to create tracing scripts than using the tools offered for writing eBPF handlers in C, Python and Lua with external libraries.

Main features:

  • Available providers:
    • cpc (CPU Performacne Counter) – obtaining performance information.
    • dtrace – handlers BEGIN, END And ERRORrun before or after other checks, as well as when errors occur.
    • fbt (Function Boundary Tracing) – tracking calls to kernel functions.
    • lockstat – tracking the status of locks.
    • pid – keeps track of function calls in processes running in user space.
    • proc – monitors process-related activity such as startup and shutdown (fork, exec, exit, fault).
    • profile – saving performance statistics at specified intervals.
    • sdt, rawtp – static kernel tracing (SDT – Statically Defined Tracing).
    • usdt: static application tracing (USDT – Userspace Statically Defined Tracing)
    • sched – tracking the allocation of CPU resources.
    • sycall – Tracks entry into and exit from system calls.
  • Aggregation collected data: the ability to use aggregate functions (avg, count, llquantize, lquantize, max, min, quantize, stddev and sum) and aggregate actions (clear, normalize, normalize, printa). Support for storing the results of applying aggregate functions in regular and associative arrays.
  • Support speculative tracing, which allows you to ephemerally monitor data, deciding which of them should be captured in the trace buffer and which should be discarded. The available functions are speculation, speculate, commit and discard.
  • Support for global and local variables, TLS (Thread-Local Storage), associative arrays and strings.
  • Providing built-in variables: arg0 – arg9, args(), caller, curcpu, curthread, epid, errno, execname, gid, id, pid, ppid, probefunc, probemod, probename, probeprov, stackdepth, tid, timestamp, ucaller, uid, uregs(), ustackdepth, walltimestamp.
  • Support actions: exit, freopen, ftruncate, mod, printa, printf, raise, setopt, stack, sym, system, trace, tracemem, uaddr, umod, ustack and usym.
  • Availability of built-in functions: alloca, basename, bcopy, copyin, copyinstr, copyinto, copyout, copyoutstr, dirname, getmajor, getminor, htonl, htonll, htons, index, inet_ntoa, lltostr, mutex_owned, mutex_owner, mutex_type_adaptive, mutex_type_spin, ntohl, ntohll, ntohs, progenyof, rand, rindex, rw_iswriter, rw_read_held, rw_write_held, strchr, strjoin, strlen, strrchr, strstr, strtok, substr.
  • Presence of data counters that are not included in the trace buffers.
  • Support for compiling D tracing scripts into BPF program form.
  • Precompiling BPF functions for library calls.
  • Option bpflog to obtain a verification log of loaded BPF programs.
  • Dynamic code generation and code precompilation for reuse in BPF programs.

Changes in the new version:

  • The pid provider, used to track processes in user space, has added support for tracking the execution of instructions by their offset in the code.
  • Added support for using the stack() and ustack() functions to aggregate collected data.
  • The ability to remove elements of any type from associative arrays by assigning the literal 0.
  • Added print action for outputting structured data with type annotations.
  • Added new built-in functions cleanpath(), d_path() and link_ntop().
  • Added the “-xcpu” parameter, which allows you to bind checks to specific CPUs.
  • Added “-xlockmem” option to limit memory size.
  • Provided the ability to save process trace data (USDT) between dtprobed restarts.
  • Support for the BTF (BPF Type Format) mechanism, which provides type checking information in BPF pseudocode.
  • Ability to use the configure script to build.

Thanks for reading: