LKET - Linux Kernel Event Trace tool based on SystemTap
The Linux Kernel Event Trace (LKET) tool is an extension to the tapsets library available on SystemTap. Its goal is to utilize the dynamic probing capabilities provided through SystemTap to create a set of standard hooks that probe pre-defined places in the kernel. It can be used to collect important information that can be used as a starting point to analyze a performance problem in the system.
The LKET tapsets are designed to only trace the events selected by the user. Once the data has been collected, it is then post-processed according to the need of the user. Trace data can be processed in various different ways to generate simple to complex reports.
By default, LKET will log the trace data in binary format.
To get a better performance for binary tracing, the "-b" option should be turned on for stap and thus -M option has to be added to stop staprun merging per-cpu files.
You could use the command lket-b2a to convert the binary trace data generated by LKET into readable data in ascii format.
lket-b2a uses the pre-cpu binary trace data files(stpd_cpu*) as inputs, and generates an output file named lket.out. or dump the trace data into MySQL database. See lket-b2a(1) manual page for more detail.
If you want LKET to log trace data in ASCII format directly, you should:
stap -D ASCII_TRACE ...
*Notes* that in order to make LKET able to work in binary tracing mode, all strings logged by LKET should be NULL-terminated, which means you have to use "%0s" instead of "%s" for both user appended extra printing statements and _lket_trace() which is called in LKET tapsets.
LKET provides a way to log the metadata of the trace data by events registering.
Two functions is provided:
event_desc is a string representation of the event, e.g: syscall.entry, scsi.iocompleted.
grpid and hookid is the groupid and hookid of the event to be registered.
fmt contains a set of fomat tokens seperated by ":". The valid format tokens are: UINT8, UINT16, UINT32, UINT64 and STRING which represents 8-bit, 16-bit, 32-bit, 64-bit binary data and NULL-terminated respectively.
names contains a set of names seperated by ":". The names contains in names should match the format tokens contains in fmt
_register_sys_event is a c function which is used to register the newly added trace hooks in LKET tapsets. For example, supposing you want to add a new event hook to trace the entry of sys_open, and you want this event hook to log the fd, flag and mode paremeters for you. You should add:
_register_sys_event("iosyscall.open.entry", _GROUP_IOSYSCALL, _HOOKID_IOSYSCALL_OPEN_ENTRY, "STRING:INT32:INT32", "filename:flags:mode");
into the function register_sys_events in LKET/register_event.stp
register_user_event is a SystemTap script function which is used for user to add extra trace data for a event hook. See the section CUSTOMIZED TRACE DATA for more detail
LKET provides a set of event hooks that log the predefined trace data for you, but LKET also make you able to log extra trace data for a event.
LKET provides a way to do this without modifying the tapset of that event hook. You can simply use printf to trace extra data. For example, supposing you want to trace sk_buff->mac_len and sk_buff->priority besides the sk_buff->len, sk_buff->protocol and sk_buff->truesize for the netdev event hooks:
probe register_event { register_user_event(GROUP_NETDEV, HOOKID_NETDEV_TRANSMIT, "INT32:INT32", "mac_len:priority") } probe addevent.netdev.transmit { printf("%4b%4b", $skb->mac_len, $skb->priority) }
Here are some examples of using LKET:
You can press "Ctrl+c" to stop the tracing. Then you will find there are one or more per-cpu data files (stpd_cpu*) on current directory. You can use lket-b2a to convert these binary trace data files into readable ascii format or dump them into database. See lket-b2a(1) man page for more detail.
The following sections enumerate the variety of event hooks implemented in LKET and their trace data format. The trace data generated by different event hooks contain common data as well as some data specific to that event hook.
the INT8, INT16, INT32, INT64 and STRING appeared in trace data format represents 8-bit, 16-bit, 32-bit, 64-bit binary data and NULL-terminated string respectively.
The data common(i.e. common_data in the following subsecions) to all event hooks is:
timestamp(INT64),(tid<<32|pid)(INT64),(ppid<<32|groupID<<24|hookID<<16|cpu_id<<8)(INT64)
Each event hook group is a collection of those hooks that have similarities of what they could trace. And the ID of each event hook (HookID) is defined in the context of its corresponding group.
Data format is:
common_data, syscall_name(STRING)
Data format is:
common_data, syscall_name(STRING)
probe begin { stoptrace_fork = 1 stoptrace_exec = 1 }
Data format is:
common_data, tid(INT32), pid(INT32), ppid(INT32), process_name(STRING)
Data format is:
common_data, new_tid(INT32), new_pid(INT32), ppid(INT32)
Data format is:
common_data, tid(INT32), pid(INT32), ppid(INT32), new_process_name(STRING)
Data format is: common_data, sig(INT8), shared(INT8), send2queue(INT8), pid(INT32)
Data format is: common_data, return(INT8)
Data format is: common_data, pid(INT32), sig(INT8)
Data format is: common_data, return(INT8)
Data format is: common_data, tid(INT32), pid(INT32), sig(INT8)
Data format is: common_data, return(INT8)
Data format is: common_data, pid(INT32), sig(INT8)
Data format is: common_data, return(INT8)
Data format is: common_data, sigset_addr(INT32), setsize(INT32)
Data format is: common_data, return(INT8)
Data format is: common_data, sig(INT8), handler(INT64)
Data format is: common_data, return(INT8)
Data format is: common_data, how(INT8), sigset(INT64)
Data format is common_data, return(INT8)
Data format is: common_data, pid(INT32)
Data format is:
common_data, elevator_name(STRING), disk_major(INT8), disk_minor(INT8), request_addr(INT64), request_flags(INT64)
Data format is:
common_data, elevator_name(STRING)
Data format is:
common_data, disk_major(INT8), disk_minor(INT8), request_addr(INT64), request_flags(INT64)
Data format is:
common_data, elevator_name(STRING), disk_major(INT8), disk_minor(INT8), request_addr(INT64), request_flags(INT64)
Data format is:
common_data, prev_pid(INT32), next_pid(INT32), prev_state(INT8)
Data format is:
common_data, current_pid(INT32)
Data format is:
common_data, disk_major(INT8), disk_minor(INT8), device_state(INT8), request_addr(INT64)
Data format is:
common_data, host(INT8), channel(INT8), lun(INT8), dev_id(INT8), device_state(INT8), data_direction(INT8), reqbuf_addr(INT64), reqbuf_len(INT32), request_addr(INT64)
Data format is:
common_data, host(INT8), channel(INT8), lun(INT8), dev_id(INT8), device_state(INT8), data_direction(INT8), request_addr(INT64)
Data format is:
common_data, host(INT8), channel(INT8), lun(INT8), dev_id(INT8), device_state(INT8), data_direction(INT8), request_addr(INT64), bytes_done(INT32)
Data format is:
common_data, memory_address(INT64), write_access(INT8)
Data format is:
common_data, netdev_name(STRING), data_length(INT32), protocol(INT16), buffer_length(INT32)
Data format is:
common_data, netdev_name(STRING), data_length(INT32), protocol(INT16), buffer_length(INT32)
All the return hooks will only log the common_data and the return value. So in the following subsections, only the entry hooks will be listed:
Data format is:
common_data, filename(STRING), flags(INT32), mode(INT32)
Data format is:
common_data, fd(INT64)
Data format is:
common_data, fd(INT64), buf_addr(INT64), count(INT64)
Data format is:
common_data, fd(INT64), buf_addr(INT64), count(INT64)
Data format is:
common_data, fd(INT64), vector_addr(INT64), count(INT64)
Data format is:
common_data, fd(INT64), vector_addr(INT64), count(INT64)
Data format is:
common_data, fd(INT64), buff_addr(INT64), count(INT64), offset(INT64)
Data format is:
common_data, fd(INT64), buff_addr(INT64), count(INT64), offset(INT64)
Data format is:
common_data, fd(INT64), offset(INT64), count(INT64)
Data format is:
common_data, out_fd(INT64), in_fd(INT64), offset_uaddr(INT64), count(INT64)
Data format is:
common_data, fd(INT64), offset(INT64), whence(INT8)
Data format is:
common_data, fd(INT64), offset_high(INT64), offset_low(INT64), result_addr(INT64), whence(INT8)
Data format is:
common_data
Data format is:
common_data, fd(INT64)
Data format is:
common_data, fd(INT64)
Data format is:
common_data, fd(INT64), operation(INT32)
All the return hooks will only log the common_data and the return value. So in the following subsections, only the entry hooks will be listed:
Data format is:
common_data, nr_events(INT32), ctxp_uaddr(INT64)
Data format is:
common_data, ctx_id(INT64), nr(INT32), iocbpp_uaddr(INT64)
Data format is:
common_data, ctx(INT64), user_iocb_uaddr(INT64), aio_lio_opcode(INT16), aio_reqprio(INT16), aio_fildes(INT32), aio_buf(INT64), aio_nbytes(INT64), aio_offset(INT64)
Data format is:
common_data, ctx_id(INT64), min_nr(INT32), nr(INT32), events_uaddr(INT64), tv_sec(INT32), tv_nsec(INT32)
Data format is:
common_data, ctx(INT64)
Data format is:
common_data, ctx_id(INT64), iocb_uaddr(INT64), result_uaddr(INT64)
All the return hooks will only log the common_data and the return value. So in the following subsections, only the entry hooks will be listed:
Data format is:
common_data, servername(STRING), prog(INT64), vers(INT8), prot(INT16), port(INT16), authflavor(INT8)
Data format is:
common_data, servername(STRING), prog(INT64), vers(INT8), prot(INT16), port(INT16), authflavor(INT8)
Data format is:
common_data, servername(STRING), prog(INT64), clones(INT16), tasks(INT16), rpccnt(INT32)
Data format is:
common_data, servername(STRING), old_prog(INT64), old_vers(INT8), prog(INT64), vers(INT8)
Data format is:
common_data, servername(STRING), prog(INT64), vers(INT8), proc(INT64), flags(INT64)
Data format is:
common_data, servername(STRING), prog(INT64), vers(INT8), proc(INT64), flags(INT64)
Data format is:
common_data, tk_pid(INT64), tk_flags(INT64)
Data format is:
common_data, sv_name(STRING), prog(INT64), prot(INT16), port(INT32)
Data format is:
common_data, prog(INT64), pg_nvers(INT8), bufsize(INT32)
Data format is:
common_data, sv_name(STRING), sv_prog(INT64), sv_nrthreads(INT32)
Data format is:
common_data, sv_name(STRING), sv_prog(INT64), peer_ip(INT64), rq_xid(INT64), rq_prog(INT64), rq_vers(INT8), rq_proc(INT8)
Data format is:
common_data, sv_name(STRING), peer_ip(INT64), rq_xid(INT64), rq_prog(INT64), rq_vers(INT8), rq_proc(INT64)
Data format is:
common_data, sv_name(STRING), timeout(INT64)
Data format is:
sv_name(STRING), peer_ip(INT64), rq_xid(INT64), rq_prog(INT64), rq_vers(INT8), rq_proc(INT64)
Data format is:
common_data, sv_name(STRING), peer_ip(INT64), rq_xid(INT64), rq_prog(INT64), rq_vers(INT8), rq_proc(INT64)
Data format is:
common_data, xid(INT64), prog(INT64), vers(INT8), prot(INT64), flags(INT64)
Data format is:
common_data, xid(INT64), prog(INT64), vers(INT8), prot(INT64), flags(INT64)
Data format is:
common_data, xid(INT64), prog(INT64), vers(INT8), prot(INT64), tk_pid(INT64), tk_flags(INT64)
Data format is:
common_data, xid(INT64), prog(INT64), tk_pid(INT64), tk_flags(INT64), delay(INT64)
All the return hooks will only log the common_data and the return value. So in the following subsections, only the entry hooks will be listed:
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), offset(INT32), origin(INR8)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64) , offset(INT64)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64) , offset(INT64)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64) , offset(INT64)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64) , offset(INT64)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), vm_start(INT64), vm_end(INT64) , vm_flags(INT32)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), flag(INT32), filename(STRING)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), ndirty(INT32)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), mode(INT16)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), ndirty(INT32)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), fl_start(INT64), fl_end(INT64), fl_type(INT8), fl_flag(INT8), cmd(INT32)
Data format is:
common_data, major_device(INT8), minor_devide(INT8), fileid(INT32), count(INT64), ppos(INT64)
Data format is:
flag(INT32)
Data format is:
fileid(INT64), rsize(INT32), page_address(INT64), page_index(INT64)
Data format is:
fileid(INT64), rpages(INT32), nr_pages(INT32)
Data format is:
fileid(INT64), wsize(INT32), page_address(INT64), page_index(INT64)
Data format is:
fileid(INT64), wpages(INT32), nr_to_write(INT64)
Data format is:
fileid(INT64), page_address(INT64), page_index(INT64)
Data format is:
fileid(INT64), page_address(INT64), page_index(INT64),offset(INT32),count(INT32)
Data format is:
page_address(INT64), page_flag(INT8)
Data format is:
page_address(INT64), page_index(INT64)
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), filename(STRING)
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), count(INT32),offset(INT64)
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), count(INT32),offset(INT64)
the entry of nfs3_proc_commit and nfs4_proc_commit
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), count(INT32),offset(INT64)
the entry of nfs_proc_read_setup, nfs3_proc_read_setup and nfs4_proc_read_setup
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), count(INT32),offset(INT64)
the entry of nfs_proc_write_setup, nfs3_proc_write_setup and nfs4_proc_write_setup
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), how(INT8), count(INT32),offset(INT64)
the entry of nfs3_proc_commit_setup and nfs4_proc_commit_setup
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), how(INT8), count(INT32),offset(INT64)
the entry of nfs_read_done, nfs3_read_done and nfs4_read_done
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), status(INT32), count(INT32)
the entry of nfs_write_done, nfs3_write_done and nfs4_write_done
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), status(INT32), count(INT32)
the entry of nfs_commit_done, nfs3_commit_done and nfs4_commit_done
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), status(INT32), count(INT32)
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), filename(STRING), flag(INT32), mode(INT32)
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), filename(STRING), flag(INT32), mode(INT32)
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), filename(STRING), mode(INT32)
Data format is:
version(INT8), major_old(INT8), minor_old(INT8), old_fileid(INT64), old_name(STRING), major_new(INT8), minor_new(INT8), new_fileid(INT64), new_name(STRING)
Data format is:
major_dev(INT8), minor_dev(INT8), fileid(INT64), version(INT8), filename(STRING)
All the return hooks will only log the common_data and the return value. So in the following subsections, only the entry hooks will be listed:
the entry of nfsd_dispatch
Data format is:
proto(INT8), version(INT8), xid(INT32), proc(INT32),client_ip(INT32)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), type(INT32), access(INT32)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), count(INT64), offset(INT64), iov_len(INT64), vlen(INT64)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), count(INT64), offset(INT64), iov_len(INT64), vlen(INT64)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), filename(STRING)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), count(INT64), offset(INT64)
the entry of nfsd_create
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), filename(STRING), type(INT32), iap_valid(INT16), iap_mode(INT32)
the entry of nfsd_create_v3
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), filename(STRING), createmode(INT8), iap_valid(INT16), iap_mode(INT32)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), filename(STRING), type(INT32)
Data format is:
old_fhsize(INT8), old_fh0(INT64), old_fh1(INT64), old_fh2(INT64), old_name(STRING) new_fhsize(INT8), new_fh0(INT64), new_fh1(INT64), new_fh2(INT64), new_name(STRING)
Data format is:
filename(STRING)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)
filename(STRING)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)
count(INT64), offset(INT64), iov_len(INT64), vlen(INT64)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)
count(INT64), offset(INT64), iov_len(INT64), vlen(INT64)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)
count(INT64), offset(INT64)
Data format is:
number(INT32)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)
filename(STRING)
Data format is:
old_fhsize(INT8), old_fh0(INT64), old_fh1(INT64), old_fh2(INT64), old_name(STRING) new_fhsize(INT8), new_fh0(INT64), new_fh1(INT64), new_fh2(INT64), new_name(STRING)
Data format is:
fh_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8) filename(STRING)
Закладки на сайте Проследить за страницей |
Created 1996-2024 by Maxim Chirkov Добавить, Поддержать, Вебмастеру |