Get full command line of a pid

[udaypalagati]

I am using a bpftrace script to fetch network usage on pid level, below article has the full code. Along with pid I want to get full command line including arguments, I could only get comm/process name since it is a built in variable. I could run "ps" commands in system function to fetch full command line but i dont think it is a good idea. I am looking for a better and optimized way, please suggest.

Article: https://www.gcardone.net/2020-07-31-per-process-bandwidth-monitoring-on-Linux-with-bpftrace/

[fbs]

we don't have a helper for that, nor seems there to be a BPF helper for it. What I'd do is try to redo the steps in bpf.

The data you want is available in /proc/pid/cmdline so a first guess would be a similar named function:

$ sudo bpftrace -l 'kprobe:*cmdline*'
kprobe:cmdline_partition
kprobe:cmdline_parts_find
kprobe:cmdline_parts_free
kprobe:cmdline_parts_parse
kprobe:cmdline_parts_set
kprobe:cmdline_proc_show
kprobe:drm_connector_pick_cmdline_mode
kprobe:drm_mode_create_from_cmdline_mode
kprobe:drm_mode_parse_cmdline_extra
kprobe:drm_mode_parse_cmdline_options.constprop.0
kprobe:get_cmdline
kprobe:get_mm_cmdline
kprobe:ima_kexec_cmdline
kprobe:proc_pid_cmdline_read
kprobe:vm_cmdline_get
kprobe:vm_cmdline_get_device
kprobe:vm_cmdline_set
kprobe:vm_unregister_cmdline_device

kprobe:proc_pid_cmdline_read seems promising, lets try:

sudo bpftrace -e 'kprobe:proc_pid_cmdline_read { printf("hit pid: %d\n", pid) }'
Attaching 1 probe...

another window:

cat /proc/$$/cmdline

hit pid: 1022199

so that seems to work.

Now lets look at the function: https://elixir.bootlin.com/linux/latest/source/fs/proc/base.c#L358

static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,
				     size_t count, loff_t *pos)
{
	struct task_struct *tsk;
	ssize_t ret;

	BUG_ON(*pos < 0);

	tsk = get_proc_task(file_inode(file));
	if (!tsk)
		return -ESRCH;
	ret = get_task_cmdline(tsk, buf, count, pos);
	put_task_struct(tsk);
	if (ret > 0)
		*pos += ret;
	return ret;
}

So it first gets the task_struct for the pid associated with the file (remember its /proc/PID/cmdline so the pid is in there) and then calls a helper to extract the cmdline. We have task_struct available as curtask to we can use that.

Inspecting the next function:

static ssize_t get_task_cmdline(struct task_struct *tsk, char __user *buf,
				size_t count, loff_t *pos)
{
	struct mm_struct *mm;
	ssize_t ret;

	mm = get_task_mm(tsk);
	if (!mm)
		return 0;

	ret = get_mm_cmdline(mm, buf, count, pos);
	mmput(mm);
	return ret;
}

So that seems like its using the mm field of the struct struct mm_struct *mm;

If we now look at https://elixir.bootlin.com/linux/latest/source/fs/proc/base.c#L256 it basically loops over arg_start so we should be able to reproduce that, lets try:

// what we attach to doesn't really matter, but this one does't trigger often and is easy to trigger as user
kprobe:proc_pid_cmdline_read {
  $mm = curtask->mm;
  $start = $mm->arg_start;
  $end = $mm->arg_end;

  if ($start >= $end) {
    printf("%d start > end\n", pid);
    return;
  }
  printf("%d %s\n", pid, str(curtask->mm->arg_start));
}

Let's run it:

$ sudo bpftrace cmdline.bt
Attaching 1 probe...
363
363

sadpanda.jpg

No data so likely that the probe_read failed, let's try again:

$ sudo bpftrace -kk cmdline.bt
Attaching 1 probe...
cmdline.bt:10:26-53: WARNING: Failed to probe_read_kernel_str: Numerical result out of range (-34)
  printf("%d %s\n", pid, str($start));
                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~
363

Digging deeper https://elixir.bootlin.com/linux/latest/source/mm/memory.c#L5133 we see: ret = get_user_pages_remote( so maybe these are not kernel mem, let's try. Change the line to:

printf("%d %s\n", pid, str(uptr($start)));

$ sudo bpftrace -kk cmdline.bt
Attaching 1 probe...
363 /lib/systemd/systemd-journald
363 /lib/systemd/systemd-journald
1023616 cat
1023616 cat
1023617 cat

Now you think you can write printf("%d %s %s\n", pid, str(uptr($start)), str(uptr($start+1))); but that won't work. Args are stored as unsigned long arg_start, arg_end so its just a start and end so to find the start of the next arg we'd need to know the amount of bytes read and add that to $start but that's information we don't have :(, str doesn't tell us how many bytes we have read.

What we can do however is use buf. We know its a linear range of mem so instead we can do:

  printf("%d %r\n", pid, buf(uptr($start), uptr($end-$start)));

$ cat /proc/self/cmdline abc banana ananas

$ sudo bpftrace -kk cmdline.bt
Attaching 1 probe...
363 /lib/systemd/systemd-journald\x00
363 /lib/systemd/systemd-journald\x00
1024384 cat\x00/proc/self/cmdline\x00abc\x00banana\x00ananas\x00
1024384 cat\x00/proc/self/cmdline\x00abc\x00banana\x00ananas\x00

So not great but definitely usable :)

[udaypalagati]

Thanks for the great explanation, looks like "kprobe:proc_pid_cmdline_read" only probes commands to read cmdline(eg: cat /proc/self/cmdline abc banana ananas or cat /proc/11922/cmdline etc ) its not probing any other commands or activity(eg: cat abcd.txt or any other commands which are running in the system) these pids are not showing up in that output.

tried to look at other probes above listed, none of them seems to be probing all the commands running in a system.

probably another way i can think of is to identify which k function populates command lines in /proc/*pid*/cmdline path and trace that

[fbs]

thats the wrong take away from this. It does not matter which probe you use as long as you have a valid curtask. You can paste these lines into opensnoop.bt or execsnoop.bt and it will work. Thats why i also added the comment // what we attach to doesn't really matter, but this one does't trigger often and is easy to trigger as user

tracepoint:syscalls:sys_enter_exec* {
  $mm = curtask->mm;
  $start = $mm->arg_start;
  $end = $mm->arg_end;

  printf("%d %r executed: ", pid, buf(uptr($start), uptr($end-$start)));
  join(args->argv);
}

1053748 -bash\x00 executed: bash -c cat execsnoop.bt
1053748 bash\x00-c\x00cat execsnoop.bt\x00 executed: cat execsnoop.bt

[udaypalagati]

Great, this works!