16

Background:

System call overhead is much larger than function call overhead (estimates range from 20-100x) mostly due to context switching from user space to kernel space and back. It is common to inline functions to save function call overhead and function calls are much cheaper than syscalls. It stands to reason that developers would want to avoid some of the system call overhead by taking care of as much in-kernel operation in one syscall as possible.

Problem:

This has created a lot of (superfluous?) system calls like sendmmsg(), recvmmsg() as well as the chdir, open, lseek and/or symlink combinations like: openat, mkdirat, mknodat, fchownat, futimesat, newfstatat, unlinkat, fchdir, ftruncate, fchmod, renameat, linkat, symlinkat, readlinkat, fchmodat, faccessat, lsetxattr, fsetxattr, execveat, lgetxattr, llistxattr, lremovexattr, fremovexattr, flistxattr, fgetxattr, pread, pwrite etc...

Now Linux has added copy_file_range() which apparently combines read lseek and write syscalls. Its only a matter of time before this becomes fcopy_file_range(), lcopy_file_range(), copy_file_rangeat(), fcopy_file_rangeat() and lcopy_file_rangeat()...but since there are 2 files involved instead of X more calls, it could become X^2 more. OK, Linus and the various BSD developers wouldn't let it go that far, but my point is that if there were a batching syscall, all(most?) of these could be implemented in user space and reduce the kernel complexity without adding much if any overhead on the libc side.

Many complex solutions have been proposed that include some form special syscall thread for non-blocking syscalls to batch process syscalls; however these methods add significant complexity to both the kernel and user space in much the same way as libxcb vs. libX11 (the asynchronous calls require a lot more setup)

Solution?:

A generic batching syscall. This would alleviate the largest cost (multiple mode switches) without the complexities associated with having specialized kernel thread (though that functionality could be added later).

There is basically already a good basis for a prototype in the socketcall() syscall. Just extend it from taking a array of arguments to instead take an array of returns, pointer to arrays of arguments (which includes the syscall number), the number of syscalls and a flags argument... something like:

batch(void *returns, void *args, long ncalls, long flags);

One major difference would be that the arguments would probably all need to be pointers for simplicity so that the results of prior syscalls could be used by subsequent syscalls (for instance the file descriptor from open() for use in read()/write())

Some possible advantages:

  • less user space -> kernel space -> user space switching
  • possible compiler switch -fcombine-syscalls to try to batch automagically
  • optional flag for asynchronous operation (return fd to watch immediately)
  • ability to implement future combined syscall functions in userspace

Question:

Is it feasible to implement a batching syscall?

  • Am I missing some obvious gotchas?
  • Am I overestimating the benefits?

Is it worthwhile for me to bother implementing a batching syscall (I don't work at Intel, Google or Redhat)?

  • I have patched my own kernel before, but dread dealing with the LKML.
  • History has shown that even if something is widely useful to "normal" users (non-corporate end users without git write access), it may never get accepted upstream (unionfs, aufs, cryptodev, tuxonice, etc...)

References:

  • 4
    One fairly obvious problem I am seeing is that the kernel gives up control about the time and space required for a syscall as well as the complexity of operations of a single syscall. You basically have created a syscall that can allocate arbitrary, unbounded amounts of kernel memory, run for an arbitrary, unbounded amount of time, and can be arbitrarily complex. By nesting batch syscalls into batch syscalls, you can create an arbitrarily deep call tree of arbitrary syscalls. Basically, you can put your whole application into a single syscall. – Jörg W Mittag Feb 26 '16 at 10:50
  • @JörgWMittag - I'm not suggesting that these run in parallel, so the amount of kernel memory used would be no more than the heaviest syscall in the batch and the time in kernel is still bounded by the ncalls parameter (which could be limited to some arbitrary value). Your are right about a nested batch syscall being a powerful tool, maybe so much so that it should be precluded (though I could see it being useful in a static file server situation - by intentionally sticking a daemon into a kernel loop using pointers - basically implementing the old TUX server) – technosaurus Feb 26 '16 at 11:48
  • 1
    Syscalls involve a privilege change but this is not always characterized as a context switch. en.wikipedia.org/wiki/… – Erik Eidt Feb 26 '16 at 15:39
  • 1
    read this yesterday which provides some more motivation and background: matildah.github.io/posts/2016-01-30-unikernel-security.html – Tom Mar 1 '16 at 22:34
  • @JörgWMittag nesting could be disallowed to prevent from kernel stack overflow. Otherwise, individual syscall will free after themselves like they normally do. There shouldn't be any resource-hogging problems with this. The Linux kernel is preemptible. – PSkocik Jun 5 '17 at 12:47
5

I tried this on x86_64

Patch against 94836ecf1e7378b64d37624fbb81fe48fbd4c772: (also here https://github.com/pskocik/linux/tree/supersyscall )

diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index 5aef183e2f85..8df2e98eb403 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -339,6 +339,7 @@
 330    common  pkey_alloc      sys_pkey_alloc
 331    common  pkey_free       sys_pkey_free
 332    common  statx           sys_statx
+333    common  supersyscall            sys_supersyscall

 #
 # x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 980c3c9b06f8..c61c14e3ff4e 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -905,5 +905,20 @@ asmlinkage long sys_pkey_alloc(unsigned long flags, unsigned long init_val);
 asmlinkage long sys_pkey_free(int pkey);
 asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags,
              unsigned mask, struct statx __user *buffer);
-
 #endif
+
+struct supersyscall_args {
+    unsigned call_nr;
+    long     args[6];
+};
+#define SUPERSYSCALL__abort_on_failure    0
+#define SUPERSYSCALL__continue_on_failure 1
+/*#define SUPERSYSCALL__lock_something    2?*/
+
+
+asmlinkage 
+long 
+sys_supersyscall(long* Rets, 
+                 struct supersyscall_args *Args, 
+                 int Nargs, 
+                 int Flags);
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index a076cf1a3a23..56184b84530f 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -732,9 +732,11 @@ __SYSCALL(__NR_pkey_alloc,    sys_pkey_alloc)
 __SYSCALL(__NR_pkey_free,     sys_pkey_free)
 #define __NR_statx 291
 __SYSCALL(__NR_statx,     sys_statx)
+#define __NR_supersyscall 292
+__SYSCALL(__NR_supersyscall,     sys_supersyscall)

 #undef __NR_syscalls
-#define __NR_syscalls 292
+#define __NR_syscalls (__NR_supersyscall+1)

 /*
  * All syscalls below here should go away really,
diff --git a/init/Kconfig b/init/Kconfig
index a92f27da4a27..25f30bf0ebbb 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2184,4 +2184,9 @@ config ASN1
      inform it as to what tags are to be expected in a stream and what
      functions to call on what tags.

+config SUPERSYSCALL
+     bool
+     help
+        System call for batching other system calls
+
 source "kernel/Kconfig.locks"
diff --git a/kernel/Makefile b/kernel/Makefile
index b302b4731d16..4d86bcf90f90 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -9,7 +9,7 @@ obj-y     = fork.o exec_domain.o panic.o \
        extable.o params.o \
        kthread.o sys_ni.o nsproxy.o \
        notifier.o ksysfs.o cred.o reboot.o \
-       async.o range.o smpboot.o ucount.o
+       async.o range.o smpboot.o ucount.o supersyscall.o

 obj-$(CONFIG_MULTIUSER) += groups.o

diff --git a/kernel/supersyscall.c b/kernel/supersyscall.c
new file mode 100644
index 000000000000..d7fac5d3f970
--- /dev/null
+++ b/kernel/supersyscall.c
@@ -0,0 +1,83 @@
+#include <linux/syscalls.h>
+#include <linux/uaccess.h>
+#include <linux/compiler.h>
+#include <linux/sched/signal.h>
+
+/*TODO: do this properly*/
+/*#include <uapi/asm-generic/unistd.h>*/
+#ifndef __NR_syscalls
+# define __NR_syscalls (__NR_supersyscall+1)
+#endif
+
+#define uif(Cond)  if(unlikely(Cond))
+#define lif(Cond)  if(likely(Cond))
+ 
+
+typedef asmlinkage long (*sys_call_ptr_t)(unsigned long, unsigned long,
+                     unsigned long, unsigned long,
+                     unsigned long, unsigned long);
+extern const sys_call_ptr_t sys_call_table[];
+
+static bool 
+syscall__failed(unsigned long Ret)
+{
+   return (Ret > -4096UL);
+}
+
+
+static bool
+syscall(unsigned Nr, long A[6])
+{
+    uif (Nr >= __NR_syscalls )
+        return -ENOSYS;
+    return sys_call_table[Nr](A[0], A[1], A[2], A[3], A[4], A[5]);
+}
+
+
+static int 
+segfault(void const *Addr)
+{
+    struct siginfo info[1];
+    info->si_signo = SIGSEGV;
+    info->si_errno = 0;
+    info->si_code = 0;
+    info->si_addr = (void*)Addr;
+    return send_sig_info(SIGSEGV, info, current);
+    //return force_sigsegv(SIGSEGV, current);
+}
+
+asmlinkage long /*Ntried*/
+sys_supersyscall(long* Rets, 
+                 struct supersyscall_args *Args, 
+                 int Nargs, 
+                 int Flags)
+{
+    int i = 0, nfinished = 0;
+    struct supersyscall_args args; /*7 * sizeof(long) */
+    
+    for (i = 0; i<Nargs; i++){
+        long ret;
+
+        uif (0!=copy_from_user(&args, Args+i, sizeof(args))){
+            segfault(&Args+i);
+            return nfinished;
+        }
+
+        ret = syscall(args.call_nr, args.args);
+        nfinished++;
+
+        if ((Flags & 1) == SUPERSYSCALL__abort_on_failure 
+                &&  syscall__failed(ret))
+            return nfinished;
+
+
+        uif (0!=put_user(ret, Rets+1)){
+            segfault(Rets+i);
+            return nfinished;
+        }
+    }
+    return nfinished;
+
+}
+
+
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index 8acef8576ce9..c544883d7a13 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -258,3 +258,5 @@ cond_syscall(sys_membarrier);
 cond_syscall(sys_pkey_mprotect);
 cond_syscall(sys_pkey_alloc);
 cond_syscall(sys_pkey_free);
+
+cond_syscall(sys_supersyscall);

And it appears to work -- I can write hello to fd 1 and world to fd 2 with just one syscall:

#define _GNU_SOURCE
#include <unistd.h>
#include <sys/syscall.h>
#include <stdio.h>


struct supersyscall_args {
    unsigned  call_nr;
    long args[6];
};
#define SUPERSYSCALL__abort_on_failure    0
#define SUPERSYSCALL__continue_on_failure 1

long 
supersyscall(long* Rets, 
                 struct supersyscall_args *Args, 
                 int Nargs, 
                 int Flags);

int main(int c, char**v)
{
    puts("HELLO WORLD:");
    long r=0;
    struct supersyscall_args args[] = { 
        {SYS_write, {1, (long)"hello\n", 6 }},
        {SYS_write, {2, (long)"world\n", 6 }},
    };
    long rets[sizeof args / sizeof args[0]];

    r = supersyscall(rets, 
                     args,
                     sizeof(rets)/sizeof(rets[0]), 
                     0);
    printf("r=%ld\n", r);
    printf( 0>r ? "%m\n" : "\n");

    puts("");
#if 1

#if SEGFAULT 
    r = supersyscall(0, 
                     args,
                     sizeof(rets)/sizeof(rets[0]), 
                     0);
    printf("r=%ld\n", r);
    printf( 0>r ? "%m\n" : "\n");
#endif
#endif
    return 0;
}

long 
supersyscall(long* Rets, 
                 struct supersyscall_args *Args, 
                 int Nargs, 
                 int Flags)
{
    return syscall(333, Rets, Args, Nargs, Flags);
}

Basically I'm using:

long a_syscall(long, long, long, long, long, long);

as a universal syscall prototype, which appears to be how things work on x86_64, so my "super" syscall is:

struct supersyscall_args {
    unsigned call_nr;
    long     args[6];
};
#define SUPERSYSCALL__abort_on_failure    0
#define SUPERSYSCALL__continue_on_failure 1
/*#define SUPERSYSCALL__lock_something    2?*/

asmlinkage 
long 
sys_supersyscall(long* Rets, 
                 struct supersyscall_args *Args, 
                 int Nargs, 
                 int Flags);

It returns the number of syscalls tried (==Nargs if the SUPERSYSCALL__continue_on_failure flag is passed, otherwise >0 && <=Nargs) and failures to copy between kernels space and user space are signalled by segfaults instead of the usual -EFAULT.

What I don't know is how this would port to other architectures, but it would sure be nice to have something like this in the kernel.

If this were possible for all archs, I imagine there could be a userspace wrapper that would provide type safety through some unions and macros (it could select a union member based on the syscall name and all the unions would then get converted to the 6 longs or whatever the architecture de jour's equivalent of the 6 longs would be).

  • 1
    Its a good proof of concept, though I would like to see an array of pointers to long instead of just an array of long, so that you could do things like open-write-close using the return of open in write and close. That would increase complexity a bit due to get/put_user, but probably worth it. As to portability IIRC some architectures may clobber the syscall registers for args 5 and 6 if a 5 or 6 arg syscall is batched ... adding 2 additional args for future use would fix that and could be used in the future for asynchronous call parameters if a SUPERSYSCALL__async flag is set – technosaurus Jun 7 '17 at 4:50
  • 1
    My intention was to also add a sys_memcpy. The user could then put it in between sys_open and sys_write to copy the returned fd to the first argument of sys_write without having to switch mode back to userspace. – PSkocik Jun 7 '17 at 6:54
3

Two main gotchas which come to mind immediately are:

  • Error handling: each individual syscall may end with an error which needs to be checked and handled by your user-space code. A batching call would therefore have to run user-space code after each individual call anyway so the benefits of batching kernel-space calls would be negated. Additionally, the API would have to be very complex (if possible to design at all) - for example how would you express logic such as "if the third call failed, do something and skip fourth call but continue with the fifth")?

  • Many "combined" calls which actually do get implemented offer additional benefits apart from not having to move between user and kernel space. For example, they will often avoid copying memory and using buffers altogether (e.g. transfer data directly from one place in the page buffer to another instead of copying it through an intermediate buffer). Of course, this only makes sense for specific combinations of calls (e.g. read-then-write), not for arbitrary combinations of batched calls.

  • 2
    Re: error handling. I thought about that and that's why I suggested the flags argument (BATCH_RET_ON_FIRST_ERR) ... a succesful syscall should return ncalls if all calls complete without error or the last succesful one if one fails. This would allow you to check for errors and possibly try again starting at the first unsuccessful call just by incrementing 2 pointers and decrementing ncalls by the return value if a resource was just busy or the call was interupted. ... the non-context switiching parts are out of scope for this, but since Linux 4.2, splice() could help those too – technosaurus Feb 26 '16 at 11:21
  • 2
    The kernel could automatically optimize the call list to merge various operations and eliminate redundant work. The kernel would probably do a better job than most individual developers at a great savings in effort with a simpler API. – Aleksandr Dubinsky Mar 3 '16 at 13:04
  • @technosaurus It wouldn't be compatible with technosaurus' idea of exceptions which communicate which operation failed (because the order of operations gets optimized). This is why exception aren't normally designed to return such precise information (also, because the code becomes confusing and fragile). Fortunately, it's not difficult to write generic exception handlers that handle various failure modes. – Aleksandr Dubinsky Mar 3 '16 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.