The first thing to understand is that you're basically needing to build a subset of all the parts that go into an operating system scheduler, including:
- instrumentation and monitoring (What resources are currently available?)
- Decision making / policy (How should resources be allocated? How will the users or administrators communicate their policy / priority desires?)
- Enforcement (How should processes be restricted from using more than their allocated amount?)
The good news is that a lot of the components, especially in the monitoring and enforcement realms, will be very thin layers on existing mechanisms such as as Linux cgroups, netstat, and tc.
There is even a chance that the policy mechanism of cgroups will be sufficient to your need. It's a lot more sophisticated than the "priorities" of historical Unix, and many commercial shops have relied on similar (albeit proprietary / single vendor) facilities such as HP PRM and AIX WPARs for application QoS management.
These typically work on some variant of fair share scheduling, which allows one to define relative priorities for different classes or groups of jobs/processes, and arrange the classes/groups with respect to one another, so that their relative importance and resource allocations are as the policy or "as the business" intended, yet without completely starving the lower-priority jobs. (If you're willing to starve lower-priority jobs completely, simpler mechanisms can be used.)
Simple forms of FCS use single-level classes, with each member in a class being equal to the others. More refined forms use hierarchical groupings, so that the "minimum 50% resources devoted to Platinum" users can be further subdivided with specific allocations or minimums for subgroups or even individual users. (Having a single app/job be the only member of a group is an easy way to give it the group's entire allocation.) This page goes into some detail about how one of these "workload managers" (Adaptive Computing's Moab) operates.
There are a variety of ways the policies can be defined. "Shares" is a relatively sophisticated way, but one that novice users often find difficult. "Percentages" or "priorities" are somewhat less precise and more perturbable, but can be more popular with less mathematical policy-setters.
And there are a lot of nuances possible. The LPAR allocations of AIX and IBM Power machines, for example, have a number of subtle options for temporary priority inversions, "stealing" cycles that would otherwise go unused, and otherwise tweaking the policies. If you want something very sophisticated, proprietary Unix systems have a number of good study points.