In a commercial context, yFiles for HTML should be pretty much exactly what you are looking for:
It comes with built-in editing capabilities that can be customized to a very high-degree. E.g. if you aren't happy with the default gestures for creating, connecting, or moving elements, if you need a different gesture for zooming or panning the viewport, all of this can be customized. Also it comes with specific support for touch devices and touch gestures.
You can attach any number and kind of data to the elements and optionally display that data in the visualization. The visualization of the diagram elements (nodes, edges, labels, ports, but also the graphical adorners like resize handles, marquee rectangles, etc.) can be fully customized. Adding new elements is easy because at its core, plain SVG is used, so you can leverage existing graphics and third party designer tools to create interactive CSS styled and animated SVG items.
Disclaimer: I work for the company that creates the yFiles library, however on SE I do not represent my employer. My posts, thoughts, and comments are my own.