The other posted answer is valid. Removing the hidden rows from the DOM will give you better performance. However, it doesn't address the underlying issue that you're passing all data to the browser before applying any filtering to it; which creates many tangential issues such as excessive bandwith usage and higher response times to get the list of items.
because users will complain about filters and sorting being applied to loaded records but not to all records
In other words, if you return page 2 (items 101 to 200), and the user has asked to filter on names containing "foo", you are applying that filter on those 100 items; instead of applying the filter on the database and then showing page 2 of the resulting set?
This is your issue. Filters must be applied before pagination. If you only perform the filtering on the webpage after all the items have loaded, then pagination must logically take place in the browser as well.
Workarounds for your current performance issue may exist (such as removing the filtered items from the DOM), but when the size of the data grows, the performance (and bandwidth) issue will pop up again in a different form (one that cannot be fixed by removing things from the DOM).
The only way to sustainably solves this is to not pass all the data to the webpage.
I thought about server sorting and filtering instead but that would just is still not suitable for working with other data manipulation functionality such as grouping.
I'm not following your explanation here. Grouping is just another step on the road to getting the output you want. There is a sequential logic about retrieving data:
Not all steps are necessary, but the order in which to execute them is usually the same (barring some fringe cases of e.g. complex group filtering - at which point grouping becomes a given and not an option as per your description)
What you're doing now is dumbing down your backend and taking on all the responsibilities in the frontend, which is simply not good design. The backend (especially a SQL-driven data storage) is tailored for efficient data retrieval.
I see no reasonable argument to shift all that responsibility to a frontend which is inherently less capable of handling large data collections and performing expensive data operations. It upsets every party involved:
- The internal network between database and backend sends way more data than the user is actually interested in.
- The backend needs to instantiate way more data than the user is actually interested in.
- The external network between backend and frontend sends way more data than the user is actually interested in.
- The frontend needs to handle way more data than the user is actually interested in.
You're making losses in network load, bandwidth usage, application performance and user experience, all for the benefit of not having to develop a method to filter/group/order/paginate data on the backend?
The pros are not outweighing the cons here.
[From a comment] there are client side functions that are driven by the users interaction that should take all data into account
You're putting the cart before the horse here. You're already calling them "client side function" which is the source of the issue. If you were to set up an interaction between the frontend and the backend so that the backend can handle the data manipulation needed, then they are no longer "client side functions".
I cannot think of any data manipulation that can only be done client-side and not server-side. I'm interested to hear an example of these "client side functions".