Hot answers tagged

15

A mainframe is designed for processing large amounts of information by the use of batch transaction processing. It particularly works well at running scalable software and dealing with massively parallel operations. Everything about mainframes is screamingly fast. Mainframes are typically built by IBM and usually run z/OS. A server (when referred to in the ...


6

It can be expensive and painful, but in the end you need to have a local "cluster". Trying to simulate race conditions, contention and the like are very hard on a single PC (my interpretation of "local development environment"). From past experience I would suggest: Push very hard to get a production level cluster into your test/dev environment, You can ...


5

I've written a lot of .NET clustered applications over the years (exclusively for Active/Passive clusters) and can share what I've learned. I've always written my apps as Windows Services and include them as clustered resources in the Windows Cluster Administrator. When the cluster fails over from Node1 to Node2, the Cluster Administrator shuts down the ...


5

Several possible downsides or issues you have to code for: Login sessions must either be stored in a central database (such as redis) that all clusters can access or connections must be made sticky so that a given client goes back to the same cluster process every time. Other server side state is either maintained separately by each clustered process or ...


5

The problem with the last option is that I would still need somewhere to store queue metadata information like creation date. Other than you may store other fields (like timestamp) in Metadata database, Metadata service should scale out well for reads and be highly available as every PUT and GET call for a message results in a call to the Metadata service. ...


3

I'd suggest to define criteria of what "dead" means, then periodically poll for the "dead" condition and perform the swing over. Perhaps "dead" gets defined as "hasn't sent any messages to any of the nodes in X seconds". Whatever decision tree a human currently follows to ascertain whether or not to flip service. It may be 1 condition, 10, or dozens. How ...


3

I can recommend the Hangfire as a solution. Key highlights for your case (extracted from the web site), Persistent Background jobs are saved into a persistent storage – SQL Server, Redis, PostgreSQL, MongoDB and others. You can safely restart your application and use Hangfire with ASP.NET without worrying about application pool recycles. Reliable Once a ...


3

If you are considering only a scaled out, clustered environment, with replicated Sessions, then you don't have a workaround. All of your objects must be Serializable. But if your application and Java EE architecture allows you to do a scaled out "clustered" environment without replicated Sessions, then you are fine. The only thing you will lose here is the ...


3

My first suggestion is to cut up your problem into two problems: first, figure out what you want and then figure out how to efficiently get what you want. You can't efficiently get something you haven't defined yet. I'll put some ideas in this answer that may help you find this definition. I suggest you make an inefficient implementation of the ideas you ...


3

For an algorithm with very little communication between computers (not sensitive to bandwidth limitations) that uses asynchronous communication (not sensitive to network latency), number of CPUs will dominate performance and networking is mostly irrelevant. For an algorithm with high communication between computers (very sensitive to bandwidth limitations) ...


3

The bandwidth is the distance/size scale of the kernel function, i.e. what the size of the “window” is across which you calculate the mean. There is no bandwidth that works well for all purposes and all instances of the data. Instead, you will need to either manually select an appropriate bandwith for your algorithm; or use an algorithm that automatically ...


2

Mbps is not the only measure to measure network speed, the other arguable more important one is setup time (how long it takes for a pair of CPUs to be ready to send a packet) which is in many cases an order of magnitude larger than straight up bandwidth. This is the reason why most algorithms try to chunk as much sending as possible This means that as soon ...


2

answer of the first question is, as you said, it will either timeout or get some error messages from server. it depends on how you design your program. I suppose some of nodes in the cluster are working as api servers as it's a reasonable configuration. If you don't have an api server in the cluster, I believe you must have to set it up in somewhere else. ...


2

Using modular decomposition you can create a tree that will contain all particles as leafs and upper nodes will cluster these. Based on that tree you can define measures that are applied to every node of it from the root to the leafs downwards. You stop this downwards traversal when the measurements reach user defined thresholds. One such measurement may be ...


2

The basic question "one or two servers" is the wrong question asked, IMHO. As I understand your question it should be more asking if the "Pull" and the "Push" functionality should be handled by the same server application. My plug would be to separate these functionalities as best as possible, makes the whole stuff easier to develop and maintain. That said,...


2

You can partition your data easily with consistent hashing, in this case you would use the entity as the hash key. The consistent hash takes a key and # of "buckets" as input and gives you back the bucket for that key. With multiple servers in mind, a simple solution would be to pick a number of partitions up front (lets say 6), which means you will have 6 ...


2

You don't need any DB operation for synchronization between the operations. If you need all the 3 operations to be performed on your data sequentially : Send your Data to Queue A, which has consumers which perform operation A and at the end of it, send it to Queue B. Consumers on Queue B will perform Operation B and send the data to Queue C where ...


2

We have a cluster-environment too. We use Hazelcast for such jobs. With Hazelcast you could embed the codeblock for updating within a "Hazelcast-Lock-Section". It is not my favourite solution, but this is how it is done in our application (and maybe suits hazelcast your needs). I opt for a smaller and easier solution: I would write a small (buzzword-alarm: ...


2

Are you losing anything ? Depends on whether you are dependant on the j2ee specs. For example JCA, If not then stay away. Stateless services are way more flexible in terms of Scalability. And more over its easy to refactor to microservices.


2

The entire point of Zookeeper (as I understand it) is to make restart AFTER the partition goes away simple by making sure that there's only ONE portion (the majority) that was changing during the partition. It can then bring the minority up to speed when they reconnect and everything is running again with no further intervention on your part. If you let two ...


2

The main reason it's a daemon is for container monitoring. You want something that occasionally checks if the container is healthy, and restarts it if it isn't. This requires some sort of process running in the background, or at least waking up periodically. Now, there's no reason that has to be one centralized system daemon. You could make smaller ones ...


2

As short outages (long enough to restart a VM) are acceptable, you don't need the immediate failover capability of an active/standby configuration. That makes the main concern that you don't get a corruption of the database storage. You have partially tacked that by having redundant storage with a main database and a backup database. What still can go wrong ...


2

The first things first. There are two completely different types of "API"s people call RESTful. First, the original meaning as described by Roy Fielding essentially describes how you create a web application. A RESTful API is basically a web-application for machines, but works exactly as a web-application for humans. That is, it starts with a ...


1

What you are describing is close to a K-means. But (classic) K-means is based on square of the distance and cluster size is not fixed. So you would need to adjust your criteria to comply with K-means or modify the algorithm. If you don't want to take the distance squared the algorithm gets more complex.


1

You dont initialize k-means using Euclidean distance. There are a range of initialization methods. The most intuitive is selecting random instances from your data and initializing there. That way each cluster in your first assignment step has at least one instance. The assignment step is where you apply a distance measure, like euclidean distance.


1

It just occurred to me that although the 2 member scenario could indeed be made more available than it is right now, it would still not be more available than the 1 member scenario, and writing is in fact slower for 2 members, because every change needs to be propagated to another member. Only reads from ZooKeeper will improve under heavy load (assuming the ...


1

I think you're after a machine learning clustering algorithm. This page from the Python SciKit Learn toolkit has pictures that suggest the DBSCAN algorithm (Wikipedia) could be what you're looking for. It seems ideal as it's input parameter is the neighbourhood size, while most other clustering algorithms want the number of clusters, which you would not ...


Only top voted, non community-wiki answers of a minimum length are eligible