Creating an expandable, cross-platform compatible program "core"

Question

Basically the brief is relatively simple. We need to create a program core. An engine that will power all sorts of programs with a large number of distinct potential applications and deployments.

The core will be an analytics and algorithmic processor which will essentially take user-specific input and output scenarios based on the information it gets, whilst recording this information for reporting.

It needs to be cross platform compatible. Something that can have platform specific layers put on top which can interface with the core.

It also needs to be able to be expandable, for instance, modular with developers being able to write "add-ons" or "extensions" which can alter the function of the end program and can use the core to its full extent.

(For instance, a good example of what I'm looking to create is a browser. It has its main core, the web-kit engine, for instance, and then on top of this is has a platform-specific GUI and can also have add-ons and extensions which can change the behavior of the program.)

Our problem is that the extensions need to interface directly with the main core and expand/alter that functionality rather than the platform specific "layer".

So, given that I have no experience in this whatsoever (I have a PHP background and recently objective-c), where should I start, and is there any knowledge/wisdom you can impart on me please?

Thanks for all the help and advice you can give me. :) If you need any more explanation just ask. At the moment its in the very early stages of development, so we're just researching all possible routes of development.

Thanks a lot

Answer 1

What you are talking about is building a rather capable framework. At a 10,000ft view, the concept is simple, but as you get closer to implementing it you will find that it will be fraught with difficulty. There's been a few different technologies that have been designed to allow cross-platform/programming language support. A couple are listed below:

1. Custom language binding -- gets you the most speed, and a library like SWIG can translate between a few low level and high level languages for you. Problem is that it can get rather complicated as you try to work out compiler specific name-mangling issues and other serious gotchas (such as interfacing a non-garbage collected language with a garbage collected language).
2. CORBA -- Let all who have messed with CORBA commence with the cursing. The concept of CORBA is the use of an Interface Definition Language (IDL) to generate the client/server bindings. The CORBA platform encodes the messages as strings and sends them through the CORBA engine from client to server and back. The advantage of CORBA is that you can split your clients and servers (users and implementations) across machines. The disadvantage is everything else--and performance.
3. Web Services -- It's an extension of the same basic concepts of CORBA, but with a lot less work. Essentially you have a request/response structure, and two major styles of interaction. SOAP based web services are almost exactly like CORBA except you use XML for your transport. REST style web services are almost exactly like serving up web pages except you send back JSON or XML (depending on the request). This of course gives you the least performance, although you gain a reasonably predictable life-cycle (unlike CORBA).

The other major complexity you are working with is the concept of an extensible UI. Essentially your framework needs to be able to map UI features dynamically based on the properties of the request and algorithm. I don't know enough about your problem domain to suggest exactly how to break that down. And if I did, I'd charge for the work (I'm not cheap). In general, you want to look at what hints you need from the algorithm itself, the input, and the type of query posed to your framework. Your UI components will also need to provide hints about what they can be bound to, and the type of object that can be visualized. From there you can draw up some rules about how to match UI bindings to the data from the algorithm.

This at least will get you thinking in the right direction. Please don't forget about security, denial of service protection, and other issues that most people forget.

Answer 2

I believe the practical technical side of your question has already been extensively covered. But, I would like to raise a more fundamental issue. A warning in advance: it could be that I am gruesomely misunderstanding you, making my contribution complete nonsense :)

<cynism>

We need to create a program core. An engine that will power all sorts of programs with a large number of distinct potential applications and deployments. The core will be an analytics and algorithmic processor which will essentially take user-specific input and output scenarios based on the information it gets, whilst recording this information for reporting.

Congratulations, you have just invented the java virtual machine.

It needs to be cross platform compatible. Something that can have platform specific layers put on top which can interface with the core. It also needs to be able to be expandable, for instance, modular with developers being able to write "add-ons" or "extensions" which can alter the function of the end program and can use the core to its full extent.

Well done. You have invented the concept of programming languages. You're a few decades late though.

</cynism>

More serious, it sounds like you're trying to invent something so amazingly generic, that it can solve practically every problem. It reminds me of, but is not the same as, the Enterprise Rules Engine.

You may spend an eternity developing this "core" and then find out that you have created extremely generic software. In real life however, we have almost no generic problems. We have sets of specific ones, some of which share properties. You may well end up with a very generic core, which is unsuitable for many of your specific problems. Basically, you'll have done it wrong and will have wasted a lot of time. In the mean time, you have no working software. Also, don't forget that the requirements of the specific problems will have changed in the mean time: requirements are never fixed.

I suggest to take it the other way around. Write the first "thing" that you would like to run on top of this core. Immediate win: working software very quickly. Then, write the next thing. Perhaps even another one. Only at that point, see what your code has in common. Extract those parts to something generic - not generic for all problems in the world, but just the two or three you're solving right now. This means you have working software fast, and a core that does exactly what it needs to do - no more, no less. As long as you write your code reasonably modular, not like a big ball of mud, and have some decent testing, that shouldn't be very difficult.

Nuancing that a bit, sometimes it's just plain obvious what the generic parts are. Go for these almost straight away. But take a moment to stop, and think whether you are absolutely sure that these are generic parts. If in doubt, don't make it generic for now: YAGNI. You can always refactor.

Answer 3

I would say if you want cross-platform-ish then you're probably wise to start with Java. It's not clear from your question whether you are talking about creating something web based and accessible from different platforms or that can be installed on different platforms by users. In the latter case Java or a scripting language available on many different types of platform is your best bet.

Given the vague brief you have it actually sounds like you're trying to create a kind of virtual machine anyway, so given that Java sort of works cross platform already ( in general it does work on a cross platform basis, but often the devil is in the details ) that seems like a sound starting point.

It seems to me that you are talking about creating a Domain Specific Language here. Assuming you're clear on the details of DSL design ( and language design in general ) and you have an absolutely detailed and unambiguous specification that should be an interesting and quite achievable project. Not necessarily a simple one, especially if you are hoping to make it easy to interface with many different languages. If you don't have existing expertise here, you may want to find someone who does and persuade them to join the team. What you're describing sounds like a considerable endeavour so there should be plenty of budget for that.

Your question does not make it clear whether you have a clear, detailed and unambiguous specification, but it is quite a vague question - if you don't have a clearer brief than the question you are asking then you need to fix that before you take things any further.

Answer 4

I've seen a team fall into this trap (well, they actively ran towards it and jumped into it, and subsequently died on the spikes).

The problem is that there is a whole lot of reinventing the wheel. Think of how much time it is going to take to develop all those plugins/user-extensible stuff that is the actual core of what you're trying to do. Now, you have this extra huge task of building a framework/core. Your actual business need (the first stuff) will be delayed by 6mos-12mos by developing the core. Plus, your core will be fresh code and you will expend considerable effort fixing bugs that don't relate to your actual business need.

The path to success is to find something that is mature (not just one guy's open source project which hasn't been updated in 3 years) and already exists and does 90%+ of what you want. Then make it do the other 10% or adjust your design to fit the constraints. For your case if you want an end user app Java is a no-brainer, or something that runs out of a browser if you want remote access. That way you're getting the millions of hours of testing and product maturity for free. And you can deliver 12 months earlier.

The danger is when people say (and sell to their bosses) that "there's nothing out there that does exactly what we want." Of course not, but that doesn't mean you reinvent the wheel, you adapt. "I don't like (one nitpick about linux) so we need to write an operating system!" It sounds silly doesn't it? Yet teams do this all the time. Not saying that this is the case for your group, just know that there are lots of dead projects on the spikes in that pit ahead.

Answer 5

I don't fully understand what you are tying to do, and your example of a web browser just confuses the matter more...

But, To me it sound like you need an SOA (Service oriented Architecture) with a central server. In which case the language you choose makes little difference.

Answer 6

Ive done it. http://members.dodo.com.au/~bpc01/index.html Has linked lists of each data type in the main class (can have as much or as little data as a class needs) Expandable using shared memory segments so that each .dll can add its own classes to the main list that classes are cloned from in the main program. Multicore enabled so that lists of classes can be run parallel. Ive added Lua as a scripting language so it can be expanded using compiled code (.dll's) or scripts. I am going to add graphics, sounds, networking and a database for more functionality