Are there types of killer applications, classes of algorithmic problems, etc., where it is better, in the long run, to create my own language?
PS: Just to be sure, I mean a new programming language and a compiler, not a new compiler for an existing language.
EDIT: Thank you for the answers. Can you provide some examples, where it is absolutly unnecessary to create a DSL or cases in which a DSL might be a good idea?
It certainly is relevant for a person to write their own language for educational purposes. To learn about programming language design and about compiler design. But real-world uses are few and far between.
In writing your own language you are:
So, if you plan to write your own language for your project then the features that it provides that other languages don't have need to offset the above costs.
Take games development for example. They often need mini-languages within their games or scripting languages. They use these languages to script out a huge amount of the in-game events that happen. However, even in this case, they almost always choose existing scripting languages and tailor them to their needs.
Let me just quote Paul Vick, ex chief developer of the VB compiler and now working on Project Oslo and the M language:
It’s mind-bendingly, stupendously difficult to build a new language, even one that’s largely based on an existing one. Yet many programmers think, “hey, I use languages, how hard can this be?” and go at it. … probably over 98% of them fail to ever gain any traction at all, but god bless the optimists because without them we’d never get the 2% of languages that succeed. I’m personally willing to sacrifice the millions of dollars and hours wasted on languages that never make it just so that we can get languages like C# and Java and Ruby and Python and so on.
So the fact that coming up with a new language is a bad idea shouldn’t dissuade people from developing new DSLs, it should just give them pause and hopefully a little humility. The key, I think, is to start small and to stay small.
When is it reasonable?
When you feel like it!
Don't listen to these people who have snarky comments that basically say:
"Don't do it because its too hard and X language is better than any language you can come up with".
The thing is, creating a DSL happens all the time. A framework is a DSL. A macro is a DSL. Every time you write a function for your program, that is part of a DSL. Sure, its within the bounds of the grammar, but vocabulary is part of a language. This is why industries often create their own vernacular: its more efficient!
If "don't do it" was the right answer, we would all be writing COBOL and Fortran.
You might want to read parts of Martin Fowler's upcoming DSL book, if you're thinking of writing you own language.
I can't really think of a business case to create a language from scratch other than it being a tremendous learning experience.
Edit: for DSL's there are plenty of business cases, but the key here is not to get carried away and Keep It Simple.
I suggest that the key questions are, "What problem am I trying to solve?" and "Who gets the ROI?"
If you're trying to build your own skills and experience, then full speed ahead, but not in a production system that's supposed to solve someone else's problem.
Seems like the main reason you'd want a new language is that you start discovering patterns in your code that existing languages don't handle well. But there are a bunch of problems with creating your own language. You'll be missing out on all the libraries and frameworks that are built up for existing languages. You'll spend a lot of time designing and implementing the new language, which is all time you don't have to spend on the real programming task. You'll spend a lot of effort convincing other developers that they should use your language. And, you'll have a hard time recruiting and training new developers.
Why not write in a language like Lisp that lets you extend the language as you discover new patterns? Then, you get all the power of a new language with all the benefits of an established language.
One reason could be to create it as an experiment to learn about language design and compiler building.
Another reason could be to build a scripting language into an application when you don't have the option to add a third-party API.
I don't think you can program without creating a new language, so it's good to realize that's what you're doing and understand the issues.
An off-the-shelf language like VB, Java, C#, etc. is just a base language. As soon as you add classes, methods, etc. to it, you have added vocabulary and semantics. There are many ways to implement languages - parsing & translating, parsing & interpreting, macros on top of an existing language, adding classes&methods to an existing language.
How do you know if you've done this? The measure I use is edit count. If one-sentence requirement A comes along, I proceed to implement the requirement in code. When I'm done & got all the bugs out, I check in the code, and the code repository gives me a list of the changes I made, B. The smaller B is, the better the language is. Averaged over the space of real & possible requirements, that measure tells me how "domain specific" the language is.
If it takes N code changes to implement 1 requirement, and you sometimes make mistakes, then the number of bugs you introduce is roughly proportional to N. In the limit where N = 1, it is almost impossible to introduce a bug without trying to.
Note that this is a direct challenge to the "code bloat" we see nowadays.
ADDED: In response to your request for an example, see differential execution. I won't say it can be understood quickly, but it does significantly reduce UI code.
It is always "feasible", to use the word in your (original) question, but it is not very often useful and very rarely optimal given the abundance of well-supported and mature languages and frameworks that exist.
It is an interesting intellectual challenge, however.
Only if your team's core business is programming languages.
I've worked on a programming language that was created in a financial company.
Clearly, for the architect himself this was a great challenge and improved his own skills.
Inevitably, the language couldn't grow or improve at anywhere near the rate that something like C# or Java could - they have teams dedicated to doing that.
The language soon stagnated as no one new wanted to take on the task of improving someone else's pet project.
The original architect left. The language withered and died after 10 years.
Those 10 years were hell for anyone who had the misfortune to work on a dead-end language.
So go ahead, create your own language but please don't ask anyone else to actually use it. Please don't expect anyone else to thank you for it.
Designing languages can be fun. But you don't have to ristrict yourself to programming languages.
If I build a moderately complex application, I like to add a kind of macro/scripting language to make it easier to execute complex repetitive tasks. Most users won't use this functionality, but the few that do use it are very grateful. Besides, i make sure it is valuable for the support people to help them fixing customer problems.
It's completely reasonable if done to extend your skills and to learn.
Other than that, if you have to ask the question, then it's not. If you're trying to figure out whether you can deal with a certain class of algorithm or a certain problem domain better than existing languages, first off you need to be an expert in the area you are addressing. You will know it is appropriate when your skills and experience tell you so.
And you could be wrong about that, too, but you'd need another expert to convince you of that (or to show you that you aren't the expert you think you are). A lively discussion that would be, not a simple Q-and-A as you'll find here.
Except for self-educational purposes, I would like to claim that there is today no need whatsoever to create your own language. In any circumstance. Ever. Regardless of what you want to do, there are boatloads of existing languages you can take/adapt to your needs.
It definitively depends on the situation. As nosklo said - If you have a good idea, a brand new concept or something like that I would highly recommend you to do that.
In general I would suggest to rely upon established technology.
But if you're interested in creating your own "language" you should check out: YACC & Lex
You can, just don't catch yourself in the anti-pattern "Recreating the square wheel".
Meaning you are recreating whats already been done, only poorer than the original(s).
Wouter was known for creating a new language for any new idea. You can draw inspiration from his work: Wouter's programming language page.
When to create your own language?
When you want to, as a large hobby project.
For a domain-specific language. These can be quite elaborate; look at what goes on in the Interactive Fiction (or text adventure) community by checking out the archive.
When your aims are very ambitious, and you think you can make a real advance, like Paul Graham's Arc project.
Also, in any sufficiently adaptable language (perhaps C++, definitely Common Lisp) in the process of developing low-level constructs.
When to avoid it like you would, I hope, avoid a cliche like avoiding it like the plague?
When it's to be the basis of continuing development for real projects. It will always wind up lagging seriously behind what's commercially available for cheap, and will cripple further development. I worked for a company with its own version of COBOL, and never want to work at another company that maintains its own language. We watched other versions of COBOL get better capabilities, and better tools, while we were stuck with the same problems. (I don't want to work with COBOL ever again, but that's another story.)
The situations in which you might create your own language don't fall into this. Hobby projects are not used for real development. Something like Arc will succeed (and get multiple implementations and further evolution and development) or fail (and nobody else will use it). A small domain-specific language is only part of a project, and since it's small it can be improved over time. A text adventure language is used to write individual games, and those games, in addition to being hobby projects, are almost never used for continuing development.
My perspective is that DSLs are generally a "weak idea", and it's more productive in the long run to use a standard language and build your domain-specific needs as a library of the "non-DSL".
However, it may turn out to be that your needs are custom enough that it's preferable to have a DSL(not just a slightly modified gcc or lisp implementation) for your company. Many companies use drop-ins of current languages that target what they are doing, without writing/maintaining their own language. E.g., I've heard that PHP has a nice drop-in; Lua is designed around being a drop-in, ModelView uses Python and AutoCAD has AutoLISP as its scripter.
There is nothing wrong with writing your own programming language if you can leverage existing tools. In todays world this would mean that you either define it in a syntax usable to an existing language (like Java or C#) or write a small transformation system (macro expander) that generate code in an existing language.
Going all the way to machine code is reinventing a LOT of wheels...
A very good reason for a DSL is to represent domain data in a succinct way. This allows domain experts to work with the data directly instead of having to go through others. The trick is then to have the resulting programs in an easy to process form.
Generally speaking the answer would be a big NO. Of the hundreds of languages out there there is usually one that will suit your problem.
However there are circumstance where it is a rational option to develop a new language:-
What language is good at is compositionality or putting the same components together in different ways.
If your domain problem just needs you to set a bunch of orthogonal switches, a language probably doesn't add much over forms, a graphic UI or a straight text-config. file. (I'm assuming here that a file full of key, value pairs isn't what you mean by a "language".)
OTOH, if your configuration is like real language eg. verbs and nouns can be put together in many different (and novel) combinations to any degree of complexity, then a language is going to become almost inevitable, because the combinatorial explosion of trying to specify what you want by any other method overwhelms.
Learning exercises aside, it is reasonable to create your own programming language only when you understand other languages, your specific problem domain, and the way that existing languages address that problem domain and this understanding is thorough enough that you know a new language is a reasonable solution without needing to ask the question.
The last time I set out to do that on a hobby project I started specifying what I wanted the syntax to look like and realized partway through I was reinventing prolog. Other languages that might be a good fit when you think you need to invent a language are lisp, lua, or something like Haskell. Basically, all those languages you ignored in college because you thought they would never be useful.
One reason is for educational purposes, as already stated. But there are more. For example, there are many research languages like Sing# on the Singularity operating system and BitC on Coyotos that have been designed because the existing languages don't offer the required features (for instance verification on a language level).
Tom Van Cutsem recently wrote an essay answer to this very question:
http://soft.vub.ac.be/~tvcutsem/whypls.html
Bullet summary (from that page):
Probably never.
Lua is the best choice you can get if you want to embed the language in basically any other language.
Small Domain Specifc Languages are currently in, and it make sense in some applications.
Other then that, reasons are mainly academical.
Creating a language when its not needed, is really a bad thing to do due to the complexity involved in developing and maintaing it. I have seen many projects that introduce some kind of scripting language specific only to that program, and it was the thing that was slowing down development of the base thing by huge amount. Good examples are for instance automation languages like Phantom, AutoHotKey, AutoIt. Those tools would be IMO much better if they used some well known emeding langauge like Lua.
Your 'edit' seems to be a substantially different question ("when should I build a DSL?" rather than the original question which people understood as "when should I build a new, general purpose programming language"). It seems people have thoroughly answered the 'original' question but there are few answers giving specific criteria for when to use a DSL. So I propose a checklist :
If all of these are true then a DSL may be appropriate.
Are there types of killer applications, classes of algorithmic problems, etc., where it is better, in the long run, to create my own language?
It depends.
Let's take our brain. It seems to be such a complex mess that we encounter borders with ANY programming language (at least now). So maybe in order to actually virtualize our brain we need other approaches and therefor other semantics and syntaxes.
Generally speaking, there are yet such complex topics which might lead to other strategies which would also include a 'better' language for a given scenario.