Let's say I want to develop an image editing program similar to Photoshop. Is it likely that certain interface patterns and approaches are patented and I can get sued if I use them in my program as well?
Do I need to take active care to avoid doing things the same way as competing software? How will I manage to make features still seem familiar to people switching from existing programs?
And how much of a difference does it make if it's non profit?
The final answer is "it depends." And whatever that answer is, it will be tempered by the attorney you need to consult before taking a product live. Laws will vary by jurisdiction, and no one here is qualified / prepared to offer you legal advice.
First, the easy part - the fact that it would be for a non-profit has almost no bearing in this matter. "Almost" because in some cases the amount of damages is proportional to the harm done against the original owner of the product. A non-profit's use or sale of the product may generate less damages because you could argue the original product would never have been used. At this level of discussion, that's equivalent to splitting hairs.
In general, you need to focus on the twins of
Innovation means "did you invent it" whereas Derivation means "did you copy it."
Wholesale copying of another program is going to get tripped up by patents, copyright, or both.
Example 1:
You make a straight copy of Photoshop but you write all the rendering algorithms yourself because you don't know what PS did. In this example, you would be infringing upon copyright but not necessarily patents held by PS. End result, your product is not in the clear.
Example 2:
You know how PS handles its rendering algorithms because you have a book "PS Algorithms in 24 Hours" that was written by that team and details how they did. It also has an explicit disclaimer that the algorithms are protected under still active patents and that you may only use the book for educational purposes. You create a brand new interface, but you blatantly copy the algorithms from your book. In this example, you will have infringed upon the PS patents and possibly some copyright.
Example 3:
Same as Example 2, but you copy the interface too. Now you definitely have infringed both their copyrights and patents.
Example 4:
You create a program that's like photoshop, but also has elements of World of War and salesforce blended into it. So you've created an online site where people consult to perform image editing and it's a massively addicting game. And you rely upon public or licensed sources for your rendering algorithms. In this final example, you are likely free and clear. While you have used other applications for inspiration, you haven't directly copied them.
So yes, you can use existing interface patterns within your software, but you need to know what is commonly used across applications (which won't be copyrighted) versus aspects that are particular to just that one application (which likely are copyrighted and maybe patented). How much of an idea you can "borrow" is a fairly subjective question and will be affected by a number of variables.
For additional information, see the links that Yannis provided in his comment.
1) You have almost no chance of threatening the dominance of Adobe's Photoshop program in the industry. As long as they're not threatened, the odds of them doing anything to you are slim. Maybe.
2) If by some miracle and decades of hard work you do manage to threaten their market share, they will almost certainly obliterate your project, regardless of how much or how little work you did the insulate yourself legally.
