Since you have a C background it is best to first look at lisp and C macros, and see what properties they share and which they are not sharing.
In C, macros are compile-time-rules for almost textual replacement of code parts by the preprocessor. C macros have a different syntax than the C programming language, they are executed by the preprocessor and emit C code that is compiled by the C compiler. So it is a preceding step to program compilation.
Lisp is a little different here. In Lisp, every code is a tree-like (directed, acyclic graph) datastructure that is written in S-Expressions (think of S-Expressions like compact XML).
((10) "a is ten")
This datastructure consists of nested list objects that offer a simple API for accessing its elements. The great thing about lisp is, that the same data structures used in a lisp program for storing lists is also used during the compilation of the Lisp code. Therefore one can write small "compiler" extensions in Lisp code that transform branches of the abstract syntax tree into other representations. Again: You use the same language that you use for programming an algorithm to transform the code.
For a moment, I would downplay the differences between a procedure and a macro call. The first real difference between a procedure and a macro is about the time when they are executed not about the nature of their abstraction in the first place.
The overall question is: How can we keep a codebase from becoming an obscure meta-langauge that is not accessible to colleagues?
Keeping programs readable
Readability of programs is a complex topic.
First of all it comes down to the experience and expectancy of the reader. Code that looks foreign is harder to understand for a programmer than code he/she is familiar with.
Then there is the problem of code-scope. When reading code we are quite local and cannot look at all parts of the source at the same time. Therefore code that is compact should be easier to read than related code that is distributed widely over the source file.
However, good abstraction helps code readability since it groups operations into logical chunks. A function composes these abstracted operations is more readable than an equivalent function that concretely implements everything on its own.
Good abstraction means separation of concerns. A function that does several unrelated things at a time is harder to understand since you have to disentangle the functionalities within the procedure.
Consider a function
write_file_and_show_results might be a bad function. Especially if it does not compose one
write_file and one
show_results function but directly opens the file, starts writing to it, shows results, writes some more....
The virtue of lisp is, that it is really good for implementing these good abstractions. This is often compared to "writing an own language for the problem". Yet it is hardly different than programming in other languages, Lisp just has more drastic means for extending syntax.
Commonly if people argue against macro use or abstraction, they point out, that a direct, concrete implementation is more readable than one that involves a few abstractions. In practice, it mostly pays off to introduce some abstraction that compacts the rest of the code. It saves LoC, maintainability is increased since you only have to perform changes once than in several parts of the problem
Architectural design: Bottom up
Paul Graham has written a nice essay on how to work when writing Lisp code. http://www.paulgraham.com/progbot.html
There is one huge problem associated with macros: Hygiene.
Take the following example of a C program to illustrate it.
#define max_defect(a,b) ((a) > (b) ? (a) : (b))
The issue at hand is, that an invocation of
max_defect(i++, 3) will increase i twice. Typically this problem is solved by introducing another binding of
b (something like
_a) and evaluate the parameters to it. Then, the bound results are used to perform the actual code.
However, what happens if you then use
_a as a parameter to the maximum-determining function? You have a name conflict at hand. In Lisp this problem has been solved by introducing gensyms. Generated symbols that are unique in the program.
However since this still leaves the possibility for errors, other ways to define macros have been developed, so called hygienic macros. Scheme does offer hygienic macros. They always guarantee correct lexical scoping.
Developing a common language
When presenting the idea of a "modifiable" programming language, people sometimes fear, that this might lead to a programmer building up "his/her own world" that is inaccessible to the outsider. While this is certainly possible, as long as you are not working in total isolation it will not be the immediate consequence. A lot of macros are already existing that encapsulate a certain abstraction, their naming schemes can be adapted, and there are also surprisingly readable macro constructs, such as the
Generally the lisp communities have slightly different traditions in naming things than other programming communities.
make-whatever for creating an object,
with-RESOURCE for RAII like resource management, etc. Getting used to this is not an issue when reading established lisp codes and will enable you to write consisten lisp code.
Disclaimer: I have mostly worked with Scheme and Clojure, not very much with Common Lisp. Clojure is probably the most readable dialect for people who have some experience with Python and/or ruby.