How large non-OO code bases are managed?

Question

I always see abstraction is a very useful feature the OO provides for managing the code-base. But how are large non-OO code bases are managed? Or do those just become a "Big Ball of Mud" eventually?

Update:
It seemed everyone is thinking 'abstraction' is just modularization or data-hiding. But IMHO, it also means the use of 'Abstract Classes' or 'Interfaces' which is a must for dependency-injection and thus testing. How non-OO code bases manage this? And also, other than abstraction, the encapsulation also helps a lot to manage large code bases as it define and restrict the relation between data and functions.

With C, it is very much possible to write pseudo-OO code. I don't know much about other non-OO languages. So, is it THE way to manage large C code bases?

Answer 1

You seem to think that OOP is the only means of achieving abstraction.

While OOP is certainly very good at doing that, it’s by no means the only way. Large projects can also be kept manageable by uncompromising modularization (just look at Perl or Python, both of which have excelled at that, and so do functional languages like ML and Haskell), and by using mechanisms such as templates (in C++).

Answer 2

Modules, (external/internal) functions, subroutines ...

as Konrad said, OOP is not the only way to manage large code bases. As a matter of fact, a rather lot of software was written before it (before C++*).

Answer 3

The modularity principle is not confined to object-oriented languages.

Answer 4

Realistically either infrequent changes (think Social Security retirement calculations) and/or deeply ingrained knowledge because the people maintaining such as system have been doing so for awhile (cynical take is job security).

Better solutions are repeatable validation, by which I mean automated test (e.g. unit testing) and human testing that follows proscribed steps (e.g. regression testing) "as opposed to click around and see what breaks".

To start moving toward some sort of automated testing with an existing codebase I recommend reading Michael Feather's Working Effectively with Legacy Code, which details approaches for bringing existing codebases until some sort of repeatable testing framework OO or not. This leads to the sort of ideas others have answered with such as modularization, but the book describes the right approach to doing so while not breaking things.

Answer 5

Though dependency injection based on interfaces or abstract classes is a very nice way of doing testing, it isn't necessary. Don't forget that almost any language has either a function pointer or an eval, which can do anything you can do with an interface or abstract class (the problem is that they can do more, including many bad things, and that they don't in themselves provide metadata). Such a program can actually achieve dependency injection with these mechanisms.

I have found being rigorous with metadata to be very helpful. In OO languages the relationships between bits of code are defined (to a degree) by class structure, in a way standardized enough to have things like a reflection API. In procedural languages it can be helpful to invent those yourself.

I've also found code generation is much more helpful in a procedural language (compared to an object oriented language). This guarantees meta-data is in synch with the code (since it is used to generate it) and gives you something a bit like aspect oriented programming's cut points - a place were you can inject code when you need it. Sometimes it's the only way to do DRY programming in such an environment that I can figure out.

Answer 6

Actually, as you recently have discovered, first order functions are all you need for dependency inversion.

C supports first order functions and even closures to some extent. And C macros are a powerful feature for generic programming, if handled with necessary care.

It's all there. SGLIB is quite a good example on how C can be used to write highly reusable code. And I believe there's a lot more out there.

Answer 7

Even without abstraction most programs are broken up into sections of some sort. Those sections usually relate to specific tasks or activities and you work on those the same way you would work on the most specific bits of the abstracted programs.

In small to medium sized projects this is actually easier to do that with a purist OO implementation sometimes.

Answer 8

Abstraction, abstract classes, dependency injection, encapsulation, interfaces and so on, are not the only way of controlling large code bases; this is just and object-oriented way.

The main secret is to avoid to think OOP when coding non-OOP.

Modularity is the key in non-OO languages. In C this is achieved just as David Thornley just mentioned in a comment:

The interface goes in the .h file, publicly available functions in the .c file, and private variables and functions get the static access modifier attached.

Answer 9

One way of managing code is to decompose it into the following types of code, along the lines of the MVC (model-view-controller) architecture.

• Input handlers - This code deals with input devices such as mouse, keyboard, network port, or higher-level abstractions such as system events.
• Output handlers - This code deals with using data to manipulate external devices such as monitors, lights, network ports, etc.
• Models - This code deals with declaring the structure of your persistent data, rules for validating persistent data, and saving persistent data to disk (or other persistent data device).
• Views - This code deals with formatting data to meet the requirements of various viewing methods such as web browsers (HTML/CSS), GUI, command line, communication protocol data formats (e.g. JSON, XML, ASN.1, etc).
• Algorithms - This code repeatably transforms an input data set to an output data set as fast as possible.
• Controllers - This code takes inputs via the input handlers, parses the inputs using algorithms, and then transforms the data with other algorithms by optionally combining inputs with persistent data or just transforming the inputs, and then optionally saving the transformed data in persistent via the model software, and optionally transforming the data via the view software to render onto an output device.

This method of code organization works well for software written in any OO or non-OO language because common design patterns are often common to each of the areas. Also, these kinds of code boundaries are often the most loosely coupled, except algorithms because they link together the data formats from the inputs to the model and then to the outputs.

System evolutions often take the form of having your software handle more kinds of inputs, or more kinds of outputs, but the models and views are the same and the controllers behave very similarly. Or a system may over time need to support more and more different kinds of outputs even though the inputs, models, algorithms are the same, and the controllers and views are similar. Or a system may be augmented to add new models and algorithms for the same set of inputs, similar outputs, and similar views.

One way OO programming makes code organization hard is because some classes are deeply tied to the persistent data structures, and some are not. If the persistent data structures are intimately related with things such as cascading 1:N relationships or m:n relationships, it is very hard to decide class boundaries until you have coded a significant and meaningful part of your system before you know you got it right. Any class tied to the persistent data structures will be hard to evolve when the schema of the persistent data changes. Classes which handle algorithms, formatting, and parsing are less likely to be vulnerable to changes in the schema of the persistent data structures. Using an MVC kind of code organization better isolates the messiest code changes to the model code.

Answer 10

When working in languages which lack inbuilt structure and organisation features (e.g. if it doesn't have namespaces, packages, assemblies etc...) or where these are insufficient to keep a codebase of that size under control, the natural response is to develop our own strategies to organise the code.

This organisation strategy probably includes standards relating to where different files should be kept, things that need to happen before/after certain types of operations, and naming conventions and other coding standards, as well as a lot of "this is how it is set up - don't mess with it!" type comments - which are valid so long as they explain why!

Because the strategy is most likely going to end up being tailored to the specific needs of the project (people, technologies, environment etc...) it is hard to give a one-size-fits-all solution to managing large code bases.

Therefore I believe the best advice is to embrace the project-specific strategy, and make managing it a key priority: document the structure, why it is that way, the processes for making changes, audit it to make sure it is being adhered to, and crucially: change it when it needs to change.

We are mostly familiar with refactoring classes and methods, but with a large codebase in such a language it is the organising strategy itself (complete with documentation) that needs to be refactored as and when necessary.

The reasoning is the same as for refactoring: you will develop a mental block towards working on small parts of the system if you feel that the overall organisation of it is a mess, and will eventually allow it to deteriorate (at least that's my take on it).

The caveats are also the same: use regression testing, make sure you can easily revert if the refactoring goes wrong, and design so to facilitate refactoring in the first place (or you just won't do it!).

I agree that is is much trickier than refactoring direct code, and it is harder to validate/conceal the time from managers/clients who might not understand why it needs to be done, but these are also the types of project most prone to software rot caused by inflexible top-level designs...

Answer 11

If you are asking about management of a large code-base, you are asking for how to keep your code base well structured on a relatively coarse level (libraries / modules / building of subsystems / using namespaces / having the right docs at the right places etc.). OO principles, especially 'abstract classes' or 'interfaces', are principles for keeping your code clean internally, on a very detailed level. Thus, the techniques for keeping a large code base manageable don't differ for OO or non OO-code.

Answer 12

How it's handled is that you find out the borders of the elements you use. For example, the following elements in C++ has a clear border and any dependencies outside the border must be carefully thought out:

1. free function
2. member function
3. class
4. object
5. interface
6. expression
7. constructor call / creating objects
8. function call
9. template parameter type

Combining these elements and regognizing their borders, you can create almost any programming style you want within c++.

Example of this is for a function would be to regognize that it's bad to call other functions from a function, because it causes dependency, instead, you should only call member functions of the parameters of the original function.

Answer 13

The biggest technical challenge is the namespace problem. Partial linking can be used to work around this. The better approach is to design using coding standards. Otherwise all the symbols become a mess.

Answer 14

Emacs is a good example of this:

Emacs Lisp tests use skip-unless and let-bind to do feature detection and test fixtures:

Sometimes, it doesn't make sense to run a test due to missing preconditions. A required Emacs feature might not be compiled in, the function to be tested could call an external binary which might not be available on the test machine, you name it. In this case, the macro skip-unless could be used to skip the test:

 (ert-deftest test-dbus ()
   "A test that checks D-BUS functionality."
   (skip-unless (featurep 'dbusbind))
   ...)

The outcome of running a test should not depend on the current state of the environment, and each test should leave its environment in the same state it found it in. In particular, a test should not depend on any Emacs customization variables or hooks, and if it has to make any changes to Emacs's state or state external to Emacs (such as the file system), it should undo these changes before it returns, regardless of whether it passed or failed.

Tests should not depend on the environment because any such dependencies can make the test brittle or lead to failures that occur only under certain circumstances and are hard to reproduce. Of course, the code under test may have settings that affect its behavior. In that case, it is best to make the test let-bind all such setting variables to set up a specific configuration for the duration of the test. The test can also set up a number of different configurations and run the code under test with each.

As is SQLite. Here is it's design:

1. sqlite3_open() → Open a connection to a new or existing SQLite database. The constructor for sqlite3.

2. sqlite3 → The database connection object. Created by sqlite3_open() and destroyed by sqlite3_close().

3. sqlite3_stmt → The prepared statement object. Created by sqlite3_prepare() and destroyed by sqlite3_finalize().

4. sqlite3_prepare() → Compile SQL text into byte-code that will do the work of querying or updating the database. The constructor for sqlite3_stmt.

5. sqlite3_bind() → Store application data into parameters of the original SQL.

6. sqlite3_step() → Advance an sqlite3_stmt to the next result row or to completion.

7. sqlite3_column() → Column values in the current result row for an sqlite3_stmt.

8. sqlite3_finalize() → Destructor for sqlite3_stmt.

9. sqlite3_exec() → A wrapper function that does sqlite3_prepare(), sqlite3_step(), sqlite3_column(), and sqlite3_finalize() for a string of one or more SQL statements.

10. sqlite3_close() → Destructor for sqlite3.

The Tokenizer, Parser, and Code Generator components are used to process SQL statements and convert them into executable programs in a virtual machine language or byte code. Roughly speaking, these top three layers implement sqlite3_prepare_v2(). The byte code generated by the top three layers is a prepared statement. The Virtual Machine module is responsible for running the SQL statement byte code. The B-Tree module organizes a database file into multiple key/value stores with ordered keys and logarithmic performance. The Pager module is responsible for loading pages of the database file into memory, for implementing and controlling transactions, and for creating and maintaining the journal files that prevent database corruption following a crash or power failure. The OS Interface is a thin abstraction that provides a common set of routines for adapting SQLite to run on different operating systems. Roughly speaking, the bottom four layers implement sqlite3_step().

A virtual table is an object that is registered with an open SQLite database connection. From the perspective of an SQL statement, the virtual table object looks like any other table or view. But behind the scenes, queries and updates on a virtual table invoke callback methods of the virtual table object instead of reading and writing on the database file.

A virtual table might represent an in-memory data structures. Or it might represent a view of data on disk that is not in the SQLite format. Or the application might compute the content of the virtual table on demand.

Here are some existing and postulated uses for virtual tables:

A full-text search interface
Spatial indices using R-Trees
Introspect the disk content of an SQLite database file (the dbstat virtual table)
Read and/or write the content of a comma-separated value (CSV) file
Access the filesystem of the host computer as if it were a database table
Enabling SQL manipulation of data in statistics packages like R

SQLite uses of a variety of testing techniques including:

Three independently developed test harnesses
100% branch test coverage in an as-deployed configuration
Millions and millions of test cases
Out-of-memory tests
I/O error tests
Crash and power loss tests
Fuzz tests
Boundary value tests
Disabled optimization tests
Regression tests
Malformed database tests
Extensive use of assert() and run-time checks
Valgrind analysis
Undefined behavior checks
Checklists

References

• Conceptual Views of Emacs' Architecture (pdf)

• The SQLite OS Interface or "VFS"

• The Virtual Table Mechanism Of SQLite

• An Introduction To The SQLite C/C++ Interface

• Emacs-Elisp-Programming · GitHub

• Tests and Their Environment - Emacs Lisp Regression Testing

• How SQLite Is Tested