SQL as a means of avoiding "releases"

Question

The system I'm working on started as a small training project within the company I work for but quickly caught interest from management as a means to help to standardize the existing excel-heavy workflows. I'm one of the lead developers on the project, despite having no lead experience prior and only being in my third year as a software engineer (NOTE: this is my first professional project).

I recently used the strategy pattern to write a validator to handle different possible validation tasks for a given object (working in Java with Spring Boot currently). My goal was to make it easy to add new validations without risking affecting other parts of the code and make it easy for other members to extend it. I thought this was a good approach, until I told my manager. He said he doesn't want new logic to be written each time there are new validation paths because he doesn't want to do an official release each time we have logic changes (we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point). He wants everything to be configurable to the point that no additional logic is needed or housed in the RDBMS so that we can just insert new records into some table or modify a stored procedure/function and call it a day. He also claims this will reduce complexity for new members.

Personally, this seems like a poor design decision. Moving everything to SQL is going to add more complexity over time, even if in the short-term things work out. Also, SQL - while powerful when it comes to data querying/organizing - doesn't seem all that great when you have to write tons of cases to account for different validation rules, etc.

Are my concerns with my manager's approach justified, or is he correct?

EDIT 2023-04-05:

I wanted to provide additional context to the validation process mentioned in the above post. The part of the application that handles the validation is new. Right now, most of the company's core data housing and management is in countless Excel sheets. Upper management has expressed a desire to move away from Excel, particularly to streamline some of the more tedious internal processes. As a part of this, one of the departments is attempting to standardize the available options in their existing excel-heavy process (previously, all of the options were free-form text fields). At the moment, not all options have been standardized, but each needs a series of validation checks if chosen. So far, they've been straightforward and amounted to various equivalence checks. However, I don't know if that precedent will hold over time because neither I nor my manager are familiar with this department and all of its processes.

Answer 1

Problem scenarios like these tend to suffer from a very confused train of thought that mixes ideas. In an attempt to answer this, I am going to have to systematically unravel all of the discrete elements here.

I'm one of the lead developers on the project, despite having no lead experience prior.

That's not necessarily a problem. You can be a lead because you are a generally experienced developer, but you can also be a lead because you know the business requirements and are able to articulate those needs well. I suspect you are in the latter category here.

I recently used the strategy pattern to write a validator to handle different possible validation tasks for a given object (working in Java with Spring Boot currently). My goal was to make it easy to add new validations without risking affecting other parts of the code and make it easy for other members to extend it.

There's not enough info for me to judge your implementation but it seems you have the right idea and goal.

He said he doesn't want new logic to be written each time there are new validation paths

Is your manager aware he is managing a software development team, and not a software development avoidance team? Joking aside, there could be some merit to his comment depending on what a "new validation path" entails.

For example, if you have validation that checks if a field contains any swear words, I do agree with your manager that you should be able to update your list of swear words without needing redevelopment and redeployment.

However, if the new validation path is algorithmically and logically different in a way that is more than just a value change for an already existing validation; then redevelopment and redeployment is very much warranted.

The rest of this answer assumes the previous paragraph is the case. If this is a matter of configurable values, your manager is correct that this should be configurable without needing development and deployment.

He wants everything to be configurable to the point that no additional logic is needed or housed in the RDBMS so that we can just insert new records into some table or modify a stored procedure/function and call it a day.

Your manager would effectively be arguing that you should be writing a Turing-complete validation rule engine so that new rules can be designed without making changes to the engine, which is a massively different remit compared to what you have described the current project to be like.
Would it be cool if you had such an engine available? Sure thing. Is your manager willing to commit the time and money needed to design such an engine? Doubt it.

because he doesn't want to do an official release each time we have logic changes (we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point)

This is the core issue. It's not a development issue, it's a management issue. The deployment process is too cumbersome, and rather than either sucking it up or improving the process, your manager is telling you to halt your development.

This is the equivalent of an inefficient hospital refusing to improve their ability to treat sick people, and instead tell the ambulance drivers that they should not be bringing in new people so frequently.

Not only is it an idiotic solution to the presented problem, it is the antithesis of what your manager should be achieving. They should be empowering their department to develop software bigger, better and faster, and instead they're undercutting the development of software so they don't have to be bothered with having to manage all this development.

He also claims this will reduce complexity for new members.

I refer back to my Turing-complete validation engine comment as a counter to their claim.

I suspect, based on my own personal history, that your manager might be more familiar with SQL (and Excel) than they are with programming languages and general software development methodologies.

In that light, SQL might be easier for them to understand. I've seen countless managers make these kinds of blanket statement subconsciously based on their own knowledge. It's usually not willful but it does still miss the mark.

Personally, this seems like a poor design decision. Moving everything to SQL is going to add more complexity over time, even if in the short-term things work out. Also, SQL, while powerful when it comes to data querying/organizing, doesn't seem all that great when you have to write tons of cases to account for different validation rules, etc. Am I crazy?

You are the only sane man (irrespective of your gender, I'm just referring to a specific idiom). If you haven't already, I feel like you're going to enjoy reading Dilbert cartoons because of its similarities to your workplace as you've described it.

Answer 2

Monkey Patching

It seems that the requirement from your boss is to have the ability to Monkey Patch the system while bypassing all other software development tools, processes and methodologies.

Monkey patching can be a workable strategy in any system when its scope is limited to things with a low business risk and business reach/impact.

There are clear advantages in some circumstances when the boss can just walk over to a developer's desk, let them know what needs doing then watch the magic happen right in front of their eyes over a cup of coffee; such an environment tends to be optimal for way-out-there crackpot innovation where any failure is easily contained and the risk to the business of the entire thing imploding on itself is something that everybody in the business understands and accepts.

Risks

If monkey patching is something which bypasses all development tooling and process, then it's important to understand everything which may potentially be sacrificed and the risks:

• Changes are not version controlled; no the history of earlier revisions, no 'revert' mechanism, no traceability of who changed something, why, when or how.
• Potential for vast differences and divergence between what developers see and what users see
• No way to prevent non-peer-reviewed changes from making their way into production.
• No static code analysis or vulnerability scanning prior to deployment.
• No automated testing prior to deployment
• No manual acceptance testing prior to deployment
• No sanity checks against potential impact to the stability of the system or integrity of the data
• Debugging and fault diagnosis happens in production.
• Faults will likely require developer tools and configuration to be deployed into production (Need to assess the potential impact of this - particularly on security and performance)
• Real, live, end-users are the QA testers - realistically they will often be the very first people to notice a new bug or when something breaks. (Are they OK with this?)
• No protection against accidental (or deliberate) misuse or damage of live data
• More humans given access to the live database only increases the potential surface for security threats to the database

Cowboy Coding

What your boss is proposing here sounds dangerously close to Cowboy Coding. It's entirely possible that your boss hasn't realised this yet, in which case I would pose the following questions about how the solution will work in practice:

• How will QA/testing work for changes going into production?
• How will developers' own local environments keep in-sync with production changes?
• Will there be test environments and test data?
• What provisions will exist for diagnosing and debugging faults?
• Will changes be source controlled?
• Will changes be properly tracked and versioned?
• Will there be a process to promote changes between environments?
• How would changes be covered by static analysis and automated tests?
• How will peer code reviews work?
• What level of risk and potential for bugs is acceptable for end-users?
• How will the data be protected against the risk of getting broken?
• Has information security been considered?

Answer 3

That code is in stored procedures inside a SQL-based database does not excuse it from version control, version management, testing, and all the usual dance steps of software development.

For validation, as for many other workflow processes, there is range of possible solutions: on one end, some values in a table (as Flater's answer pointed out); on the other end, code. Somewhere along the line there are complex embodiments of code in data.

Perhaps you need an ERP system? Those are plagued with code-as-data. And implementing one in a company is no trivial matter. But they already have solutions for many many company problems.

Answer 4

we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point

While you added it as a passing comment, this is the most important part of your question. Continuous integration is a fundamental part of software development today - so fundamental that in my opinion, a company that lacks it cannot reasonably be called a software development company. And if they don't have CI, they almost certainly have other fundamental problems - likely they don't use a reliable systems development lifecycle, which implies poor code quality and insufficient oversight. Essentially, a "cowboy coder" environment.

As its name suggests, such an environment breeds cowboys, so-called because they're "quick on the draw" to fix problems that arise in their projects. This makes them extremely popular with management, who perceive this reactive nature as a positive and reward the cowboys. The actual issue is that a lack of an SDLC is what is causing the problems!

Thus, any attempt to introduce an SDLC into that environment will receive massive pushback from the cowboys, because they understand that making the development process more predictable and transparent threatens their good reputation with management. Especially since most cowboys are also middling to poor developers, an SDLC represents a massive threat to them - and they'll use their political clout with management to resist it.

As for management, you need to remember that their job first and foremost is to avoid risk. But the fact that SDLC-less development is so fraught with bugs, SNAFUs, and broken releases means it has a lot of risk, so their inherent risk aversity means they come to see any possible new risk in development - like, say, using the strategy pattern vs SQL - as unacceptable.

In short, if you don't have CI, you almost certainly don't have other fundamentals like an SDLC - and if you don't have these things, you as an ordinary developer are probably not going to be able to implement them, due to the aforementioned politics. You're also unlikely to get your "new-fangled" ideas approved by management.

If the above sounds like your company, you need to get out. Seriously. The chance that you as a developer, are going to be able to change things for the better, are as close to zero as they can be without actually being zero. Wasting your time and effort in an attempt to fix the company's problems simply isn't worth it - take that advice from someone who has tried, and failed.

Your alternative is to simply shut up and do what you're told by management, but your post indicates to me that you're not developing software because it pays the bills, but rather because you are indeed a software engineer in the true sense of the term. Your "only" three years of experience is irrelevant, what matters is your desire to build software well - and that desire shines through strongly in your writing. If you are a software engineer, then forcing yourself to do things the wrong way will eventually break you. Again, I speak from experience.

Looking for a good new job is difficult and scary. Staying in a bad one is ultimately much, much worse - both professionally and personally.

Answer 5

You are both right, on different levels.

Your boss is right that given the constraints that full releases are slow and time-consuming, it makes sense to move business rules into the database, which can be updated on the fly. (Assuming you do need to be able to quickly add or change business rules.)

But you are correct that it complicates the system to move business rules into stored procedures if they are better expressed in regular code.

So the question is if the current deployment procedure is set in stone or if you can push for more CI/CD, making code deployments less expensive. This is probably a question of organizational culture and may not be something you or even your boss can change.

You probably don't need to go full Kubernetes. It is easier to introduce changes gradually, e.g. by replacing some of the manual deployment steps with batch scripts.

Alternatively, if you can't introduce CI/CD, you might consider using some kind of scripting language for some of the business logic, where the scripts can be updated on the fly.

Answer 6

because he doesn't want to do an official release each time we have logic changes (we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point)

This sounds like you are being enlisted in an interdepartmental politics war to help fight via technical means.

Probably, The intent is to lever out, sabotage, or undermine a change management process which is broken or perceived as broken or simply counter to your department's interest.

Don't consider technical pros and cons as first priority. Understand the underlying politics first, and who on whose side has which kind of clout. You might be being used, or you might have been enlisted to fight on the winner's side.

In case the change management process is due to regulatory requirements, stop and really find out what is going on before you shoot yourself or the whole team or company in the foot with a cannon.