Do I have the wrong idea about software engineering? [closed]

Question

I have a question which has been raised by my latest job (rather intern).

Just to put things into context - I'm 21 and I've finished my 2nd year of university before that I've had around 2 years of experience doing sys admin/QA jobs and basically I can say that I've seen how different IT sectors operated. Flash forward to present times and here's me landing an interning job at one of the premier research institution in the UK.

What I have to do is create some internal tools using a mix of technologies - mainly AWS/Java/Bash - you get the picture. Everything is OK, I'm doing my job BUT I'm not happy. Why is that - because I'm expected to work in an ad-hoc matter. That is create things quickly, without spending time on designing. My manager explicitly said that it was expected to "rush" through problems as they arise and we essentially. As a consequence it turned out that things had to be re-done and re-engineer and they are still not perfect. As far as testing is concerned - keep it to a minimum, as long as it looks working then it is OK.

Am I at fault to disagree with this way of conducting work? Is it wrong to want to think over the system as a whole, then focus on different components and see how they might inter-operate, to zero in on different "key points" which might turn out to be problematic in the future? Is it a crime to want to do a good job and not a "quick job"? Is it a mistake or wrong attitude to want to research the data structures applicable to a problem so that you can choose the best depending on a particular problem set? To the best of my understanding the "Engineering" bit in "Software Engineering" has got to do exactly with this - research your problem domain and come up with a informed solution then refine as necessary?

I've been to an interview at an Arm's office in the UK and they showed me their SCRUM room and it looked they had pretty good idea as to how to manage their project - they had a backlog, they had metrics as to how long each issue might take to resolve - the usual things for SCRUM - completely different than the way things are run "here"

Have I built a wrong idea about the software industry in general? I'd like to hear your input on that. I mean I "entered" software development purely because I want to create things - plain and simple, but I want to create quality things. I want to see my software used in various scenarios, I want to see it bullet proof - isn't that the driving force for all software engineers? I think everyone can be a programmer/coder by just learning the syntax but for me where the real fun begins is when you actually have to come up with a design which is doable in real world.

I used to do my university assignments by just looking at them and directly start coding and could easily get marks above 75% and never really appreciated the "software development lifecycle" module. But now when I saw in the real world how bad it is to work without any formal process and the frustration that is inherent in situation where you don't know if the requirements are going to change tomorrow (oh, did I say that we don't have clearly defined requirement analysis?)

I really like to believe I just landed a position where some people just needed a code monkey to do their dirty work and this is not the case how the software world operates at large.

Answer 1

Making software re-usable and bullet proof is not the driving force of software engineering. Engineering is about solving real world problems optimally within real world constraints. Most engineers would prefer to work on a Ferrari - but a station wagon needs just as much engineering, and the reason a station wagon doesn't perform as well (in some ways) is due to more difficult design constraints, not due to worse engineering.

When you say you want to do a "good job" rather than a "quick job", most engineers do, but sometimes part of what defines good is how quick it is to completion. So it's not right to think of "good" and "quick" as opposing choices. Or to think you are doing a bad job, or are just a "code monkey", for doing the best possible job in the time available.

Of course, it's quite possible that the process isn't optimal, and would do better with a little more design up front. The acid test would be, is the current way of doing things creating more problems than it solves for the users, or is it just bugging the developers who have to work that way? If it's actually causing problems for the users, part of your job is to try to demonstrate that this is the case, and try to win people over to a slightly more controlled process.

Answer 2

Actually, this bothers me. You're in a profession where you develop tools for research scientists, correct. However, you're told to make these programs quick and have them appear to work minimally. Surprise surprise. This is simply the researcher's typical approach to programming with the buck passed off to an actual programmer.

The main concern here being that lack of testing in particular can be ethically dubious if the tools have an important purpose. If one isn't sure the software has defects because one is restricted to minimal testing time, this means NO ONE is held liable to the working condition of the software, and Atlas shrugs.

Let's stop and think about this for a second though. What kind of tools are you developing? If you're developing software that models data, there's a big ethical dilemma here. In some situations, scientific research leads to decisions that affect a lot of people on a big scale.

Suppose for an instant, the controversial topic of man-made climate change. Let's say they placed the same standards on the modeling software that they use to come to the conclusions they have today. The topic has a big impact on how we approach correctly managing the environment, and international policy.

Is it not ethical to ensure that the modeling software doesn't have major problems with its predictions?

The whole problem isn't that greenhouse gases warm the earth. The problem is whether the net result of the feedback effects is an accelerating gain in temperature, which after breaking a threshold would no longer be reversible.

If said gain was occurring, the evidence of a net result would be marginal, probably within the err range.

So, slight miscalculations, even methodology involving data storage and retrieval on the back end, could result in either ignoring a serious environmental problem on one end of failure, or international policy that affects a lot of people (destroys jobs, destroys pensions, etc.) on the other.

So, yes, you are right. I don't care what the pace of research is... If researchers want to rely on software tools to manage data and perform calculations for them, they need to learn to wait on software done right. Otherwise this software becomes a vulnerability point in their theories that they aren't held accountable to, resulting in ethic misconduct.

Answer 3

You don't have the wrong idea about what software engineering is. However, you're missing a very important aspect of it: this is a service industry. Some of us get to work on a product for years, and go through design and then many iterations before it's in v1. Others have to produce something in 3 hours. It depends on who you're servicing and what the purpose is.

If you can produce an application in 3 hours (or days) that does what it's meant to do, why spend more time on it up to design up front? You're just wasting money. Wasting money is general no a Good Idea™.

Answer 4

As others have already said, a big part of what software engineering is about are "extrinsic constraints". eg. Time, budget, service, support, satisfying irrational idiotic demands, etc.

A lot of us (myself included) got into programming thinking it's all about the programming itself - coding beautiful and elegant pieces of software in a vacuum (or a relative vacuum at least). It rarely is. There might be some rare academic or R&D software jobs which come close to it, but for the most part, the overwhelming majority of software development work is nothing like that. Especially in the maintenance phase - which is typically 90%+ of a product's lifetime - and the bread and butter everyday routine of most permanent commercial software work.

For a long time, I had an internal conflict about this which often made me unhappy about my work (and it's part of what eventually lead to a burnout last year). I always felt that a job sucks if it isn't all about creating beautiful code and taking the time to do it propertly. But really, this is the reality - and some people actually thrive on a very service-oriented work flow. It's what makes them feel pragmatic and useful. Even if the actual "pure software engineering" aspects of a project get relatively rushed and sloppy.

Anyway, it's good that you're questioning this now. This is one of those things that they never really explain properly at school. And companies love to pretend that they follow very good engineering practices even if they don't. Hint: most don't.

All that said, things vary. Certain companies (mostly those for whom software is their core business, and those who work on highly safety-critical software such as medical gear) do follow a very strict engineering process. But overall, yes, I'll tell you point blank now that most commercial software work is relatively sloppy. There is usually some formal process, but adhering to it strictly almost always takes a back seat to reacting to client input and other commercial pressures. It's not really "sloppiness" per se, it's just plain pragmatic usefulness. The trick is to find your niche, and look at a role from the point of view from what service it provides, rather than how cool the "pure programming" aspect of it is.

EDIT: I think I might have sounded too one-sided in my initial assessment. I'd like to add that often there are also genuine problems with things being too sloppy, and lack of good process - to the point where it's driving the project into technical debt and is actually bad for the business. But seeing this comes with experience. The initial point still basically stands: most commercial software work today is not as rigidly engineering oriented as purists might like.

Answer 5

What an excellent question! Sometimes you can make something valuable by being fast. This is typically the case in a research lab where the faster we can learn what we don't know, the better off we will be. The software you produce exists only to answer questions. It is "throw away code". This is also the case with startups that don't know what the customers really want. Also, the first time you make something, it will be crappy. Read The Mythical Man-Month.

Sometimes you can make something valuable by being good. This is typically the case with shrink-wraped software like Adobe Photoshop. The research has already been done years ago and now the question is how to add the list of features that customers want in a way that doesn't introduce bugs. It is a matter of architecture, design, and testing, testing, testing. The code itself is what is valuable, not what you learn from it.

If you are not happy with research (making the first of something, learning new things that no one knew before) then give shrink-wrapped software a try. In fact, at your age you should try as many things as possible. Go take risks! You will learn a lot and you will be better off in the long run.

Answer 6

I think you've noticed very early in your career that doing things quickly, without a proper design or proper testing, tends to come back to bite you. You obviously don't like this and you have good reason not to. It's ridiculous to be expected to "rush through problems", especially if you have to revisit them later when the initial solutions are incorrect or incomplete. You can only provide solutions to problems if you understand them completely, and that takes time and careful planning.

My suggestion to you is to let your superiors know that this bothers you and to suggest to them a better approach to doing your work. If they do not agree and want you to continue to "rush" through your work, I would start looking for work elsewhere. There is no sense in doing things in a manner that is not up to your own standards, let alone a standard of software development quality the industry expects.

Answer 7

Here is my advice based on my experiences, I’m 20 and currently on placement working for major financial institution in the UK and had the same feelings you had a few months ago, what I noticed is that this maybe be due to the nature of the work you are doing.

What I mean by that is you said:

“What I have to do is create some internal tools using a mix of technologies - mainly AWS/Java/Bash”

I also had to create internal tools to help manage and automate certain processes and the fact is that in a fast paced environment “small” things are required to be implemented quickly. I didn’t have the luxury of applying most of the software engineering or algorithms and data structure principles I was taught in my 2nd year because a working version of the tool was required in a few weeks.I was very frustrated with this however it was not all bad as I did learn how to write better more readable code.

I had to be patient and I recently rotated to a new team that is working on a new iteration of in-house built system used by 10K+ users and I can assure you the software engineering aspect of it is taken very seriously. I have been told that I will gain exposure to the complete software life cycle from requirements capture/analysis all the way to implementation and testing. I believe I’m going to gain this experience because I’m not working on internal tools but that I’m working on a full scale system with a large user base.

What I recommend is that you be patient, finish creating the tools and make a very good job at it so that your supervisors gain more trust in you and assign you more challenging tasks that will require the use of software engineering principles. Gain extra knowledge from doing some extra reading and apply that knowledge to what you are currently doing, I remember ransacking the entire e-book library at the company just to further my knowledge muhahah, from all the books I read I felt that effective java was really good book that helped me a lot.

Just be patient, invest heavily in you own knowledge and apply that knowledge where possible. If you are doing a very good job someone will soon notice.

Answer 8

I'd agree that the way your current work operates is suboptimal, yes. However, if you want to say that it doesn't work at all, I'd disagree with you there as there are various results and the institution is still around.

My main question back to you is to what extent are you dealing with fires that require immediate solutions done quickly similar to giving a medical patient first aid versus requests that could be set up as projects and handled on a much different scale similar to the medical patient having to schedule tests and various procedures that aren't necessary to do immediately but in the near term.

Taking the time to do a job well depends a bit on the maturity of the organization along with how important is it for something to be done well versus a claim of being done.

The question on researching the data structures is how long are wanting to do this. If you want to take a decade to research a data structure that is quite different from wanting a couple of hours. While I can appreciate the desire to get the best answer, there is something to be said for diminishing returns after spending some time on a problem,e.g. could you spend hours finding a solution to FizzBuzz as you could try to solve it in various languages on various hardware to optimize just how fast could it run.

While it can be important to research it is also important to deliver something. If you don't deliver something, how good are you really? Duct Tape Programmer would be more of an example on the getting things done side here.

Scrum is a specific methodology that you possibly could try to adopt at your current workplace. Don't think Scrum is a silver bullet though. Under What Circumstances Can Projects Under Scrum And Agile Fail? would be a blog post on that subject that may be of interest.

My guess is that you aren't seeing how informal your current place's processes are and thinking the grass is immensely greener on the other side where there is a formal methodology. While it may be better there, some people may prefer what you have now with the massive freedom of being a cowboy.

Answer 9

I think your situation is still on the Real World scale towards less emphasis on the quality side. Your preference is on the other end of the real world. Specs change, get over it. Things need to get done.

Consider ways to identify these types of companies when you apply for your next job. Few places have a business model where they can afford to have developers analyze their designs forever (Even professors have to teach.). Clients rarely pay if your work doesn't leave the dry erase board. Hate to see you drive yourself crazy so early in your career.