It’s good to sit back and reflect from time to time. Lucian Radu Teodorescu does just that and reports back.
15th of August 2002 – the date I started working as a professional software developer. I was still 18 at that time. I have been in this profession more than half my life. Enough time to hope that one can learn a thing or two about what it means to be a software engineer. This article explores 20 lessons that I learned (or I wish I had learned better) during my 20-year career.
Far be it from me to provide a list of clear guidelines for young software engineers based on my professional experience. But I do believe this could be a nice occasion to share a list of items that are hard to master; and, yes, I must confess that I am not an expert in most of the items below. However, I do believe that the sooner a software engineer connects with these items, the easier it is to acquire the needed skills. These items are not necessarily the end goal; they can be the means for improving oneself to be a better software engineer. An aspect on which I am continuing to work, as despite over-used clichés, I consider that learning must be a never-ending process.
1. Reading more software engineering literature
Software engineering should be based on science. Science is based on knowledge.1 Knowledge is best obtained through reading books or journals. Reading about software engineering is thus essential for being a good professional.
Repeatable experiments are key to good science, but not every scientist and engineer needs to repeat all the relevant experiment; we should know the ‘state of the art’ in our field and not reinvent the wheel.
This is important, especially in our era. We are now bombarded by too many shortcuts for getting to needed information. There are wikis, small articles, small YouTube videos, and countless tweets. All of these seem to give the audience condensed information. That can be helpful sometimes, but it is not proper knowledge. To properly assimilate information and transform it into knowledge, one often needs to know the context around that information and to be able to fully reason about it.
For example, saying that QuickSort is O(n log n) on average misses a lot of the context where QuickSort can be used and what the guarantees are about it. One may need to know that the worst case is O(n2), that there are tricks to improve its performance, that it can be faster than other O(n log n) sorting algorithms, or that it is usually slower than an insertion sort for small number of elements, etc.
With all the benefits of being able to access information quickly, we tend to lose a lot of the things that form knowledge. New media simply doesn’t allow the authors to expand on the context for the information they try to convey.
Looking from another perspective, literature can be more trustworthy than various blogs found on the internet. Good publishing houses have thorough review processes and try to keep high standards for everything they publish. Of course, books can sometime contain errors/misinformation, and getting quick access to information is sometimes good enough, but the general idea holds in many contexts.
2. Read more literature, in general
Reading about software engineering is good, but reading general literature (i.e., fiction) should not be ignored either. It can help a lot in self-development. In a way, it allows one to live more than one life. It promotes the development of one’s understanding skills, it makes one much more capable of mastering various languages, and it contributes significantly to building knowledge in general.
The human mind does not work like a set of drawers (or like a hard disk) in which we put information in different compartments, disallowing any interaction between them. It’s more like a complicated web of interactions between different parts. If a programmer looked at the brain, they would probably describe it as the biggest spaghetti code that ever existed.
Acquiring knowledge in one domain helps in other domains too. Learning to learn and to reason will help software engineering a lot, as our field can be described as applied epistemology (see below).
For example, looking at how many attempts we had before we sent people to the moon can provide good insights on how software engineering needs multiple iterations and refinement to solve complex problems.
3. Context is king
During my 20 years of professional activity, I often searched for good solutions that could be applied to all types of problems. For example, trying to search for the best programming paradigm, the best style of writing programs, the best way to approach concurrency, or the best way to architect a software system. But all my attempts were in vain; for each of these subjects, there isn’t a general best solution.
All solutions depend on the context of the issue they try to address. Changing the context of the best solution can make the solution pretty bad compared to other solutions.
For example, there are cases in which monoliths are superior to microservices, despite the popular trend of moving from monolithic applications to microservices.
“It depends!” is often the right answer.
4. Everything is a tradeoff
Even if the context of a problem is well understood, there isn’t such a thing as a perfect solution. Every solution has downsides. I believe that part of our role as software engineers is to recognise these tradeoffs and provide the best solution, the one that best matches the goals of the project.
We often improve performance of a system by degrading its modifiability. Similarly, security is usually improved at the expense of usability. We frequently want to reduce coupling in an application, but the complete absence of coupling means that the two components cannot be used together; we need to have a tradeoff.
At the organisation level, there is always a tradeoff between being too conservative or being too progressive. A conservative organisation works with known tools and can be more predictable; however, it typically stays behind in the innovation game. A progressive organisation will use the latest version of each tool (even beta versions), and can innovate faster; on the other hand, it needs to spend too much time learning new things, that are then thrown away. The organisation must have a compromise between the two extremes.
I often say that, if you can’t argue on both sides on a technical topic, you are probably just confused.
Drawing a parallel from philosophy, Aristotle founds his ethics on the principle of the golden mean, which is a tradeoff between two extremes [Aristotle]. He says:
virtue is a mean, […] a mean between two vices, the one involving excess, the other deficiency
For example, Aristotle argues that the virtue of courage is the mean (to be read tradeoff) between cowardliness and recklessness.
And maybe this is just a small example of how general literature can help with new perspectives on software engineering. The truth is that we ought to properly see the tradeoffs with each solution we provide.
Maybe it’s also worth mentioning that tradeoffs change over time. Well, to be more exact, it’s not the tradeoff that changes over time, but the value that we associate with the alternatives involved in the tradeoff. In any case, the change over time is often important to be remembered; see also item 9 (‘Document the decisions’). 2
5. Software Engineering is applied epistemology
I learned this fact in the last few years from Kevlin Henney [Henney19], and it entirely changed my view of software engineering.
It may sound a bit precious to begin with, but once one thinks more about it, it makes sense. In our field, the main problem is not how to write code that machines understand, but rather to write code that humans understand; to be able to keep track of all the different parts of a complex system, to reason about it, and to ensure that the system grows according to the expectations. The main bottleneck is our mind, not the typing speed.
Thus, our main concern should be to organise knowledge in a structured and meaningful way. That is applied epistemology.
Taking this together with the previous item, it makes sense to say that software engineering can be closer to philosophy than we might naively acknowledge.
6. Software engineering != programming
For many years, I described myself as being a professional programmer. I don’t do that anymore. We are (or should be) software engineers, not programmers. I found that there needs to be a big distinction between software engineers and programmers.
Similarly to the distinction between a carpenter and a mechanical engineer, and to the distinction between an electrician and an electrical engineer, we must have a distinction between a programmer and a software engineer. The main job of the engineers is to design, while the non-engineers typically just execute. The non-engineers might design small-scale things, but they don’t have a structured approach to design.
I believe that being good software engineers really entails the following:
- basing our decisions on knowledge
- having a structured approach to design
- using empirical methods
- using iterations to improve knowledge
As Mary Shaw remarks in Progress Toward an Engineering Discipline of Software [Shaw15], one of the purposes of good engineering is to allow regular persons to do what previously could only be done by virtuosos.
And, let’s not forget that engineers should solve real problems. That may seem obvious, but oftentimes we spend a lot of effort solving problems that we don’t have (overgeneralization or problem misunderstanding). Talking to the customer for a couple of minutes can make the difference between implementing what the customer really wants and what we assume they want.
7. Apply knowledge and good reasoning in day-to-day work
When we design software in a certain way, we should not do it just because it feels right. We should make informed decisions, and we should be able to defend our decisions.
For example, we might design a certain system with a few classes that follow the clean code doctrine. The code looks right to us, and it looks right to our peers. But we should also be able to explain why that’s the case. We should probably be able to explain that our system has low coupling, high cohesion, and it follows the SOLID principles.
As another example, if we decide to use a NoSQL database solution for a web service, we should have made a comparison with SQL databases and be able to show why the chosen solution is superior. Just using NoSQL because it is trendy is not a good argument. See also item 4 (‘Everything is a tradeoff’).
This type of reasoning needs to be applied to the important parts when designing a system, but it also has to be applied in the day-to-day job. This can be a hard thing to do, but that’s the mark of a good engineer. Whenever one writes a new function, changes a class, uses an algorithm, it must follow the same knowledge-based reasoning approach. This process may not need to be shared with other engineers, but it has to happen in the head of the person writing the code.
8. Know the implications of the design
Having a design that solves a particular set of concerns is not enough. The design must perform well for any other concerns that might apply to the software system.
This phrasing is a bit abstract, so let’s take an example. One might design a software sub-system to process some images. The business rules might be complex, so the design will focus on meeting those constraints. But, even if performance is not an explicit constraint of the problem, the engineer needs to be aware of the performance implications of the new design.
If one chooses an algorithm to solve a problem, one has to know the conditions in which the algorithm can operate, the performance characteristics, and the potential difficulties that using the algorithm might entail.
If one chooses a particular database engine, one must understand the performance characteristics, the functional capabilities, the scalability implications, the costs for using that database, and, of course, the development cost for using that database.
Every decision has a set of implications, and these should be well understood when taking the decision.
9. Document the decisions
Software development is applied epistemology, so one can consider it to be a large collection of decisions. The rationale behind the decisions might be forgotten with time. Furthermore, not all the engineers working on the project were involved with these decisions. Thus, it’s important to document the decisions.
When documenting these decisions, one should document the context in which the decision was taken, the alternatives considered, tradeoff analysis between different alternatives and the impact of the decision.
The context is essential; if the context changes, then the decision may not be valid anymore. If we don’t document the context, we never know when a decision is not applicable anymore.
While writing too much documentation can be a big burden, good judgement needs to be taken to document just the things that make sense to be documented. In my personal experience, I would document all architectural decisions, and other decisions that may not be intuitive, or for which the context may not be immediately apparent.
10. Find a way to explain technical details to newcomers
Explaining complex technical details to newcomers is an essential skill for software architects and a good-to-have skill for any software engineer. One should be able to explain technical details to new engineers on the team, to seasoned engineers that were not part of the design, to quality engineers, to management and sometimes to end users. All these stakeholders should have the same understanding of the problem/solution.
One can consider this to be a communication ability. But probably what’s more important is the skill of abstracting out the unimportant details and focusing on the important aspects. Having good abstraction skills is required to be a good engineer, and tailoring your explanation to your audience helps develops those skills.
11. Design the system for others to understand it
In our profession, we often work in teams. We don’t just have to build personal knowledge on the system we are designing, but instead we should build collective knowledge about it. This means that it’s often more important for others to understand a design than the person doing the design.
And, considering the amount of information we subject our brain to, we soon start to forget why and how we designed the system the way it is. We often become the ‘other’.
A good approach when designing a software system is to ask whether a Jon Doe can easily understand the design. If the answer is yes, then we probably did a good job designing.3 If the answer is no, we should consider what needs to change to make it easier for another person to understanding it. Sometimes this means changing the design, sometimes it means documenting important aspects of the design, and sometimes this requires having discussions with other people to understand the pain points. Regardless of the solution, we should make sure the design can be easily understood by other people.
12. Testing is essential
Just to be clear, this item doesn’t say that we need a Software Quality department in addition to the software engineers. Any software engineer needs to spend considerable amount of time performing testing activities.
I’m not saying this because I’m a sold TDD fan. I’m saying this because testing is a core aspect of engineering. As part of being empirical, we must consider that our hypotheses are wrong, and try to test them. The more empirical we are, the more testing we are doing.
There are multiple ways of testing that we can employ in software engineering (unit testing, integration testing, load testing, etc.), and typically more than one testing method is used in a given project. The mix of testing methods depends on the specifics of the software projects. But regardless of the type of project, performing the required testing is the mark of being a good engineer.
13. More solving problems than writing code
Our job as software engineers is to solve software problems. Not to write code. Sometimes, removing code is the best solution for a certain problem. Sometimes, changing the configuration solves the problem. Other times, we can just prove that the problem is just apparent, and that this is actually the best behaviour for the users (in the given context). Occasionally, it’s just making sure that other engineers/teams have the required information to implement a simple solution on their end.
A typical engineer writes about 300 lines of code per day. If we just consider the typing part, this can be done in less than 5 minutes. That is, about 1% of the total work time for a software engineer. We need to be aware that the other 99% is dedicated to solving problems.
One important part of that 99% is design, but there are other activities that need not be neglected: communicating with others, writing documentation, analysing empirical data, creating models to better understand the consequences of a certain design, exploratory experimentation, etc.
14. Knowing the algorithms and data structures
Coming back to what it means to be a good software engineer, we need to base our work on prior knowledge. Knowledge about algorithms and data structure constitute a significant part of software engineering’s body of knowledge.
Knowing algorithms and data structures is like knowing the vocabulary of a language. The better one knows the vocabulary, the better one can communicate in that language; better capture the intended meaning, and be more precise.
15. Innovate on the small items
We tend to associate innovation with significant changes in our industry, like the launch of the iPhone, the launch of iPad, the advancements in deep learning, etc. But innovative products are not built out of thin air. They often require a culture of innovation; this culture is typically built on small-scale innovations.
Innovation is improving a product or a process compared to a de facto standard.
I frequently give the following example: if the common practice in a team is to get all the emails into a single inbox folder, adding rules to automatically move some emails to dedicated folders can be a small-scale innovation. Previously, one had to manually sort emails, and after a certain number of emails received daily, this can become a burden, and a source of defocus. If all the emails on a particular subject would go to a dedicated folder, then the person is freed of some manual work, and, moreover, can be more focused on reading emails.
If one gets in the habit of improving small processes, then one gets the innovation habit. Sooner or later, this person will participate in larger innovations.
16. Learn by doing and do by learning
Learning is quintessential to software engineering. After all, we argued that we are doing applied epistemology; moreover, we are constantly building new things, increase complexity, and adopt new technologies. I don’t believe that there will be a point in the career of a software engineer in which one can stop learning.
Reading books, watching YouTube lectures is a way of learning. But one needs also applied learning. Thus, one needs to learn by doing. After all, engineering has to be empirical.
However, I believe that the opposite can also be true. We can do spectacular stuff by learning. If we develop good learning methods, if we apply sound empirical processes, a learning experience can also lead to good software. Approaching new topics with intense curiosity increases our creativity. This can lead to innovation.
17. Learn from mistakes to get things right
It’s hard to not make mistakes in our field. Be it estimates that are too optimistic, consequences that were not anticipated, unintentional bugs, or something else, we all make mistakes. One should not be afraid to make mistakes, as the mistakes are key to progress.
One problem with trying very hard to avoid mistakes is to enter analysis-paralysis; that is, to continuously delay the point at which the solution is considered ready. This increases the time needed to build software a lot, and cannot eliminate mistakes. If we are entirely ignoring the severity of the mistakes, having a failure rate of 5% with a speed of 100 features per year is more profitable than having a failure rate of 1% with just 10 features completed per year. In the first case, the engineer delivers 95 good features in a year, while in the second case they deliver just 9.9 good features per year (on average).
The other key aspect of making mistakes is the impact of these mistakes. One needs to make sure that their impact is kept under control. Thus, when implementing a feature, the engineer must assess and track the most important risks of the feature. If these are kept under control, the possible mistakes have small impact.
Mistakes are not entirely negative. They tend to help us learn faster. Empiricism is based on the fallibility of hypothesis; we make a prediction, and if that turns out to be false, we learn something new. Mistakes prove that our model is wrong, which often leads to improving our model and our understanding. This was the key to success in natural science, and can be the key to success in software engineering as well.
In short, if their impact is controlled, making mistakes fast can lead to good software and improved knowledge.
18. Being skeptical with everything, including self
Engineering is based on science, and according to Karl Popper, science needs to be built on falsifiable propositions. That is, we should assume that all propositions can be false. We should be skeptical about all the predictions we make about our software.
Being skeptical allows us to incorporate failure in our processes. As previously discussed, this allows us to continuously improve.
But just being skeptical isn’t enough. We also need to measure key aspects of the software we are building. Without this measuring, we are blindly navigating a territory full of traps and dead ends. Continuous measurement and frequent iterations allows us to improve our models and build quality software.
Looking at the moon-landing for example, we did not achieve success at the first attempt. There were several dozen missions – some of them successful, some of them failures – that led us to expand our knowledge. With each mission, we learned something new, so that incrementally we build the right technology to accomplish the moon-landing goal. The key behind the iterations was a large dose of skepticism. We had to assume that we might be wrong, to carry on an experiment.
This skepticism should also apply to yourself. You are also fallible, and you should acknowledge this. Be your number one critic. Spotting your mistakes first is extremely beneficial for your personal growth, and it also gives others less chance to criticise you.
19. Own biases are problems that need to be managed
Continuing on the previous item, you should acknowledge your biases. You have to be an attentive observer of self, and then work out what your biases are. Knowing your biases allows you to compensate them so that you get the best out of you.
Personally, I’m an introvert. Although it doesn’t come naturally, I learned to force myself to communicate even when I would rather just run away. I was also fearful of making mistakes; I learned that I should try more often, and that I should engage others to criticise me as early as possible so that I don’t get the overall feeling that I completed something just to find out it was a mistake. Being too optimistic in estimating simple tasks is also a bias that I have (for complex tasks, I usually overestimate).
Think of your biases as problems that need to be tackled even if, most probably, you will not be able to eradicate them. Then apply empirical methods to improve yourself.
20. Never being done
People have claimed for centuries that physics is almost done. And yet, we continue to have big revolutions in physics.
Civil engineering (and its precursors) is an old discipline. By now, we expect that most of the things have already been tried, and somehow there is not much left for us to do in civil engineering. And yet, now and then we have innovation in this field.
We can’t expect software engineering to be done soon. After all, this is about complexity, and we cannot find a way to simplify all the complexity involved.
We should not stop learning, experimenting, and improving ourselves.
Acknowledgement
During my 20 years as a software engineer, I have met many people who contributed in a significant and positive manner to my professional development. I always will be grateful for what they taught me, directly or indirectly. Still, I want to take this opportunity to express my appreciation for the very first person who, in a country which at that time did not encourage young students to take jobs, being rather reluctant to this idea, believed in my passion and offered me my first job. Thank you, Gina, for giving my career a wonderful start.
References
[Aristotle] Aristotle, Nicomachean Ethics, translated by W. D. Ross, http://classics.mit.edu/Aristotle/nicomachaen.mb.txt
[Henney19] Kevlin Henney, ‘What Do You Mean?’, ACCU 2019, https://www.youtube.com/watch?v=ndnvOElnyUg
[Shaw15] Mary Shaw, ‘Progress Toward an Engineering Discipline of Software’, GOTO 2015, https://www.youtube.com/watch?v=lLnsi522LS8
Footnotes
- The word science comes from Latin sciencia which means knowledge. Nowadays, we often define science as the application of scientific method; this is a method of acquiring knowledge that involves making hypothesis (or models), driving predictions based on these hypotheses, and then verifying the predictions against reality. However we look at it, knowledge is at the heart of science.
- Or maybe this paragraph was just the result of my mind trying to argue both sides of the argument.
- Here, we just focus on the human understating aspect of the design. We take for granted the fact that the design meets all its goals (functional, quality attributes, constraints).
has a PhD in programming languages and is a Staff Engineer at Garmin. He likes challenges; and understanding the essence of things (if there is one) constitutes the biggest challenge of all.
