Questions on the Form of Software

2020 is ending. Australian bushfires, Covid-19 pandemic outbreak, Black Lives Matter protests, Beirut explosion, US West Coast wildfires, and a lot of other catastrophic events. Quite a year! At the end of the year, and the beginning of the new year, it’s often the time for more reflection, and less action. In this spirit, I will put on hold my intent of writing another article on threading; with all these disasters, the least thing we need is yet another article showing how disastrous are the usual approaches on threading.

Instead, I’ll try a reflective, more philosophical article. We will start from Brook’s ‘No Silver Bullet’ article [Brooks86, Brooks95], discuss the essentialist and metaphysical views expressed by the article and consider some questions that may arise from them. I do not have an answer for any of these questions; providing such an answer would probably be one of the biggest advances in Software Engineering. As a software engineer passionate about understanding the essence of things, it is natural for me to ponder over these questions, even if the answers seem far away.

Starting point: essential and accidental software properties

I would argue that ‘No Silver Bullet’ [Brooks86, Brooks95] is one of the most fundamental articles written in software engineering. It defines the main problems in software engineering, and simultaneously it defines the limits of the field. To prove its point, Brooks makes a metaphysical inquiry in software engineering.

The main conclusion of the article is:

There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.

This statement was first made in 1986, and 9 years later Brooks confirms that it was a true prediction¹. Although there were some claims that the prediction is not true², there is no generally accepted position that there is a silver bullet in software engineering from the time that Brooks predicted this.

For the present article, the method through which Brooks arrived at this conclusion is far more important than the conclusion itself. Let’s outline his reasoning.

Brooks starts from looking at the difficulties of software engineering. Following Aristotle, he divides them into essential and accidental difficulties³. We have complexity, conformity, changeability and invisibility as essential difficulties; all the other difficulties (tooling, use of high-level languages, processes, etc.) are accidental.

Here is, for example, a famous passage from the article [Brooks86, Brooks96]:

Software entities are more complex for their size than perhaps any other human construct, because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into one, a subroutine, open or closed.

After having a discussion on these four essential difficulties of software, Brooks argues that, due to their nature, one cannot find any single method to considerably improve on these difficulties. He goes over great lengths and show that all the major promises of software engineering attack accidental difficulties and not essential ones⁴.

The point that Brooks makes is that if most of the promises are related to accidental difficulties, while the essential difficulties are predominant⁵, then there is no way for a promise to deliver more than 10× in terms of productivity, reliability or simplicity. Even if we were to completely eliminate accidental difficulties, we will not be able to achieve one order-of-magnitude improvement.

Like Sisyphus, software engineers are cursed forever to have to struggle with complexity, conformity, changeability. There is no silver bullet and no spell to set them free.

Background

Essentialism in Brooks, essentialism in a software

Merriam-Webster [MW] defines essentialism as:

: a philosophical theory ascribing ultimate reality to essence embodied in a thing perceptible to the senses

: the practice of regarding something (such as a presumed human trait) as having innate existence or universal validity rather than as being a social, ideological, or intellectual construct

Reading No Silver Bullet one cannot only but remark the strong essentialism present in the article. That is, Brooks believes that there is an idea or a form of software engineering activities, that is more than just a generalisation of the practices seen so far. The laws of the Universe are made in a way that software engineering is essentially difficult.

I’m going to take this idea further and apply it to actual software. That is, for any particular software problem that needs to be solved, we have essential difficulties of the software (e.g., there is an essential complexity of the software), and then we have accidental difficulties involved in making that software (e.g., the software took more time to develop, the design is not ideal, we have technical debt, etc.)

Even though Brooks never said this, based on the essentialism emanated from his article, I can perfectly picture him saying:

Within every software there is a simpler software striving to be free.

In this paradigm, every software has an essence, and that would be the ideal software, or the Form of the software. Now, software is typically more complex than it needs to be, it has conformity problems (i.e., bugs), but also changeability and invisibility issues; that is, essentially is less ideal. We call such a software a less-ideal software. In addition to that, software typically has many accidental issues (e.g., tooling-related issues) – this is the real-life software. Figure 1 tries to graphically show these three types of software, by adding deformity to an ideal form (circle).

Figure 1

Philosophical views of essentialism

Before going forward with our questions on the Form of software, we need to take a very short trip through some philosophical views of essentialism. We’ll briefly present the metaphysical views of Plato, Aristotle and Ockham⁶. Although Brooks explicitly mentions Aristotle, we will start from Plato, Aristotle’s teacher.

Plato (429?-347 B.C.) is one of the greatest philosophers in the western world, often named the founder of western philosophy [Kenny10]⁷. Due to the large variety of topics approached by Plato, Alfred North Whitehead noted [Whitehead78]:

the safest general characterization of the European philosophical tradition is that it consists of a series of footnotes to Plato

Plato’s most important contribution to philosophy is the theory of Forms, which is exactly what we are interested in for our article. According to Plato, the reality consists of Forms (or Ideas) which are outside our material world. The material world consists in shadows (or copies) of these abstract Forms. Our senses cannot interact with the world of Forms but only with these shadows. The Forms are eternal and unchanging, while the things in our material world are always changing. For example, all the humans in the world are just shadows of a Human Form; similarly, all the good things in the world are just shadows of the Goodness Form; there are Forms for shortness, justice, redness, humanness, etc. This view about the existence of Forms in an ideal, abstract world is called Platonic realism.

So far, we have used the term Forms in the context in which we meant abstract ideas; this was done on purpose, to refer to these Platonic Forms, the most extreme form of idealism.

Aristotle (384–322 B.C.) was a student of Plato at the Academy. If Plato is credited for founding western philosophy, Aristotle is often credited as being the first scientist, with the systematic way that he approached all the branches of knowledge. He is widely accepted as the inventor of logic and of the deductive reasoning. [Kenny10]

Aristotle was Plato’s first true critic. He exposed a very detailed critique of the theory of Forms, arguing its inability for expressing change and its inability for explaining how the Forms are related with the real-world. Aristotle replaced Plato’s extreme idealism, with a moderate realism; universal properties exist, but only in as much as they exist in our world.

Being able to distinguish between accidental and essential properties of objects is paramount to being able to identify universal properties in the world. Having a human-like body is an essential property that makes Socrates a human, but the colour of his skin is just an accidental property of Socrates. Also, Aristotle bases a lot of his metaphysics on the notions of potentiality and actuality; with time the set of humans change, and we can also think of the potential humans and apply the same reasoning to them.

Both Plato and Aristotle argued that there are universal concepts like Human (either in an ideal world or in the material world). There are, however, philosophical views that deny the existence of these universals. Probably the best known theory is the Nominalism theory of William of Ockham (1287–1347)⁸.

Ockham argues that no universal exists outside our mind; everything in the world is singular. That is, there is no concept of Human in this world, or any other world that somehow influences our world. The world contains only human instances, and nothing more; the Human concept is just present in our minds.

Let’s transpose these three theories in software engineering:

1. Plato would argue that there is a Form of Software Engineering in a parallel universe, and all instances of software engineering in our world are imperfect copy of that Form.
2. Aristotle would argue that there is a universal called Software Engineering that can be analysed by properly looking at its essential properties (and ignore accidental properties).
3. Ockham would argue that there is no such thing as software engineering; there are just many activities that have nothing truly in common, and it is only in our minds that we call them software engineering.

Brooks seems to share the same views as Aristotle.

With these three views covered, we are now finally ready to reach the questions, the essentials of this article.

Q1: Does a Form of a software exist?

Let us take as an example a text editor software (with a known set of requirements). We call this TextEditorSw. The question then becomes whether a Form of TextEditorSw exists, and how real is this.

According to the three philosophical views, we have three potential answers:

1. Yes, there is a Form of TextEditorSw in an ideal world, and all our real instances of text editors are just imperfect copies of it.
2. Yes, there is a universal for TextEditorSw, which is present (at least in potentiality) in this world; we just have to enumerate the essential characteristics of this universal to better understand it.
3. No, there is no such universal; everything is in our head.

If the answer is one of the first two, then there is an ‘essence’ of TextEditorSw. Furthermore, it probably makes sense to assume that this ‘essence’ will be the least complex, the one that is the most possible conformant, with no changeability and no visibility issues. That is, the software would be perfect.

If we can identify the perfect software (at least for a particular type of software), then that would be the Holy Grail of software engineering. Trying to understand this perfect software, and the means that we need to take to reach it would be the most important activities of software engineering.

If we would have a clear proof (or at least a strong enough argument) that this perfect software exists, I can imagine a lot of research being founded to get us closer to this perfect software. We can probably start developing a method/program to dramatically improve software engineering, similar to Hilbert’s program in mathematics⁹. This ultimately would lead us to the silver bullet of software engineering.

If the answer is yes with an extreme idealist form (Plato’s theory of Forms) it means that the perfect software is somewhere inaccessible to us, but we can get near it by pure reasoning. One can, theoretically, just by thinking, arrive close enough to the Form of the perfect software. In the process it might uncover other Forms of what a good software means, and thus, we can improve our software engineering practices.

If we take an Aristotelian view, then the perfect software might exist in the real world (albeit it can be hard to define); if it does not actually exist, then it is certainly possible for it to exist. We can certainly use empirical inquiries to find out more about the properties of the software, and thus to continuously improve our software engineering practices. This time, the attack on software engineering difficulties must be targeting only essential difficulties.

On the other hand, if we believe that a nominalism view would be more appropriate, and we conclude that the third answer is the right one, then we would be hopeless in trying to identify what good software engineering means. All our systematic approaches in improving software engineering will inevitably fail. We can never know what makes a software a good software; we just have different instances of software, and all the generalisations are just in our head. All the improvements we can hope for in software engineering will then come from our ability to improve how our mind looks at these problems.

Note that there are multiple other nuances between these three possible answers. I’m just trying to lay out the main answers according to the main theories related to the presence of universals.

Q2: How close can we get to the Form of a software?

Provided that we exclude the nominalist view, and we have enough confidence that there is a (almost) perfect Form of the software (either in this world or outside it), can we go near it? Or, lowering the bar a bit, can we go to something like 10% close to it?

Now, for the sake of argument, let’s assume that if we get close enough to that software, we get 90% of the benefits of having a perfect software, and we just have something like 10% of difficulties. If that would be the case, then, this again would probably count as the silver bullet. We eliminated most of the complaints that Brooks had on why software engineering is difficult.

More than that, we have a model to study on what the almost-perfect software would look like. If we know how an almost-perfect TextEditorSw would look like, then we can probably generalise this knowledge to other types of text editors. And, then, even further, to other types of software products.

Continuing this line of thought we would probably end up with a systematic way of ensuring that we ‘properly’ build software, that ensures that we get very close to ideal software.

It is almost like we gain the same level of trust that bridge building has from structural engineering. Software engineering would not be such a costly activity anymore, as we would know how to get around most of the difficulties. We can probably also automate it in a large proportion (i.e., robots writing code).

This question is far too important to be left unanswered, or at least to be left without any attempts to be answered.

Q3: Do we have a way to measure the distance to the Form of a software?

This questions is mainly a continuation of the previous question. We assume that there is a Form of software, and we may or may not know how to get to it.

In the best case we can have a test that would give us a numeric distance between a particular software and its ideal form. But, if that is not possible, a test that would tell them whether the two are close enough is also a significant addition to our toolset. This would be similar to the inability of finding a numeric measurement to indicate the ‘distance’ between an object and a chair, but we can easily test if the given object is a chair or not.

If we would have a numeric measurement, then arriving to (or close to) the ideal form of a software is relatively easy. If we don’t have a systematic way, then we can apply some form of learning (similar to how we do it in Machine Learning) to arrive close to the ideal Form.

If we would only have a binary test, then we can approach this problem the brute-force way. Doing this enough times, for different types of problems, would probably lead us to some conclusions, and thus we can learn by repeated experience how to design the perfect software (I’m assuming that there are some universal laws that can be approximated, and the perfect software is not just subjected to randomness).

Thus, if we have such a measurement, we can empirically derive an algorithm to get our software to the perfect state, and, moreover, we can hope that we can learn the best way to do this for all software problems.

Q4: Can we distinguish between accidental and essential complexity?

One thing that puzzles me in Brooks’s description is that it is not apparent what essential complexity is. On one hand, Brooks puts complexity as a whole as an essential difficulty to software engineering, but then he goes on and argues that the use of higher-level programming languages are just covering for accidental difficulties.

Let us consider an algorithm over a collection of data that can be written in 10 lines of a high-level programming language (either OO or functional). Let’s assume that this algorithm does some sorting, some mapping and some filtering, etc. If we were to translate this into assembly it would probably far too big for a person to understand it in a day, not to mention writing it. So, there are at least a couple of orders of magnitude between the time spent in understanding the high-level code and understanding the assembly one (not to mention that assembly can be considered high-level compared to the processes that happen at the hardware level). Do we really think that understanding assembly is just accidental difficulty? Probably not.

On the other hand, one can write the same problem in multiple ways using a high-level language. And most of the variants are having roughly the same complexity. For example, breaking a private 10 lines of code function into two separate function is most probably not changing the complexity of the solution. This surely cannot be considered as being essential traits for the software.

So then, what is essential, and what is accidental? Although we do have the tools to properly make this distinction, I don’t think we know how to do it.

If we would answer this question, then probably we could easily derive reasoning that would tell us what a good software is, and what is not.

Q5: Is technical debt an essential difficulty?

Most often when software developers discuss difficulties in their day to day work, the phrase technical debt pops up. Symptoms of technical debt include: large number of bugs in a particular area, ad hoc design, difficulty of changing the software, lack of or inappropriate documentation, lack of testing, etc.

There are cases in which technical debt can be a good thing for the project lifetime (i.e., for proof-of-concepts, for features that need to make a deadline, for delaying some effort that are not sure that are really needed, etc.), but most of the time technical debt is considered a bad thing; thus a difficulty in the sense we are discussing here.

The main question that arises then is whether technical debt is an essential difficulty or not.

On one hand, one can argue that it is one of the major sources of complexity in a project, and as complexity is an essential difficulty of software engineering, so technical debt is an essential difficulty. On the other hand, if some technical debt can be good for the project, then it cannot be an essential difficulty.

But maybe, technical debt is somewhere in the middle. Sometimes, over certain limits it is an essential difficulty, and sometimes, when kept under control, it is not an essential difficulty (either an accidental difficulty or a good thing).

If we can find out the limits after which technical debt is an essential difficulty, then maybe we can put processes in place to prevent it from crossing those limits.

It’s essential to understand essential difficulties (pun intended).

Final words

In our post-modern world, people are always looking for solutions to problems. Sometimes we even invent solutions to problems that we never had. But, if it is said that asking questions is more important than answering them, perhaps the best solutions can be found only after trying hard to ask the right set of questions.

This is my attempt to ask some questions that I feel are important for the field of software engineering. Maybe they cannot be properly answered, maybe they are not the most important questions that we have to ask, but it’s clear that they gravitate around the essence of software engineering (if there is an essence to it).

I argue that answering any of these questions will be a significant step forward in software engineering. Probably it would be the mythical silver bullet. But, if it’s not, it would most likely lead to alleviating some difficulties in our field.

If these cannot be properly answered, a good answer approximation would probably still advance the fundamental research in software engineering.

If that doesn’t happen either, then I hope at least I was able to detract the reader from this cruel reality, and make them enjoy this short detour in philosophising about software engineering.

References

[Aristotle84a] Aristotle, ‘Topics’ in The Complete Works of Aristotle, Volume 1: The Revised Oxford Translation (vol 1), edited by Jonathan Barnes, Princeton University Press, 1984

[Aristotle84b] Aristotle, ‘Metaphysics’ in The Complete Works of Aristotle, Volume 2: The Revised Oxford Translation, edited by Jonathan Barnes, Princeton University Press, 1984

[Brooks86] Frederick P. Brooks Jr., ‘No Silver Bullet – Essence and Accidents of Software Engineering’, Proceedings of the IFIP Tenth World Computing Conference, edited by H.-J. Kugler, 1986

[Brooks95] Frederick P. Brooks Jr., The Mythical Man-Month (anniversary ed.), Addison-Wesley Longman Publishing, 1995

[Kenny10] Anthony Kenny, A New History of Western Philosophy In Four Parts, Clarendon Press, Oxford, 2010

[MW] Merriam-Webster, ‘essentialism’, accessed Oct 2020, https://www.merriam-webster.com/dictionary/essentialism

[Seemann19] Mark Seemann, ‘Yes silver bullet’, 2019, https://blog.ploeh.dk/2019/07/01/yes-silver-bullet/

[Wikipedia] Wikipedia The Free Encyclopedia, Hilbert’s program, accessed Oct 2020, https://en.wikipedia.org/wiki/Hilbert%27s_program

[Whitehead78] Alfred North Whitehead, Process and Reality, New York: The Free Press, 1978.

Footnotes

1. See the ‘ “No Silver Bullet” Refired’ chapter in The Mythcal Man-Month. Essays on Software Engineering, Anniverary Edition [Brooks95].
2. For a more recent claim, please see Yes silver bullet by Mark Seemann [Seemann19].
3. The distinction between essential and accidental plays an important role in Topics [Aristotle84a] to make the distinction between the definition of a thing and a property of a thing, and in Metaphysics [Aristotle84b] to analyse the essence of being.
4. Interestingly enough, Object-Orientation is among the promises that Brooks analyses and finds that it cannot achieve much; this happens before OOP became mainstream.
5. Brooks actually clarifies this better in the follow-up ‘ “No Silver Bullet” Refired’ chapter of the 1995 edition of The Mythical Man-Month book [Brooks95]; he believes that until that date most pressing accidental difficulties are solved, and that the ratio between accidental and essential difficulties cannot be greater than 9 to 1.
6. In Latin, Ockham is spelled Occam.
7. Plato also created the Academy, which is considered the first higher learning institution in the western world.
8. The famous Occam’s razor is most of the time attributed to Ockham; the principle says that “entities should not be multiplied without necessity”, and sometimes it’s paraphrased as “the simplest explanation is most likely the right one”.
9. Hilbert proposed a program to ensure the existence of a method to prove all propositions in mathematics [Wikipedia]; Gödel showed later that such a program is unattainable.

Lucian Radu Teodorescu has a PhD in programming languages and is a Software Architect at Garmin. In his spare time, he is working on his own programming language and he is improving his Chuck Norris debugging skills: staring at the code until all the bugs flee in horror.