Dependencies Flow Down Abstractions

Dependencies are the bane of software development. They make code rigid, difficult to understand, and hard to test. If we want to do better we need to deal better with dependencies.

Of course quite some efforts have been made in this direction. The SOLID principles are trying to give good advice: Use the Dependency Inversion Principle! And patterns like „MVC“ or „layered architecture“ or „Hexagonal Architecture“ or „Clean Architecture“ are trying to do good by prescribing a certain use of dependencies.

This is all good and well - but unfortunately misses the point, I fear. And that’s why even code that’s based on SOLID and one of these patterns quickly becomes hard to change.

I don’t know when things started to go bad (or why they never really started out good) with dependencies. Maybe it’s rooted in decades of scarcity in memory and computing power? Maybe the kind of spaghetti code we see despite all these efforts is a kind of premature optimisation? I don’t know.

But one thing has become clear to me: We need to get to the bottom of this. We need to stop kidding ourselves. SOLID is definitely not enough - or even outright harmful.

So much for my rant to unsettle you ;-) Now let me substantiate my view.

What kind of dependencies?

First let me make clear what kind of dependencies I’m talking about.

Software has two sides: behaviour and data. Behaviour is about action, about transformation. Behaviour is what you can observe. When you enter data into a program it responds with data. That’s the software’s behaviour. It’s always about input-to-output transformation. Object-orientation has not changed a bit of that.

Software is a diligent servant. It executes our requests stated in the form of data by transforming the data, by getting additional data from resources, by sending data to resources, and finally by responding with some data.

Data in itself is passive. It does not work, but is worked on and with. Data flows through processing steps and gets transformed. Data sits around waiting as state. It’s read from (re)source, it’s written to resources/sinks.

Dependencies can exist between a) parts of behaviour and between b) parts of data. And they can exist between c) parts of behaviour and parts of data.

• Example for a): Function f() calls function g()
• Example for b): Data structure p{} contains a reference to data structure a{}.
• Example for c): Function f() receives data of structure p{} as input or function g() access global state of structure a{}.

Dependency means, one part cannot do its job without the other. Or one part cannot be fully understood without the other. Or one part is incomplete without the other.

These are tangible, physical dependencies. But there are also intangible, logical dependencies. For example: Function f()might depend on the data in structure p[] to be sorted. This is important to show correct behaviour at runtime, but at compile time it is hardly visible and cannot be checked. Even if you write a test for that it’s not as clear cut as a physical dependency - because you simply can forget to write this test.

Now, what I’m (primarily) concerned with here are tangible dependencies between behavioural parts of software. That’s the realm of SOLID and design patterns of all sorts.

I want to focus on functions calling other functions, and modules containing functions calling functions from the same or other modules.

The purpose of dependencies

How are dependencies used in behaviour creation today? The first description coming to my mind is: unsystematically.

A developer lets a function call another function whenever she likes. Maybe the motivation is re-use: some problem has already been solved and the solution wrapped into a function - why not call thi function instead of solving the problem again? Maybe the motivation is better understandability: the logic inside a function for solving a problem becomes too convoluted, so the developer extracts a part of it into a separate functions which the previous function then calls.

Although these and other reasons for letting functions call other functions are noble, I think such use is still unsystematic. My reason for that: there is no clear understanding of what the purpose of behavioural dependencies is. What are they there for in the first place? Decades ago they were means to save memory. But today?

The purpose of a wall is to separate space. The purpose of a lightbulb is to provide visibility of the surroundings. The purpose of glue is to keep things stuck together. What’s the purpose of behavioural dependencies?

Here’s my answer:

The purpose of behavioural dependencies is to abstract by integration (or composition).

Let me explain:

When you see an apple, an orange, and a cherry you see… fruits. „Fruit“ is an abstraction. It’s an abstraction by aggregation from a set of concrete things (or details) to help you make sense of the world. „Fruit“ is a category; it’s not inherent in an apple or a cherry. You make it up on the basis of some similarities you see between apple, orange, and cherry.

The category „fruit“ now depends on its elements. Without elements it vanishes. Should the perceived similarities be a misunderstanding the category does not provide any help anymore.

When you see a car you see… a whole. You don’t usually see tires, hull, motor, seats, steering wheel etc. as separate parts. Rather you see the car as the relevant and useful thing it is to you. And that’s an abstraction by integration. The term „car“ abstracts from all the different parts and their relationships. The car is a composition of compositions of compositions which as a whole provides some service to you. It makes something possible (or at least easier) which a heap of parts wasn’t able to. A car is more than the sum of its parts. That’s what the power of integration is!

Aggregation just draws a line around stuff. „Fruit“, „Seed“, „Vegetable“ are just heaps to put on this or that according to some similarity.

But integration not just draws a line around stuff, it also connects stuff. „Car“ describes more than a heap. It also stands for certain relationships between what’s on the heap. A car needs to be assembled.

Aggregations assemble very similar parts into a whole. But integrations assemble very different parts into a whole. That’s the point of integration: it creates something new by connecting what has been assembled.

Integration is a tool of creation. Aggregation is a tool of analysis.

But of course, as aggregations only make sense with parts on their heap, so make integrations only sense with correctly assembled parts. Integrations depend on their parts.

To make it very clear let me abstract by distillation from this. Let me extract the essence from aggregation and integration for my purpose:

In abstractions dependencies point from the whole to the parts

Or to say it the other way around: the purpose of dependencies in abstractions is to tie the parts to the whole.

The part - an apple or a tire - exists perfectly well without any abstraction. It does not depend on it. But the category „fruit“ or the tool „car“ don’t make sense, are not useful, cannot exist without their parts. Thus dependencies point downwards from the whole, from the abstraction towards the parts, to the details.

And there are no dependencies between the parts either! An apple exists perfectly well without an orange or a cherry. Also a spark plug does not need for its existence or even functionality a steering wheel or a trunk or a tail light.

There is no existential dependency between parts of a whole. They only come together with regard to a whole. The whole spans an umbrella of meaning over them.

What a spark plug depends on is a current as input. But it does not care where that’s coming from. Likewise a tire depends on some force to turn it. But it does not care where that’s coming from or how it’s generated.

Let me emphasise it again:

In abstractions there are only dependencies from the top of the whole down to the parts. There are no dependencies between the parts.

Integration is how we build tools. Progressive integration is how things become easier and easier to use. That’s what I see around me every day.

Hence I strongly believe: In order to build good software we should stick to this principle found in cars, computers, drills, blenders, fountain pens, and even organisms. Let me call this the

Dependency Follows Abstraction (DFA) principle

Levels of abstraction

Abstraction, or to be more precise: integration is all around us. Integration (or composition) is what delivers ease of use every day. You don’t have to care about a gazillion „moving pieces“! Just step into your car, turn the key, and drive off. Whatever is happening behind the scene is abstracted away. You’re interacting with integrations of integrations of integrations… It’s compositions all the way down.

And that’s the point: there is not just one level of abstraction. Today’s useful and convenient integrations are deep, deep hierarchies of dependencies.

At the top level you might interact with a car. That’s the whole integrating all the parts like tire, seat, steering wheel, cigarette lighter, engine, transmission etc.

But those parts are not monolithic. A car manufacturer might buy them as smaller wholes and assemble them into a new whole, the car. But for the manufacturer of a seat the seat is not a monolith, but again a whole consisting of parts.

The car seat abstraction might depend on parts like head rest, back rest, cushion etc.

And a head rest might be an integration itself. It depends on guides, screws, cover, a tube etc.

You see what I mean? It’s integrations integrating integrations across many levels. And only at the bottom of this integration tree sit some monoliths.

Those atomic elements don’t consist of any manufactured parts. Their elements are molecules and atoms - but that’s not relevant from the integration point of view of a car manufacturer, for example.

Take this integration hierarchy:

• W
  • P1
    • SP1.1
    • SP1.2
  • P2
    • SP2.1
      • SSP2.1.1
      • SSP2.1.2
    • SP2.2
    • SP2.3
  • P3

A Whole consist of Part 1 to 3. But Parts 1 and 2 are smaller wholes by themselves: they consist of Sub-Parts. Those sub-parts are of no interest to the manufacturer of W. W is only concerned with integrating Part 1..3. And Sub-Part 2.1 again consists of Sub-Sub-Parts!

Whatever the purpose of W is it is fully represented by W. W has the highest level of abstraction. It hides all the details (parts) which are necessary to achieve the purpose.

But the purpose is also present on the next lower level: P1+P2+P3 together also represent the purpose - but on a lower level of abstraction. P1+P2+P3 plus their relationships serve the purpose like W does. But who wants to deal with so much detail?

On the next lower level of abstraction are SP1.1+SP1.2 + SP2.1+SP2.2+SP2.3 + P3. They together with their relationships also represent the purpose. In fact it’s them „working together“ which do the job.

Or the lowest level of abstraction, where most detail is present: SP1.1+SP1.2 + SSP2.1.1+SSP2.1.2 +SP2.2+SP2.3 + P3.

Abelson/Sussman call these levels of abstraction strata. Hierarchies of this kind show a stratified design: more useful „stuff“ is built from parts which is less useful because of its larger generality.

Heterogeneous, but complementary parts are assembled in a manner as to yield some larger whole serving a more specific purpose than the parts: a car has much more specific use cases than a screw in one of its seats or the rubber in one of its tires.

A perfect example of this principle of building in software is a communication stack like the OSI model: on every level (stratum) the purpose is the same; they are all about communication. But with each lower the means of communication become more fine grained. From level to level the abstraction decreases and the number of details to know increases.

Today communication between two processes, even on two machines across the world is easy: just issue a HTTP call. In C# it might look like this:

WebClient client = new WebClient();
var downloadString = client.DownloadString("http://www.gooogle.com");

Think about the huge amount of details hidden by this abstraction! Think about what you would have had to do 10 or 20 years ago. At the lowest level of abstraction things might not have changed that much since then: streams are involved, TCP connections are necessary etc. But over the years integration layers were heaped on integration layers to finally make it that easy.

That’s abstraction I like! That’s a use of dependencies which makes a lot of sense to me!

Higher levels of abstractions using parts from lower levels of abstraction to provide useful services. Great!

Please note: „using“ implies knowing which implies depending on. The high level (of abstraction) class WebClient{} builds on the lower level class HttpWebRequest{}, i.e. a WebClient{} object depends on a HttpWebRequest{} object. And that in turn depends on even lower level objects and so on. All the way down to byte streams and TCP sockets. And below that hardware.

Dependencies are inevitable to achieve this. Dependencies pointing down from higher strata to lower strata are necessary, they are good, we need to keep them. They are even easy to understand.

But now see how behavioural dependencies are used in software development…

Dysfunctional dependencies

Let me remind you: what is more abstract depends on parts which are less abstract. But these less abstract parts do not (!) depend on each other. Parts assembled into a whole just depend on certain input. And they produce certain output. Input might be fuel or electricity or a force, output might be heat or pressure or sound. But no screw „knows“ about another screw, no tire „knows“ about another tire, no head rest „knows“ about a steering wheel etc. There are no (!) dependencies between parts of an integrating whole. That’s how our mechanical and electrical tools are design. From bicycle to hair blower to train to chemical plant.

Just software development begs to differ. Just in software development we think we should not follow this time tested principle.

Look at the dependencies in these popular patterns:

Model-View-Controller (MVC)

Image source: Wikipedia

The Controller depends on the Model which it manipulates. The Model depends on the View which it updates. Each of the boxes knows about the next one. They are connected by dependencies. That’s what the arrows are about. They don’t show data flowing, but runtime dependencies, which in OO means object references.

But do those dependencies follow the DFA? I don’t think so. The reason is simple: there is no whole tangible for the user. The whole of her user experience entails interaction with two boxes: Controller - to issue requests, and View to observe responses. The whole lives just in the head of the user. But there is no whole in the code. Rather, Controller, Model and View are complementary parts of the conceptual whole. Hence they should not depend on each other.

Or ask yourself: Does a Controller do the same as a Model does, just on a higher level of abstraction? Does a Model have the same purpose as a View, just on a higher level of abstraction? To me the answer to both questions is a wholehearted No. Controller, Model, and View are all very different. And it’s this difference which makes them parts of some whole. Call that the application, if you like. But please realise there is no representation of an application in the MVC model; let alone an application which as a whole has dependencies to Controller, Model, and View as its parts.

And so the dependencies to me are dysfunctional. They create pain: they make all dependent parts more difficult to understand and harder to test.

Layered architecture

Image source: Scott Hanselman at Microsoft

Upper layers depend on lower layers. Layers depend on cross-layer concerns. I added the red arrows to make these dependencies very clear.

Do those dependencies follow the DFA? I don’t think so. Again, ask yourself: Does a box like Business Component do the same as for example Security - just on a higher level of abstraction?

There are just two boxes of which I might think of as a whole: UIC and UIP. Behaviour is created by processes, i.e. by stepwise transformation from input to output. Behind each button or menu item a user can click such a process sits and waits to execute a request.

The UIC could be viewed as the top representation of what a software can do. It’s all the details hidden behind a button.

Then on the next level multi-step processes represents what a software can do. That’s a more detailed view of how the purpose is fulfilled.

But beneath that I don’t see further levels of abstraction but just parts for the process level of abstraction. Even more so with the orthogonal concerns.

Bottom line: most of the dependencies in the layered architecture are dysfunctional. They make understanding and testing more difficult. No small wonder they have to be defused by applying the DIP (and then Inversion of Control (IOC)) - which creates accidental complexity.

Clean Architecture

Image source: Robert C. Martin

Outer circles depend on inner circles. Technology depends on domain; let the pretty unmovable and crucial not be tainted by the fleeting concrete. That sounds reasonable. At least until you compare it with the DFA.

Do Devices or the Web do the same as Entities, just on a higher level of abstraction? I don’t think so. Or if they did the same on different levels of abstraction, the Entities would need to depend on devices and/or web. Because Entities are more abstract than a DB.

Or are Presenters on a higher level of abstraction than Use Cases? I don’t think so, either.

The whole, i.e. purpose of the software, to me is only represented by the Use Case circle. Use Cases are like processes: they tie together different parts to create a larger behaviour. A single intent of the user is implemented by a Use Case. But the rest then is all details, details, details residing on lower, and still lower levels of abstraction and to be hidden beneath.

The Clean Architecture is well intentioned, of course. It tries to improve on the layered architecture and unify different concentric approaches.

Unfortunately it does not solve the problem which has riddled software development for ages: how do arrange dependencies in a reasonable manner. Most of the behavioural dependencies even in the Clean Architecture are dysfunctional - which leads even to more accidental complexity from the application of DIP/IoC. (I might even say that the purpose behind the Clean Architecture like the Hexagonal Architecture before it is to let dependencies contradict the DFA. Both clearly state that the concrete, the details should depend on the essential. That both focus on compile time dependencies and not runtime dependencies does not mitigate the contradiction.)

But don’t take my word for it. Even Robert C. Martin says in his book Clean Architecture about a class diagram he presented: „Much of the complexity in that diagram was intended to make sure that the dependencies between the components pointed in the correct direction.“

If the DFA was observed this would not happen, at least not to this degree. The reason: the direction of dependencies would be natural and helpful.

Moving towards stratified designs

Robert C. Martin did not only come up with the Clean Architecture, but also with what he called screaming architecture.

Your architectures should tell readers about the system, not about the frameworks you used in your system.

I like that idea of his very much. First and foremost architecture should express the purpose of a software system. That is, I think, in perfect alignment with the DFA and stratified design.

The patterns presented, though, pretty much contradict this idea. Because in them frameworks are precisely positioned, be it the View in MVC or the Presentation Layer or Devices in the Clean Architecture. The first thing you’re told when looking for guidance on how to structure your software is that you need to find a place for all the frameworks. If screaming architectures are to be desired, then why put frameworks so squarely into the patterns?

Sure, the pattern advice continues by telling how those frameworks are best connected to other parts of the software. And some other parts are even mentioned as to not be forgotten. The Clean Architecture puts the purpose even in the center of its depiction.

Nevertheless what’s completely forgotten is that first and foremost it’s about „the system“ as Robert C. Martin calls it. Or I’d say „the purpose“ or „the behaviour“.

A fundamental pattern for software architecture should not even mention any frameworks. Or if they are mentioned, then lump them all together. „Yes, there are frameworks you’ll need to use, but focus on the purpose first! Focus on abstractions!“

A fundamental pattern for software architecture should emphasise what has been the formula for success in all the rest of the world: abstraction. Or more specifically: abstraction by integration. Because only through abstraction by integration (or composition) large systems are built from small systems. Coarse grained specific „components“ on higher levels of abstraction get assembled from finer grained more general „components“ on lower levels of abstraction.

Think of the OSI layers. Think of the layers of abstraction of code itself: on the lowest level there is microcode, above that machine code, above that some virtual machine code (e.g. .NET IL, Java JVM), above that 3GL code, above that maybe even some Domain Specific Language. What a success this is: building easier stuff on top of more difficult stuff! Build new behaviour as a whole from „older“, partial behaviour as parts.

And the dependencies only and always point down, down, down from abstraction to detail, from whole to part.

That (!) to me needs to be expressed by a fundamental pattern for software architecture:

Please note how all specific concerns like View, Presenter, Entities are gone! That’s not a bug, but the main feature of this pattern. This way the focus is on what’s paramount for clean code development: find good abstractions.

And by that I don’t mean a couple of data structures loaded up with functionality like in mainstream object-orientaction. No, I mean real levels of abstractions in terms of the solution.

Pick out one problem from the many that a software facing the user should solve. Example: A Tic Tac Toe game application needs to solve the problem of a player making a move. Upon the player pointing to a cell on the game board, where she wants her next token to appear, the software should behave appropriately.

Image source: Wikipedia

• The top level abstraction might be called „Make move“.
• The process behind this abstraction then could consist of three stages: accept the coordinates for the token from a player, execute the request to place the token, show the updated game.
• Each processing step then is not only a part, but a whole again. Because each step might be made up of yet smaller steps on a next lower level of abstraction. For example request execution could be made up of: validate the move, place the token, check for a game-over situation, determine the next player.

That’s stratified design! That’s a design where problems are solved on successively more detailed levels in a very systematic way. It even naturally follows the Single Level of Abtraction principle.

Every stratum mirrors the whole purpose - although in various degrees of granularity.

Of course frameworks in the end have their place in such architectures. But they don’t depend on higher level „components“, nor do „components“ within the same stratum depend on each other.

Let me show you what that would mean for the functional „components“ of the above patterns:

In the MVC pattern a new „authority“ would need to be introduced to represent the overall purpose which then ties together the pattern’s concerns.

A simplified layer architecture would morph into this:

Since the concerns are all on the same level of abstraction another level needs to be put on top of them to tie them together - and to make them independent of each other.

And finally a somewhat simplified Clean Architecture:

Some dependencies look like in the original definition, eg. Use Cases depending on Entities. But others would look differently, e.g. Presenters not depending on Use Cases or Use Cases depending on Data Access.

The sum of Controllers, Use Cases, and Presenters has the same purpose as the top stratum. And the sum of Entities and Data Access has the same purpose as the Use Cases. Whereas the integrating „component“ always adds something to the sum of its parts: relationships between the parts.

But these relationships are not dependencies! Like a spark plug does not depend on a distributor or a valve or a piston. All the parts are just fit together in a productive way by a completely separate whole.

Summary

The point of the DFA is very systematically about abstraction. No top-down, clockwise, or outside-in dependencies serving some arbitrary shape. The DFA just cares about what’s more abstract vs more concrete, what’s a whole, what’s a detail? Use any shape you like to arrange your „components“ - but focus on the purpose.

As you see the fundamental pattern for software - or maybe I should call it a meta-pattern? - can be used to accommodate different ideas as to what kind of functional „components“ software should consist of.

Here’s a stratified design for the Tic Tac Toe game above with a number of „components“ instead of processing steps. For some people it’s more natural to think of software first in terms of nouns instead of verbs (even though software is dynamic and about behaviour).

This I’d call a screaming architecture (albeit a small one). Because not only is the focus on abstractions, but also the size of the boxes somewhat mirrors the amount of code needed to deliver on a respective purpose.

Here’s a challenge for you: The next time you look at your code and see a dependency between classes or functions think about the DFA. Does the dependency point down from higher abstraction to lower? Or is it a pointer between parts on the same level of abstraction?

Becoming aware of the DFA is a first step towards cleaner code. That’s my strong belief.