Sacred Cows? There are no Sacred Cows!

Often on projects there are areas that individuals or teams will tell you are fine, or are things that you shouldn't look at all for very plausable reasons; these are called Sacred Cows.

It could be their favourite developer tool, their source code management system, a particular back-end or front-end system - whatever the tool, you are told that is out of scope of examination.

However, work on enough projects and you soon realise that it is the Sacred Cows that will slow you down or sink your project - that is typically why they are Sacred Cows, everyone knows that they don't work, or don't work well but they are the favorites of someone more powerful in the organisation or they have been burned by other tools.

Never ignore a Sacred Cow!

Having said that, how do you deal with Sacred Cows? It depends on what the Sacred Cow is but above all know about them, understand them inside and out, their weaknesses and problems. Come up with a set of work-arounds, tools, standards, project management techniques, enact those that you are able to and have the others in your back-pocket so that when the issues hit you can quickly mobilise to work around the Sacred Cow.

Think about the projects you have worked on, think about those that have failed or had significant issues, think of the Sacred Cows you have come across and discuss in the comments below.

Complexity in Software

I get to work with numerous clients in a variety of different industries at varying levels of maturity and I see a number of themes which lead to software complexity:

• Complex business domain
• Large legacy of software and systems which must be integrated
• Mismatched developer experience
• Policy

I wanted to point out why these areas cause complexity and how you can overcome them to deliver quality software.

Complex Business Domain

You typically can't change a complex business domain, except in a scenario where you see that there is an overly complex business process and there is an opportunity for you to propose simplification of the process using the software as a driver. If you can simplify the business process before you implement the software - even better because it will reduce the risk of implementation.

Having said that you can't change the complex business domain, you shouldn't verbatim create that complexity in the software, you should use your kitbag of techniques to simplify the design of the software such as using:

• Domain model
• Service layers
• Software Layering
• Object Oriented development
• Service Oriented development
• Componentisation
• Aspect oriented software development

Use techniques like Single Responsibility Principal (SRP) for classes and methods as well as IoC, and make sure that you have the right unit and integration tests in place.

By using these techniques you can spread the complexity across a number of layers, components, classes or services - just be careful you introduce the right level of abstractions that you don't overcomplicate the software by using too many of the techniques or spreading the complexity too thinly.

Complex integration environment

Again, this is not something you can typically get away from, any business of some age or size will have a legacy of systems that will typically have evolved and you will just have to deal with. You can't expect them to throw away everything they have and start again - and if they did, you would just have another type of complexity on your hand - project management complexity.

To deal with a complex integration environment, your approach should be:

• Understand the software you are integrating with early and very deeply
• Make sure you understand how you need to integrate with it
• Consider if an integration layer (e.g. BizTalk) will simplify the integration or make it more complex
• Have well defined interfaces (not just data and operation, but all of the other non-functional aspects of software)
• Have a good stub strategy
• Integrate as early as possible
• Test as early as possible
• Make sure your developers are testing against real services
• Make sure your continuous integration builds are testing against real services

Mismatched Developer Experience

Complexity is often introduced by developers working on the project not having the right level of knowledge of experience, and it isn't just about getting better software developers. Too inexperienced and the developers will use inappropriate technology to solve the problem which will introduce complexity through choosing lots of technologies or the wrong one. An example of this I found was using Word on a server to generate letters which caused layers of complexity which nearly derailed a multi-million pound development project.

Go out and hire some very good developers and you might find that they do and build in complexity by selecting too many technologies for the job, selecting the perfect technology for each task and leaving you with a bunch of technologies you don't need. On another project I was on had 3 IoC containers because different developers liked different technologies.

Choosing a range of developers that work together well and understand how to make pragmatic decisions will be invaluable to ensuring a solution which is matched to your needs, far more than highly skilled developers with large egos.

Another important decision when developing the software is ensuring that the people who will maintain the software can understand it - a high-powered development team are likely to move onto other projects and technologies leaving you with software that you can't understand or maintain.

Policy

Often organisations will implement policies to try to manage software development, if the organisation doesn't have much experience in software development or they have had poor experiences when developing software will introduce more and more layers of policy and control without improving the outcome of software development. One organisation I worked with had 80 quality gates, would take a long time to produce simple software and would often fail to deliver working software.

Other policies I have seen fail is selection of software for strategic platforms without properly understanding the technology or selecting technology because it was by the organisation favoured by senior management - regardless of whether that was right for the organisation or right-sized for the project.

Policy should support the development process and shouldn't be so detailed as to hamstring projects by forcing them to pass irrelevant quality gates with documentation which adds no value and is never looked at again or using technology which has to be shoehorned in to make it work in the way which supports the project.

Choose processes which support a competent development team and allow them to make the right decisions to deliver the software. Allow them to choose the right software to support the project and perform neutral evaluation. Documentation should have a purpose and help support the aim of developing high-quality software, not cover for poor management.

Get a management team who have a proven record in quality software delivery projects which are on time and meet the business need as well as being supportable in the long term. Often managers who have delivered projects which are sold as products will understand this better than manager who have been sheltered in IT departments for the whole of their careers.

The overwhelming message here is that policy cannot fix a broken project, poor technology selection or inexperienced developers, don't add layers just because of an issue which was experienced once.

Summary

So where are we with these observations and recommendations? Select the right people as your number one priority. Don't add policy to make up for bad management or poor technology selections, don't be dogmatic about your technology selections and be prepared to revise your decisions as you get more information - this could happen throughout the project. Make sure that you build continuous integration environments, have a good stub strategy and push the testing as early as possible and make sure your developers are doing it.

Make sure that you use good software development factoring so that you use the right mix of complexity, layering and componentisation techniques which ensure that the software isn't overly complex either by having too much logic in one place or spreading the logic over too many code files or classes.

Experience counts - make sure the people you have are experienced but have also demonstrated pragmatism in their previous projects.

Loosly coupled systems

Ricco recently posted an article: Less Loosely Coupled Than Meets The Eye which talks about something I have been thinking about for a long time from another perspective. We all try to write loosly coupled systems, and to some extent this is possible, for example using interfaces in code or Web Services to de-couple things.

This works to some extent, and is getting better with features such as Code Contracts in .NET or Data Contracts in Web Services and WCF, however there are always unwritten or assumed behaviours of an API which create dependencies that the architect or developer are unaware of and could cause you to have to re-write and recompile parts or all of your application because of that implied behaviour.

Ways around this can be by using Unit Tests to assert various expectations on the behaviour of dependent code. Mocking frameworks are useful in this situation, but often don't help you to identify when a subtle behaviour in the upstream or downstream system or dependency has changed. Integration testing can certainly help, but it can't ever truely capture all of these complex behaviours in systems.

Ricco says:

"I don’t know that it is possible to write anything like a unitary software
system in a way that is truly loosely coupled. It’s not that you can’t make
boxes and lines that are cleanly separated in all sorts of pretty ways, though
that’s hard enough. The problem is that even if you manage to do that, what you
end up with is actually still pretty tightly coupled.


What do I mean?


Well, let me use Visual Studio as an example. It’s made up of all kinds of
extensions which communicate through formal interfaces. Putting aside the warts
and just looking at how it was intended to work it seems pretty good. You can
make and replace pieces independently, there is a story for how now features
light up; it’s all pretty good. Yes, it could be better but let’s for the moment
idealize what is there. But is it loosely coupled, really?


Well the answer is heck no."

He is correct in that unitary software (e.g. software hosted on the same machine or process or even on the same network) can not be isolated by impact from another part of the system becoming errant and impacting on your perfectly tested, perfectly working software.

The closest we can probably get at the moment with the tools, processes and software we have is to isolate key parts of the system or other systems behind a buffered queue and having data and operation contracts which provide isolation of data changes, to some extent behavioural changes, process memory requirements and CPU time as well as timing and race conditions by ensuring that we only depend on the behaviour of the messaging system (still we are relying on subtle behaviours of the message queue technology).

Does it matter?

In simple systems, probably not. Just ensure that your classes have low cohesion by using interfaces and dependency injection along with unit testing and a mocking framework to verify the state and behaviour of the system.

For medium sized projects you are going to have to do all of the above, plus ensure that you have low cohesion between the systems involved as well as defining your data, operation and version contracts.

For large systems, these are always going to require complex testing of end-to-end systems. However where I think we can make advances is in the testing of the Interface specification.

In interfaces today we go away and write a specification, then each end of the integration go away and write their part, each side will use a stub or driver to ensure that it works while the other side completes their work. Then comes the big-bang integration which costs lots of money and time and usually only results in a system which works for the tested scenario. This is why systems integration projects are so difficult.

Think of an analogy to this: HTML and CSS. This is a 20 year old spec which has gone through five versions and yet browser manufacturers still render the same HTML very differently. They are moving in the right direction with the ACID and the W3C test suites which allow the implementations to be tested.

So this presents a way forward. Write a spec for your interface; define the data and operation contracts as well as the versioning story. Then write a program which can sit between the sides of the interface and can provide a common set of behaviours between the two. At least then the systems will be dealing with the same "bugs" in the behaviour. Don't have each implementor write a stub or driver which will differ, but provide something they can actually run against.