Sunday 12 December 2010

Turning DTOs into a domain layer

Following on from this blog: Do WCF Data Contracts have relevance outside of WCF where I talk about using WCF Data Contracts outside of service calls. This is part of a strategy for re-using DTOs for more than just a transport layer and building them for use across all layers.

Obviously you will instantly think about domain object behaviours, Object Oriented Design etc. In my experience you want to separate out the behaviour of a class in different layers. For example you will want to bind your View to a Model in the UI or use persistent objects in the Data layer, additionally you will want to have business behaviours in the domain layer. What if you could share the underlying shape of the data across all of these tiers.

In this article I will talk about reusing DTOs as a domain model. The obvious benefit is that you don't need to maintain 2 sets of objects and a mapping layer between them.

The main premise of this model is to have the DTO model representing the data shape of the domain object and adding behaviours through the use of extension methods to add this behaviour. For example you might have a Person DTO:


    [DataContract]
    public class Person
    {
        [DataMember(IsRequired=true)]
        public string Title { get; set; }

        [DataMember(IsRequired = true)]
        public string FirstName { get; set; }
       
        [DataMember(IsRequired = true)]
        public string LastName { get; set; }

        [DataMember(IsRequired = true)]
        public DateTime DateOfBirth { get; set; }

        [DataMember(IsRequired = true)]
        public Address Address { get; set; }

        [DataMember(IsRequired = true)]
        public PersonSex Sex { get; set; }

        [ContractInvariantMethod]
        private void EnforceInvariant()
        {
            Contract.Invariant(this.DateOfBirth < DateTime.Now);
            Contract.Invariant(!string.IsNullOrWhiteSpace(this.FirstName));
            Contract.Invariant(!string.IsNullOrWhiteSpace(this.LastName));
            Contract.Invariant(!string.IsNullOrWhiteSpace(this.LastName));
            Contract.Invariant(this.Address != null);

            Contract.Invariant(Enum.IsDefined(typeof(PersonSex), this.Sex));
        }
    }

Note that this uses the code contract model that I describe in Combing Code Contracts with DataContracts for better contract expressiveness and quality

To build a domain layer on top of this you can add a set of extension methods like this to add some business logic to the class:

    public static class PersonBusinessLayerExtensions
    {
        public static bool IsOver18(this Person person)
        {
            return (person.DateOfBirth <= DateTime.Now.AddYears(18));
        }

        public static bool LivesInUK(this Person person)
        {
            Contract.Requires(person.Address != null);

            return (string.Compare(person.Address.Country, "UK", StringComparison.OrdinalIgnoreCase) == 0);
        }

        public static bool CanVoteInUKElection(this Person person)
        {
            return (IsOver18(person) && LivesInUK(person));
        }
    }



You can see from this that we've added a bunch of methods which query some semantic value on top of the raw data, but you could easily add some modification of the state.

Remember that you can combine this with the Transaction Script pattern for longer business logic transactions spanning DTOs or other calls.

One of the elements that makes it powerful is that you can add polymorphic behaviour in different layers and protect the business logic from the presentation or database layers or change the behaviour. This is enforced by deploying the appropriate assembly in the layer.


    public static class PersonUIExtensions
    {
        public static string GetFullName(this Person person)
        {
            return string.Format(CultureInfo.InvariantCulture, "{0} {1} {2}", person.Title, person.FirstName, person.LastName);
        }
    }


A potential disadvantage with this model is polymorphism of behaviours through abstract and virtual methods on the domain classes. However this can still be simulated via inheritance of the DTOs and extension methods on the base classes with extended behaviours on derived classes. This will require different names, but that can be a good thing, as sometimes polymorphism can be abused by overloading the behaviours in ways which aren't described by the method name.

Your thoughts on this approach?

Wednesday 8 December 2010

Sacred Cows? There are no Sacred Cows!

Often on projects there are areas that individuals or teams will tell you are fine, or are things that you shouldn't look at all for very plausable reasons; these are called Sacred Cows.

It could be their favourite developer tool, their source code management system, a particular back-end or front-end system - whatever the tool, you are told that is out of scope of examination.

However, work on enough projects and you soon realise that it is the Sacred Cows that will slow you down or sink your project - that is typically why they are Sacred Cows, everyone knows that they don't work, or don't work well but they are the favorites of someone more powerful in the organisation or they have been burned by other tools.

Never ignore a Sacred Cow!

Having said that, how do you deal with Sacred Cows? It depends on what the Sacred Cow is but above all know about them, understand them inside and out, their weaknesses and problems. Come up with a set of work-arounds, tools, standards, project management techniques, enact those that you are able to and have the others in your back-pocket so that when the issues hit you can quickly mobilise to work around the Sacred Cow.

Think about the projects you have worked on, think about those that have failed or had significant issues, think of the Sacred Cows you have come across and discuss in the comments below.

Sunday 7 November 2010

Combing Code Contracts with DataContracts for better contract expressiveness and quality

I'm going to write some more about Code Contracts (http://research.microsoft.com/en-us/projects/contracts/) soon, but wanted to show you a neat example of using in combination with DataContracts.

This shows the power of CodeContracts in enforcing more semantic information about a DataContract than is currently expressible in WSDL or DataContracts.

For example take a look at this DataContract:

   [DataContract]
   public class Address
   {
       [DataMember(IsRequired=false)]
       public int? HouseNumber { get; set; }
       [DataMember(IsRequired = false)]
       public string HouseName { get; set; }
       [DataMember(IsRequired = true)]
       public string Street { get; set; }
       [DataMember(IsRequired = true)]
       public string City { get; set; }
       [DataMember(IsRequired = true)]
       public stringRegion { get; set; }
       [DataMember(IsRequired = true)]
       public string Country { get; set; }
       [DataMember(IsRequired = false)]
       public string PostCode { get; set; }
   }

Then we can see that there are several attributes which indicate that they are optional, for example the House Number and House Name. In the UK then many houses will have either one or the other or both. However no address should be missing both, the problem with this contract is that this isn't clear or enforceable.

If we enhance this DataContract with CodeContracts:

   [DataContract]
   public class Address
   {
       [DataMember(IsRequired=false)]
       public int? HouseNumber { get; set; }
       [DataMember(IsRequired = false)]
       public string HouseName { get; set; }
       [DataMember(IsRequired = true)]
       public string Street { get; set; }
       [DataMember(IsRequired = true)]
       public string City { get; set; }
       [DataMember(IsRequired = true)]
       public stringRegion { get; set; }
       [DataMember(IsRequired = true)]
       public string Country { get; set; }
       [DataMember(IsRequired = false)]
       public string PostCode { get; set; }
       [ContractInvariantMethod]
       private void EnforceInvariant()
       {
           Contract.Invariant(HouseNumber > 0);
           Contract.Invariant(!(HouseNumber == null && string.IsNullOrWhiteSpace(HouseName)));
           Contract.Invariant(!string.IsNullOrWhiteSpace(Street));
           Contract.Invariant(!string.IsNullOrWhiteSpace(City));
           Contract.Invariant(!string.IsNullOrWhiteSpace(Region));
           Contract.Invariant(!string.IsNullOrWhiteSpace(Country));
       }
   }

Here you can see I have added a method that is decorated with a ContractInvariantMethod attribute. When you run the code through the Code Contract post compiler then this will ensure that at each point that the class changes that these invariants remain true. I've also added some invariants that ensure that the values are not null, empty or whitespace strings and that the house number is greater than 0. This can be analysed with a static analysis tool, with PEX and CHESS tools for testing and runtime checks.

When I am working with WSDL produced from .NET WCF services then I like to get the DataContract assemblies so that these checks are possible. It is also better to go to the source rather than generating these from the WSDL - as long as the people that wrote the contracts have done the decent thing and put their DataContracts in a separate assembly to isolate it from the other layers and behaviours.

What do you think of this approach - do you think it enhances the data contract? Leave some comments below.

Thursday 4 November 2010

Complexity in Software

I get to work with numerous clients in a variety of different industries at varying levels of maturity and I see a number of themes which lead to software complexity:
  • Complex business domain
  • Large legacy of software and systems which must be integrated
  • Mismatched developer experience
  • Policy
I wanted to point out why these areas cause complexity and how you can overcome them to deliver quality software.

Complex Business Domain

You typically can't change a complex business domain, except in a scenario where you see that there is an overly complex business process and there is an opportunity for you to propose simplification of the process using the software as a driver. If you can simplify the business process before you implement the software - even better because it will reduce the risk of implementation.

Having said that you can't change the complex business domain, you shouldn't verbatim create that complexity in the software, you should use your kitbag of techniques to simplify the design of the software such as using:
  • Domain model
  • Service layers
  • Software Layering
  • Object Oriented development
  • Service Oriented development
  • Componentisation
  • Aspect oriented software development
Use techniques like Single Responsibility Principal (SRP) for classes and methods as well as IoC, and make sure that you have the right unit and integration tests in place.

By using these techniques you can spread the complexity across a number of layers, components, classes or services - just be careful you introduce the right level of abstractions that you don't overcomplicate the software by using too many of the techniques or spreading the complexity too thinly.

Complex integration environment

Again, this is not something you can typically get away from, any business of some age or size will have a legacy of systems that will typically have evolved and you will just have to deal with. You can't expect them to throw away everything they have and start again - and if they did, you would just have another type of complexity on your hand - project management complexity.

To deal with a complex integration environment, your approach should be:
  • Understand the software you are integrating with early and very deeply
  • Make sure you understand how you need to integrate with it
  • Consider if an integration layer (e.g. BizTalk) will simplify the integration or make it more complex
  • Have well defined interfaces (not just data and operation, but all of the other non-functional aspects of software)
  • Have a good stub strategy
  • Integrate as early as possible
  • Test as early as possible
  • Make sure your developers are testing against real services
  • Make sure your continuous integration builds are testing against real services

Mismatched Developer Experience

Complexity is often introduced by developers working on the project not having the right level of knowledge of experience, and it isn't just about getting better software developers. Too inexperienced and the developers will use inappropriate technology to solve the problem which will introduce complexity through choosing lots of technologies or the wrong one. An example of this I found was using Word on a server to generate letters which caused layers of complexity which nearly derailed a multi-million pound development project.

Go out and hire some very good developers and you might find that they do and build in complexity by selecting too many technologies for the job, selecting the perfect technology for each task and leaving you with a bunch of technologies you don't need. On another project I was on had 3 IoC containers because different developers liked different technologies.

Choosing a range of developers that work together well and understand how to make pragmatic decisions will be invaluable to ensuring a solution which is matched to your needs, far more than highly skilled developers with large egos.

Another important decision when developing the software is ensuring that the people who will maintain the software can understand it - a high-powered development team are likely to move onto other projects and technologies leaving you with software that you can't understand or maintain.

Policy

Often organisations will implement policies to try to manage software development, if the organisation doesn't have much experience in software development or they have had poor experiences when developing software will introduce more and more layers of policy and control without improving the outcome of software development. One organisation I worked with had 80 quality gates, would take a long time to produce simple software and would often fail to deliver working software.

Other policies I have seen fail is selection of software for strategic platforms without properly understanding the technology or selecting technology because it was by the organisation favoured by senior management - regardless of whether that was right for the organisation or right-sized for the project.

Policy should support the development process and shouldn't be so detailed as to hamstring projects by forcing them to pass irrelevant quality gates with documentation which adds no value and is never looked at again or using technology which has to be shoehorned in to make it work in the way which supports the project.

Choose processes which support a competent development team and allow them to make the right decisions to deliver the software. Allow them to choose the right software to support the project and perform neutral evaluation. Documentation should have a purpose and help support the aim of developing high-quality software, not cover for poor management.

Get a management team who have a proven record in quality software delivery projects which are on time and meet the business need as well as being supportable in the long term. Often managers who have delivered projects which are sold as products will understand this better than manager who have been sheltered in IT departments for the whole of their careers.

The overwhelming message here is that policy cannot fix a broken project, poor technology selection or inexperienced developers, don't add layers just because of an issue which was experienced once.

Summary

So where are we with these observations and recommendations? Select the right people as your number one priority. Don't add policy to make up for bad management or poor technology selections, don't be dogmatic about your technology selections and be prepared to revise your decisions as you get more information - this could happen throughout the project. Make sure that you build continuous integration environments, have a good stub strategy and push the testing as early as possible and make sure your developers are doing it.

Make sure that you use good software development factoring so that you use the right mix of complexity, layering and componentisation techniques which ensure that the software isn't overly complex either by having too much logic in one place or spreading the logic over too many code files or classes.

Experience counts - make sure the people you have are experienced but have also demonstrated pragmatism in their previous projects.

Sunday 31 October 2010

Hierarchyid column type in SQL 2008

If you have ever tried to create an org chart, document structure or other type of hierarchy in SQL you will be aware of the self-join table pattern which works at a basic level but is hard to have arbitrary flexibility in the number of levels and selecting everything below a particular node.

It is interesting to note that Microsoft added a new column type in SQL 2008 which allows you to represent this relationship natively in the database. I was aware of it but hadn't had much use for it or delved too deeply until I read this article:

http://blogs.msdn.com/b/simonince/archive/2008/10/17/hierarchies-with-hierarchyid-in-sql-2008.aspx

This is a very powerful feature in particular the T-SQL extensions for placing the row in the hierarchy (such as hierarchyid::GetRoot()) and also the ability to select all nodes below a particular sub-tree, so finding all reports of a particular manager (direct and in-direct) is easy.

SELECT
    Id,
    Id.ToString() AS [Path],
    Id.GetLevel() AS [Level],
    Name
FROM EmployeeWithHierarchyID
WHERE Id.IsDescendantOf('/5/') = 1
This does not replace the need for a hierarchical database, but it does make SQL server more versatile and meets a key need for document repositories. I wonder if and when SharePoint will use this for a range of features such as document repositories.

Friday 1 October 2010

Do WCF Data Contracts have relevance outside of WCF

In working with WCF you will come across DataContract attributes which decorate DTOs. DataContracts are great because they allow you to specify the WCF contract and are much more flexible than the XML serializer used by the original web services in .NET in that you can specify the attributes on prvate fields so you don't need to break encapsulation. It also supports versioning, optional parameters and a much fuller XSD model

However, can we take this feature which appears to have been created for WCF and use it more generically?

My view and way that I use this is for any sort of data contracts between layers of an application, so between the UI and the business tier, between services, between service layers and even into serialized persistent stores (e.g. XML datatype in the database).

Why?

  • It is flexible in terms of versioning, optionality and data encapsulation
  • It is faster than Binary or XML Serialization (in my testing)
  • It creates compact, concise output
It seems Microsoft had this in mind when they created the class as it is placed in the System.Runtime.Serialization namespace.

So I would recommend that you use data contracts when passing data between any layer of an application and enforce that contract to ensure compatibility.

This relates to a number of upcoming articles, so stick with me while I evolve the story around DataContracts and how you might use them across the application from the data tier, to the UI and avoid writing lots of mappers between domain objects, DTOs and persistent entities.

Feel free to give your views on this, either positive or negative in the comments below

Wednesday 29 September 2010

Loosly coupled systems

Ricco recently posted an article: Less Loosely Coupled Than Meets The Eye which talks about something I have been thinking about for a long time from another perspective. We all try to write loosly coupled systems, and to some extent this is possible, for example using interfaces in code or Web Services to de-couple things.

This works to some extent, and is getting better with features such as Code Contracts in .NET or Data Contracts in Web Services and WCF, however there are always unwritten or assumed behaviours of an API which create dependencies that the architect or developer are unaware of and could cause you to have to re-write and recompile parts or all of your application because of that implied behaviour.

Ways around this can be by using Unit Tests to assert various expectations on the behaviour of dependent code. Mocking frameworks are useful in this situation, but often don't help you to identify when a subtle behaviour in the upstream or downstream system or dependency has changed. Integration testing can certainly help, but it can't ever truely capture all of these complex behaviours in systems.

Ricco says:
"I don’t know that it is possible to write anything like a unitary software
system in a way that is truly loosely coupled. It’s not that you can’t make
boxes and lines that are cleanly separated in all sorts of pretty ways, though
that’s hard enough. The problem is that even if you manage to do that, what you
end up with is actually still pretty tightly coupled.


What do I mean?


Well, let me use Visual Studio as an example. It’s made up of all kinds of
extensions which communicate through formal interfaces. Putting aside the warts
and just looking at how it was intended to work it seems pretty good. You can
make and replace pieces independently, there is a story for how now features
light up; it’s all pretty good. Yes, it could be better but let’s for the moment
idealize what is there. But is it loosely coupled, really?



Well the answer is heck no."

He is correct in that unitary software (e.g. software hosted on the same machine or process or even on the same network) can not be isolated by impact from another part of the system becoming errant and impacting on your perfectly tested, perfectly working software.

The closest we can probably get at the moment with the tools, processes and software we have is to isolate key parts of the system or other systems behind a buffered queue and having data and operation contracts which provide isolation of data changes, to some extent behavioural changes, process memory requirements and CPU time as well as timing and race conditions by ensuring that we only depend on the behaviour of the messaging system (still we are relying on subtle behaviours of the message queue technology).

Does it matter?

In simple systems, probably not. Just ensure that your classes have low cohesion by using interfaces and dependency injection along with unit testing and a mocking framework to verify the state and behaviour of the system.

For medium sized projects you are going to have to do all of the above, plus ensure that you have low cohesion between the systems involved as well as defining your data, operation and version contracts.

For large systems, these are always going to require complex testing of end-to-end systems. However where I think we can make advances is in the testing of the Interface specification.

In interfaces today we go away and write a specification, then each end of the integration go away and write their part, each side will use a stub or driver to ensure that it works while the other side completes their work. Then comes the big-bang integration which costs lots of money and time and usually only results in a system which works for the tested scenario. This is why systems integration projects are so difficult.

Think of an analogy to this: HTML and CSS. This is a 20 year old spec which has gone through five versions and yet browser manufacturers still render the same HTML very differently. They are moving in the right direction with the ACID and the W3C test suites which allow the implementations to be tested.

So this presents a way forward. Write a spec for your interface; define the data and operation contracts as well as the versioning story. Then write a program which can sit between the sides of the interface and can provide a common set of behaviours between the two. At least then the systems will be dealing with the same "bugs" in the behaviour. Don't have each implementor write a stub or driver which will differ, but provide something they can actually run against.

Monday 27 September 2010

First Great Western ticket machines rooted?

I was in Paddington Station recently and went to pick up tickets at one of the machines. I noticed that it was out of service but there was an interesting window in the bottom right hand corner:


Interesting I thought, maybe a standard virus scan, so I took a closer look:


They are running a Downadup (otherwise known as Confiker) removal tool, I watched for a little bit longer and noticed that someone was running pcAnywhere, and there was a real human running this remotely. They rebooted after the scan and I noticed that they were running Windows 2000.

So here is what I see wrong with this:
  1. They are running Windows 2000 (not embedded Windows, but full Windows)
  2. They obviously have had infections because they are not scanning, but running a removal tool
  3. The machines can be remotely accessed via a remote control software
  4. These machines have personal information in them and in particular: they have a credit card reader with PIN pad.
This means that some of these machines have been rooted by a trojan horse and Windows 2000 can't be patched because it is not supported by Microsoft.

Thursday 23 September 2010

New begininings

Welcolme to the new veitch.co.uk. I've had this website for over 10 years and it has been through various incarnations, however it has been pretty stagnant for the last 5 years.

From today it will have a new life hosting my blog where I will be talking about the various aspects of what I am up to and anything interesting I find along the way. I won't be naming any of my clients or revealing any secrets, but I hope to spread some of what I learn.

So, watch this space for upcoming new material!