Hi, Docker and Ubuntu

I was watching #MSBuild2017, looking over the balcony, up to the cloud, a voice whisper in my head, Oh man, you are far behind. What a moment! So many new things, I decided to pick Docker as a starting point. Honestly, I had no clue where to start. However, I did know that I had to start somewhere. A wise man said start small. Docker has just that, a hello world application.

Whenever I want to learn something, I head over to Pluralsight. Here we go. Give me Docker

Pluralsight Give Me Docker
Pluralsight Give Me Docker

I decided to take the Getting Started with Docker, then later Docker Deep Dive (still enjoying this course). They are awesome. The author, Nigel Poulton, has a good sense of humor.

Docker on Windows

I have been using Windows since I first met a computer. It is fairly simple to run and try out docker on Windows. Head over to Docker on Windows, download and follow the instruction.

After installation completed, fire up PowerShell, your very first command is docker version.

Your very first docker command

Have not used Command Line Interface for years, that output gave me a strong feeling. I uttered “wow it is cool“.

You cannot imagine my feeling when I ran this command

Hello World from Docker

From that moment, I know that I can do many things with the abilities Docker gives.

Docker on Linux

Here is another fact. I do not know Linux. I have not used any Linux system. Years ago, it was hard to setup a Linux lab. What is now?

Hyper-V The power of virtualization

With the power of Hyper-V, I can have as many machines as I wish. I decided to download and try with Ubuntu server (The course suggested it as well).

Installing Ubuntu on Hyper-V is simple. Download Ubuntu Server here. Hello Ubuntu.

Hello Ubuntu

Let try docker

Hello World on Ubuntu

Such an amazing moment 🙂 They are exactly the same in both versions.

Values?

Sound a trivial task. However, there are many things that I gained during the weekend.

  1. It triggers my learn process. I feel energized again after years of writing code.
  2. It opens opportunities. The more I know, the better I am. The better I am, the more opportunities I see.

What I showed here is the end result. Just like anything else in the world, the devil is at the detail. When you actually get your hands dirty, you will meet the roadblocks. Solving them gives me some Linux pieces.

Some Linux (Ubuntu Server) Command Line I learned

sudo su: Run as administrator (mapped to Windows environment).

ifconfig: Same as ipconfig in Windows.

ip addr show: Display the IP addresses with other information

route -n: Display kernel routing table.

apt-get install [name]: Install a package. For example: apt-get install docker.io will install docker on your Ubuntu

apt-get update: Install all updates.

service [name] status: View status of a service [name]

Not a bad result for my weekend. Not mention that I, now, love the Command Line approach.

 

So far so good. It is enough for me to move on. My next challenge is how to take advantages of them. It would be fun.

 

 

Performance Improvement – From Thousands Queries Down To 1 (or 10)

One day, you got a performance bug report. The report said that the customer got a timeout issue. Because the system database is SQL, you open SQL Profiler and monitor what happened inside the SQL server when a single button clicked.

Right in front of you opened eyes, there are more than 7.000 (yes, seven thousands) queries executed to serve a single request. The request is to search a list of records, and there are more than 2.000 records returned__seven thousands queries for two thousands records.

The system is a typical client – server application. The server side is built with WCF + NHibernate (SQL). The system has the notion of CQRS. NHibernate is used to manage domain object with mapping to the database. Therefore, the request (search records) belongs to Query side.

Root Cause

Lazy Loading! It is not hard to figure out the root cause. We use NHibernate to conduct a search query and load all the found records. Here is a simple overview of the architecture.

Architecture Overview
Architecture Overview

The problem happens at the map between domain and view.

Solution

Quite simple. Follow these steps

  1. Define the Model that contains enough data for the View. Note that I am talking about Model not Domain.
  2. Define a SQL View that collects all the data to serve the Model.
  3. Use NHibernate to load data from SQL View instead of Domain.

The general idea is that we should get enough data in a single or two queries.

Lessons Learned

Command and Query Separation

Even the architecture says CQRS notion, there is no real separation of Command and Query. They share the same data model (the domain). The Domain should only appear and use in Command side only. And the Query side, there should be a notion of Model. Domain holds the business logic of the system. Whereas, Model models the read data of the system.

Martin Fowler Command Query Separation.

Overused Tools (NHibernate)

NHibernate is a great tool. However, just like other tools, it has good and bad scenarios. I am sure NHibernate is not a good fit for searching data just to returning partial data for clients. In the example, NHibernate is used to search for list of records, take some of information, and transfer to a view.

However, if we decide to use NHibernate to for this job, then we should read about Select N+1 problem.

Utilize Our Tools (SQL)

We tend to depend so much on ORM framework that we forget how powerful SQL Server is. By defining a SQL view for search request, we can utilize the SQL server. Later we can ask DBA experts to improve SQL performance without redeploying the application.

Final Thoughts

Today, there are thousands of tools at our disposal. We should choose them wisely. Otherwise, there is a big payback in the near future. Every tool has its advantages and disadvantages. You do not have to know them all. However, you should consider studying them before using.

The more I code the more I love CQRS principle. At the highest, it is about Separation of Concern.

I have introduced some concepts in this post. Just in case you have not known about them yet, here are some references:

CQRS on MSDN

CQRS Martin Fowler

SO Select N+1 Problem

CodeProject Select N+1 Problem

 

 

 

My Thoughts Unit Test Is an Art

It started with my Unit Test Mock and State Live Together. My friend asked me to explain a bit more about my statement

Unit Test is an art.

 

It was such an interesting question when he asked: Why do you have this statement?

He made a good point. I promised to give him my thoughts on it.

Things have no meaning unless you label them.

“What UT is” does not matter much. What you think it is matters.

Unit Test (UT) is Code

First and foremost, UT is code. We write a piece of code to test our production code. When comes to code, we have to

  1. Design
  2. Consider best practices
  3. Consider maintainability
  4. Decide what tools to use
  5. … Thousands of other things

As a developer, I say: Code is an Art.

Point of Views

When working in an organization, when talking about Unit Test, each level has a different view. I will give some of my observations. Please notice that I do not claim anything here. I give my observations.

Non – Technical Manager

He assumes UT is a piece of cake, that we should write UT when there is time.

Problem: There is always NO time.

Half – Technical Manager

Maybe he knows some about UT. He knows the important of UT. He also knows the difficulty of writing UT. Regardless of those he still wants to push for code coverage, and think UT is kind of easy.

Developer

I have not seen many developers have the right mindset about UT. And depending on the technical skill, their UT code quality is less than or equal the production code.

I have not seen many developers study UT seriously. To them, it is a nice-to-have thing.

Because of that mindset, their UT code is a maintenance nightmare. Think about when you have to change your production code; when you have to add new code.

Some developers think UT is a joke, a waste of time.

QA

Here is a funny part. Sometimes I heard a story that in a company the manager expects QA to write UT. Oh God!

 

Many parties just forgot (or pretended to) the fact that UT is code.

When people come to an Art gallery, looking at the same picture. Will they say the same thing? I do not think so.

So What?

People look at things depending on their point of views in life. The same event two people might have different opinions. Sometimes, in order to improve something, we simply need to change our view.

Who should change their view? Developers, in my opinion. Developers get the best advantages out of UT.

Here are a few things I suggest you, developers, to begin with

  1. Do a serious research about UT: What is it? What is it for? What are the best resources to learn it? …
  2. Start today build the habit of writing UT. The key word is TODAY.
  3. After a week, review all UT you wrote. You can ask an experienced developer to give you advice.

And you should not

  1. Study “How to” books at the beginning. “How to” are good only when you know what and why.
  2. Start TOMORROW.

 

Final Thought

What you think about UT will determine how good you are as a developer. When reading this post, some might think of TDD. But NO. I am not talking about TDD. I highly recommend you start small.

By no mean, I am a UT expert. I am just trying to make things better every day.

Want to learn more about UT? This The Art of Unit Testing by Roy Osherove is a good resource.

Unit Test Mock and State Live Together

Unit Test is an art. There is no silver bullet for what to test, how to test, what tools to use. In my opinion, everything depends on the kind of projects, the maturity of the projects, and the skill of developers in the projects.

The Story

Given this piece of code. It is not a real production code nor production quality code. I made it up to demonstrate in this post.

    public class Account
    {
        public virtual bool IsClosed { get; set; }
        public virtual decimal Amount { get; set; }
        public virtual void Withdraw(decimal? amount)
        {
            Amount -= amount.Value;
        }
    }

    public class AccountManager
    {
        private readonly Account _account;

        public AccountManager(Account account)
        {
            _account = account;
        }

        public void Withdraw(decimal? amount)
        {
            if (_account.IsClosed)
                return;
            if (amount == null)
                throw new ArgumentNullException(nameof(amount));
            _account.Withdraw(amount);
        }
    }

This is not a design or best practice post. Let’s focus on the business logic and what/how to test.

There is an Account with 2 properties:

  1. IsClosed (true/false): If closed, cannot withdraw money.
  2. Amount (decimal): The current total amount left in the bank Account. The amount is reduced by the Withdraw method.

I keep the Account class pretty simple.

To manage an account, we have AccountManager. What it does are

  1. If the account is closed, nothing happens.
  2. If not closed, and the amount is null (to demonstrate the point that it is invalid withdrawn amount), throw ArgumentNullException.
  3. If all is valid, the account’s amount is reduced by the withdrawn amount.

How are we going to test that 3 expected behaviors?

The Analysis

I believe at this phase, we have to depend on the mind of the developer. First, we need to recognize what kind of test we should perform.

#1: Look like an interaction test. We want to verify a certain behavior: Nothing happens.

#2: Verify input valid which expects an exception to be thrown.

#3: Look like a state test. We want to verify the account’s state. Why? Because we do not really care how the money is withdrawn. We only care about the amount left.

The Technology

Let bring in the tools we know to test. At this phase, depend on organizations; size and kind of projects. I work on .NET stack. I will use MS Test + RhinoMock (for interaction test).

#1: When the account is closed, nothing happens.

The account class might have many properties (now or later), but I only care about “IsClosed” property.

    [TestClass]
    public class AccountManagerTest
    {
        [TestMethod]
        public void Withdraw_AccountClosed_DoNothing()
        {
            // Arrange
            var account = MockRepository.GenerateMock<Account>();
            account.Expect(x => x.IsClosed)
                .Return(true);
            account.Expect(x => x.Withdraw(Arg<decimal?>.Is.Anything))
                .Repeat.Never();
            var sut = new AccountManager(account);
            // Act
            sut.Withdraw(null);
            // Assert
            account.VerifyAllExpectations();
        }
    }

The main point is at the “Arrange” phase. I expect these behaviors

  1. The “IsClosed” must be called
  2. The “Withdraw” must not be called. It means: Do nothing.

I am happy to see that test passed.

#2: Throw ArgumentNullException

        [TestMethod]
        [ExpectedException(typeof(ArgumentNullException))]
        public void Withdraw_AccountNotClosedAndAmountNull_ThrowArgumentException()
        {
            // Arrange
            var account = new Account {IsClosed = false};
            var sut = new AccountManager(account);
            // Act
            sut.Withdraw(null);
            // Assert
        }

I can use either Mock or real object at this test as far as ensuring the IsClosed is false.

#3: If all is valid, account’s amount is reduced by the withdrawn amount

        [TestMethod]
        public void Withdraw_AllValid_AmountReducedByWithdrawnAmount()
        {
            // Arrange
            var account = new Account
            {
                IsClosed = false,
                Amount = 100
            };
            var sut = new AccountManager(account);
            // Act
            sut.Withdraw(30);
            // Assert
            Assert.AreEqual(70, account.Amount);
        }

At this test, I do not care how the system implemented. Given a valid account, and a valid amount to withdraw, my account must have the right amount left.

Be very careful, fellows. If you use mock here, you will end up with interaction test.

 

As I said in the beginning, unit test is an art. You have to research, learn about it every day. Watch other people do it. Make up your own mind. During my career, I found out that developers have a tendency to jump directly to “The Technologies” phase. “The Analysis” phase is very important.

Happy coding and write more unit test.

The Fruit of Refactoring a Better Way of Unit Test

A while ago, I wrote about refactoring 6K LoC class The main benefit of such a refactoring is to have a maintainable codebase.

Last week, I fixed a small bug. The bug itself is trivial. However, the way I enjoyed the new code and wrote unit test was more fun to share. I called it a beautiful solution. And I loved that work.

The Story

In the Refactoring post, I introduced the School and Student to demonstrate the problems. One day, I got a bug report (speak in the blog language, not my real project language)

When removing a student by student’s id, an exception was thrown if the id is invalid.

The expected behavior is not throwing an exception if id is invalid (or not found). It is just a normal business rule in the application.

To demonstrate the story here is the simplest code

    public class School
    {
        protected internal IList<Student> _students = new List<Student>();
     
        private readonly StudentManager _studentManager;

        public School()
        {
            _studentManager = new StudentManager(this);
        }

        public virtual void AddStudent(Student student)
        {
            _studentManager.Add(student);
        }

        public virtual Student FindStudent(Guid id)
        {
            throw new ArgumentException($"The student you are looking for does not exist {id}", nameof(id));
        }
        
        public virtual void Remove(Student student)
        {
            _students.Remove(student);
        }
    }

    public class Student
    {
        public virtual Guid Id { get; protected set; }
    }

And the consumer code (a simple console application)

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Welcome to the test");
            try
            {
                var school = new School();
                var studentId = Guid.NewGuid();
                var student = school.FindStudent(studentId);
                school.Remove(student);
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
            Console.WriteLine("Press key to exit.");
            Console.Read();
        }
    }

Once you run the code

Run Client
Run Client

The Manager

Remember that School is a ghost class. And the student management is delegated to StudentManager. Any attempt to fix directly in School class is a big risk.

There is a small issue in the consumer code. The client has to do 2 things

  1. Find a student by id
  2. Ask School to remove that student

They are too much for a client to know and make decisions. What the client wants is the ability to remove a student by id without throwing an exception.

    public class StudentManager
    {
        private readonly School _school;

        public StudentManager(School school)
        {
            _school = school;
        }

        public void Add(Student student)
        {
            _school._students.Add(student);
        }

        public bool TryRemove(Guid studentId)
        {
            var student = _school._students.FirstOrDefault(x => x.Id == studentId);
            if (student == null)
                return false;
            return _school._students.Remove(student);
        }
    }

The StudentManager will take care of the logic.

The Client will be happy

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Welcome to the test");
            try
            {
                var school = new School();
                var studentId = Guid.NewGuid();
                var studentManager = new StudentManager(school);
                var result = studentManager.TryRemove(studentId);
                Console.WriteLine($"Are you removed? {result}");
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
            Console.WriteLine("Press key to exit.");
            Console.Read();
        }
    }

Why? Because it just needs to ask StudentManager to handle the logic (in a single method call) and deal with result in its own way.

The Unit Test

So far I am very happy with the fix without touching the existing giant class (School class). Good!

Good! How am I going to test this implementation?

The system data access layer uses NHibernate which requires all domain properties virtual. That design brings us a big benefit in term of unit testing. We can either mock or substitute objects.

To understand more about unit test I highly recommend this course from Pluralsight Writing Highly Maintainable Unit Tests.

Due to the complexity of the legacy code, I tended to use Mock (which is an interaction test) rather than Substitute (which is a state test).

Ok, what do I mean by Interaction Test and State Test in this example? We will test the StudentManager.TryRemove method.

What To Test?

From the business logic, we will test at least 2 things

  1. A student is removed if the student id is valid.
  2. If a student id is not valid, do not throw an exception.

How To Test?

The #2 logic should be simple to test. This code will ensure that

    [TestClass]
    public class StudentManagerTests
    {
        [TestMethod]
        public void TryRemove_InvalidStudentId_NotThrowException()
        {
            // Arrange
            var school = new School();
            var sut = new StudentManager(school);
            // Act
            var result = sut.TryRemove(Guid.NewGuid());
            // Assert
            Assert.IsFalse(result);
        }
    }

However, #1 is not straight away depending on the mind of the developer. Here are at least 2 ways to think about it

  1. The implementation has to find a student, then remove him if found. Keywords here are the 2 verbs (find and remove). Some might refer to verify those 2 actions invoked. That is the interaction test.
  2. The other might simply think: it does not matter what it does (the implementation) as far as a student is removed. Before the school has 10 students. After invoking the method, it has exactly 9 students, and cannot find that student in the school anymore. That is the state test.

I would go for the state test.

Everything should be easy if all properties are public. But that is not the case. School holds an internal protected student list. Student has a protected Id setter. And we do not want to expose them as public.

The key is to Substitute them. In the Unit Test we create derived classes and override the method we want to hook in or bypass.

Prepare school and student substituted.

    internal class StudentSchool : School
    {
        public StudentSchool(IEnumerable<Student> students)
        {
            _students = students.ToList();
        }
    }

    internal class IdStudent : Student
    {
        public IdStudent(Guid id)
        {
            Id = id;
        }
    }

They are internal (only visible in the unit test assembly).

Without any change to the production code, I have tests covered

    internal class StudentSchool : School
    {
        public StudentSchool(IEnumerable<Student> students)
        {
            _students = students.ToList();
        }

        public int StudentCount => _students.Count;

        public bool Exists(Guid studentId)
        {
            return _students.Any(x => x.Id == studentId);
        }
    }

    internal class IdStudent : Student
    {
        public IdStudent(Guid id)
        {
            Id = id;
        }
    }
    [TestClass]
    public class StudentManagerTests
    {
        [TestMethod]
        public void TryRemove_InvalidStudentId_NotThrowException()
        {
            // Arrange
            var school = new School();
            var sut = new StudentManager(school);
            // Act
            var result = sut.TryRemove(Guid.NewGuid());
            // Assert
            Assert.IsFalse(result);
        }

        [TestMethod]
        public void TryRemove_ValidStudentId_CountReduce1AndCannotFindAgain()
        {
            // Arrange
            var batman = new IdStudent(Guid.NewGuid());
            var superman = new IdStudent(Guid.NewGuid());
            var spiderman = new IdStudent(Guid.NewGuid());
            var school = new StudentSchool(new[] {batman, spiderman, superman});
            var sut = new StudentManager(school);
            // Act
            var result = sut.TryRemove(superman.Id);
            // Assert
            Assert.IsTrue(result);
            Assert.AreEqual(2, school.StudentCount);
            Assert.IsFalse(school.Exists(superman.Id));
        }
    }

Oh yeah! Superman is removed 🙂 Who needs Superman btw 😛

 

There are pain and joy in refactoring + unit testing code. The more you code with the improvement mindset, the more joy you have. It is a journey where you become better and better after each line of code. Keep going my friends.

Happy weekend fellows.