The toolsmith

Frederick P Brooks, Jr quoted / explained a theory explained by Mills and Baker¹ around 1971-72:

Chapter 3: The Surgical Team

[…] but the team be organized like a surgical team rather than a hog-butchering team. That is, instead of each member cutting away on the problem, one does the cutting and the others give him[her] every support that will enhance his[her] effectiveness and productivity.

[…]

Brooks, F. - The Mythical Man-Month, chapter 3: The Surgical Team

(Text between square brackets is mine)

The author goes on to explain the job descriptions for the surgeon, copilot, administrator, editor, secretaries, program clerk, toolsmith, tester and the language lawyer.

The one that will be covered in this post is the toolsmith:

The toolsmith. File-editing, text-editing, and interactive debugging services are now readily available, so that a team will rarely need its own machine and machine-operating crew. But these services must be available with unquestionably satisfactory response and reliability; and the surgeon must be sole judge of the adequacy of the service available to him. He needs a toolsmith, responsible for ensuring this adequacy of the basic service and for constructing, maintaining, and upgrading special tools –mostly interactive computer services– needed by his team. Each team will need its own toolsmith, regardless of the excellence and reliability of any centrally provided service, for his job is to see to the tools needed or wanted by his surgeon, without regard to any other team’s needs. The tool-builder will often construct specialized utilities, cataloged procedures, macro libraries.

Brooks, F. - The Mythical Man-Month, chapter 3: The Surgical Team

Even though the chapter reads a bit outdated, especially with the part about genders and methodologies (i.e., waterfall), the part about roles is interesting. The surgical team, as the author presents will be composed by 9 1/2 (nine and a half) as the language lawyer can be shared among two or three teams.

Day to day practices

So, one full-time job is to build tools to service the surgeon and the copilot (1:8,5 ratio), or approximately 12% of a full-time position.

These tools can make the surgeon and the copilot more effective, ranging from a single digit percentage to orders of magnitude in other tasks.

As agile teams no longer have a toolsmith and the scrum master is not necessarily a technical expert who can build tools for the whole team, the tasks of creating tools must be taken care by everyone. No one better than yourself can know your needs and tailor solutions for them.

The only downside is you have to invest ~5h (for a full time position) per week into creating and polishing tools for your work: it is a need for achieving more productivity when working.

Nowadays that IDEs are configurable, this time might be invested on learning more advanced features as other shortcuts, refactoring techniques, other design patterns, etc. I like having my own live templates (see description), especially for tests, as I saw on the Crafting Code workshop by Sandro Mancuso

It may also include other activities as setting up your bash alias, cleaning your desk, trying a new UML diagram modeler, etc. Anything that improves the productivity of your main task.

If we consider our experience and knowledge part of our tools (as we are knowledge workers), training could be considered part of polishing our tools. Short, daily or weekly training sessions (even internal) can make a great impact on breadth (not depth) of knowledge among your teammates. This shallow knowledge can open doors for more knowledge to be gained outside of company time, or used to practice and test other methods.

This training is so useful for everyone (“when one teaches, two learn”) that I recommend my clients to do it. This has helped them cope with changing technologies, knowledge transfer and even some team building.

Company provided or not?

Should the company provide for this time to improve? Although I don’t know the answer, it would greatly affect the programmers' productivity to do so. I would say, provide this time as long as what is produced is useful. How do you measure useful? This is a subjective opinion, so it depends on each one.

Some companies acknowledge that this is so important as to invest part of their resources (0-2 hours per week) into training.

Either way, should your company provide it for your or not, you are the one in charge of your career (from Mancuso, Software Craftsmanship - now called The Software Craftsman). If you are not allowed to invest company time for your own benefit, you could use some of your personal time to do so. It is not required, but you will soon enjoy its profits.

Examples

• Every sprint, one of my clients (as of 2015-05) wants to review the work done using some slides and a live demo. The ‘sample’ slides with the format, layout is useful. Everything that should be replaced (timestamp, contents, etc) is marked with a TODO
• Every day, I commit plenty of times to the control version system (i.e., git). I have an alias for commit, diff, status, etc, that saves me some keystrokes on the most common commands
• See the post “A basic working environment”
• Deployment, even on local, is automated: a script for getting the latest branch, develop or master code, build the Docker container, start the docker image, notify me the code is ready
• Pipeline: I have a pipeline, in local, where I can push and the tests will be executed automatically. If the build fails (usually BDD tests), I can push -f without affecting to the repo history.

Mills, H., “Chief programmer teams, principles, and procedures,” IBM Federal Systems Division Report FSC 715108, Gaithersburg, Md., 1971.

Baker, F. T., “Chief programmer team management of production programming,” IBM Sys. J., 11, 1 (1972).

This is one of the best legacy code introductions I’ve seen:

The Grand Redesign in the Sky

Eventually the team rebels. They inform management that they cannot continue to develop in this odious code base. They demand a redesign. Management does not want to expend the resources on a whole new redesign of the project, but they cannot deny that productivity is terrible. Eventually they bend to the demands of the developers and authorize the grand redesign in the sky.

A new tiger team is selected. Everyone wants to be on this team because it’s a greenfield project. They get to start over and create something truly beautiful. But only the best and brightest are chosen for the tiger team. Everyone else must continue to maintain the current system.

Now the two teams are in a race. The tiger team must build a new system that does everything that the old system does. Not only that, they have to keep up with the changes that are continuously being made to the old system. Management will not replace the old system until the new system can do everything that the old system does.

This race can go on for a very long time. I’ve seen it take 10 years. And by the time it’s done, the original members of the tiger team are long gone, and the current members are demanding that the new system be redesigned because it’s such a mess.

Robert C. Martin in the chapter 1: Clean Code from the book by the same name

Also, why should you estimate the new tiger team abilities better than the old tiger team? Is it because the language, tools, or just because they are “plain better”?

What guarantees that the resulting software is going to be better than the old one?

Another interesting definition of legacy code is Michael Feather’s one (code without tests).

I’m currently reading the Apprenticeship patterns book, out of order (explained here: TODO).

I’m writing down the read chapters in a text file, to measure progress and also be able to backtrack if necessary. This also makes reading the book a “choose your own adventure”

After reading approximately half the chapters, the navigation wasn’t so easy using only the “see also” part. So I decided to break free from that constraint and start reading other chapters. But how do you know which chapters you have left to read? Knowing the whole set (all) and the read set (read) this is just the all - read.

Coming to a more practical way, how to know this is an automated (non-manual) way? There are m in read and n in all, where m <= n.

The simplest approach would be:

for each current in read:
  remove current from all

as read is is no particular order, that would yield some O(m*n), worst case scenario (m==n), O(n^2).

That is quite expensive, especially if you have to do it manually

If we use the same algorithm but sort the read set (no longer a set, as sets have no order). Also, converting it to a O(1) lookup table (e.g., a map), that would yield some O(n log2 n + m) that is already better. Taking into account that the sorting can be automated by the computer, it would yield a “manual O notation cost” of O(1 + m). So, relative to the amount of chapters you have read.

But we can even do better in terms of this “manual big o notation”:

read <- sort read into a map ; cost O(1)
all <- sort all into a map ; cost O(1)

unread <- all - intersection(all, read) ; cost O(1)

And this is where the power tools come into place: if we use comm or diff, the computer will create the unread set for us, in “manual” cost of O(1)

Example using vim, sort and diff

This is the read set (for the moment):

the long road
draw your own map
find mentors
the deep end
create feedback loops
the white belt
practice, practice, practice
concrete skills
your first language
use the source
reflect as you work
record what you learn
kindred spirits
nurture your passion
study the classics
expose your ignorance
dig deeper
be the worst
rubbing elbows
unleash your enthusiasm

and this is the all set (copy-paste from the book):

INTRODUCTION 1 What Is Software Craftsmanship? 3 What Is Apprenticeship? 8 What Is an Apprenticeship Pattern? 9 Where Did the Patterns Come From? 9 Where Do We Go from Here? 10
2 EMPTYING THE CUP 11 Your First Language 13 The White Belt 18 Unleash Your Enthusiasm 22 Concrete Skills 24 Expose Your Ignorance 25 Confront Your Ignorance 28 The Deep End 30 Retreat into Competence 32 Wrapping Up 34
3 WALKING THE LONG ROAD 37 The Long Road 38 Craft over Art 40 Sustainable Motivations 43 Nurture Your Passion 45 Draw Your Own Map 47 Use Your Title 50 Stay in the Trenches 52 A Different Road 53 Wrapping Up 55
4 ACCURATE SELF-ASSESSMENT 57 Be the Worst 58 Find Mentors 61 Kindred Spirits 64 Rubbing Elbows 66 Sweep the Floor 68 Wrapping Up 71
5 PERPETUAL LEARNING 73 Expand Your Bandwidth 74 Practice, Practice, Practice 77 Breakable Toys 79 Use the Source 82 Reflect As You Work 85 Record What You Learn 87 Share What You Learn 89 Create Feedback Loops 91 Learn How You Fail 94 Wrapping Up 95
6 CONSTRUCT YOUR CURRICULUM 99 Reading List 100 Read Constantly 102 Study the Classics 104 Dig Deeper 105 Familiar Tools 109 Wrapping Up 111

The first line is not interesting to our purposes, so delete it with:

go to first line (1G), then delete (dd)

After a quick inspection, there’s a number at the end of the desired line, so replace it with newline:

%s/\d\+/\r/g (a snippet below):

EMPTYING THE CUP 
 Your First Language
 The White Belt 
 Unleash Your Enthusiasm 
 Concrete Skills 
 Expose Your Ignorance
 Confront Your Ignorance 
 The Deep End 
 Retreat into Competence 
 Wrapping Up 


 WALKING THE LONG ROAD 

The title lines are all uppercase, so delete them with :%g/^\s*\u\u/d:

 Your First Language 
 The White Belt 
 Unleash Your Enthusiasm
 Concrete Skills 
 Expose Your Ignorance
 Confront Your Ignorance
 The Deep End
 Retreat into Competence
 Wrapping Up 
 
 
 The Long Road 

explanation:

• : : command
• % : in all the file
• g : do (I remember it as ‘go’)
• ^ : starting at the beginning
• \s*: whitespace, as much a possible
• \u\u: exactly two uppercase letters. Can also be expressed as \u\{2}
• d : delete

for more information on vim’s regex, go here

but there are empty lines. Delete them with: %g/^$/d:

 Your First Language
 The White Belt
 Unleash Your Enthusiasm
 Concrete Skills
 Expose Your Ignorance 
 Confront Your Ignorance
 The Deep End 
 Retreat into Competence
 The Long Road

there is a space at the beginning of the line, delete it with :%s/^ //:

Your First Language 
The White Belt 
Unleash Your Enthusiasm 
Concrete Skills 
Expose Your Ignorance  
Confront Your Ignorance 
The Deep End  
Retreat into Competence 
Wrapping Up  
The Long Road  

there are upper and lowercase letters. You could find a way of doing it with vi, but I prefer something that I can remember: save the document, then process it with awk and sort

cat all.txt | awk '{print tolower($0)'}| sort > all_sorted.txt

(snippet):

a different road 
be the worst 
breakable toys 
concrete skills 
confront your ignorance 

If needed, sort and downcase the read.txt too.

cat read.txt | awk '{print tolower($0)'}| sort > read_sorted.txt

I tried using comm, but it wasn’t so useful (because of whitespace).

Tried diff:

diff read_sorted.txt all_sorted.txt  -bBw > to_read.txt

0a1
> a different road 
1a3
> breakable toys 
2a5,6
> confront your ignorance 
> craft over art 
5a10
> expand your bandwidth 
6a12
> familiar tools 
8a15

to know which are the remain chapters:

cat to_read.txt | grep ">"

to count the number of remaining chapters:

cat to_read.txt | grep ">" | wc -l

The lesson learned here is to master your tools. This would have cost me a few minutes if done manually as the list was short, too much time if long (>1000 lines).

On a funny note, it has taken me approximately 2 min to do this processing but ~45 minutes to write this blog post. So I don’t recommend you write as many posts as processings you do.

To quote Apprenticeship patterns, A different road:

If you walk away from software development, you will find that the habit of rigorous thinking and automating tasks involving large volumes of data will still be useful wherever you go

D Hoover, A Oshineye

This week we had a great discussion about Behavior Driven Development (BDD). We have explained it as a way of developing software based on requirements, via automatically tested specifications. For more information, see the wikipedia’s article on the subject

I forgot to mention that this is the perfect start to TDD, as this is usually called the double-loop TDD. See a post on it on coding is like cooking

In this double-loop TDD, the first thing is to create a BDD scenario, run it (red-1), create a unit test that reflects this red (red-2), pass it, refactor; go to red-1 as many times as you need, doing TDD cycles. When you think the feature is done, execute the BDD scenarios and go to green (outer layer) and refactor.

(Our) Common setting

Going back to BDD, a common scenario would be the following:

Layers of software:

• Domain-Specific Language (DSL). See wikipedia’s article
• Test adapter: an interpreter for this DSL
• Test software: test fixtures, methods, structures to arrange, act, and assert on the sut
• Production software (system under test - SUT): thing being tested

A BDD test vs A TDD test

Let’s imagine a BDD test for a linked list:

Given I have an empty list
When I add an element
Then I have an element on the list

now, the same test in TDD [using java, junit 4, hamcrest]

@Test
public void add_an_element_to_an_empty_list(){
	MyLinkedList linkedList = new MyLinkedList();

	linkedList.add("something");

	assertThat(linkedList.size(), is(1));
}

In some aspects, a BDD and a TDD test are similar:

• the given is the arrange, i.e., the prerequisite
• the when is the act, i.e., the command
• the then is the assertion, i.e., the postcondition

(for more information on this, see Hoare’s triple)

But in others, they are not. In BDD you do not specify the APIs, methods, etc. You only care about behavior.

Conclusion

Benefits:

• everyone might involved in the specification by example workshops (source: ATDD by example)
• live specifications. managers can now understand the real state
• regression tests are much cheaper (from automation)

Gotchas:

• Test only the happy paths
• This has a cost, do not overdo it
• Doesn’t remove manual tests
• How do you manage multiple scenarios
• How fast should it be?

This was a low-hanging fruit as a team member who specialises in QA complained about testing in the last moment during the last sprint’s retrospective.

Did the grunt job of connecting the dots and configuring the maven project (using jbehave). Also, got the inspiration from a tutorial.

Announced it only as it was in place and QA approved of it

This grunt job clicked with Sweep the floor

Let’s see how the sprint goes and what are the pain points during this sprint

At a client, I organized an open discussion on code reviews. We had a great conversation.

The main idea was to discuss about it and share the ideas each one had. I didn’t want it to turn into a masterclass (see the white belt)

Benefits

These are the main benefits we saw in it:

• Increased trust
• Learning from others, other approaches
• Less defects, more quality
• Increased bus factor, decreased information silos
• Also:
  • Getting out of your comfort zone
  • Communicating more often (code style, edge cases, complaining, etc)

Pair programming

Then we discussed about the topic of code reviews and pair programming:

• How do they mix

  • Is the need for code review reduced when doing pair programming?
  • It is cheaper to catch defects when pairing than code review (e.g., less to modify, mental caches are hot, …), so why do code reviews? J. B. Rainsberger has an article about it

• Does it give you the same benefits?

  • I argued that it does, some of my colleagues argued against as one is “while doing” and the other is after.

Practice, Practice, Practice

This is also a reference to a chapter by the same name (here)

We proposed this problem:

Receive a list of numbers to a command-line (CLI) application and print their sum

I wrote simple code listing and we reviewed it. The code is here as a gist

(Just in case, I wrote this listing with some defects / smells on purpose)

Here are the comments:

• Not all parameters will be added, only the first three

• The parameters are out of order, overcomplicating things. Is it due to something? Should I be careful about it?

• There is duplication (i.e., parsing from string)

• Bad naming: what is a, b, c?

• There are no tests for it. No manual / javadoc either. Therefore, it’s difficult to maintain

• Difficult to test: a smell for bad design?

  • How to test the adding by itself? The output is to the console, so we have to capture it (at GMaur we published this tool: legacyUtils)
  • How to test the parsing if there is no mock to be injected? Only via state tests, therefore “end to end” tests as it includes the adding

• Mixed concerns / responsibilities. Parsing the numbers and adding them are different concerns and should not be together

Conclusion

• No one was against it
• We all agreed that is important to do it even in time-constrained environments
• It is also important to review our code reviews to make the most of it. The article by J. B. Rainsberger is quite useful for that
• Even small codes can be bad and need to be reviewed (as the example)
• Quick and dirty proof of concept do not mix well with code reviews