Motivation

At a client, one of the projects has a long building process and the tests are mostly slow, so I use a local building pipeline, an example of the Pipes and Filters pattern.

This allows for executing manually only the fast unit tests, then automatically (no user intervention, no time spent) executing the rest of them before pushing. In case the latter fails, it is possible to do git push -f to the pipeline without corrupting the central repository (origin) history, possibly disturbing others.

This strategy also allows for parallel modification of sources: you can continue working on your IDE while the compiler is working on the other working directory. Should you introduce any syntax / logical error on your working code, the compiler is not affected, as it has a working copy just for itself.

Implementation

This requires two git repositories:

• local or working copy. Configure it so it has a remote called pipeline and the origin, the repository you cloned from. This is a non-bare repository.
• pipeline, used for building. This also is a non-bare repository.

In the local you can do the development and local commits.

When you’re done, instead of

git push origin $branch

do

git push pipeline $branch

After the git hook is installed, this will trigger the pipeline execution.

Git hook

In the pipeline, in the .git/hooks/post-receive file:

#!/bin/bash
chmod +x pipeline.sh
while read oldrev newrev refname
do
  # whatever you want to execute
   branch=$(git rev-parse --symbolic --abbrev-ref $refname)
  ./pipeline.sh $branch
done

In the above script, we’re telling git to execute the pipeline.sh with the received branch as argument.

Pipeline executor

In the pipeline repository, in .git/pipeline.sh file:

#!/bin/bash

set -e
set -o pipefail

function cleanup {
  git checkout develop
  git pull develop
}

# upon failure, tell the user
function err {
  cleanup
  growlnotify "pipeline fails"
}

# trap signal ERR, executing function 'err'
trap "err" ERR

branch=$1

if [[ -z $branch ]]; then
    echo "need to specify a branch"
    exit -1
fi

git checkout $branch
mvn clean install | tee output.log
git push --set-upstream origin $branch
cleanup

Pipeline executor explanation

• We prepare the bash environment:

  • -e: fail the script when a command fails
  • -o pipefail: fail the script when some command fails in a pipe
  • err and trap: create a hook to be executed when the signal is trapped

• We require a branch to execute this script.

• Checkout to that branch

• Clean, compile & execute tests

• Push to origin

• Clean up

• In case this fails, the script will stop and notify the user with growl and clean up.

Clean up: checkout to develop (or any other branch that always exists), leaving the system prepared to execute again.

Notifications: telling the user

The program growlnotify is a CLI notifier to growl (windows, linux)

Conclusions

Ideally, the tests should be faster, and executing them locally should always be possible, maybe in the pre-commit hook. Whenever this is not possible, a local pipeline can reduce the time spent waiting for test execution and remove the lock on the working directory while the compiler is working.

This pipeline aims to be simple, without many customizations and being single-user. For more complex workflows and other restrictions, it might be better to drop this project and start investigating continuous integration (CI) tools such as e.g., Jenkins, Travis, Bamboo

Further work

The jobs in the pipeline could be queued, so it is possible to push to the pipeline before the previous job has started. See reference below.

Reference

• Trap
• Set Builtin
• Sample bash error handling
• Queuing bash jobs

Appendix

This script will push to the pipeline:

#!/bin/bash

git push pipeline  2>&1 > /dev/null &

JVM default encoding as UTF-8

Make the JVM use UTF-8 character encoding by default

Set up the system property JAVA_TOOL_OPTIONS to -Dfile.encoding=UTF8

There are times when you only want to have the production code:

• downstream job after a successful build
• compiling in local after checkout from DVCS
• spiking (possibly the tests are broken)
• speeding up the process

For that, maven allows to skip the tests, selectively:

• -DskipTests compiles the tests, but skips running them
• -Dmaven.test.skip skips compiling the tests and does not run them. This can also be written as -Dmaven.test.skip=true

In case some of your tests depend on tests from another artifact (e.g. Object Mother, test infrastructure) and the latter has changed, you must compile the tests. You need to not execute the tests

For more information:

• this stackoverflow post is useful
• Documentation for skipping tests

In bash, the usual behavior is to return the last status code:

bash-3.1$ tr
tr: two strings must be given when translating
bash-3.1$ echo $?
1

But this also applies to pipes:

bash-3.1$ tr|tee 1.txt
tr: two strings must be given when translating
bash-3.1$ echo $?
0

Should you want to pass the non-zero (error) status code to the next operation, use this feature toggle: set -o pipefail

bash-3.1$ set -o pipefail
bash-3.1$ tr|tee 1.txt
tr: two strings must be given when translating
bash-3.1$ echo $?
1

This might be useful if you stop on a non-zero status (set -e).

Set Builtin Reference

Reference from the Set Builtin:

-e

-e

Exit immediately if a pipeline (see Pipelines), which may consist of a single simple command (see Simple Commands), a list (see Lists), or a compound command (see Compound Commands) returns a non-zero status. The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test in an if statement, part of any command executed in a && or || list except the command following the final && or ||, any command in a pipeline but the last, or if the command’s return status is being inverted with !. If a compound command other than a subshell returns a non-zero status because a command failed while -e was being ignored, the shell does not exit. A trap on ERR, if set, is executed before the shell exits.

This option applies to the shell environment and each subshell environment separately (see Command Execution Environment), and may cause subshells to exit before executing all the commands in the subshell.

If a compound command or shell function executes in a context where -e is being ignored, none of the commands executed within the compound command or function body will be affected by the -e setting, even if -e is set and a command returns a failure status. If a compound command or shell function sets -e while executing in a context where -e is ignored, that setting will not have any effect until the compound command or the command containing the function call completes.

-o pipefail

-o pipefail

If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully. This option is disabled by default.

Blazing Fast HTML: Virtual DOM in Elm

I’ve read this article explaining how Elm works in relation to the virtual DOM. By Evan Czaplicki.

Tags: elm, dom, virtual-dom, comparison, speed, haskell, evan-czaplicki

Why you should never, ever, ever use MongoDB

I’ve read this article stating the downsides of using MongoDB. Written by Sven Slootweg

Tags: mongodb, downside, sven-slootweg, json, document-store, postgresql, acid

Do you really get classloaders?

I’ve watched this video by Jevgeny Kabanov on classloaders: what they are, how they work nad some tips to solve common issues.

My notes:

• the classloading is lazy: the resources are not loaded until they are mentioned
• every classloader has a parent
• every class has a classloader associated with it
• every object has a class associated with it
• classloading delegation: the parent is consulted before loading a class. if it has it, it will load it. If not, I will delegate it

There is also this article on the same video

Tags: jevgeny-kabanov, classloader, java,

Truth table

I’ve read the wikipedia article about the truth table and the 16 possible binary boolean operators:

• Contradiction
• NOR
• Converse nonimplication
• Negation (of p)
• Material nonimplication
• Negation (of q)
• XOR (exclusive disjunction)
• NAND
• AND (conjunction)
• XNOR (biconditional)
• Projection (of p)
• Implication
• Projection (of q)
• Converse implication
• OR (disjunction)
• Tautology

7 lines of code, 3 minutes: Implement a programming language from scratch

I’ve read this article by Matt Might about the benefits of implementing a language (from scratch but on top of another existing language used as framework). Chooses lambda calculus as the desired DSL

Tags: matt-might, lambda-calculus, racket, domain-specific-language

Turing completeness

I’ve read this wikipedia article (both main and discussion) about this subject.

It has sounded strange to me the reference to S-expressions:

Data languages The notion of Turing-completeness does not apply to languages such as XML, HTML, JSON, YAML and S-expressions because they are typically used to represent structured data, not describe computation […]

If data (S-expression) is represented as the same format as code (e.g., Lisp’s S-expression) and the latter is used to represent a computation, shouldn’t be “data languages that also are used as computation languages” excluded from that list?

As a side note, Lisp is Turing Complete (see Turing Complete#Examples)

The Prize Is Won; The Simplest Universal Turing Machine Is Proved

I’ve read this article by Stephen Wolfram about the proof of a Turing Machine 2,3 being universal by Alex Smith.

Tags: stephen-wolfram, turing-machine, universal-turing-machine, proof, paper, alex-smith

Making a successful commitment in each Sprint

I’ve read this article by Nikolaos Raptis on committing to work in each sprint.

Tags: nikolaos-raptis, sprint, agile, commitment

No increment operator (++) in Ruby?

I’ve read this stackoverflow question about the lack of post-increment operator (i++) in Ruby:

++ and – are NOT reserved operator in Ruby.

Mailing list from Yukihiro Matsumoto aka matz

Code is Data, Data is Code

I’ve read this article by James Donelan on the equivalence of code and data in homoiconic languages, including Clojure.

The article talks about homoiconicity, manually creating Abstract Syntax Trees (AST) out of data and the macro environment. Also, compares the difference between macros and functions.

Tags: code=data, data=code, james-donelan, clojure, homoiconicity, abstract-syntax-tree, ast, macro, function

Agile in Practice: Test Driven Development

I’ve watched this video introducing TDD by the Agile Academy. Explains its motivations and the basic cyclical fashion.

Tags: agile-academy, video, tdd, introduction

Agile in Practice: Pair Programming

I’ve watched this video by Agile Academy on practicing pair programming. Explains the destination and how to get there.

Tags: agile-academy, pair-programming, video, introduction

A low cost approach to working on side projects

I’ve read this article by Gregory Brown on how to apply the MVP and Lean Startup principles to side projects (a.k.a. pet projects). Try to start with a brainstorming session of all the features you want your product to have, then prioritise, picking the most important ones.

Most projects' needs can be solved with a minimal version, in the form of a script that took half an hour to develop.

Tags: gregory-brown, mvp, lean-startup, script, proof-of-concept, priority, backlog, side-project, pet-project, low-cost, cost-reduction

Extreme Programming

I’ve watched this video by STAR Videos on the basic principles behind Extreme Programming and how to use it to improve software quality

Tags: star-videos, extreme-programming, xp, software-quality

Disruptor: High performance alternative to bounded queues for exchanging data between concurrent threads

I’ve read this white paper describing the disruptor framework: a way of exchanging data between concurrent threads.

It uses a ring buffer and other techniques to eliminate write contention, reduce read contention and produces good results. It was developed for a financial exchange environment but it’s general purpose.

Tags: martin-thompson, dave-farley, michael-barker, patricia-gee, andrew-stewart, trisha-gee, disruptor, contention, framework, concurrency, parallelism, white-paper, comparison, arrayblockingqueue, concurrentlinkedqueue, doug-lea, benchmark, lmax, low-level-optimization, optimization

Dissecting the Disruptor: What’s so special about a ring buffer?

I’ve read this article about the ring buffer data structure. A FIFO data structure where elements get overwritten to reduce pressure on the garbage collector. Every message gets sent to every consumer.

Tags: patricia-gee, trisha-gee, disruptor, ring-buffer, data-structure

The Dependency Inversion Principle

I’ve read this paper about the Dependency Inversion Principle (DIP) written Robert C. Martin.

It explains the theory, philosophy behind it. Also, creates an example with a copy buffer and different output locations

Tags: robert-martin, dependency-inversion-principle, dip, example, comparison, object-oriented, principle, solid, design-principle

DIP in the Wild

I’ve read this article about the Dependency Inversion Principle (DIP) written by Brett L. Schuchert.

It is explained non-canonical representations of this principle, how in other cases he has solved it and conclusions of using it day to day.

Tags: brett-l.-schuchert, dependency-inversion-principle, dip, example, object-oriented, solid, design-principle

The New New Product Development Game

I’ve read this paper about the holistic approach to product development, considered seminal for Scrum and other Agile methodologies. Written by Hirotaka Takeuchi and Ikujiro Nonaka

In it, they describe how teams are distributed, the responsibilities for each one, and the degrees of freedom.

Cites the six basic characteristics:

• built-in instability
• self-organizing project teams
• overlapping development phases
• multilearning
• subtle control
• organizational transfer of learning

Tags: hirotaka-takeuchi, ikujiro-nonaka, scrum, agile, product-development, management, 1986

Books I’ve read this quarter¹:

• The Craftsman by Richard Sennett. Posts under the tag the-craftsman
• Software Craftsmanship: The New Imperative by Pete McBreen. Posts under the tag software-craftsmanship-book
• Lisp Hackers by Vsevolod Dyomkin
• The Passionate Programmer: Creating a Remarkable Career in Software Development by Chad Fowler.

Books started, not yet finished (WIP):

• Haskell Programming, By Chris Allen and Julie Moronuki. This book is still in progress and I’m reviewing it
• Functional Programming Patterns in Scala and Clojure by Michael Bevilacqua-Linn
• Pragmatic Thinking and Learning: Refactor Your Wetware by Andy Hunt

Books I want to finish:

Books that have entered the queue:

• Clojure for the brave and true
• How to Solve It: A New Aspect of Mathematical Method by George Pólya
• Conceptual Mathematics: A First Introduction to Categories by Lawvere and Schanuel
• Understanding Computation: From Simple Machines to Impossible Programs by Tom Stuart
• Programming in Haskell by Graham Hutton