Motivation
At a client, one of the projects has a long building process and the tests are mostly slow, so I use a local building pipeline, an example of the Pipes and Filters pattern.
This allows for executing manually only the fast unit tests, then automatically (no user intervention, no time spent) executing the rest of them before pushing. In case the latter fails, it is possible to do git push -f
to the pipeline
without corrupting the central repository (origin
) history, possibly disturbing others.
This strategy also allows for parallel modification of sources: you can continue working on your IDE while the compiler is working on the other working directory. Should you introduce any syntax / logical error on your working code, the compiler is not affected, as it has a working copy just for itself.
Implementation
This requires two git repositories:
local
or working copy. Configure it so it has a remote called pipeline
and the origin
, the repository you cloned from. This is a non-bare repository.
pipeline
, used for building. This also is a non-bare repository.
In the local
you can do the development and local commits.
When you’re done, instead of
do
git push pipeline $branch
After the git hook is installed, this will trigger the pipeline execution.
Git hook
In the pipeline
, in the .git/hooks/post-receive
file:
#!/bin/bash
chmod +x pipeline.sh
while read oldrev newrev refname
do
# whatever you want to execute
branch=$(git rev-parse --symbolic --abbrev-ref $refname)
./pipeline.sh $branch
done
In the above script, we’re telling git to execute the pipeline.sh
with the received branch as argument.
Pipeline executor
In the pipeline
repository, in .git/pipeline.sh
file:
#!/bin/bash
set -e
set -o pipefail
function cleanup {
git checkout develop
git pull develop
}
# upon failure, tell the user
function err {
cleanup
growlnotify "pipeline fails"
}
# trap signal ERR, executing function 'err'
trap "err" ERR
branch=$1
if [[ -z $branch ]]; then
echo "need to specify a branch"
exit -1
fi
git checkout $branch
mvn clean install | tee output.log
git push --set-upstream origin $branch
cleanup
Pipeline executor explanation
-
We prepare the bash environment:
-e
: fail the script when a command fails
-o pipefail
: fail the script when some command fails in a pipe
err
and trap
: create a hook to be executed when the signal is trapped
-
We require a branch to execute this script.
-
Checkout to that branch
-
Clean, compile & execute tests
-
Push to origin
-
Clean up
-
In case this fails, the script will stop and notify the user with growl
and clean up.
Clean up: checkout to develop (or any other branch that always exists), leaving the system prepared to execute again.
Notifications: telling the user
The program growlnotify
is a CLI notifier to growl
(windows, linux)
Conclusions
Ideally, the tests should be faster, and executing them locally should always be possible, maybe in the pre-commit
hook. Whenever this is not possible, a local pipeline can reduce the time spent waiting for test execution and remove the lock on the working directory while the compiler is working.
This pipeline aims to be simple, without many customizations and being single-user. For more complex workflows and other restrictions, it might be better to drop this project and start investigating continuous integration (CI) tools such as e.g., Jenkins, Travis, Bamboo
Further work
The jobs in the pipeline
could be queued, so it is possible to push to the pipeline before the previous job has started. See reference below.
Reference
Appendix
This script will push to the pipeline:
#!/bin/bash
git push pipeline 2>&1 > /dev/null &
Blazing Fast HTML: Virtual DOM in Elm
I’ve read this article explaining how Elm works in relation to the virtual DOM. By Evan Czaplicki.
Tags: elm, dom, virtual-dom, comparison, speed, haskell, evan-czaplicki
Why you should never, ever, ever use MongoDB
I’ve read this article stating the downsides of using MongoDB. Written by Sven Slootweg
Tags: mongodb, downside, sven-slootweg, json, document-store, postgresql, acid
Do you really get classloaders?
I’ve watched this video by Jevgeny Kabanov on classloaders: what they are, how they work nad some tips to solve common issues.
My notes:
- the classloading is lazy: the resources are not loaded until they are mentioned
- every classloader has a parent
- every class has a classloader associated with it
- every object has a class associated with it
- classloading delegation: the parent is consulted before loading a class. if it has it, it will load it. If not, I will delegate it
There is also this article on the same video
Tags: jevgeny-kabanov, classloader, java,
Truth table
I’ve read the wikipedia article about the truth table and the 16 possible binary boolean operators:
- Contradiction
- NOR
- Converse nonimplication
- Negation (of p)
- Material nonimplication
- Negation (of q)
- XOR (exclusive disjunction)
- NAND
- AND (conjunction)
- XNOR (biconditional)
- Projection (of p)
- Implication
- Projection (of q)
- Converse implication
- OR (disjunction)
- Tautology
7 lines of code, 3 minutes: Implement a programming language from scratch
I’ve read this article by Matt Might about the benefits of implementing a language (from scratch but on top of another existing language used as framework). Chooses lambda calculus as the desired DSL
Tags: matt-might, lambda-calculus, racket, domain-specific-language
Turing completeness
I’ve read this wikipedia article (both main and discussion) about this subject.
It has sounded strange to me the reference to S-expressions:
Data languages
The notion of Turing-completeness does not apply to languages such as XML, HTML, JSON, YAML and S-expressions because they are typically used to represent structured data, not describe computation […]
If data (S-expression) is represented as the same format as code (e.g., Lisp’s S-expression) and the latter is used to represent a computation, shouldn’t be “data languages that also are used as computation languages” excluded from that list?
As a side note, Lisp is Turing Complete (see Turing Complete#Examples)
The Prize Is Won; The Simplest Universal Turing Machine Is Proved
I’ve read this article by Stephen Wolfram about the proof of a Turing Machine 2,3 being universal by Alex Smith.
Tags: stephen-wolfram, turing-machine, universal-turing-machine, proof, paper, alex-smith
Making a successful commitment in each Sprint
I’ve read this article by Nikolaos Raptis on committing to work in each sprint.
Tags: nikolaos-raptis, sprint, agile, commitment
No increment operator (++) in Ruby?
I’ve read this stackoverflow question about the lack of post-increment operator (i++) in Ruby:
++ and – are NOT reserved operator in Ruby.
Mailing list from Yukihiro Matsumoto aka matz
Code is Data, Data is Code
I’ve read this article by James Donelan on the equivalence of code and data in homoiconic languages, including Clojure.
The article talks about homoiconicity, manually creating Abstract Syntax Trees (AST) out of data and the macro environment. Also, compares the difference between macros and functions.
Tags: code=data, data=code, james-donelan, clojure, homoiconicity, abstract-syntax-tree, ast, macro, function
Agile in Practice: Test Driven Development
I’ve watched this video introducing TDD by the Agile Academy. Explains its motivations and the basic cyclical fashion.
Tags: agile-academy, video, tdd, introduction
Agile in Practice: Pair Programming
I’ve watched this video by Agile Academy on practicing pair programming. Explains the destination and how to get there.
Tags: agile-academy, pair-programming, video, introduction
A low cost approach to working on side projects
I’ve read this article by Gregory Brown on how to apply the MVP and Lean Startup principles to side projects (a.k.a. pet projects). Try to start with a brainstorming session of all the features you want your product to have, then prioritise, picking the most important ones.
Most projects' needs can be solved with a minimal version, in the form of a script that took half an hour to develop.
Tags: gregory-brown, mvp, lean-startup, script, proof-of-concept, priority, backlog, side-project, pet-project, low-cost, cost-reduction
Extreme Programming
I’ve watched this video by STAR Videos on the basic principles behind Extreme Programming and how to use it to improve software quality
Tags: star-videos, extreme-programming, xp, software-quality
I’ve read this white paper describing the disruptor framework: a way of exchanging data between concurrent threads.
It uses a ring buffer and other techniques to eliminate write contention, reduce read contention and produces good results. It was developed for a financial exchange environment but it’s general purpose.
Tags: martin-thompson, dave-farley, michael-barker, patricia-gee, andrew-stewart, trisha-gee, disruptor, contention, framework, concurrency, parallelism, white-paper, comparison, arrayblockingqueue, concurrentlinkedqueue, doug-lea, benchmark, lmax, low-level-optimization, optimization
Dissecting the Disruptor: What’s so special about a ring buffer?
I’ve read this article about the ring buffer data structure. A FIFO data structure where elements get overwritten to reduce pressure on the garbage collector. Every message gets sent to every consumer.
Tags: patricia-gee, trisha-gee, disruptor, ring-buffer, data-structure
The Dependency Inversion Principle
I’ve read this paper about the Dependency Inversion Principle (DIP) written Robert C. Martin.
It explains the theory, philosophy behind it. Also, creates an example with a copy buffer and different output locations
Tags: robert-martin, dependency-inversion-principle, dip, example, comparison, object-oriented, principle, solid, design-principle
DIP in the Wild
I’ve read this article about the Dependency Inversion Principle (DIP) written by Brett L. Schuchert.
It is explained non-canonical representations of this principle, how in other cases he has solved it and conclusions of using it day to day.
Tags: brett-l.-schuchert, dependency-inversion-principle, dip, example, object-oriented, solid, design-principle
The New New Product Development Game
I’ve read this paper about the holistic approach to product development, considered seminal for Scrum and other Agile methodologies. Written by Hirotaka Takeuchi and Ikujiro Nonaka
In it, they describe how teams are distributed, the responsibilities for each one, and the degrees of freedom.
Cites the six basic characteristics:
- built-in instability
- self-organizing project teams
- overlapping development phases
- multilearning
- subtle control
- organizational transfer of learning
Tags: hirotaka-takeuchi, ikujiro-nonaka, scrum, agile, product-development, management, 1986