A Work in Progress ...

Reasons to love Python: Validating Dataclasses (Part 1)

2026-10-16T00:00:00+02:00

I love Python. Sure it’s not the fastest language in the world, and the package management can sometimes be painful, but I still love it.

I often reach for Python when developing the scripts that wrap systems together. Read a configuration file, process that data and do something with it.

I find myself reaching for a Pydantic BaseModel but I don’t want to create a virtualenv just for the script (uv script mode nonwithstanding), and so I just make it a dataclass and spend the next 20 minutes fighting type errors in my data.

What if there was a better way? What if I could re-implement just enough of a BaseModel in little enough code that I can just paste it into the script?

Python is marvelous language. It is full of features that feel like they must be intrinsics of the compiler, but if you peek under the hood, the machinery is there, in all it’s sometimes hacky glory, ready to be inspected.

Dataclasses are one of them. Add a few annotations, and hey-presto! The magic is performed and your bare class now has an __init__, a __str__, even __eq__ if you ask for it!

Unpacking YAML into a dataclass

Let’s start with a simple example: we want to define a model, and populate it with data from a .yaml file.

from __future__ import annotations

from dataclasses import dataclass
from typing import TextIO

import yaml


@dataclass(frozen=True)
class Person:
    name: str
    age: int

    @classmethod
    def load(cls, file: TextIO) -> Person:
        return cls(**yaml.safe_load(file))


if __name__ == "__main__":
    with open("person.yaml") as f:
        person = Person.load(f)

    print(person)

So far so good. The script opens the data file, loads it into a dict and uses ** to splat it into Person.__init__, the constructor generated by the dataclass decorator.

$ python load-dataclass.py
Person(name='Thibaud', age=35)

Un fortunately, this only checks that the data file contains the right fields, not that the fields have the correct types. If we add an extra field, Person.__init__ receives an extra argument and we get an exception:

>>> Person(name="Thibaud")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Person.__init__() missing 1 required positional argument: 'age'

And if we are missing a field:

>>> Person(name="Thibaud", age=35, nationality="FR")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Person.__init__() got an unexpected keyword argument 'nationality'

And if we specify all the arguments, but one of them is the wrong type we do not get an error.

>>> Person(name=12, age="None of your business!")
Person(name=12, age='None of your business!')

Validating Scalar Fields

To ensure that the value that we assign to each field has the correct type, we can introspect the dataclass to read out the type hint for each field, and then check that the value has correct type (or can be casted to the correct type).

Python gives us a very convenient get_type_hints function for this that takes care of the nitty gritty details, hands us a dictionary of names, and the associated annotations.

from __future__ import annotations

from dataclasses import fields, is_dataclass
from typing import get_type_hints, Any, Callable, Type, cast


def _coerce[T](
    field_type: Type[T], value: Any, field_name: str
) -> T:
    # If the value has the correct type, move on
    if isinstance(value, field_type):
        return value

    # Otherwise, try to cast it to the correct type
    try:
        # Tell the checker the class accepts the raw value.
        return cast(Callable[[Any], T], field_type)(value)
    except (ValueError, TypeError):
        err = (
            f"Invalid value {value!r} for '{field_name}'. "
            f"Expected '{field_type.__name__}'."
        )
        raise ValueError(err) from None


def hydrate_dataclass[T](cls: type[T], data: dict[str, Any]) -> T:
    assert is_dataclass(cls)
    result = data.copy()
    hints = get_type_hints(cls)

    for f in fields(cls):
        # Skip fields that are not present in the source dictionary.
        # When constructing the dataclass, these will become missing
        # arguments.
        if f.name not in result:
            continue

        result[f.name] = _coerce(
            hints[f.name], result[f.name], f.name
        )

    # Look for fields that are not defined in the dataclass.
    # Allows us to make a better error message than the TypeError.
    unknown = set(data) - {f.name for f in fields(cls)}
    if unknown:
        raise ValueError(f"Unknown fields: {sorted(unknown)}")

    # Build the dataclass from the dictionary and return it
    return cls(**result)

If we run this script with a data file that has all correct types, the dataclass is built as before. If the data file has the wrong types though, we get nice error messages:

>>> hydrate_dataclass(Person, {"name": "Thibaud", "age": "Not saying!"})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "hydrate_dataclass.py", line 29, in hydrate_dataclass
    raise ValueError(err) from None
ValueError: Invalid value 'Not saying!' for 'age'. Expected 'int'.

Converting Enums on the Fly

With what we have so far, we can validate scalar types, but what about enumerations? Can we magically convert values from our input files into the correct enumeration members?

class TeamRole(StrEnum):
    Principal = "Team Principal"
    Driver = "Driver"
    PitCrew = "PitCrew"

@dataclass(frozen=True)
class Person:
    role: TeamRole
    name: str

In python, enumerations are classes derived from the Enum class. A simple is_subclass(x, Enum) check is enough to detect them.

Once we find an enumeration, we look up it’s members, and return the correct enumerator.

def _coerce[T](
    field_type: Type[T], value: Any, field_name: str
) -> T:
    # If the value has the correct type, move on
    if isinstance(value, field_type):
        return value

    if isinstance(field_type, type) and issubclass(field_type, Enum):
        # The enumerator in the data should be a string,
        # and it should be an enumerator on the Enum.
        if (
            isinstance(value, str)
            and value in field_type.__members__
        ):
            return field_type[value]

        # Enumerator not found. Raise an error.
        err = (
            f"Invalid value {value!r} for '{field_name}'. "
            f"Expected one of : {', '.join(field_type.__members__)}"
        )
        raise ValueError(err) from None

    # Otherwise, try to cast it to the correct type
    try:
        # Tell the checker the class accepts the raw value.
        return cast(Callable[[Any], T], field_type)(value)
    except (ValueError, TypeError):
        err = (
            f"Invalid value {value!r} for '{field_name}'. "
            f"Expected '{field_type.__name__}'."
        )
        raise ValueError(err) from None

That’s all it took, one is_subclass check and we’re spitting out enumerations!

>>> hydrate_dataclass(Person, {"name": "Thibaud", "role": "PitCrew"})
Person(role=<TeamRole.PitCrew: 'PitCrew'>, name='Thibaud')

If we try to pass in a value that is not a valid enumerator for the enumeration, we get a descriptive error.

>>> hydrate_dataclass(Person, {"name": "Marcus", "role": "Boss"})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "hydrate_dataclass.py", line 37, in hydrate_dataclass
    raise ValueError(err) from None
ValueError: Invalid 'Boss' for 'role'. Expected one of: Principal, Driver, PitCrew

Loading Nested Dataclasses

Validating one dataclass is a nice start, but sometimes our data is nested, and better represented as a series of nested dataclasses.

@dataclass(frozen=True)
class Person:
    name: str


@dataclass(frozen=True)
class Volunteer:
    identity: Person
    email: str

What happens if we try to load this with our hydrate_dataclass function?

>>> hydrate_dataclass(Volunteer, {"identity": {"name": "Thibaud"}})
Volunteer(
    identity=Person(name={'name': 'Thibaud'}),
    email='hello@example.com'
)

That looks good at first sight. But wait, why do we have name={‘name’: ‘Thibaud’} instead of just name=’Thibaud’?

Our code is smart enough to detect enumerations, and special case them, but it doesn’t (yet) know anything about dataclasses. Since isinstance(value, field_type) returns false, the value is converted by

result[f.name] = field_type(value)

Which is functionally equivalent to running

result[f.name] = Person(name={'name': 'Thibaud'})

When what we need to do is

result[f.name] = Person(**{'name': 'Thibaud'})

It’s an easy fix, we just need to special case dataclasses. Once again, Python gives us exactly what we need: is_dataclass.

def _coerce[T](
    field_type: Type[T], value: Any, field_name: str
) -> T:  # If the value has the correct type, move on
    if isinstance(value, field_type):
        return value

    if isinstance(field_type, type) and issubclass(field_type, Enum):
        # The enumerator in the data should be a string,
        # and it should be an enumerator on the Enum.
        if (
            isinstance(value, str)
            and value in field_type.__members__
        ):
            return field_type[value]

        # Enumerator not found. Raise an error.
        err = (
            f"Invalid value {value!r} for '{field_name}'. "
            f"Expected one of : {', '.join(field_type.__members__)}"
        )
        raise ValueError(err) from None

    if is_dataclass(field_type):
        if not isinstance(value, dict):
            err = f"Invalid {value!r} for '{field_name}'. Expected dict."
            raise ValueError(err)

        return hydrate_dataclass(field_type, value)

    # Otherwise, try to cast it to the correct type
    try:
        # Tell the checker the class accepts the raw value.
        return cast(Callable[[Any], T], field_type)(value)
    except (ValueError, TypeError):
        err = (
            f"Invalid value {value!r} for '{field_name}'. "
            f"Expected '{field_type.__name__}'."
        )
        raise ValueError(err) from None

And with that, we can load nested dataclasses:

>>> hydrate_dataclass(Volunteer, {"identity": {"name": "Thibaud"}})
Volunteer(
    identity=Person(name='Thibaud'),
    email='hello@example.com'
)

The Full Script

In about 50 lines of code, we can validate a fairly useful set of dataclasses. We can’t handle lists, dictionaries, optional values, and unions yet, but they are surprisingly easy to implement.

That will be for a future post.

In the meantime, you can find the full script that we have worked through in this post here:

hydrate_dataclass.py

The case against Makefiles

2026-10-05T00:00:00+02:00

You don’t need a Makefile. Whatever your use-case, there is a better tool.

Makefiles are archaic technology, living on by force of inertia and notalgia. The syntax is opaque and the error messages are unhelpful. It’s time to move on.

As far as I can see, there are two remaining use cases for Makefiles in the wild:

building C and C++ code;
running arbitrary tasks that have dependencies between each other.

Makefiles are no longer the best solution for either of these use cases,

Use-case 1: Building C and C++ code

Makefiles are the original build system. The make program was born in 1976 at Bell Labs, out of a frustration with existing build scripts (piles of sh). The concepts it pionneered (_e.g._ the build graph) are still the basis of modern build systems, but the ergonomics of build systems have improved a lot since then.

Aside: Build scripts, makefiles, and other build systems

Consider a script that builds a pile of source files (without using any fancy shell features that would have been unavailable in 1976)

#!/bin/sh

CFLAGS="-Wall"

# Compile the individual source files to object files
cc $CFLAGS -c -o main.o main.c
cc $CFLAGS -c -o lib.o lib.c

# Link the object files into an executable
cc -o app main.o lib.o

Forget to add a cc command to build a source file? Now you’re linking with an old version of the file that you compiled by hand.

Using modern bash features, you could improve the script significantly.

#!/bin/bash

CFLAGS="-Wall"
SRCS = "main.c lib.c"
OBJS = ${SRCS/.c/.o}

for file in $SRCS; do
        cc $CFLAGS -c -o ${file/.c/.o} $file
done

cc -o app $OBJS

Using a bash variable expansion to replace .c with .o allows us to use re-use the list of sources in both the compile commands and the linker command. For a small example program, this approach works, but it doesn’t scale well.

An equivalent makefile would be

CFLAGS = -Wall
SRCS = main.c lib.c
OBJS = $(SRCS:.c=.o)

app: $(OBJS)
        $(CC) -o $@ $(OBJS)

%.o: %.c
        $(CC) $(CFLAGS) -c -o $@ $<

The similarities are quite obvious. The most significant difference between the script and the makfile is what we describe. The script is an imperative procedure: it lays out the steps that need to be accomplished to build the final binary, and executes them all in sequence, every time. The makefile describes the dependencies between inputs and outputs. This allows the makefile to skip steps that are redundant because the input has not changed.

The cost of this change is syntax. The build script is quite readable but the makefile is full of _magic_ tokens. What are $@, and $<? To an experienced practitioner, they are obvious. For me, they require a trip to google each time (or if I actually remember, a trip into The Sum of All Knowledge, my knowledgebase in`Obsidian`_).

Compare this with an equivalent CMake script

cmake_minimum_required(VERSION 3.10)
project(app C)

add_executable(app main.c lib.c)

Without knowing anything about CMake, the goal of the script is quite clear. The knowledge of how to actually build an executable or a library is hidden which is a shame, but the intent is clear.

Make’s mistake

The fundamental mistake of Make is that it describes the edges of the build graph, not the nodes. It describes the operations required to transform one file into another, but never describes the goal of the transformation.

%.o: %.c
$(CC) $(CFLAGS) -c -o $@ $<

To build a file, we say that there exists a node %.c and a node %.o, and that to go from %.c to %.o, it is necessary to apply $(CC) $(CFLAGS) -c -o $@ $< to %.c. A no point does this say anything about what we are trying to achieve.

In CMake, to build an executable, we use add_executable(app main.c lib.c). This describes the end goal, not the transformation. This is not to say that CMake is the perfect build-system, far from that, but it is at least easier to follow.

Build-systems that describe intent, and not transformations, have one property that most Makefiles in the wild lack: they work reliably when run in parallel. Most makefiles I have had the misfortune of encountering fail when run in parallel. To build reliably, build scripts become

make -j$(nproc) all || make -j$(nproc) all || make -j$(nproc) all

This is definitely a skill issue. But a tool too complicated for the average C programmer inside a boring company is no longer a good tool.

Re-inventing the wheel with make

Make aficionados love to tell new victims how great their makefiles are: they are invariably better than whatever sub-par build system they have begrudgingly accepted to use at their workplace.

However, if my experience is representative, their makefiles still suffer from the edges-not-nodes issue discussed above, but more problematically: they have accumulated just as much complexity as other build systems, whilst having written none of the documentation that would be required to explain their bespoke build system to others.

Need to add a new library target? Need to add a custom target for code-generation, doxygen, a formatter, a linter, … ? jump on a call and ask them to walk you though it.

And after all that, you still end up rm -rf build/ at every other build because you don’t trust the damn thing …

And because it doesn’t actually work when run in parallel, the compile times are agonising …

Use case 2: Running tasks

The concept of a dependency graph generalises well to many other problems, not just compiling code.

Using a makefile to express that the publish-docker rule depends on the build-docker rule seems natural. Sure you have to mark the rule .PHONY to tell make that the outputs are not actual files … but it works.

.PHONY: publish-docker
publish-docker: build-docker
        docker image push my-image:latest

.PHONY: build-docker
build-docker:
        docker image build -t my-image docker/my-image

Sure, it took you a couple of tries because you forgot to change your editor settings to use tabs for indentation instead of spaces, but it works.

But now you have a second image you want to build. You copy the rule, but feel icky. The whole point of using a makefile was to avoid this kind of duplication.

.PHONY: publish-docker-foo
publish-docker-foo: build-docker-foo
        docker image push my-foo-image:latest

.PHONY: build-docker-foo
build-docker-foo:
        docker image build -t my-foo-image docker/my-foo-image

.PHONY: publish-docker-bar
publish-docker-bar: build-docker-bar
        docker image push my-bar-image:latest

.PHONY: build-docker-bar
build-docker-bar:
        docker image build -t my-bar-image docker/my-bar-image

Instead of copying the rule, you’d like to re-use and somehow pass the image name as a parameter … and now you’re stuck. A make wizard would spit out a magic command that generates the rules with more make magic but how maintainable is that?

You’ve hit the limits of make. It isn’t made for this. You need an actual task runner. Something that doesn;t think in files, but in actual tasks, that might have arguments. There are many out there, but my personal go-to is just. The equivalent justfile is

build-docker-foo: (build-docker "my-foo-image")
publish-docker-foo: (publish-docker "my-foo-image")

build-docker-bar: (build-docker "my-bar-image")
publish-docker-bar: (publish-docker "my-bar-image")

publish-docker {{ IMAGE_NAME }}:
        docker image push {{ IMAGE_NAME }}

build-docker IMAGE_NAME:
        docker image build -t {{ IMAGE_NAME }} docker/{{ IMAGE_NAME }}

Isn’t that better? Out-of-the-box we also get a list of rules (and a documentation comment for each one) with —list, the ability to group them in the output, the ability to import other justfiles without recursive makefile hacks, the ability to execute the body of a rule as a script (no need to add a \ at the end of each line like in a makefile) …

It’s just a better tool. There are plenty of others out there that might fit your aesthetic better, just don’t use make for this.

What Makes a Job Worthwhile

2026-06-26T00:00:00+02:00

We spend 8 hours a day working. For software engineers, currently in the enviable position to pick their workplace, instead of needing to accept whatever job offer that comes their way, the question is: what makes a job worthwhile?

What makes a job something worth dedicating half of our waking hours (and a larger portion of our productive hours) to?

The answer to the question is deeply personal.

For some, the comfort of a stable paycheck, and the devil you know, is a a sufficient motivation to stay in place. For others, the potential for professional growth, or career growth are more important.

Having seen varying levels of dysfunction throughout the years, I think that, for me, the answer boils down to

the potential for growth;
the potential for change;
feeling valued as an individual.

The potential for growth

Growth is an overloaded term, and what matters most will change from person-to-person. Is growth personal growth, like improving your communication, or conflict management skills? Is growth professional growth in the sense of getting better at your job, developing a good intuition for how to solve problems, or create maintainable systems? Is it about professional growth: seeing your job title change from junior, to senior, to manager, … Or is it a completely different axis?

The meaning of growth, and what actually matters to me, has changed oer time. As a junior, the most important thing was learning. Now, growth tends towards helping others improve.

In unstructured environments, it’s been easy to pull a junior under my wing, and watch them grow. More structured environments make this harder: everybody has their own work, nicely siloed up, making it harder to understand other people’s work well enough to help.

The Potential for Change

All code is legacy code. The moment code gets merged into the main branch, it becomes legacy code that needs to be supported. The most important thing from that point on is whether or not the processes allow for incremental improvement on the existing code.

Refactoring and bug-fixes should not be work that is snuck in under the pretense of doing feature development. More broadly, as development practices evolve, the processes around the code should change.

For example, it seems unthinkable, in 2026, to start a new project without code-formatters, linters, continuous-integration, … So why is still so common to see older projects that lack these basic building blocks of software quality?

Environments that refuse to make the time to modernise processes make simple maintenance harder, and make it easier to introduce errors. I never thought that I would see a project on which “Who broke the build?” is still a common question.

Hearing a colleague, explain to me, with a straight face, that “the industry has moved away from code reviews because they add no value”, was a shock.

Learning that the we cannot prohibit direct pushes to the master branch because an older colleague would not accept code reviews, was unpleasant.

If there is potential for change, the situation can be salvaged. But if current practices are stuck in stone, staying in place is equivalent to slowly de-skilling myself.

Without code reviews, I can only learn from my mistakes if I stay long enough to see their effects.

With code reviews, I can learn from the mistakes my colleagues have made in the past.

With linters, I can learn from my entire industry’s mistakes.

Without any of those, I am stuck in place, slowly being losing ground to others, in more progressive environments.

Feeling Valued as an Individual

Companies will preach values, and forget them the moment the slideshow is over. I’ve seen companies praise their efforts to create a friendly office environment, simultaneously not intervene when a senior engineer repeatedly threatened to throw me out of the window, in a I’m joking but deep down I really mean it kind of way.

I’ve seen companies explicitly say that they hire juniors and foreigners because they’re inexpensive, and that they have no intention of keeping them longer than two years.

When I found a company that paid above market rate for talent, and offered reasonable perks to retain the staff they had, I was skeptical. In practice, they seemed to put effort into improving the workspace, improving the working environment, and motivating employees to stay,

When I reached out to HR, they said and did the right things.

When employees wrote an open-letter to management, they said and did the right things.

So maybe they really cared.

Yet here I was, scrolling Linkedin jobs, daring myself to click that Easy Apply button again.

What makes it all worthwhile?

Nostalgia is rose-tinted glasses, but some of the most fun I’ve had has been in a workplace where the software team was a small island of sanity, surrounded by socially inept project managers, and where a significant part of the staff was somewhat incompetent. Some of the most soul-crushing work I’ve had, was in a worksplace where the work was technically challenging, but the colleagues were so distant that, other than their names, I knew basically nothing about them.

Is there a workplace that ticks all three boxes? Is that actually a pre-requisite for me to enjoy my time there long-term?

I still don’t know, and leads me back to the age-old question: is the grass greener somewhere else?

Finding Meaning in Work

2026-06-21T00:00:00+02:00

Given the amount of time and energy we expend in our workplaces, it makes sense to look for some kind of meaning in the work we do. We cannot all be Jonas Salk (inventor of the Inactivated Polio Vaccine), or Frederick Banting (first to isolate insulin, and origin of the Miracle at the Toronto General Hospital), but we can at least hope that our work is making the world a slightly better place.

However, if I were to judge the output of my career so far, I’m not sure that any of it has contributed in a meaningful way to improving anything.

Working in Academic Research

My career started in an academic research lab, working on medical robots (think Da Vinci Surgical System). The project aimed to improve patient outcomes by adding virtual walls to the robot workspace to protect important anatomical structures. This required precise registration of pre-operative imagery with the operating field.

This registration was impossible. The pre-operative images were MRI scans, taken with the patient lying horizontally, but in the operating theatre, the patient was tilted about 10° downwards (with their head lower than their feet), and their abdominal cavity was inflated. This pushed all the organs away from their natural positions, and gave the surgeons more room to operate. It also made the pre-operative imagery useless since everything had moved.

After a year, we parted ways.

Working in EU Projects

My next job was for a company that made most of its money on European Union Funded projects, and had never made a single product in its entire lifetime. The organisation would bid on robotics projects, hack together a solution that worked just well enough to make a demo, and then move on to the next project. The strategy worked well, but the lab was full of abandoned prototypes of things that might have been viable products.

After four years of never finishing anything, I grew frustrated, and was not-so gently shown the door.

Working in the European Space Agency

I wanted to work somewhere where there was an actual long-term goal, and I thought I had found it when I joined the European Space Agency. The lab had an illustrious history of ground breaking work, but by the time I joined, it was rudderless. The project that was driving the work was at best an engineering demo, not a serious research goal.

Within 18 months, I was gone.

Working in Industry

I was done with research. I wanted real products, real customers.

I joined a small company making components for small satellites (less than 50 kg). I had high expectations, and was quickly disappointed. The company had survived on a steady diet of juniors, left without guidance, destined to repeat the mistakes of their predecessors. The code was cobbled together by electrical engineers, as a means to an end, mashing the keyboard until they thought it worked.

The firmware was brittle, assumed that everything would forever stay on the happy path, had no tests, or standards, or even a hint of good practices. Repositories were usually not up to date, and picking up a new project usually meant asking the head engineer to search through his virtual machines for his uncommitted changes. The existence of an On Orbit Hardware-In-The-Loop Test frame in the communication protocol said a lot about the overall lack of quality. [1]

To some extent, I am proud of the work I did there. Every day I would work on hardware that was on its way to space! I did my best to impose standards in the codebase, but I wonder how long they persisted after my departure.

But was the work actually useful?

The products we worked on were of questionable quality. One in-development product could not use the external oscillator for the microcontroller clock. The engineer who had designed it was long gone, and the project was too late to respin the board. As a result, the control loop could only use 0.5 % of the available control action.

The customer, who bankrolled the entire development effort, would have been better off killing the project, and buying off the shelf hardware from somebody competent.

Working in Defense

After moving to a new country, I decided to change industry, and stepped into the world of defense. Learning little from my past experience, I had high expectations.

Aside from the habitual bureaucratic incompetence of large organisations, I was struck with confusion over the goals of the team. Had anybody asked me what the priorities for the next quarter, or the next year were, I would have been at a loss. Had anybody asked me what the most pressing issues with the hardware were, I would have been similarly stumped.

As I tried to onboard onto the project, trying to make a mental model of this undocumented monolith, I struggled to see how I would make an impact. I was hired with lofty goals: to become a bridge between their grumpy senior engineer and the junior programmers tasked with making his vision reality, but I was left chasing down the typos of a prolific, but clearly dyslexic programmer.

I entered the defense industry looking for purpose. I expected to be able to relate my work directly to current geopolitical priorities, but by the time the work landed on my plate, it had been stripped of all context, and purpose was lost with it.

What if the Journey was the Purpose?

Looking back at the last 10 years of my life, I could not point to a single project I worked on that improved the world around it in any measurable way.

As I pondered this unpleasant thought, another one came into focus. I have always enjoyed teaching, and I had the opportunity to mentor quite a few interns and juniors along the way.

Even interns whom I thought I failed, seemed happy with what they learned working with me. Juniors I mentored grew into engineers I would hire again in a heartbeat.

If I failed to have a positive impact through my projects, maybe they would.

On Disappointment

2026-06-19T00:00:00+02:00

Every new job comes with some amount of trepidation: how dysfunctional will this one be? Yet, I was excited about this one. I was moving back into aerospace: the land of exacting standards, and attention to detail. Having seen the chaos of New Space, I wanted the calm competency of Old Space.

First Contact

The hiring process had been slow, but fair. The interviews had been challenging and interesting, so I was quite eager to start.

By the end of the first morning, I had a laptop (Windows unfortunately), and login information for outlook and teams. A low bar, but one many companies fail to meet. Laptop in hand, I went to my new manager, and things took a turn for the worse.

“You’re going to read documentation for the first month” he said.
“No, I’m not!” I responded, reflexively.

He assigned me documents to read, and I hit my first roadblock. I did not have access to the document management system, and could not access any of the assigned documents.

Things did not get better after that. After a week of reading increasingly repetitive and irrelevant documentation, my brain was melting: I could no longer focus on the handful of remaining pages. As my gaze passed over them, my eyes glazed up, and the content passed through me without ever encountering a conscious thought.

I begged for access to GitLab to at least take a look at the software for which I was reading the documentation. I was met by a certificate warning. I tried to clone the repository, and spent the rest of the afternoon looking for the root-certificate and installing it.

When I finally looked at the code, I was struck by cursed formatting styles, applied haphazardly, and without any uniformity.

This was not the high-quality codebase I hoped to see.

My first month crawled by: I was required to come to the office every day, and would sit in front of my computer, glumly, silently, only meeting colleagues at lunch, before sitting in traffic for an inordinate amount of time on my commute home.

I was miserable.

Somehow, it gets worse

With my first month up, I was assigned my first real work. Software testing!

Aerospace is big on paperwork, and one big chunk of paperwork is proving that you did enough testing. Important systems require 100% unit-test coverage (or more accurately: require you to provide some kind of documentation showing that you have 100% test coverage). The quality of tests themselves is not examined, just the number.

Once those pass, a bevy of integration tests are required, showing that core requirements are addressed.

In a reasonable environment, this would be a handful of automated test runs. Run a handful of commands, get the logs, and attach them to the report. In a well-oiled system, the tests would be run automatically by the continuous-integration as part of a tag pipeline, and the certification work would be limited to downloading the automatically-generated test report.

This was not the case. Two weeks were budgeted for me to learn how to run the tests, followed by two weeks to actually run them. Another month of my life, commuting to work every day, for a job that had explicitly advertised 50% home-working.

This may have been the most difficult period of my entire professional career to that point. The steps to run the manual tests were often wrong, or incomplete. Worse, many of the tests had unclear pass/fail criteria, and in many cases there was no way to differentiate between a test that passed successfully, and a test that had been executed incorrectly.

The most egregious test though were the automated tests. Hours-long test suites, that failed to check any pre-conditions. Forget to run a service: half of the tests will fail with obscure error messages, for something that should have been detected in the first 5 seconds.

I was going insane.

Somehow this agonising period also came to end, and I was promised my first real project. I was hopeful again.

All hope is lost

I cloned the repository, ran the build commands in the README, and was immediately stopped by a wall of errors.

The project had been entirely staffed by external contractors, working more-or-less independently, until that point. I was the first internal person to look at the code in a year.

I asked questions, found some answers, and eventually got the project building, but somehow, nobody was able to tell me, in any useful detail, what the short-term goals were, what was actually working, or not working, or even find references for the core algorithms that were required for the project.

As I looked deeper, I only grew more worried, the documentation was scarce, and most of what I could see was a half-baked, under-development, in-house framework, for … message passing between threads?

Where was the actual meat of the project?

Where were the tests?

Where was the continuous integration?

Where was the documentation?

I grew more concerned.

I dug in some more, found some trivial race conditions.

I pushed deeper, and started to notice a total lack of error handling.

This was not what I expected. Instead of a carefully architected system, designed with the expertise and caution most people would associate with an aerospace company, what I had in front of me was an unnecessary framework, rapidly growing more warts as the core design problems became more apparent.

Here I was, on a project staffed by contractors, very much on the outside. Desperately trying to figure out what was going on, without even having access to the issue tracker, or an invite to the weekly progress meeting.

What the f… was I supposed to do now?

Pushing Through

Not one to give up easily, I rolled up my sleeves and tried to understand the writhing monster of a framework, growing every week, with undocumented, and untested code. At each refactor, the framework resisted, mutating under my fingers: a new push, attempting to fix another issue that should have been apparent from the start, haphazardly adding parameters to function I had removed.

Nobody wants to throw away a year of work, but faced with the realization that understanding undocumented code is harder than writing it from scratch, and absent any signs of an actual grand vision for the software architecture, maybe that would be the best path forwards.

I groaned silently. That would be a hard sell.

Ensure that Git Submodules are Initialized

2026-03-10T00:00:00+01:00

You’ve checked out a new repository, try to build it, and are confronted with a wall of errors.

The CI is green and your colleagues have assured you that all the dependencies are included in the repository. Why is the build failing?

You scratch your head — why is your environment different?

You look through the CMakeLists.txt, do some grepping, but you can’t find those missing dependencies. Puzzled you look at the README, and there you see it, you forgot to clone the submodules, …

Facepalm …

After making this mistake when cloning a new project too many times, I resolved to find a solution.

The idea is simple, check if the project has submodules, and make sure that each submodule is properly initialised.

In git we can list submodules with git submodule. Helpfully, the command tells us the state of each submodule.

$ git submodule status

 -xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx path/submodule_path (branch)
 +xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx path/submodule_path (branch)
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx path/submodule_path (branch)

A line starting with - denotes a submodule that is not checked out yet, a line starting with + denotes a submodule where the currently checked out commit does not match the hash of the commit from the .gitmodules file (the submodule has changed, but the parent repository has not been updated).

Enter EnsureSubmodulesInitialised. Just include it in your main CMakeLists.txt, and if you ever forget to initialise a submodule, you’ll get a nice helpful error message.

# Ensure that all submodules have been initialised.
#
# This is helpful in repositories that include dependencies as
# submodules since it is easy to forget to initialise them on
# clone, or to forget to initialise a new dependency after a pull.

function(ensure_submodules_initialised)
    # Check if the submodule is initialized.
    #
    # If the repository was cloned without `--recursive`, the
    # submodule directory will be empty. If it has been
    # initialised, it will contain a `.git` directory. We use
    # it's present as a marker to ensure that the submodule has
    # been initialised.

    execute_process(
        COMMAND git submodule
        OUTPUT_VARIABLE _git_submodule_status
        OUTPUT_STRIP_TRAILING_WHITESPACE
        WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}
        COMMAND_ERROR_IS_FATAL ANY
    )

    # Replace newlines with `;` to make the result into a list
    string(REGEX REPLACE "\n" ";"
        _git_submodule_status "${_git_submodule_status}")

    set(_has_uninitialised_submodule FALSE)
    foreach(_submodule_status_line ${_git_submodule_status})
        # examples of status lines
        #
        # -ac9dac21a68eacaa826c69a3b42d0aa69a3c68cf
        #      path/submodule_path (branch)
        # +ac9dac21a68eacaa826c69a3b42d0aa69a3c68cf
        #      path/submodule_path (branch)
        #  ac9dac21a68eacaa826c69a3b42d0aa69a3c68cf
        #      path/submodule_path (branch)
        #
        # A `-` in the first position indicates that the
        # submodule is not initialised.

        # Remove the leading space. It will only complicate
        # parsing for us.
        string(STRIP ${_submodule_status_line}
            _submodule_status_line)

        # Make the string into a list so that we get easier
        # access to the different parts
        string(REPLACE " " ";"
            _submodule_status_line ${_submodule_status_line})
        list(POP_FRONT _submodule_status_line
            _submodule_hash _submodule_path _submodule_branch)

        string(FIND ${_submodule_hash} "-" _submodule_not_initialised)
        if("${_submodule_not_initialised}" EQUAL 0)
            message(STATUS "Submodule ${_submodule_path} is not initialised.")
            set(_has_uninitialised_submodule TRUE)
        endif()
    endforeach()

    if(_has_uninitialised_submodule)
        message(FATAL_ERROR
            "Some submodules were not initialised. \n"
            "Run 'git submodule update --init --recursive' to "
            "initialise them."
        )
    endif()

endfunction()

It Wasn’t Making me Happy - On Changing Jobs

2026-03-06T00:00:00+01:00

I started a new job in January, and left in February. It wasn’t making me happy.

It started like any other interview process: a video interview, followed, a few weeks later, by a site visit. They showed me their hardware, talked about the challenges they were facing, how they were trying to improve precision, trying to extend their system to do new things, reacting to market needs. I was impressed.

It felt like they had an appreciation for quality.

It felt like they had hard engineering challenges to solve.

I felt like I might find my people there.

I was optimistic.

Feeling small

Most companies like to say that their biggest strength is their people. Few can actually articulate how they ensure that they keep the best people, but most at least try to give the impression that they try by having free coffee, or free snacks, or free something (as long as it’s cheap).

I had never worked anywhere that actively worked to give me the opposite impression.

The parking is a vacant lot, next to a gutted-out building, covered in shards of glass.

It took 4 days to get a laptop and an email login.

The laptop came with a pair of monitors. Two 1080p slabs, with pixels large enough to count.

The docking station was a USB-C hub from a no-name Chinese brand.

I asked for a QWERTY keyboard without a numpad. It was denied. The resulting email change went all the way up the org chart.

But most damning, the laptop didn’t charge over USB-C.

Feeling lost

Starting in a new team means discovering a new codebase. Somebody points you at some out-of-date documentation full of wishful thinking and high hopes, and you’re off spelunking.

Unfortunately, reading is all I could do. The hardware is large, and there’s only a handful of units to use. There is no simulator for the system, so I could not actually run the code, and the tests are limited; large parts of the codebase are too tightly coupled to be easily unit tested.

If I can’t run the code, I can at least study it. If only I could figure out what the relevant parts of it are. The class names are unhelpful, and the code itself is mostly lacking documentation. I stare at my open editor tabs: 5 of them are called Controller. All of them do different things.

I ask for whatever documentation there is, and I am met with embarrassed smiles.

This codebase is hundreds of thousands of lines, takes a good 10 minutes to build on my beefy development machine, and I have no idea what it even does.

I’m lost, I wasn’t expecting a reverse-engineering job.

Feeling Unimpressed

My first task, and the only code I successfully merged in my time there, was easy enough, take a pattern from one file, and apply it to another. Add some unit tests. Watch the CI go green.

My second task was a simple change, it amounted to adding a single line to a function.

A good first issue on its face.

After some time finding my bearings, I add the required line, and try to add a unit test.

I failed.

Oh yeah, that part of the code can’t be unit tested. We’ll just have to test on the actual hardware.

We go to test the change on actual hardware.

Connect the special cable to the special connector, grab the ssh key to log on to the onboard computer, overwrite the binary on the target system, and … SEGFAULT.

Oh that happens every now and then. Look at the backtrace and just comment that line out.

I look on bewildered and confused. This isn’t the highly resilient codebase I was expecting.

I duly do as I’m told, find the line, comment it out, rebuild, reflash, and … the system still fails to initialise.

We look at the logs, scroll up, scroll down, scroll back up some more, ponder …

After a brief moment, a pronouncement:

The motors are acting up. There’s nothing more we can do today.

We make plans to return another day to test on another system.

A few weeks pass before we try again. This time, the code runs and the station boots. When we change a configuration file to see if the fix I had implemented works, another error comes up. Another subsystem is broken by the configuration change, my code never even ran.

I look on confused. Another pronouncement:

The fix won’t work. This issue can only be handled at a much lower level.

I drop my changes. The best I can do now is merge the cleanups I did.

I am lost, but, more than anything, I feel helpless. How am I supposed to become productive in this code base?

I do not understand what the code actually does.

I cannot even describe the high-level architecture of the code.

I cannot even list out the main components of the code.

I’m totally lost. Fuck.

Feeling Powerless

If I can’t land any changes to the actual software, I can at least try to improve the process. One of the quick wins would be to get people to actually write documentation.

In the past, I’ve had good success with a merge request template that simply asks if the change requires new documentation.

When creating a new merge request, you now have to explain why the documentation does not need to be updated. Quite a few times, as I was writing the rationale, I realised what documentation actually needed to be updated.

The file, including comments, is about 20 lines long.

That should be a quick win.

I bring it up with the team lead, and he asks me to demo it to him. He’s enthusiastic, and sends an email to the other team leaders asking them for their opinions.

Nobody answers.

It dies there.

The merge request that actually builds the documentation with doxygen, dead too.

The change to actually run the formatter in the CI pipeline. Also dead.

How dispiriting.

Making Up My Mind

Career progression is about learning. Every job I’ve had so far, even the ones that did not make me happy, had opportunities to learn more about the craft of software, the stack, or the culture.

I was hoping to see a quality-focused environment, obsessively eking out the last drips of performance from the control loops, but seeing the hardware judder around killed that notion.

I was expecting to find a culture of quality, striving to create a reliable system, but seeing the system crash on a regular basis absolved me of those expectations.

I left every day, more and more confused as to how I would ever become productive in the code base, and what I had to learn from it.

The decision was made, I would find something else.

Model-View-ViewModel: Structuring UI Code

2026-03-01T00:00:00+01:00

Your user interface started off clean, but now the code is a rat’s nest of special cases, and no two parts of the UI can agree on what the data they should be displaying is?

Maybe you need MVVM, or Model-View-Viewmodel.

MVVM is an application design pattern often used in user‑interface programming. It provides a structure for organising applications that is particularly useful when multiple views need access to the same underlying data.

MVVM defines the following components:

Model: Provides access to the underlying data that should be displayed. This can be a direct reference to the data, or mediated by a data‑access‑layer (e.g. an ORM).
View: Responsible for actually drawing the contents onto the screen. It displays a representation of the Model, receives user-input, and forwards the handling of these events to the ViewModel.
View Model: The ViewModel provides an interface to access and mutate the Model.

The Problem - Multiple Views bind to the same Model

Imagine an application in which we need to display a Person in multiple places.

In a naive approach, where each View directly references the Model, there is no easy mechanism for a View to notify other Views of the same Model that it changed.

Consider a TUI application (in this example, written with textual), that displays the same Person in two different PersonView widgets.

When pressing i (a keybind registered on the PersonView), the focused PersonView is updated, but the other one is not. The data is out-of-sync: one view shows Alice, age 34, and the other shows Alice, age 30.

#!/usr/bin/env -S uv run
# /// script
# dependencies = ["textual"]
# ///

from dataclasses import dataclass
from textual.app import App, ComposeResult
from textual.widgets import Static, Footer
from textual.binding import Binding


@dataclass
class Person:
    name: str
    age: int


class PersonView(Static):
    BINDINGS = [Binding("i", "increase_age", "Increase Age")]
    can_focus = True

    def __init__(self, person: Person):
        super().__init__()

        self.person = person
        self.update()

    def update(self) -> None:
        super().update(f"{self.person.name}, age {self.person.age}.")

    # Called when `i` is pressed, must manually update the view.
    # Does not have a mechanism to notify other views of the change.
    def action_increase_age(self):
        self.person.age += 1
        self.update()


class PersonApp(App):
    CSS = """
    PersonView {
        text-align: center;
        content-align: center middle;
        height: 1fr;
    }
    """

    # Creates two ``PersonView`` over the same ``Person`` instance
    def compose(self) -> ComposeResult:
        person = Person(name="Alice", age=30)
        yield PersonView(person)
        yield PersonView(person)
        yield Footer()


if __name__ == "__main__":
    app = PersonApp()
    app.run()

A PySide application that displays the same Person object in two different PersonView objects.

When the increase_button on one PersonView is pressed, the age on the underlying Person object is increased, but there is no mechanism to notify the other view. The age shown there never changes.

#!/usr/bin/env -S uv run
# /// script
# dependencies = ["PySide6"]
# ///

from dataclasses import dataclass
from PySide6.QtWidgets import (
    QApplication,
    QMainWindow,
    QWidget,
    QVBoxLayout,
    QHBoxLayout,
    QLabel,
    QPushButton,
)
from PySide6.QtCore import Qt
import sys


@dataclass
class Person:
    name: str
    age: int


class PersonView(QWidget):
    def __init__(self, person: Person):
        super().__init__()
        self.person = person

        layout = QVBoxLayout()

        self.label = QLabel(
            f"{self.person.name}, age {self.person.age}."
        )
        self.label.setAlignment(Qt.AlignmentFlag.AlignCenter)
        layout.addWidget(self.label)

        increase_button = QPushButton("Increase Age")
        increase_button.clicked.connect(self.increase_age)
        layout.addWidget(increase_button)

        self.setLayout(layout)

    def increase_age(self):
        self.person.age += 1
        # Only this view updates - the other view won't!
        self.update_display()

    def update_display(self):
        # Must manually update the view
        self.label.setText(
            f"{self.person.name}, age {self.person.age}."
        )


class PersonApp(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Person Model-View Example")

        self.person = Person(name="Alice", age=30)

        central_widget = QWidget()
        main_layout = QVBoxLayout()

        # Create two views bound to the same person
        views_layout = QHBoxLayout()
        self.view1 = PersonView(self.person)
        self.view2 = PersonView(self.person)
        views_layout.addWidget(self.view1)
        views_layout.addWidget(self.view2)

        main_layout.addLayout(views_layout)

        central_widget.setLayout(main_layout)
        self.setCentralWidget(central_widget)
        self.resize(400, 250)


if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = PersonApp()
    window.show()
    sys.exit(app.exec())

If we increment the age, the widget is not automatically re-drawn. The developer must manually update the widgets each time the values change. This is error prone.

The Solution - A View binding to a ViewModel

A ViewModel can provide an interface between the Model and the View that ensures that when the Model is updated, the View reflects this change.

Textual and pyside provide slightly different mechanisms for this.

The ViewModel binds to the Model and exposes it’s attributes as reactive attributes. Other widgets can be notified when the change with self.watch(Widget, reactive_attribute, callback).

#!/usr/bin/env -S uv run
# /// script
# dependencies = ["textual"]
# ///

from dataclasses import dataclass
from textual.app import App, ComposeResult
from textual.reactive import reactive
from textual.widgets import Static, Footer
from textual.widget import Widget
from textual.binding import Binding


@dataclass
class Person:
    name: str
    age: int


class PersonViewModel(Widget):
    age: reactive[int] = reactive(0)

    def __init__(self, person: Person):
        super().__init__()
        self.person = person
        self.age = person.age

    # Called automatically when self.age changes
    def watch_age(self, new_age: int) -> None:
        self.person.age = new_age

    def increase_age(self):
        self.age += 1


class PersonDisplayView(Static):
    BINDINGS = [Binding("i", "increase_age", "Increase Age")]
    can_focus = True

    def __init__(self, view_model: PersonViewModel):
        super().__init__()

        self.view_model = view_model
        self.update()

    def update(self) -> None:
        super().update(
            f"{self.view_model.name}, age {self.view_model.age}."
        )

    # Called automatically when widget is added to the app
    def on_mount(self) -> None:
        self.watch(self.view_model, "age", self.update)

    def action_increase_age(self):
        self.view_model.increase_age()


class PersonMVVMApp(App):
    CSS = """
    PersonDisplayView {
        text-align: center;
        content-align: center middle;
        height: 1fr;
    }
    """

    def compose(self) -> ComposeResult:
        person = Person(name="Bob", age=25)
        view_model = PersonViewModel(person)
        yield PersonDisplayView(view_model)
        yield PersonDisplayView(view_model)
        yield Footer()


if __name__ == "__main__":
    app = PersonMVVMApp()
    app.run()

PySide uses a more verbose mechanism, that functions in conceptually the same way. The ViewModel binds to the Model and exposes it’s attributes as @Property (note the capital P).

When the Property is changed, the associated signal is triggered (via the notify= attribute of the Property).

The Views subscribe to changes by connecting to the Signal. When the Signal is triggered, the callback passed in the connect call is invoked.

#!/usr/bin/env -S uv run
# /// script
# dependencies = ["PySide6"]
# ///

from dataclasses import dataclass
from PySide6.QtWidgets import (
    QApplication,
    QMainWindow,
    QWidget,
    QVBoxLayout,
    QHBoxLayout,
    QLabel,
    QPushButton,
)
from PySide6.QtCore import Qt, QObject, Signal, Property
import sys


@dataclass
class Person:
    name: str
    age: int


class PersonViewModel(QObject):
    name_changed = Signal(str)
    age_changed = Signal(int)

    def __init__(self, person: Person):
        super().__init__()
        self.person = person

    @Property(str, notify=name_changed)
    def name(self):
        return self.person.name

    @name.setter
    def name(self, value: str):
        if self.person.name != value:
            self.person.name = value
            self.name_changed.emit(value)

    @Property(int, notify=age_changed)
    def age(self):
        return self.person.age

    @age.setter
    def age(self, value: int):
        if self.person.age != value:
            self.person.age = value
            self.age_changed.emit(value)

    def increase_age(self):
        self.age = self.person.age + 1


class PersonDisplayView(QWidget):
    def __init__(self, view_model: PersonViewModel):
        super().__init__()
        self.view_model = view_model

        layout = QVBoxLayout()

        self.label = QLabel(
            f"{self.view_model.name}, age {self.view_model.age}."
        )
        self.label.setAlignment(Qt.AlignmentFlag.AlignCenter)
        layout.addWidget(self.label)

        # Add increase age button to each view
        increase_button = QPushButton("Increase Age (i)")
        increase_button.clicked.connect(self.view_model.increase_age)
        increase_button.setShortcut("i")
        layout.addWidget(increase_button)

        self.setLayout(layout)

        # Connect to view model changes - views automatically update!
        self.view_model.name_changed.connect(self._update_display)
        self.view_model.age_changed.connect(self._update_display)

    def _update_display(self):
        self.label.setText(
            f"{self.view_model.name}, age {self.view_model.age}."
        )


class PersonMVVMApp(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Person MVVM Example")

        person = Person(name="Bob", age=25)
        person_view_model = PersonViewModel(person)

        # Create two views bound to the same view model
        central_widget = QWidget()
        main_layout = QVBoxLayout()

        views_layout = QHBoxLayout()
        views_layout.addWidget(PersonDisplayView(person_view_model))
        views_layout.addWidget(PersonDisplayView(person_view_model))

        main_layout.addLayout(views_layout)

        central_widget.setLayout(main_layout)
        self.setCentralWidget(central_widget)
        self.resize(400, 250)


if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = PersonMVVMApp()
    window.show()
    sys.exit(app.exec())

Now, instead of incrementing the age directly on the Model (which knows nothing about the user‑interface library we are using), we increment the age on the ViewModel, which can inform other interested parties of the change.

Our UI is now sync.

You Should: Make Meringue

2026-02-22T00:00:00+01:00

You really should be making Meringue. It’s bloody easy, and will make you popular with whomever you share them with (friends, family, colleagues, …). I’d show you a picture, but I ate them all already.

You need, egg whites (as many as you have), and 50 g of white sugar per egg white.

Put the egg whites in a bowl, and mix them with an egg beater. Once the whites start to become firm, add half of the sugar. Once they are firm to add the rest of the sugar. The mix is firm enough when you can flip the bowl upside down without the mix falling out.

Take a baking tray, and put a sheet of baking paper on it. If you have a piping bag, pipe out nice circular meringues, otherwise, use a spoon to drop dollops onto the sheet.

Stick the tray in the oven at 100 °C and leave them there for at least 3 hours. You don’t need to pre-heat the oven, they are staying in there so long that it won’t make a difference.

Meringue isn’t really cooked. It’s dehydrated. That’s why it needs such a low temperature. You just need to sweat them until there isn’t a drop of moisture left in them.

Debugging Your Failures

If your meringue is sticky, you just didn’t cook them long enough. They need to cook for a long time because you want to get absolutely all the moisture out of them. Just stick them into the oven again for an hour.

This even works to revive your meringue if it’s been left out for a while and has absorbed humidity from the environment. Just stick them back in the oven to dehydrate them some more.

If your meringue is yellowish, the temperature was too high. Next time, lower the temperature.

You Should Add FeatureSummary to your CMake Projects

2026-01-13T00:00:00+01:00

Here’s a common problem: your cmake build is failing with linker errors, and now your’re looking through your CMake logs and reading the compilation database to see what libraries were actually included.

What if I told you that there was a better way? Enter Feature Summary.

Basic Usage

The FeatureSummary module has two parts

add_feature_info(<name>, <condition>, <description>) to add entries to the summary.
feature_summary(WHAT ALL) to print the summary.

In short, add include(FeatureSummary) to make the commands available to downstream modules, then when declaring options, add them to the summary with add_feature_info.

cmake_minimum_required(VERSION 3.28)
project(FeatureSummaryExample)

include(FeatureSummary)

# Will be reported in the feature summary without further intervention
find_package(Threads REQUIRED)
find_package(Catch2 3)

# Create an option, and add it to feature summary
option(WERROR "Enable -Werror to turn warnings into errors" False)
add_feature_info(
    Werror WERROR "Enable -Werror to turn warnings into errors"
)

option(RUN_CLANG_TIDY "Run clang-tidy as part of the build" False)
add_feature_info(
    "Run Clang Tidy" RUN_CLANG_TIDY "Run clang tidy as part of the build"
)

# Print the feature summary
feature_summary(WHAT ALL)

When you run cmake, the options you set (or did not set) will be printed out:

-- The following features have been enabled:

 * Werror, Enable -Werror to turn warnings into errors

-- The following REQUIRED packages have been found:

 * Threads

-- The following features have been disabled:

 * Run Clang Tidy, Run clang tidy as part of the build

-- The following OPTIONAL packages have not been found:

 * Catch2 (required version >= 3)

-- Configuring done (0.4s)

Making it Less Verbose

Having to declare an option and then calling add_feature_info each time quickly becomes tedious. A quick script can improve this significantly.

include(FeatureSummary)

function(add_feature_option
    _variable_name
    _option_description
    _option_default
)
    if(NOT DEFINED default)
        set(default OFF)
    endif()

    # option(<variable> "<help_text>" [value])
    option(${_variable_name} ${_option_description} ${_option_default})

    # add_feature_info(<name> <enabled> <description>)
    set(_option_value ${${_variable_name}})
    add_feature_info(
        "${PROJECT_NAME}::${_variable_name}"
        ${_option_value}
        ${_option_description}
    )
endfunction()

This function declares the option, and adds it to the feature summary in a single pass, saving us quite some typing (and ensuring that all options are always in the feature summary).

cmake_minimum_required(VERSION 3.28)
project(FeatureSummaryExample)

include(FeatureSummary)
include(AddFeatureOption)

# Will be reported in the feature summary without further intervention
find_package(Catch2 3 QUIET)

# Create an option, and add it to feature summary
add_feature_option(
    WERROR "Enable -Werror to turn warnings into errors" False
)
add_feature_option(
    RUN_CLANG_TIDY "Run clang-tidy as part of the build" False
)

# Print the feature summary
feature_summary(WHAT ALL)

A Justfile for Docker

2026-01-10T00:00:00+01:00

Each new project I start ends up needing a dedicated container for CI. Some then accumulate additional containers for other CI jobs.

Remembering the commands to build and publish these containers usually became a game of grepping the shell history in search of the correct commands. After doing this too many times, I now have a Justfile to store these commands.

I usually create a docker/ directory in the root of the repository, and put my dockerfiles there.

<repo-root>
    - justfile
    - docker/
        - docker.justfile
        - pre-commit.dockerfile

The docker.justfile is included in the main Justfile with import “docker/docker.justfile”. This helps to keep the length of main Justfile manageable.

Without further ado, here is the Justfile

#
# Top-Level Rules
#

[group("Build Docker Images")]
build-docker: \
    (build-docker-pre-commit)

[group("Publish Docker Images")]
publish-docker: \
    (publish-docker-pre-commit)

#
# Image-specific rules
#

[group("Build Docker Images")]
build-docker-pre-commit: (_build-docker "pre-commit")

[group("Publish Docker Images")]
publish-docker-pre-commit: \
    build-docker-pre-commit \
    (_publish-docker "pre-commit")

#
# Helper rules
#

GITLAB_REGISTRY_URL:= "registry.gitlab.com"
GITLAB_REGISTRY_NS := GITLAB_REGISTRY_URL + "/com_kieffer/comkieffer.com"

GIT_HASH := `git describe --always --dirty`

_build-docker image_name:
    @echo "Tagging {{ image_name }} with {{ GIT_HASH }}"
    docker image build                                                 \
        --file docker/{{ image_name }}.dockerfile       \
        --tag {{ GITLAB_REGISTRY_NS }}/{{ image_name }}:{{ GIT_HASH }} \
        docker/
    docker image tag                                                   \
        {{ GITLAB_REGISTRY_NS }}/{{ image_name }}:{{ GIT_HASH }}       \
        {{ GITLAB_REGISTRY_NS }}/{{ image_name }}:latest

_publish-docker image_name:
    docker image push {{ GITLAB_REGISTRY_NS }}/{{ image_name }}:{{ GIT_HASH }}
    docker image push {{ GITLAB_REGISTRY_NS }}/{{ image_name }}:latest

Starting from the bottom, we have a pair of rules: _build_docker, and _publish_docker. Since they start with an underscore, they are private rules, and they are not shown in the just —list output.

The _build_docker rule, takes a single argument: the name of the image (without the .dockerfile extension). This name will become the container name. Once the image is built, it receives the latest tag, as well was being tagged with the current commit short hash. This makes it easier to find out when an image was built and what it actually contains.

Make sure to edit the GITLAB_REGISTRY_NS to give the images the correct name.

The _publish_docker command just publishes the images to the registry.

These rules give us an easy way of building containers. For each container we created, we add a build-docker-<container-name> and publish-docker-<container-name> to build and publish the container.

Finally, we add these rules to the main build-docker and publish-docker rules.

In general, running just publish-docker is enough to build and publish all the images.

From Regolith to Cosmic DE

2026-01-10T00:00:00+01:00

I recently bought a new laptop, and since I have some time on my hands, I decided to explore the landscape of tiling window managers again. I’ve been using Regolith Linux for the past 5 years, and I’ve developed a fair bit of muscle memory for it’s shortcuts.

Regolith combines Gnome with i3 (or Sway on Wayland). You get a good tiling window manager, but also the collection of tools that make a desktop environment (lock screen, monitor configuration, desktop backgrounds, notifications, …). Instead of needing to assemble all of these components yourself, Regolith gives you a working system out-of-the-box.

I don’t have many gripes about it. It generally works, and the defaults are fine. Still, I wanted to take stock of the existing landscape and see if anything better has popped up.

In 2025, the options seem to be

Cosmic DE is a tiling window manager made by System 76. It started as an extension for Gnome Shell, and eventually became a stand-alone desktop environment.
Omarchy an Arch-based Linux distribution using hyprland.
KDE + i3/sway

Whilst Omarchy seems interesting, the entire desktop configuration is text-based, and I enjoy not needing google in order to accomplish a task like positioning my external monitor.

KDE and i3/sway seem to play nicely together, but I just never enjoyed KDE.

This leaves us with Cosmic.

After using it for a few weeks, I can say that it is quite a polished product. Some things are still missing (_e.g._ setting up the fingerprint reader) but it is quite usable.

At this stage, a couple of things are missing to make me stay with it.

Easy workspace manipulation

In my Regolith setup, I often move entire workspaces from one output to the another with a single shortcut. I have not been able to get this to work in Cosmic. For example, I often have Sublime Merge (my GUI client of choice), and a terminal open on a secondary monitor. When I need to commit/fixup.rebase/… I often move the entire workspace to my main monitor. With Cosmic, I would need to move each window individually.
Unique workspace numbers across outputs

In i3, workspace numbers are unique across all outputs. Using Meta + 1 moves focus to that workspace, whatever output it is on. In Cosmic, this takes me to the first workspace on the current output.

This is especially important to me, because I tend to keep applications on specific workspaces. For example, workspace 1 is my main browser window, workspace 2 is chat and email, workspace 4 is my git tool, workspace 5 is my main IDE window, … I often move the workspaces from one output to another depending on the task at hand. My git tool is usually on a secondary display, but if I need to dive into the repository history looking for a change, I will bring it onto the main display. Whatever display it is on though, it will always be workspace 4.

Not having this makes jumping to a specific application slightly slower.

All-in-all, I enjoy working with Cosmic, but the paradigm is just different enough from i3 to feel foreign. I might revisit it in a year or two, but in the short term, I’ll be re-installing Regolith.

Automatically compiling PySide UI and resource files

2025-08-31T00:00:00+02:00

The common workflow when working with Pyside is to build the UI with PySide Designer which outputs an xml file (.ui`). This file is then fed into pyside6-uic` to generate the python code that actually the window.

Resource files work in a similar manner. A resource file describes resources (e.g. fonts and images) that should be included into the application. These are described in an xml file (.qrc`) and compiled into Python by pyside6-rcc.

Keeping the xml and compiled files synced by hand is tedious. A common option is to use a makefile to automatically re-generate the python files when the xml files are updated.

A Simple makefile

Here is a simple Makefile to compile your .ui and .qrc files

# Find all .ui files and generate corresponding .py target names
UI_FILES := $(shell find src/ -name "*.ui")
UI_PY_FILES := $(patsubst %.ui,$(dir %)ui_$(notdir $(basename %)).py,$(UI_FILES))

# QRC file and target
QRC_FILE := rcc/resources.qrc
QRC_PY_FILE := src/quinspect/resources.py

# Default target
all: compile-ui-files compile-qrc-files

# Convenience targets
compile-ui-files: $(UI_PY_FILES)
compile-qrc-files: $(QRC_PY_FILE)

# Rule to compile .ui files to .py files
src/%/ui_%.py: src/%/%.ui
    @echo "Compiling $< ..."
    .venv/bin/pyside6-uic --output $@ $<

# Rule to compile .qrc file to .py file
$(QRC_PY_FILE): $(QRC_FILE)
    @echo "Compiling $< ..."
    .venv/bin/pyside6-rcc --output $@ $<

A Simple Justfile

Makefiles may be powerful, but the syntax can be hard to grasp (and remember). For simple tasks, I like to use just <https://just.systems/>`_, a simple task runner.

The big difference between the Justfile and the Makefile is that make` will only run a rule if the input is newer than the output. With just`, rules are always run, so we need to do this ourselves.

Here is our Justfile:

default:
    just --list --justfile {{ justfile() }}

# Compile .ui files if the generated code is out-of-date or missing
[group("Generators")]
compile-ui-files:
    #!/bin/bash

    for ui_file in $(find src/ -name "*.ui"); do
        ui_file_name=$(basename $ui_file)
        py_file="$(dirname $ui_file)/ui_${ui_file_name/%.ui/.py}"

        if [[ ! -f $py_file || $ui_file -nt $py_file ]]; then
            echo "Compiling "$ui_file" ..."
            .venv/bin/pyside6-uic --output $py_file $ui_file
        fi
    done


# Compile .rc files if the generated code is out-of-date or missing
[group("Generators")]
compile-qrc-files:
    #!/bin/bash

    rc_file=rcc/resources.qrc
    py_file=src/quinspect/resources.py

    if [[ ! -f $py_file || $rc_file -nt $py_file ]]; then
        echo "Compiling "$rc_file" ..."
        .venv/bin/pyside6-uic --output $py_file rc_file
    fi

This Justfile defines three rules: default, compile-ui-files, and compile-qrc-files.

The default rule is called when just is invoked without arguments, and simply lists the supported commands:

 $ just
just --list --justfile justfile

Available recipes:
    default

    [Generators]
    compile-qrc-files # Compile .rc files if the generated code is out-of-date or missing
    compile-ui-files  # Compile .ui files if the generated code is out-of-date or missing

The compile-ui-files looks for .ui files, inside the src/ directory, and runs a command for each one.

for ui_file in $(find src/ -name "*.ui"); do
    # contents of for loop elided

For each files we find, we want to generate the compiled Python version in the same directory, just prefixed with ui_ to make it identifiable.

# Get the file-name (stripping out the leading directories)
ui_file_name=$(basename $ui_file)

# Create the file python file in the same directory (dirname $ui_file), add the `ui_` prefix to the file name, and the replace the ``.ui`` extension with `.py`
py_file="$(dirname $ui_file)/ui_${ui_file_name/%.ui/.py}"

Finally, if the python file does not exists, or the .ui file has been modified more recently, re-generate the python file.

To check if the file exists, we use ! -f $py_file, where !` negates the “-f` condition (argument is a file).

To compare the modification times, we use the -nt (or newer than) operator to [[. If $ui_file is newer than the $py_file, them we run the pyside6-uic compiler.

if [[ ! -f $py_file || $ui_file -nt $py_file ]]; then
    echo "Compiling "$ui_file" ..."
    .venv/bin/pyside6-uic --output $py_file $ui_file
fi

The compile-qrc-files` rule follows the same logic. However, since we only have one .qrc` file, and the location of the output file is less straightforward, we hard-code it into the rule.

Make or Just ?

So, should you use a Makefile or a Justfile?

If you already have a Makefile, by all means use a make to update the .ui files. You can see how much easier it is.

If you already have a Justfile, you might want to keep adding to it. I personally think that the syntax of the Justfile is easier to read, even if the rules end up being more verbose. However, in the age of coding assistants that can generate makefiles and explain them, this might not be as large an issue as it used to be.

Of course, you can also use your justfile to run your makefile. That might be the best of both worlds.

Bonus Content: Use fzf to select the file to edit

Since writing the first draft of this article, I have extended the justfile with the following rule.

# Select a .ui file to edit with PySide6 Designer
edit-ui:
    fdfind --extension ui --type f \
    | fzf --layout=reverse \
        --preview 'batcat --style=numbers --color=always {}'  \
        --select-1 --exit-0 \
    | xargs -r poetry run pyside6-designer

    just --justfile {{ justfile() }}  compile-ui-files

When run, it shows a list of ui files (with a preview of the file contents), and opens the selected file in PySide Designer. When the designer is closed, the compile-ui-files command is run to recompile the changed files.

A small change, but a definite time-saver.

A better cv::waitKey function

2017-12-08T00:00:00+01:00

The deceptively name OpenCV cv::waitKey in OpenCV is the centre of any use of the highgui module. This module makes it easy to create simple user interfaces and the waitKey function, in addition to waiting for keyboard events, is responsible for updating the message pump for all the windows.

The prototype of the function is:

int cv::waitKey(int mills);

Where millis is the amount of time to wait. If millis is 0 the functions waits until a key is pressed. Otherwise, if millis is greater than 0 the function will return the keycode for the pressed key (a positive integer) if a key was pressed within the allotted time or a negative number.

The usual use of cv::waitKey is to read keypresses in order to drive the user interface. This is usually done as:

auto keycode = static_cast<char>(cv::waitKey(100));
switch(keycode) {
    case 'q':
    case 27: // Escape
        ...

    default:
        ...
}

This is not very easy to use. Finding the right keycode usually involves looking up old pieces of code I wrote to get the mapping from the specific key we are interested in to the keycode. Let’s design a better interface.

In other libraries with keyboard handling components such as SFML keys are represented by a KeyCode struct. Let’s start with that:

enum class CVKeyCode {
    Backspace = 8,
    Tab = 9,
    Enter = 12,

    /* snip, snip */

    Num0 = 48,
    Num1 = 49,
    Num2 = 50,
    Num3 = 51,

    /* snip, snip */

    A = 97,
    B = 28,
    C = 99,
    D = 100,
    E = 101,

    /* snip, snip */
}

We also need to wrap the cv::waitKey function to make it return a KeyCode:

inline KeyCode waitKey(int millis) {
    return static_cast<KeyCode>(cv::waitKey(millis));
}

Since we used a strongly typed enum class instead of a raw enum we cannot convert the int implicitly to a CVKeyCode, we need to cast explicitly with static_cast<CVKeyCOde>( … ). Using a strongly typed enumeration in this fashion makes it easier to identify the possible values that the enum can take which allows the compiler to warn use when we use them wrong.

Quick-Tip: Check that your documentation is up-to-date with clang

2017-09-13T00:00:00+02:00

It is too easy to forget to update documentation when refactoring code. I often change the parameters of a function and forget to update the doc doc comment which lives on, wrong and misguiding. In c++-land doxygen is the standard documentation generator and it can be configured to warn when a parameter of a function is not documented. This helps but documenting every single parameter of every single function is hard, especially when developing new features which leads to a high false positive rate.

A finer grained approach that only warns if the documentation comment and the function disagree would be valuable.

Our knight in shining armour today is the -Wdocumentation flag for clang. With this flag enabled the compiler will parse the documentation comments and warn if the method signature does not match the documentation every time you rebuild your project, not simply when you rebuild your documentation.

To enable this feature you can add the following snippet to your CMakeLists.txt file:

if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
    add_definitions("-Wdocumentation")
endif()

This snippet checks that the compiler is clang and enables the flag.

Alternatively to add the flag to a single target you can use target_compile_options($TARGET PRIVATE -Wdocumentation). This will append the specified flags to the compiler invocation when building the specified target.

Fixing Nodelet does not exist errors in ROS

2017-09-06T00:00:00+02:00

This error is caused by the fact that ROS does not know about the nodelet. To be sure we can check what nodelets ROS knows about by running:

$ rosrun nodelet declared_nodelets

If the nodelet you are trying to run is not listed in the output then we have confirmed that ROS does not know about it. If you are trying to run a nodelet from another package this usually means that the package itself is not in your ROS_PACKAGE_PATH. This can happen if, for example, the package is not correctly installed.

If you are seeing this error for a nodelet that you wrote some possible reasons are:

In the package declaring your nodelets the exports block does not link to your nodelet_plugin.xml
Your package.xml does not list nodelet as a dependency

If the nodelet is listed in the declared_nodelets but still cannot be launched some possible reason are:

The class name specified in the nodelet_plugins.xml file does not match the name of the class implementing the nodelet. This usually happens to me when I forget which namespace the nodelet is in.
You forgot to add the ‘PLUGINLIB_EXPORT_CLASS macro to the .cpp file
Even dumber, you forgot to add the implementation of the nodelet to your CMakeLists.txt file

A CMake Include Rule for IDEs

2017-08-04T00:00:00+02:00

When creating a CMakeLists.txt file we would like to only list the .cpp files in the add_executable and add_library commands. Whilst this is sufficient to compile the code (and should be enough for any IDE) most IDE’s today will exhibit limited functionality unless the header files are also included in the add_executable and add_library commands.

A simple workaround that preserves the best of both worlds is to create a dummy rule that includes all the headers but doesn’t actually do anything. better yet, we can make the rule locate all the headers with a simple GLOB expression:

# Dummy target for IDE's
FILE (GLOB_RECURSE all_headers_for_ides
        ${CMAKE_SOURCE_DIR}/**/*.hpp
        ${CMAKE_SOURCE_DIR}/**/*.h)
ADD_CUSTOM_TARGET(ide_includes SOURCES ${all_headers_for_ides})

Enable locate on systems with and encrypted home directory

2017-08-04T00:00:00+02:00

If you have an encrypted home directory on a linux system chances are that you are using ecryptfs. This is the simplest way of encrypting the sensitive parts of your computer. You may also have noticed that the locate command does not find files in your home directory any more.

This behaviour is intentional. locate does not search the disk directly, instead it queries a database file generally called mlocate.db generated by the updatedb tool. This file is updated by the updatedb tool which is usually run with cron and it contains information about every file and folder on your system. This might not seem like a big deal but it contents could provide a potential attacker valuable information about the information stored on your computer. Worse, this file is stored in /var/lib, outside the confines of your encrypted home directory.

We can however work around this and have the best of both worlds. Maintain the security of the encrypted home and the convenience of locate. To do this we need to create a new mlocate.db“file just for our home directory and tell “locate to use it too.

To achieve the first step we will add a new task to our crontab. To edit it run crontab -e as yourself. Do not use sudo since that would cause you to edit the crontab of the root user. Add the following line to the bottom of the file:

00 01 * * * username updatedb --require-visibility 0 --output $HOME/.local/lib/mlocate.db --database-root $HOME/

This will run updatedb every every hour. Normally updatedb would refuse to run if the owner of the mlocate.db file is not slocate. The —require-visibility 0 tells it to sip that check which allows us to run the command as our own user. The —output option specifies the location where the database should be saved and the —database-root makes updatedb only search the specified folder.

Now that we have a database file we need to help locate find it. We can do this with the LOCATE_PATH environment variable. Add the following line to your .bashrc

# Tell locate to use our very own mlocate database including files indexed here
export LOCATE_PATH="${HOME}/.local/lib/mlocate.db"

There you have it. We’ve got locate working again.

The development of the telegraph - An Incomplete time-line

2017-05-06T00:00:00+02:00

Last week I was in London and, true to habit, I went to visit the London Science Museum. I spent a lot of time in the Information Gallery, specifically the exhibits dedicated to the development of the telegraph.

The telegraph, for the first time, made it possible to communicate almost instantly over long distances. In addition to speeding up the distribution of information it created a host of other problems. Things that we take for granted today like time zones can be traced directly to the telegraph.

The rest of this post is (an incomplete) timeline of the development of the telegraph. Don’t take this to be a definitive history, it is simply a re-transcription of the notes I took during whilst exploring the exhibit.

1835

Morse makes his first telegraph.

1840

The printing telegraph appears.

In a printing telegraph, instead of having one button the operator would use a piano like keyboard to send a series of pulses along the cable. A machine on the receiving end would then decode the pulses and print out the text of the message directly without requiring an operator to translate it from Morse code.

1850

First cross channel telegraph wire laid.

A telegraph cable is laid from Britain to France. It was only about two centimetres in diameter and failed within an hour of completion. A second attempt in 1851 was successful and was used until 1875.

1852

The telegraph is used to propagate railway time in England.

Until railways brought towns closer together in an interconnected mesh each would have it’s own time derived from the local solar time. Whilst not an issue for towns close to each other it was a problem when the task of expressing the timetables for trains in distant areas of the country.

The solution was railway time. The railway company expressed all the timetables in London time and kept “railway time” clocks in each station. However, precise mechanical clocks were still expensive and keeping the clocks synchronised throughout the country was a real challenge.

Enter the telegraph. Dedicated telegraph lines were used to connect stations to central clock. Every hour, on the hour, an electrical pulse would be sent on the wires. This allowed distant clocks to be regularly re-synchronised as they drifted away from the official time.

1857

The first transatlantic cable is laid.

No ship in the world at the time was large enough to carry the entire length of cable required to cross the Atlantic ocean so instead, two ships were converted, one in England and one in the United-States: the HMS Agamemnon and the USS Niagara. They set sail and met in the middle of the Atlantic to join their cables together. The first two attempts were failed when the cable broke and could not be retrieved.

A third attempt in 1858 was more successful however, when the two cables were about to be joined it was discovered that the armouring around the cables had been wound in opposite directions and, if directly connected they would unravel each other. A new splicing technique had to be improvised then and there to allow the two cables to be joined. The cable however failed after three weeks. The splice, fortunately, was not at fault.

In those early days it was impossible to amplify the signal which made the signal incredibly faint even when extremely voltages were used to send messages. To read such faint signals an ingenious invention was used: the TODO: (add wikipedia link) mirror galvanometer. This contraption uses a mirror suspended in the centre of a coil of wire. When a current passes through the coil a magnetic field is generated which deflects the mirror. When a light is shone on the mirror the deflection will cause a large change in the reflection of the light.

1858

Punched Tape Systems.

Very quickly humans were identified as a bottleneck. The telegraph lines could be better be put to use if only messages could be sent faster. This led to the development of mechanical machines to replace telegraph operators. In the punched tape systems an operator would encode a message onto a narrow strip of paper which would then be read by an automated machine. On the receiving end another machine would receive the telegraph and print it out on another strip of paper ready to be decoded by another operator.

1866

First successful transatlantic cable laid.

After the failures of 1857 and 1858 the interest for transatlantic cables waned. It was only in 1866 that a new cable was laid. This time a single ship, the SS Great Eastern was used. It was a passenger liner specially converted for the mission of cable laying and the only ship afloat capable of carrying the entire spool of cable.

1884

International Meridian Conference.

The telegraph has made the world smaller, distant areas can now communicate in close to real time and it becomes important to standardise time not only in countries but also world-wide. At the International Meridian Conference the world accepts the concept of 24 hour days based on the Greenwich meridian. This is the birth of GMT time.

1897

Time division multiplexing is used to improve the throughput of individual telegraph lines

Even with mechanisation there were still ways to increase the amount of messages that could be sent over a single telegraph line. One possibility was a time division multiplexing scheme in which several operators would share a single line. Their consoles were connected to a machine that connected them to the actual cable one by one at fixed intervals. In a scheme with 5 operators for example, each would receive 200 milliseconds every second in which to transmit their message. A machine on the other end would perform the opposite operation to decode the messages.

List included directories of a target with CMake

2017-04-28T00:00:00+02:00

When debugging a CMake build file it is often useful to print out all the directories included by a target. The following snippet does just that. Simply replace ${target_name} and run cmake to see the list of included files.

get_property(includes
        TARGET ${target_name}
        PROPERTY INCLUDE_DIRECTORIES
        )

message(STATUS "Included directories for target ${target_name}")
for (dir in ${includes})
    message(STATUS "  - ${dir}")
endfor()

A very non-comprehensive introduction to ROS package format 2

2017-04-25T00:00:00+02:00

The ROS package format 2 is the new recommended package format for ROS projects. It supersedes the previous package format with a handful of improvements. Let’s take a look at how to structure the new file.

To specify that you are using format 2 you must change the opening <package …> tag in your package.xml to

<?xml version="1.0"?>
<package format="2">
    ...
</ package>

The changes from format 1 to 2 only affect the tags specifying dependencies. The old <build_depend> and <run_depend> tags are gone now. In there place we have a handful of more explicit tags.

<build_depend>: This tag is used for any system or ROS packages required to build your package. This means that any -dev or -devel ubuntu package required to compile should be listed here.
<build_export_depend>: This is the list of packages that other packages that use yours should need. If for example one of your interfaces uses a type defined in another library your users will need to include the headers for that library in their project. This is where you specify that.
<exec_depend>: This tag is for packages that provide executables or shared libraries required by your program. This is where you specify all the bits and pieces that your program needs to run.
<doc_depend>: This tag is for any packages required to build your documentation (e.g. doxygen).

This set of tags is more explicit and solves the ambiguity around the previous <run_depend> tag.

Lobster, a simple template engine for MATLAB

2017-04-17T00:00:00+02:00

Lobster is a template engine for MATLAB with a syntax quite similar to Jinja, a template engine implemented in python. A template engine is useful if you need to create documents with a repetitive syntax. I use them for example to

generate C++ to convert enumeration fields to and from strings,
generate HTML content
generate latex contents from MATLAB variables.

Let’s start with a simple example of the syntax. Create a file called template.tpl with the following contents

Welcome {{ FirstName }},

Your grocery list contains:

{% for item in groceries %}
- {{ item.name }} {% if item.quantity > 1 %}(x{{ item.quantity}}){% endif %}
{% endfor %}

If you’ve downloaded Lobster and added it to your path then using the template is as simple as:

context = struct( ...
    'FirstName', 'Thibaud', ...
    'Groceries', {'Milk', 'Eggs', 'Bacon', 'Nutella'} ...
);

template = Lobster('template.tpl').render(context);

The rendered template is stored in the template variable. What you do with it is up to you. The full documentation can be found in the github repo.

Matlab, Linux & Serial Ports

2017-04-17T00:00:00+02:00

MATLAB on linux follows Ubuntu LTS releases and as such can be quite behind on more recent changes in the linux ecosystem. One recent-ish change that MATLAB still struggles with is the new naming convention for USB-Serial ports. In previous versions of Ubuntu these emulated serial ports were named /dev/ttyUSBX where X is an integer. In newer Ubuntu versions however, these ports are named /dev/ttyACMX, a change that MATLAB has not caught on to yet.

This seemingly small change means that, out-of-the-box, USB-Serial ports will not work at all in MATLAB. The instrhwinfo(‘serial’) command will never see the serial port. Fortunately the fix is quite simple if obscure.

You must create a java.opts file in the same directory as your MATLAB binary with the following contents:

-Dgnu.io.rxtx.SerialPorts=/dev/ttyACM0:/dev/ttyACM1:/dev/ttyACM2

This simply tells MATLAB that serial ports are named ttyACM0, ttyACM1 and ttyACM2.

If you installed MATLAB globally (i.e with sudo) the MATLAB executable should be in /usr/local/MATLAB/R20XXx/bin/glnx64/. This is the directory in which to put the java.opts file. Remember to change the path to point to your specific MATLAB version.

You might also want to check out my post on the first things I do after installing MATLAB on a linux computer here (short version: install matlab-support, fix shortcuts, autosave, date format, default browser & last working folder).

CLion & ROS

2017-04-16T00:00:00+02:00

CLion is a great C++ IDE. It also has the advantage of being cross platform and working on Linux. It has become my IDE of choice on that platform. In this post I’ll be sharing a couple of tips to use more productively with ROS.

Sourcing the ROS environment

Unfortunately CLion does not natively understand ROS. This can however be solved quite easily, we just need to change the .desktop file used to launch it so that it sources the ROS workspace. You can find it in ~/.local/share/applications/jetbrains-clion.desktop. If the file does not exist you can generate it from CLion by clicking Create Desktop Entry in the Tools menu.

The launcher should look like this:

[Desktop Entry]
Version=1.0
Type=Application
Name=CLion
Icon=/home/YOUR-HOME-DIR/ ... /.icon.svg
Exec="/home/YOUR-HOME-DIR/ ... /bin/clion.sh" %f
Comment=A smart cross-platform IDE for C and C++
Categories=Development;IDE;
Terminal=false
StartupWMClass=jetbrains-clion

To let CLion know about ROS we need a way to pass it the environment variables that get set by ROS. If you source a ros setup.bash file in your .bashrc then we just need to run CLion through bash to make it work. In this case just modify the Exec= line adding bash -i -c to the beginning.:

Exec=bash -i -c "/home/YOUR-HOME-DIR/ ... /bin/clion.sh" %f

If you installed CLion using the jetbrains toolbox app the toolbox will overwrite the launcher at every boot. There are two workarounds to this:

make the launcher read-only with chattr +i ~/local/share/applications/jetbrains-clion.desktop
make a copy of the launcher with

opted for the second option. I made a copy of the launcher, changed the Exec line and changed the Name line to Name=CLion (ROS Kinetic)

If you don’t source the ROS setup.bash file from your .bashrc you can still get this to work. We just need to sourced it manually as part of the Exec command. If your ROS workspace is in ~/development/ros_ws/ then change the Exec line to :

Exec=bash -i -c "source ~/development/ros_ws/devel/setup.bash; /home/YOUR-HOME-DIR/ ... /bin/clion.sh" %f

You might have to logout and login again before the changes take effect.

Registering .launch files as xml

To get good syntax highlighting and completion in .launch files you will need to Clion that these are actually xml files. You can do this by opening File > Settings and looking for File Types in the Editor menu. Simply add *.launch to the XML section.

Setting up CMake and Doxygen

2017-04-16T00:00:00+02:00

Setting up CMake to build your documentation with Doxygen is really quite simple. All you really need is

find_package(Doxygen)
if(${DOXYGEN_FOUND})

        add_custom_target(doc ALL
            COMMAND ${DOXYGEN_EXECUTABLE} Doxyfile.h
            WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
            COMMENT "Generating API documentation with Doxygen"
            VERBATIM)

endif() # Doxygen Found

This adds a doc command that builds the documentation and includes it in the ALL rule so that it will be built by default. However, we can make this much smarter. For instance, we can easily make doxygen entirely optional by only enabling it with a CMake option.

Here is the snippet I use in all of my projects now. It uses CMake to generate the Doxyfile with:

a version number set by the VERSION_MAJOR, VERSION_MINOR and VERSION_PATH variables.
Only enable Doxygen if explicitly requested by setting the -DBUILD_DOCUMENTATION option on the command line
An install target

To use it, just copy and paste it into your CMakeLists.txt file.

# The project version number.
set(VERSION_MAJOR   0   CACHE STRING "Project major version number.")
set(VERSION_MINOR   1   CACHE STRING "Project minor version number.")
set(VERSION_PATCH   0   CACHE STRING "Project patch version number.")
mark_as_advanced(VERSION_MAJOR VERSION_MINOR VERSION_PATCH)

find_package(Doxygen)
option(BUILD_DOCUMENTATION
        "Create and install the HTML based API documentation (requires Doxygen)" ${DOXYGEN_FOUND})

if(BUILD_DOCUMENTATION)
    if(NOT DOXYGEN_FOUND)
        message(FATAL_ERROR "Doxygen is needed to build the documentation.")
    endif()

    set(doxyfile_in ${CMAKE_CURRENT_SOURCE_DIR}/Doxyfile.in)
    set(doxyfile ${CMAKE_CURRENT_SOURCE_DIR}/Doxyfile)

    configure_file(${doxyfile_in} ${doxyfile} @ONLY)

    add_custom_target(doc ALL
            COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile}
            WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
            COMMENT "Generating API documentation with Doxygen"
            VERBATIM)

    install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/html
            DESTINATION share/doc)
endif()

Useful snippets for ROS

2017-04-11T00:00:00+02:00

Set CMake options in catkin build

To pass CMake options to the catkin build tool simply use the —cmake-args command line switch. For example to set the CMAKE_BUILD_TYPE I would use:

catkin build --cmake-args -DCMAKE_BUILD_TYPE=Debug

To save the so that they are added for every call you can use the —save-config command line option :

catkin build --save-CONFIG --cmake-args -DCMAKE_BUILD_TYPE=Debug

Build a single package

To build a single package just specify its name at after catkin build, e.g.

catkin build my_package

There is also an even more convenient form. If you are in the package directory then you can simply use

catkin build --this

To just re-build the package without touching any of its dependencies use:

catkin build --this --no-deps

The —this switch can also, quite conveniently, be used for catkin clean.

Create a new package

To create a new ROS package with all the boilerplate just use:

catkin create pkg PACKAGE_NAME

Most of the options of the package.xml file can also be set from the command line.

-l	Add licence
-a	Add author in “First Name, Last Name” email format
-m	Add maintainer

An example invocation using all of these parameters is:

catkin create pkg demo -l MIT -a "Thibaud Chupin" thibaud.chupin@mail.com

Change the console log format

The default format string for logs in ROS is quite poor. I usually want to know what node produced each message for example. Changing it is quite simple fortunately, we just need to override the ROSCONSOLE_FORMAT shell variable. The format string I use now is:

export ROSCONSOLE_FORMAT='[${severity}] <${time}> ${node} @ ${file}:${line}
    ${message}'

Note the new line! This produces logs that have the following appearance.

[ INFO] <1494230701.053693261> /camera_left_extrinsic @ /home/me/Projects/Nearlab/ros_ws/src/aruco_extrinsic/src/ArucoBoardDetector.cpp:26
    Listening for camera images on /lappycam/image_rect_color
[ INFO] <1494230701.060647944> /camera_left_extrinsic @ /home/me/Projects/Nearlab/ros_ws/src/aruco_extrinsic/src/ArucoBoardDetector.cpp:43
    Publishing camera pose to /lappycam/pose
[ INFO] <1494230701.063317878> /camera_left_extrinsic @ /home/me/Projects/Nearlab/ros_ws/src/aruco_extrinsic/src/ArucoBoardDetector.cpp:49
    Publishing detection images on /lappycam/detections
[ INFO] <1494230701.356161276> /camera_left_extrinsic @ /home/me/Projects/Nearlab/ros_ws/src/aruco_extrinsic/src/ArucoBoardDetector.cpp:55
    Publishing detected marker poses and ids on /lappycam/markers
[ INFO] <1494230701.357764129> /camera_left_extrinsic @ /home/me/Projects/Nearlab/ros_ws/src/aruco_extrinsic/src/ArucoBoardDetector.cpp:59
    Aruco board detector ready.
[ INFO] <1494230701.362489914> /camera_left_extrinsic @ /home/me/Projects/Nearlab/ros_ws/src/aruco_extrinsic/src/ArucoBoardDetector.cpp:66
    Aruco board parameters: size = 6x4, length = 0.0222, sep = 0.004, dictionary = 1

The full list of variables that can be used in the ROS console format string cane be found in the documentation.

Debugging a node launched with rosrun

To run gdb on a node launched with rosrun simply add —prefix ‘gdb -ex run’ to the command line. This will start gdb and run the node.

A complete invocation might look something like this:

rosrun --prefix 'gdb -ex run' my_pkg my_node

Debugging a node launched with roslaunch

With roslaunch we don’t control the command line like when we start the node with rosrun instead we can pass gdb command with the prefix parameter in the <node> declaration.

<node pkg="my_pkg" type="my_node" name="node" prefix="gb -ex run">
</node>

Check out the documentation for node to see the full list of parameters it accepts.

Common shortcuts for Bash

2017-02-07T00:00:00+01:00

Keys	Action
Ctrl R	reverse-i-search
Ctrl I	i-search
Ctrl A	Move to start of line
Ctrl E	Move to end of line
Ctrl B	Move backward one word
Ctrl F	Move forward one word
Ctrl K	Delete everything from the cursor to the end of the line
Ctrl U	Delete everything from the cursor to the start of the line

Matlab Live Script Keyboard Shortcuts

2017-02-07T00:00:00+01:00

Keys	Action
Ctrl Alt L	Insert Title (#)
Ctrl Alt H	Insert Heading (##)
Ctrl Alt Enter	Insert Section Break (%%)
Ctrl I	Italic
Ctrl B	Bold
Ctrl U	Underline
Ctrl M	Monospaced
Ctrl K	Hyperlink
Ctrl Alt G	Latex/Equation
Ctrl Alt Enter	Toggle Code/Text Mode

Title, Heading and Section Break can be triggered without the shortcut by typing #, ## and %% respectively.

Fixing the MEvent. CASE! error in MATLAB

2016-11-24T00:00:00+01:00

This rather annoying issue in MATLAB causes the string MEvent. CASE! to be repeatedly printed to the command window every time you scroll any window in the application. The suggested solution is to run

!synclient HorizTwoFingerScroll=0

as part of your startup file to disable horizontal scrolling. This however does not work on more recent linux versions because the synaptics touchpad driver is being deprecated in favour of libinput. The new solution to this problem is slightly more complex however.

First we need to find the id of the touchpad device with the xinput list command from a terminal (not the MATLAB command window). You should see something like:

~$ xinput list
⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPad                id=13   [slave  pointer  (2)]
⎜   ↳ ELAN Touchscreen                          id=11   [slave  pointer  (2)]

We are interested in the SynPS/2 Synaptics TouchPad which in this case has id=13. We can see the configuration options supported by this device by running :

~$ xinput list-props 13

Remember to change 13 to the id of the touchpad on your machine ! In the output you should see a line like:

Synaptics Two-Finger Scrolling (283):   1, 1

This tells you that two finger scrolling is enabled in the vertical and horizontal directions. To change this run :

~$ xinput set-prop 13 "Synaptics Two-Finger Scrolling" 1 0

If you couldn’t find the “Two-Finger Scrolling” line all is not lost. Look for a line related to horizontal scrolling. In my case that was:

libinput Horizontal Scroll Enabled (266):    1

and the command used to disable this property is:

~$ xinput set-prop 13 "libinput Horizontal Scroll Enabled" 0

To have this run automatically every time you run MATLAB you can add

!xinput set-prop 13 "libinput Horizontal Scroll Enabled" 0

to your startup file.

Tips for MATLAB on Linux

2016-11-19T00:00:00+01:00

Setting up MATLAB can be a hassle. On Linux systems some of the defaults settings can be rather baffling. Here are the first things I do after installing.

Install the matlab-support package

After installing MATLAB you may be frustrated to find that it doesn’t appear in the launcher. To run it you need to open a terminal and manually run the matlab command. On Ubuntu-derived systems the matlab-support packages solves this. It creates the missing .desktop files to let the launcher know about MATLAB and offers to solve some common problems with MATLAB on Linux. Pi To install it simply run

sudo apt install matlab-support

Change the keyboard shortcuts

On Linux systems MATLAB uses emacs-like keyboard shortcuts. This might be great for emacs power users but makes life harder for the rest of the us. To switch back to more common shortcuts

Open MATLAB preferences.
In the MATLAB tree open Keyboard then Shortcuts and change the Active Settings dropdown from Emacs Default Set to Windows Default Set.

To change the default open Preferences. In the MATLAB tree open Fonts and change the Desktop Code Font

Change the autosave location

MATLAB will helpfully autosave our files for us. By default it does this by creating a copy of the file and appending a ~ to its name. These files however can create clutter your working directory. An alternative is to instruct MATLAB to save them all in a single folder.

Open MATLAB preferences
In the MATLAB tree open Editor/Debugger then Backup Files.
Change the Location option from Source file directories to Single directory and set the target directory in the text box. I usually use a hidden folder in my MATLAB home directory so I set it to ~/home/Documents/MATLAB/.autosave.

Changing the date and time formats

Whilst America seems proud of its imperial units and weird way of writing dates the rest of world can’t help but look on with a puzzled expression. By default MATLAB seems to completely disregard de system locale when presenting dates.

Whilst we can’t force America to adopt the glorious metric system we can however instruct MATLAB to present dates in a reasonable manner.

Open MATLAB preferences
In the MATLAB tree open Command Window
Check the Locale option in the Datetime Format section. If MATLAB was able to locate a sane locale changing it should be enough. In my case MATLAB was unable to locate a normal locale so I had to manually set the datetime and date formats.
Set the Default date and time format to uuuu-MM-dd HH:mm:ss and the Default date only format to uuuu-MM-dd.

Forcing MATLAB to use the default browser for html pages

If you frequently use the Publish as HTML feature of MATLAB to share scripts and their results you may be frustrated that MATLAB doesn’t us your browser of choice. To force to always use the default browser if you change it later on you can tell MATLAB to use the xdg-open command instead. This command always uses the system default apps to open any file type.

Open MATLAB preferences
In the MATLAB tree open Web and in System Web Browser set Command to xdg-open. You can also set it as the PDF Reader command.

Last working folder

It can be frustrating to have to manually navigate to your current project’s working folder every time you re-open MATLAB. MATLAB can remember your current working folder and re-open it automatically.

Open MATLAB preferences
In the MATLAB tree open General and change the Initial Working Folder to Last working folder from previous MATLAB session.

startup.m files

One of the most underused features of MATLAB is startup.m files.

When MATLAB starts up it will look for a file named startup.m in the current working directory and run it automatically. You can use it to set project specific settings easily.

For example, I use it to disable some warnings caused by some tools I use (and can’t modify) to avoid extraneous noise. Another common use it to load all the subdirectories that contain tools I am using in the current project instead of having to addpath() them manually.

You can find the startup file I put in the root of my MATLAB projects here

Virtual Reference Feedback Tuning

2016-04-29T00:00:00+02:00

Virtual Reference Feedback Tuning is a data driven method used to design a controller for an unknown plant based on input/output measurements. The method uses a single set of input/output data generated by the plant without the need for specific experiments or model identification.

This is the first post in a series of posts detailing the method and explaining how to use it in MATLAB.

Virtual Reference Feedback Tuning (this post)
VRFT in MATLAB - Coming Soon

How Does It Work ?

The general idea behind VRFT is to transform the control problem of designing the controller into an identification problem. To use the VRFT method we need:

A reference Model. This is a model that describes the ideal behaviour of the closed loop system.
One set of input-output data on the plant. The input does not need to have any special characteristics.

Block diagram showing the closed loop control system and the reference model.

Note that the output of the reference model and the control loop are identical.

The output of the plant is the known from the I/O data that was previously collected. If we consider these measurements as the output of the reference model then we can build an input signal $r(t)$ such that:

\begin{equation*} y(t) = M_R(z) r(t) \end{equation*}

This signal is the virtual reference. We want to make the closed loop behaviour of the system as close as possible to the reference model so we use this virtual reference as the set-point for the control loop. The output of the controller is the plant input $u(t)$. This is the other known signal.

The input of the controller can easily be calculated:

\begin{equation*} e(t) = r(t) - y(t) \end{equation*}

Now that both the input and the output ($u(t)$) of the controller are known the control problem can be reduced to an identification problem.

A More Rigorous approach

It is assumed that the plant is a linear SISO discrete-time dynamical system described by a rational transfer function ($P(z)$).

We choose a reference model $M_R(z)$ that describes the desired behaviour of the closed loop control system.

We determine a class of controllers $C(z; \theta)$ that depend linearly on the parameter vector $\theta$ such that:

\begin{equation*} C(z; \theta) = \beta(z)^T\theta^T \end{equation*}

Where:

$\beta = [\beta_1(z), \beta_2(z), ..., \beta_n(z)]^T$ is a vector of linear transfer functions
$\theta = [\theta_1, \theta_2, ... \theta_n]^T$ is the vector of parameters

The control objective is the minimisation of the following criterion:

\begin{equation*} \begin{aligned} J_{MR}(\theta) &= \left| \left( \frac{P(z)C(z; \theta)}{1 + P(z)C(z; \theta)} - M(z) \right) W(z) \right|_2^2 \\ &= \left| \left( T(z) - M(z) \right) W(z) \right|_2^2 \end{aligned} \end{equation*}

which is equivalent to minimising the difference between the closed loop system and the reference model scaled by an appropriate weighting function. The choice of this weighting function $W(z)$ is up to the system designer and can be chosen to emphasize or de-emphasize performance in certain frequency bands.

An optimal choice of $C(z; \theta)$ will produce a closed loop system whose transfer function is $M(z)$. Thus, if the closed-loop system is fed a reference signal $r(t)$ its output will be :

\begin{equation*} y(t) = T(z)r(t) = M(z)r(t) \end{equation*}

The VRFT method works by using the measured output ($y(t)$) to generate a reference signal $\bar{r}(t)$ such that:

\begin{equation*} M(z)\bar{r}(t) = y(t) \end{equation*}

This is our so-called virtual reference, it does not exist in the physical world and is only used a tool to generate the optimal controller. The input of the controller is:

\begin{equation*} e(t) = \bar{r}(t) - y(t) \end{equation*}

The optimal controller must be such that when fed the virtual reference it produces the measured input ($u(t)$). Thus:

\begin{equation*} u(t) = C(z; \hat{\theta}) \bar{r}(t) \end{equation*}

Where $\hat{\theta}$ is the optimal parameter vector.

To calculate the optimal parameter vector we can start by attempting to solve a slightly easier problem.

We apply a filter $L(t)$ to the data. I will explain the reason for this filter and it’s form soon but bear with me in the meantime.

\begin{equation*} e_L(t) = L(z)e(t) \quad, \quad y_L(t) = L(z)e(t) \end{equation*}

Now, consider the following performance index:

\begin{equation*} \begin{aligned} J^N_{VR}(\theta) &= \frac{1}{N} \sum_{t=1}^{N} \left( u_L(t) - C(z; \theta) \ e_L(t) \right)^2 \\ &= \frac{1}{N} \sum_{t=1}^{N}\left( u_L(t) - \beta^T(z)\theta \ e_L(t) \right)^2 \\ &= \frac{1}{N} \sum_{t=1}^{N}\left( u_L(t) - \varphi_L^T(t)\theta \right)^2, \quad \varphi_L = \beta^T(z) \ e_L(t) \end{aligned} \end{equation*}

Since this equation is quadratic in $\theta$ the optimal value $\hat{\theta}_N$ is an explicit function of the data:

\begin{equation*} \hat{\theta}_N = \left[ \sum_t \phi(t) \phi(t)^T \right]^{-1} \sum_t \phi(t)u(t) \end{equation*}

However, this doesn’t seem to bring us any closer to solving our original problem of minimising $J_{MR}$. This method though is useful because it is based purely on the data and is relatively easy to compute.

Before continuing we need to make a digression. Let’s rewrite $J_{MR}$ in the frequency domain:

\begin{equation*} \begin{aligned} J_{MR}(\theta) &= \frac{1}{2 \pi} \cdot \int_{-\pi}^{\pi} \left| \frac{PC(\theta)}{1 + PC(\theta)} - M \right|^2 \left| W \right|^2 d\omega \\ &= \frac{1}{2 \pi} \cdot \int_{-\pi}^{\pi} \left| \frac{PC(\theta)}{1 + PC(\theta)} - \frac{PC_0}{1 + PC_0} \right|^2 \left| W \right|^2 d\omega \\ &= \frac{1}{2 \pi} \cdot \int_{-\pi}^{\pi} \left| \frac{PC(\theta)[1 + PC_0] - PC_0[1 + PC(\theta)]}{[1 + PC(\theta)][1 + PC_0]} \right|^2 \left| W \right|^2 d\omega \\ &= \frac{1}{2 \pi} \cdot \int_{-\pi}^{\pi} \left| P \cdot \frac{C(\theta) - C_0}{[1 + PC(\theta)][1 + PC_0]} \right|^2 \left| W \right|^2 d\omega \\ \end{aligned} \end{equation*}

We can now go back to $J_{VR}$.

If $u(t)$ and $y(t)$ can be considered realisations of stationary stochastic processes then as the amount of data grows ($N \rightarrow \infty$) the following holds:

\begin{equation*} J^N_{VR}(\theta) \rightarrow J_{VR}(\theta) \end{equation*}

And:

\begin{equation*} \begin{aligned} J_{VR}(\theta) &= E \left[ \left( u_L(t) - C(z; \theta)e_L(t) \right)^2 \right] \\ &= E \left[ L(z) \cdot \left( u(t) - C(z; \theta)e(t) \right)^2 \right] \end{aligned} \end{equation*}

We would like to get rid of $e(t)$ in the equation to make it depend only on our initial data:

\begin{equation*} e(t) = \bar{r}(t) - y(t) = \bar{r}(t) - M_R(z)\bar{r}(t) = \left( 1 - M_R(z) \right) \cdot \bar{r}(t) \end{equation*}

And since:

\begin{equation*} r(t) = \frac{y(t)}{M_R} = \frac{P}{M_R} \cdot u(t) \end{equation*}

We can rewrite the criterion as:

\begin{equation*} \begin{aligned} J^N_{VR}(\theta) &= \frac{1}{N} \sum_{t=1}^{N} \left[u_L(t) - C(z; \theta) \cdot \left( 1 - M_R(z) \right) \cdot \frac{P}{M_R} \cdot u_L(t) \right]^2 \\ &= \frac{1}{N} \sum_{t=1}^{N} \left[ \left( L(z) \left( 1 - C(z; \theta) \cdot \frac{1 - M_R(z)}{M_R(z)} \cdot P \right) u_L(t)\right)^2 \right] \end{aligned} \end{equation*}

We also know that, by definition, $M_R$ is:

\begin{equation*} M_R(z) = \frac{PC_0}{1 + PC_0} \quad \Rightarrow \quad 1 - M_R = \frac{1 + PC_0 - PC_0}{1 + PC_0} = \frac{1}{1 + PC_0} \end{equation*}

Which brings us to:

\begin{equation*} \begin{aligned} 1 - C(\theta) \cdot \frac{1 - M_R}{M_R} \cdot P &= \frac{1}{M_R} \left( M_R - PC(\theta) \left[ 1 - M_R \right] \right) \\ &= \frac{1}{M_R} \left( \frac{PC_0}{1 + PC_0} - \frac{PC(\theta)}{1 + PC_0} \right) \\ &= \frac{1}{M_R} \left( P \cdot \frac{C_0 - C(\theta)}{1 + PC_0} \right) \\ &= \frac{1}{M_R} P \cdot \left( C_0 - C(\theta) \right) \left( 1 - M_R \right) \end{aligned} \end{equation*}

By substituting this result back into $J_{VR}$ we obtain:

\begin{equation*} J^N_{VR}(\theta) = E \left[ \left( \frac{L}{M_R} \cdot \left( C_0 - C(\theta) \right) \cdot \left( 1 - M_R \right) \right)^2 \right] \end{equation*}

The frequency-domain representation of this criterion is:

\begin{equation*} J_{VR}(\theta) = \frac{1}{2\pi} \int_{-\pi}^{\pi} \left| P \cdot \left( C_0 - C(\theta) \right) \left( 1 - M_R \right) \cdot \frac{L}{M_R} \right|^2 \Phi_u \ d\omega \end{equation*}

Where, $\Phi_u$ is the power density of $u(t)$.

If $C_0(z) \in C(z; \theta)$ and $J_{VR}(\theta)$ has a unique minimum then minimizing $J_{VR}(\theta)$ will always yield ${C_0(z)}$ independently of the value of $L(z)$.

Notice the similarity between the expressions of $J_{MR}$ and $J_{VR}$. With a proper choice of the pre-filter $L(z)$ we can make them equal.

If:

\begin{equation*} \left| L \right|^2 = \left| \frac{M_R W}{1 + PC(\theta)} \right|^2 \cdot \frac{1}{\Phi_u} \end{equation*}

Then,

\begin{equation*} \begin{aligned} J_{VR}(\theta) &= \frac{1}{2\pi} \int_{-\pi}^{\pi} \left| P \cdot \frac{C_0 - C(\theta)}{1 + PC_0} \cdot \frac{W}{1 + PC(\theta)} \right|^2 \ d\omega \\ &= \frac{1}{2 \pi} \int_{-\pi}^{\pi} \left| P \cdot \frac{C(\theta) - C_0}{[1 + PC(\theta)][1 + PC_0]} \right|^2 \left| W \right|^2 \ d\omega = J_{MR}(\theta) \end{aligned} \end{equation*}

Unfortunately this pre-filter is not practical since it depends on knowing the plant model $P$. However, if $C(z; \theta)$ is a good approximation of $C_0(z)$ then:

\begin{equation*} \left( 1 - M_R \right) = \frac{1}{1 + PC(z; \theta)} \approx \frac{1}{1 + PC_0(z)} \end{equation*}

Thus, $L(z)$ can be written as:

\begin{equation*} L(z) = \left| \left(1 - M_R \right) \cdot M_R \cdot W \right|^2 \cdot \frac{1}{\Phi_u} \end{equation*}

This formulation of the pre-filter is completely known. This proves that $J_{MR}$ and $J_{VR}$ are equivalent and that by solving the simpler problem ($J_{VR}$) we also solve the original problem ($J_{MR}$).

Recap

In conclusion, the VRFT method can be broken down into a set of relatively simple steps:

Choose a reference model $M_R$ and class of controllers $C(z; \theta)$
Calculate the pre-filter $L(z)$
Compute the optimal parameter vector $\hat{\theta}_N$

The VRFT method can be extended to cases where the signal is noisy, we will discuss this in a later post. The VRFT method has also been extended to cascade control systems. This too will be the subject of a later post.

Open In File Browser for Linux

2016-03-08T00:00:00+01:00

On windows when right clicking in the file browser pane you can select Open in file browser to open the current folder in windows explorer. On linux you will simply get an error message stating that this action is not supported on this operating system (as of MATLAB 2016a).

The following script provides the same functionality:

%
% Open the specified path in the default file browser (linux only)
%
% The Unix version of MATLAB does not support 'Open in File Browser' so I
% implemented a basic version here. All modern linux distributions should
% support `xdg-open`
%
% To open the MATLAB userpath directory (Documents/MATLAB) run:
%
%   >> open_in_file_browser userpath
%

function open_in_file_browser(path)
    if ~exist('path', 'var'); path = '.'; end

    if strcmp(path, 'userpath')
        upath = userpath();
        path = upath(1:end-1);
    end

    if ~exist(path, 'dir')
        warning('%s is not a folder', path)
        return
    end

    % To properly redirect stdout and stderr we need to detect the current
    % shell. Currently we support bash/sh and fish.
    output_redirect_syntax = '> /dev/null 2>&1';
    if strcmp(shell(), 'fish')
        output_redirect_syntax = '> /dev/null ^&1';
    end

    [status, stdout] = system(['xdg-open ', path, ' ', output_redirect_syntax]);
    if status ~= 0
        warning('xdg-open failed.\n  Status code: %i\n  stdout: %s', status, stdout);
    end
end

% Detect the shell that the command will be run in.
function shellname = shell()
    % Use -n to not emit a '\n' charracter
    [status, out] = system('echo -n $SHELL');

    if status ~= 0; warning('Unable to detect shell type.'); end
    [~, shellname, ~] = fileparts(out);
end

The script leverages xdg-open to open the current folder in the default file browser. This should work on any desktop linux distribution.

Quickly changing projects in MATLAB

2016-03-08T00:00:00+01:00

When working on several different projects in MATLAB changing from one project folder to another can be a hassle. Some projects are in a shared dropbox folders, others are in my main MATLAB directory. Most powerful text editors offer a way of quickly changing project directories without having to navigate the folder structure by hand.

This is my solution for MATLAB.

%
% Hacky way of making projects in MATLAB
%
% One hassle of MATLAB is that moving from one project to another is a hassle.
% With this script you only have to type:
%
%   open_project 'my project'
%
% to move from the current directory to the specified directory. Projects are
% simply folders in the one of the 'project_folders'.
%
% To list all the currently detected projects use:
%
%   open_project --list
%
% To move to your MATLAB user directory (~/Documents/MATLAB on linux) use:
%
%   one_project mlhome
%

function open_project(project_name)

    project_folders = {
        '/home/tibo/Documents/MATLAB/scripts'
        '/home/tibo/Dropbox/PoliMi'
    };

    if strcmp(project_name, '--list')
       list_projects(project_folders);
       return
    end

    if strcmp(project_name, 'mlhome')
        fprintf('Changing directory to %s\n', userpath);
        % We need to strip the trailing ':' from the userpath
        user_path = userpath();
        cd(user_path(1:end-1));
        return
    end

    for k = 1:length(project_folders)
        path = fullfile(project_folders{k}, project_name);

        if exist(path, 'dir')
            fprintf('Found project. Changing directory to %s\n', path');
            cd(path);
            return;
        end
    end

    fprintf('Unable to locate project %s\n', project_name);
end

function list_projects(project_folders)
    for k = 1:length(project_folders)
       fprintf('Projects in <%s>:\n', project_folders{k});

       folders = dir(project_folders{k});

       for j = 1:length(folders)
          if folders(j).isdir &&  ~strcmp(folders(j).name, '.') && ~strcmp(folders(j).name, '..')
            fprintf('    %s\n', folders(j).name);
          end
       end
    end
end

After populating the project_folders variables with the directories you use to store your projects you can use open_project project_name to move from on project folder to another. To see the list of directories detected by the tool run open_project —list.

This will only change the current directory to the project directory. It will not close existing editor tabs or clear the workspace.

Quick Tip: Receive a pushbullet notification when a ssh login is successful

2016-01-19T19:32:00+01:00

Pushbullet is a service that allows you to push notifications to your devices. We will be using it to send us a notification any-time somebody successfully connects to our server with ssh.

The cleanest way to run a script when on ssh logins is through PAM. PAM or “Pluggable Authentication Module” is a library called by all the authentication related functions in Linux that provides a way to hook into the authentication process.

The pam_exec module gives us easy access to this system. It allows us to run scripts to react to events like user logins.

Save this script to /opt/ssh_notify/notify.sh. The script uses the Pushbullet API to send a notification to your phone with the username and IP address for the user that has just signed on.

#!/bin/bash

PB_TOKEN="------REPLACE-ME-----------------"

if [[ "${PAM_TYPE}" == "open_session" ]]; then
    PAYLOAD=$(printf '{
            "title": "New shh connection",
            "body": "User <%s>, from %s successfully connected at %s to %s.",
            "type": "note"
        }' "${PAM_USER}" "${PAM_RHOST}" "$(date +%H:%M)" "$(hostname)" \
    )

    curl --header "Access-Token: ${PB_TOKEN}" \
         --header 'Content-Type: application/json' \
         --data-binary "${PAYLOAD}" \
         --request POST https://api.pushbullet.com/v2/pushes
fi

Before you can start sending notifications you need to retrieve and API access token from the Pushbullet website. Go to your profile and click on CreateAccessToken. Copy the token into the space marked PB_TOKEN in the script.

Now we need to tell pam about our script. Add the following line to the end of /etc/pam.d/sshd:

session optional pam_exec.so /opt/ssh_notify/notify.sh

To test your changes open a new terminal and ssh into your machine. You should receive a notification.

The pam_exec environment exposes several other variables that you can use to build a more descriptive message. You can check out the complete list here.

Exporting MATLAB figures to LaTeX

2015-10-14T14:31:00+02:00

The standard way of exporting figures from MATLAB to LaTeX is to use the builtin print command to export the figure to an eps file using:

print -depsc $filename

Unfortunately figures exported this way generally do not look like the original MATLAB figures. The aspect ratio is different and the fonts are too small. This can be overcome by tweaking the paper size, the font size and the line thickness but the process is fiddly and resulting eps files are hard to use since alternative TeX compilers like XeTeX and LuaTeX do not support eps files out of the box.

It is possible to save MATLAB figures as pdf files using print -dpdf $filename. This allows you to use XeLaTeX and LuaLaTeX but the other issues are not solved.

MATLAB can also save figures as png files. The resulting files actually look like the figures they were saved from but this is not recommended for publication quality graphics as high resolution images are often larger than their pdf/eps equivalents.

Enter matlab2tikz. This package translates MATlab figures into tikz graphics using pgfplots for direct inclusion into LaTeX documents. tikz is a TeX package used to generate graphics whilst pgfplots is a LaTeX package that provides convenient primitives for drawing plots. The resulting files are easy to use and modify. Additionally since they are just LaTeX code they are typeset with the rest of document and use the same fonts and font sizes. This makes them fit in a lot better and makes changing the document font or style easy since it doesn’t require you to also edit your figures to change the font ! As an added bonus this also allows us to change the size and aspect ratio of our plots without having to regenerate them !

Basic Usage

To use matlab2tikz you need to download the package and place it on you MATLAB path. Once MATLAB can find it saving your figures is as simple as calling matlab2tikz(filename). This will save the current figure. Your LaTeX distribution should be smart enough to do that for you once you’ve added the proper \usepackage directives to your preamble.

To add the figure to your LaTeX document you need to add the pgfplots package to the preamble of your LaTeX document:

\usepackage{pgfplots}
\pgfplotsset{compat=newest}
\pgfplotsset{plot coordinates/math parser=false}

To add the figure to the LaTeX document simply insert it using \input{} into a figure environment:

\begin{figure}
        \centering

        \input{filename}
        \caption{My cool figure}
        \label{fig:my_figure}
\end{figure}

If your documents contains more than a couple tikz graphics you will probably encounter the following error:

! TeX capacity exceeded, sorry [main memory size=3000000].

There are several work arounds to this problem. The simplest is to reduce the size of the generated graphics. When matlab2tikz exports your graphs it saves every single point in the dataset even if they are so close that only a fraction of them will actually be visible. The cleanfigure command provided with matlab2tikz will reduce the number of datapoints uused to draw the graph to the bare minimum required. Sometimes however it needs help: the minimumPointsDistance option tells it how far away two points must be and discards all the points in between. Obviously, this value depends on your dataset. If correctly used you can significantly reduce your file size.

If this isn’t enough, the matlab2tikz wiki has some other options.

To export a figure simply run:

cleanfigure();
matlab2tikz('tikz/filename.tikz');

Advanced Usage

Editing your figures

The biggest advantage of this approach over the pdf/png options is that modifying your graphs after the fact is easy. Getting the exact look you want in MATLAB can be an exercise in frustration. pgfplots by contrast is a pleasure to work with. Most of the syntax is self explanatory and the documentation is well written.

In one of my figures for example I needed to move the legend down because it was covering the axis labels. I just had to change the line

legend style={at={(0.5,-0.03)},anchor=north}

legend style={at={(0.5,-0.2)},anchor=north}

Replacements like this can even be automated using MATLAB with very little code, just open the file, find the string you want to replace and close it again:

% fix the positioning of the legend in the tikz file:
src = fileread('tikz/filename.tikz');
new_src = strrep(src, ...
    'legend style={at={(0.5,-0.03)},anchor=north}', ...
    'legend style={at={(0.5,-0.2)},anchor=north}'   ...
);

fid = fopen('tikz/filename.tikz', 'w');
fwrite(fid, new_src);
fclose(fid);

Changing the aspect ratio of a figure

If you export your figures as png setting the aspect ratio is as easy as changing the resolution of the generated image. Changing the aspect ratio is not as easy when export ing them as eps/pdf files. I often had to fiddle with pbaspect and subplots to get the right result. With matlab2tikz this is quite easy.

You can pass the desired height and width of your figure to the matlab2tikz as LaTeX commands. By default the script tries to make reasonable assumptions about the width and height of your figures but this can lead to inconsistent results in some cases. If this is the case you can specify the width attribute:

matlab2tikz('filename', 'width', '\textwidth');

This also allows you to set the height of the figure as a function of the width:

matlab2tikz('filename', 'width', '\textwidth', 'height', '0.7\textwidth');

Being so explicit about the figure width is bad practice. If we decide that we want smaller or bigger figures in our document we have to go back to the script that generates the figures and change the sizes manually. A better option is to create a special length just for our figures. That way changing the size of the figures post-facto is as easy as redefining the length in our LaTeX document. To do this we need to add the following lines to the preamble:

\newlength\figureheight
\newlength\figurewidth
\setlength\figureheight{6cm}
\setlength\figurewidth{\textwidth}

This defines the lengths \figureheight and \figurewidth. We just need to tell matlab2tikz to use these instead of trying to guess:

matlab2tikz('filename', 'width', '\figurewidth', 'height', '\figureheight');

These lengths can also be redefined for just one figure in our document. If for example we want on figure to be taller we just need to redefine the length in the figure block:

\begin{figure}
        \setlength\figureheight{8cm}

        \centering
        \input{file.tikz}
\end{figure}

Closing Notes

Hopefully you are now able to produce nicer MATLAB figures with less work. If you find yourself repeating the same code often to export your figures consider wrapping your call to matlab2tikz in a helper function.

You can find mine at github