Category: Learning and Development

Strategies for continual learning and training in software development, as well as techniques for identifying and resolving bugs and other issues, and effective approaches for solving problems and overcoming obstacles in software development.

  • Quickly Resolving “no url found for submodule path” Problems

    “No url found for submodule path” errors can be tricky to track down; if you think you have all your submodules set up correctly. I ran into this very issue today! I cloned one of my team’s repositories and could not then pull the submodules due to this error. After some searching, I was able to find a simple solution to resolve the error from incorrectly adding submodules to your repository.

    man in white shirt using macbook pro
    Photo by Tim Gouw on Pexels.com

    The problem I was running into was this:

    $ git submodule update --init --recursive
    fatal: No url found for submodule path 'common_source' in .gitmodulesCode language: JavaScript (javascript)

    I was left scratching my head because I did in fact have my .gitmodules set correctly. I should not have been seeing this “No url found for submodule path” error.

    The normal resolution did not help:

    git rm --cached modules/common_source
    git add git@<my-common-source-submodule-repo-url>/common_source.git modules/common_sourceCode language: HTML, XML (xml)

    This would resolve my issue for my local repository, but if I then cloned it anew I would run into the same problem.

    I finally found the magic sauce from this post on Netlify. (Note: Gitlinks, which are submodules, have mode 160000).

    $ git ls-files --stage | grep 160000
    14:160000 9e05e5731986604b50a1bf4a6201aec642f76a27 0	common_source
    270:160000 9e05e5731986604b50a1bf4a6201aec642f76a27 0	modules/common_source

    Somehow, I’m not sure how, but somehow the submodule got added at the wrong path initially and then was not correctly removed. Using the git ls-files command, I was able to see that the submodule was indeed listed twice. That was what was causing the error.

    To fix this once and for all, I did this:

    $ git rm --cached common_source

    And, viola! Problem solved!


    I hope that someone else can find this helpful!

  • Streamlining Your Code Review Process with Bitbucket Pipelines and clang-format

    As developers, we are constantly striving to write clean, maintainable code. But as codebases grow and evolve, it can be challenging to ensure consistency across the entire project. Consistent code formatting can make it easier to read and understand code, and can even reduce bugs and errors.

    To address this issue, many teams use code formatters like clang-format to automatically enforce a consistent style across their codebase. But how can we ensure that our code is properly formatted before it even reaches the code review stage? This is where Bitbucket Pipelines comes in.

    By integrating clang-format checks into your Bitbucket Pipeline, you can automatically test that your code is properly formatted on every push or pull request. This helps catch formatting errors early in the review process, making it easier to maintain a consistent codebase and ultimately reducing technical debt.

    In this blog post, we’ll walk through how to set up a Bitbucket Pipeline step to test code formatting with clang-format. We’ll also discuss best practices for code formatting and how to integrate these checks into your development workflow. So, let’s get started!

    Add a consistency check step to your pipeline

    To test that formatting is correct with clang-format in a Bitbucket Pipeline, you can add a step to install clang-format and then run a check against your code files. Here’s an example of how you could do this:

    definitions:
      steps:
          - step: &Check-Formatting
              name: Check formatting
              image: atlassian/default-image:4
              script:
                - apt-get update && apt-get install -y clang-format
                - clang-format --version
                - find . \( -name '*.h' -o -name '*.c' -o -name '*.hpp' -o -name '*.cpp' \) -exec clang-format -style=file --output-replacements-xml {} \; | grep "<replacement " && exit 1 || exit 0
    pipelines:
      pull-requests:
        '**':
          - step: *Check-Formatting
    Code language: Lua (lua)

    Here’s what this step does:

    1. Installs clang-format by running apt-get update and apt-get install -y clang-format.
    2. Prints the version of clang-format by running clang-format --version. This is useful for debugging and logging information in your pipeline.
    3. Runs a find command to search the repository for all C/C++ source files (extensions *.h, *.c, *.hpp, and *.cpp). If you have code in files with other extensions, you can add them to the search. Just add -o -name '*.<your extension>' after the '*.cpp' search term. Mind the space between the last term and the closing ')'!
    4. Runs clang-format against the found code files with the -style=file flag, which tells clang-format to use the formatting style specified in the .clang-format file in the root of your repository. The command generates an XML report of the formatting changes.
    5. Pipes the XML report to grep "<replacement ", which searches for any lines in the report that contain the <replacement> tag. This tag indicates that clang-format made a formatting change to the code. If any replacements are found, the pipeline will exit with an error code (exit 1). Otherwise, it will exit successfully (exit 0).

    The pipeline step will pass if this returns with exit code of 0 and fail otherwise. Place this step as the first step in your pull request pipeline and when a pull request is opened it will fail quickly if there are formatting issues. This indicates to the developer they need to fix those issues before continuing.

    Note that this checks the entire repository, not just the changes!

    Best Formatting Practices

    Ensuring consistent formatting is definitely a good practice to work into your development practice. To make it easy for developers to do this, many IDEs support formatting files per some standard. For example, Eclipse allows you to define a style and easily format highlighted sections or the entire file. VS Code has options to format your code when you save a file, in addition to formatting specific sections of code you are working on.

    Formatting options in VS Code.

    Outside of your IDE, my favorite option is to use a git hook to format your patches as you commit them to the repository. By using the pre-commit hook and some clang-format scripts, you can ensure that any new code or changed code gets formatted properly before getting pushed to the server. Read more about how to set that up in my other post.

    Best practices for enforcing consistent formatting comes down to two approaches, in my opinion, and depends on whether you are working with an established code base or a new one.

    Consistent Formatting for Established Code Base

    For established code bases, it may not be desirable or feasible to format all the code. That would require a huge amount of retest and revalidation that just may not be possible. In that case, set up your pipeline to just check the formatting on patches in the pull request. As developers make changes to fix bugs or add features, those new changes will go through the testing and validation process and also get the correct formatting applied. Over time, the consistency of the formatting will increase as more and more of the code gets updated.

    To help developers automate this, set up the development environment to format changes with a git pre-commit hook. This will cut down on pipeline failures which enforce that patches are formatted properly on pull requests.

    An example pipeline that enforces formatting on just patches is shown here:

    definitions:
      steps:
          - step: &Check-Patch-Formatting
              name: Check code formatting with clang-format-diff
              image: atlassian/default-image:4
              script:
                # Install clang-format-diff
                - apt-get update && apt-get install -y clang-format
                # Run clang-format-diff on modified files
                - if [[ $(git diff -U0 --no-color --relative origin/master...origin/${BITBUCKET_BRANCH} | clang-format-diff -p1) ]]; then exit 1; fi
    pipelines:
      pull-requests:
        '**':
          - step: *Check-Patch-FormattingCode language: PHP (php)

    Consistent Formatting for New Code Base

    When working with a new project and a new code base, there is no reason why you can’t enforce formatting right off the bat.

    First, set up your IDE to format your files on save, to ensure they stay properly formatted. In addition, ensure you have the pre-commit hook installed to validate formatting (and to fix it) before committing. Finally, set up the pipeline with two steps: one for validating the patch on each PR, and another that checks the entire code base as part of a nightly (or weekly) build process. The first pipeline in this post shows this type of pipeline step (for a pull-request).

    Conclusion

    In conclusion, maintaining consistent code formatting is crucial for writing readable, maintainable code. With Bitbucket Pipelines and clang-format, we can automate the process of checking our code formatting and catch errors before they make it into production. By setting up a simple Pipeline step to run clang-format checks, we can ensure that our codebase stays tidy and easy to read.

    Remember that consistent code formatting is just one piece of the puzzle when it comes to writing high-quality code. It’s important to also focus on good design principles, writing clear and concise code, and properly testing our applications. By making conscious decisions and striving to improve our development practices, we can create software that is not only functional, but also maintainable and sustainable in the long run.

    Additional Resources

    1. Bitbucket Pipelines documentation: The official documentation provides a detailed guide on setting up and using Pipelines, including how to integrate with external tools like clang-format.
    2. clang-format documentation: The official documentation provides a comprehensive guide on using clang-format to format code, including style options and command line usage.
    3. GitHub Action for clang-format: If you use GitHub instead of Bitbucket, you can check out this GitHub Action which can help you integrate clang-format checks into your workflow.
    4. Better coding practices for your organisation using clang-tidy and clang-format: This Medium post provides a practical guide to using clang-tidy and clang-format to format C++ code in a consistent and effective manner.
  • Design Patterns in C/C++: Streamlining Your Code and Enhancing Your Development Skills

    A lot has been written on design patterns, but they were woefully absent in my education as a developer. I learned about them after a few years as a developer, and they were game-changing for me. In this post, I want to share with you a few of the most useful design patterns for C/C++ that I often use in my own designs.

    summary design patterns for software development
    Image courtesy of Jason McDonald

    What are Design Patterns?

    When I first learned about design patterns, they were presented to me as the proper way to organize classes to solve specific types of problems. In fact, the first design pattern I was introduced to was the state pattern for implementing state machines. Up until that point, I was resorting to complex case statements with lots of conditional checks to implement my state machine logic, simply because that was all I knew.

    I was taught that I could encapsulate all the transition logic into state classes. I was blown away by how much that single refactor could simplify my code, and how much easier it was to add and remove states.

    Over time, I started to find that the state pattern applied to so many other problems, and then that got me into trouble. I had this shiny new tool in my toolbox and I just wanted to use it for everything!

    When Should Patterns Be Applied?

    Thanks to some great mentors, I was able to step back and start looking at other patterns that existed — the factory pattern, adapters, singletons, and strategies. All of these different tools could be used in my designs to solve different problems.

    And that is the key — design patterns are tools, nothing more. As with any tool in your toolbox, there is a time and place to use it. Design patterns are no different. For C/C++ developers, they can lead to some strikingly simple and elegant solutions, but sometimes they can be a rabbit hole that will just add unnecessary abstraction to your code when a simple, straightforward solution would do.

    As conscious developers, our goal is to design amazing applications, using the best tools available to us. We understand that if a certain tool does not work for a given problem, we move on. We don’t try to force a tool on a design, which is one of the major pitfalls for new developers once they learn about design patterns.

    Types of Design Patterns

    The literature on design patterns typically breaks all the common patterns down into three primary categories, each of which deal with a different aspect of the development process:

    1. Creational Patterns
    2. Structural Patterns
    3. Behavioral Patterns

    I’m going to share with you a few of my favorite design patterns from each category. You can find tons of information about design patterns through a simple search, however one resource I have found extremely valuable is refactoring.guru. That site has lots of good information, all of which can be applied to being a conscious developer!

    Creational Design Patterns

    Creational design patterns are used in software development to provide a way to create objects in a flexible and reusable way. They help to encapsulate the creation process of an object, decoupling it from the rest of the system. This makes it easier to change the way objects are created or to switch between different implementations.

    Creational design patterns provide a variety of techniques for object creation, such as abstracting the creation process into a separate class, using a factory method to create objects, or using a prototype to create new instances. These patterns are useful when the creation of objects involves complex logic or when the object creation process needs to be controlled by the system.

    Some examples of creational design patterns include the Singleton pattern, which ensures that only one instance of a class is created, and the Factory pattern, which provides a way to create objects without specifying the exact class of object that will be created.

    Overall, creational design patterns help to improve the flexibility and reusability of software systems by providing a more structured and standardized approach to object creation.

    Singleton Creational Pattern

    The Singleton design pattern is a creational pattern that ensures that a class has only one instance and provides a global point of access to that instance. This pattern is useful in situations where only one instance of a class should exist in the program, such as managing system resources or ensuring thread safety. However, it should be used with caution, as it can introduce global state and make testing more difficult.

    UML describing the Singleton Creational Pattern

    In C++, this can be achieved by doing two things:

    1. Make the constructor for your class private.
    2. Provide a static method to access the singleton instance.

    The static method creates the singleton instance the first time it is called and returns it on subsequent calls. Ultimately, this static method typically returns either a reference to the newly created class, or a pointer to it. I prefer to use references because they are safer, but there have been cases where I create a std::shared_ptr in my static function and return that to the caller to access the singleton.

    C++ Example of the Singleton Creational Pattern

    To implement the Singleton pattern in C++, a common approach is to define a static method in the class definition. Then, in the implementation file, define the static method and have it define a static member variable of the class type. The static method will initialize the singleton instance if it has not been created yet, and return it otherwise. Here’s an example:

    class Singleton {
    public:
      static Singleton& getInstance() {
        static Singleton instance; // The singleton instance is created on the first call to getInstance()
        return instance;
      }
    
    private:
      Singleton() {} // The constructor is private to prevent typical instantiation
      Singleton(const Singleton&) = delete; // Delete the copy constructor to prevent copying
      Singleton& operator=(const Singleton&) = delete; // Delete the assignment operator to prevent copying also
    };
    
    Code language: C++ (cpp)

    In this example, the getInstance() method returns a reference to the Singleton instance, creating it on the first call using a static variable. The constructor is private to prevent instantiation from outside the class. The copy constructor and assignment operator are deleted to prevent copying.

    Your getInstance() method can take arguments as well and pass them along to the constructor, if that is desired. You need to be aware that those are only used to create the object the first time, and never again. For this reason, I consider it to be best practice for getInstance() to not take any arguments to not confuse a user making a call.

    Abstract Factory Creational Pattern

    The Abstract Factory design pattern is a creational pattern that provides an interface for creating families of related objects without specifying their concrete classes. This pattern is useful in situations where there are multiple families of related objects, and the actual type of object to be created should be determined at runtime. For example, in a messaging application, there might be multiple types of messages that need to be created. Each type of message should be consistent in interface, but with specific behavior.

    UML describing a modified Abstract Factory Creational Pattern

    C++ Example of the Abstract Factory Creational Pattern

    In C++, this can be achieved by defining an abstract base class for each family of related objects. Then define concrete subclasses for each type of object in each family. Here’s an example from a recent project of mine:

    struct Event {
      std::string m_uuid;                  // define a UUID for the event
      std::string m_msg;                   // message related to the event
      std::chrono::time_point m_timestamp; // time event occurred
    
      virtual ~Event () {}
      virtual void handle() = 0;
      virtual void clear() = 0;
    };
    
    struct ErrorEvent : public Event {
      uint32_t m_id;  // Error ID for the specific error that occurred
    
      void handle() override {
        // Perform "handling" action specific to ErrorEvent 
      }
      void clear() override {
        // Perform "clear" action specific to ErrorEvent 
      }
    };
    
    struct StateChange : public Event {
      uint32_t m_state; // state identifier
    
      void handle() override {
        // Perform "handling" action specific to StateChange 
      }
      void clear() override {
        // Perform "clear" action specific to StateChange 
      }
    };
    
    class Component {
    public:
      template <
          class T,
          std::enable_if_t<std::is_base_of_v<Event, T>, bool> = true>
      std::shared_ptr<T> GetEvent(void)
      {
        return std::make_shared<T>();
      }
    };
    Code language: C++ (cpp)

    In this example, the Event base class defines virtual methods for handling the event and clearing the event. The ErrotEvent and StateChange classes are concrete subclasses that implement these methods for the specific events. The Component class defines a GetEvent() methods for creating Events. Now, when I want to create a new event, I just inherit the Event base class and can call GetEvent() to create a new instance of the event.

    Pay special attention to lines 37. This ensures that the compiler will give an error if I try to call GetEvent with a data type that is not derived from my Event class. And that is ideal — turning what were once run time errors into compiler errors.

    This is just one use of the Abstract Factory pattern. There are others that allow you to define specific behavior for different platforms or architectures as well!

    Structural Design Patterns

    Structural design patterns are used in software development to solve problems related to object composition and structure. These patterns help to simplify the design of a software system by defining how objects are connected to each other and how they interact.

    Structural design patterns provide a variety of techniques for object composition, such as using inheritance, aggregation, or composition to create complex objects. They also provide ways to add new functionality to existing objects without modifying their structure.

    Some examples of structural design patterns include the Adapter pattern, which allows incompatible objects to work together by providing a common interface, and the Decorator pattern, which adds new behavior to an object by wrapping it with another object.

    Overall, structural design patterns help to improve the modularity, extensibility, and maintainability of software systems by providing a more flexible and adaptable way to compose objects and structures. They are particularly useful in large and complex software systems where managing the relationships between objects can become challenging.

    Adapter Structural Pattern

    The Adapter design pattern is a structural pattern that allows incompatible interfaces to work together. In C++, this pattern is used when a class’s interface doesn’t match the interface that a client is expecting. An adapter class is then used to adapt between the two interfaces.

    UML describing the Adapter Structural Pattern

    C++ Example of the Adapter Structural Pattern

    To use this in C++, define a class that implements the interface that the client expects, and internally use an instance of the incompatible class that needs to be adapted. Here’s an example:

    class ExpectedInterface {
    public:
      virtual ~ExpectedInterface() {}
      virtual void request() = 0;
    };
    
    class IncompatibleInterface {
    public:
      void specificRequest() {
        // Perform some specific request
      }
    };
    
    class Adapter : public ExpectedInterface {
    public:
      Adapter(IncompatibleInterface* adaptee) : adaptee_(adaptee) {}
    
      void request() override {
        adaptee_->specificRequest();
      }
    
    private:
      IncompatibleInterface* adaptee_;
    };
    
    Code language: C++ (cpp)

    In this example, the ExpectedInterface class defines the interface that the client expects, which is the request() method. The IncompatibleInterface class has a method called specificRequest() that is not compatible with the ExpectedInterface. The Adapter class implements the ExpectedInterface and internally uses an instance of the IncompatibleInterface class to make the specificRequest() method compatible with the ExpectedInterface.

    Using the Adapter pattern allows us to reuse existing code that doesn’t match the interface that the client expects, without having to modify that code. Instead, we can write an adapter class that mediates between the incompatible interface and the client’s expected interface.

    Decorator Structural Pattern

    The Decorator design pattern is a structural pattern that allows adding behavior or functionality to an object, without affecting other objects of the same class. This pattern is commonly used to attach additional responsibilities to an object by wrapping it with a decorator object.

    UML describing the Decorator Structural Pattern

    C++ Example of the Decorator Structural Pattern

    To use this pattern in C++, define an abstract base class that defines the interface for both the component and the decorator classes. Then define a concrete component class that implements the base interface and a decorator class that also implements the same interface and holds a pointer to the component object it is decorating. Here’s an example:

    class Component {
    public:
      virtual ~Component() {}
      virtual void operation() = 0;
    };
    
    class ConcreteComponent : public Component {
    public:
      void operation() override {
        // Perform some operation
      }
    };
    
    class Decorator : public Component {
    public:
      Decorator(Component* component) : component_(component) {}
    
      void operation() override {
        component_->operation();
      }
    
    private:
      Component* component_;
    };
    
    class ConcreteDecoratorA : public Decorator {
    public:
      ConcreteDecoratorA(Component* component) : Decorator(component) {}
    
      void operation() override {
        Decorator::operation();
        // Add some additional operation
      }
    };
    
    class ConcreteDecoratorB : public Decorator {
    public:
      ConcreteDecoratorB(Component* component) : Decorator(component) {}
    
      void operation() override {
        // Do not call base class operation() to remove that functionality from this instance
        // Add some different additional operation
      }
    };
    Code language: C++ (cpp)

    In this example, the Component class defines the interface that the concrete component and decorator classes will implement. The ConcreteComponent class is a concrete implementation of the Component interface. The Decorator class is an abstract class that also implements the Component interface and holds a pointer to a component object it is decorating. The ConcreteDecoratorA and ConcreteDecoratorB classes are concrete decorators that add additional behavior to the ConcreteComponent object by calling the Decorator base class’s operation() method and adding their own additional behavior.

    Note that if you only needed a single decorator (i.e., you did not require decorators A and B), you can get away with simply adding the necessary additional operations directly to the Decorator class in the example. However, following the pattern as shown requires not much more work immediately and will make it easier when you need to define additional concrete decorators down the road.

    Using the Decorator pattern allows us to add or remove behavior from an object without affecting other objects of the same class. It also allows us to use composition instead of inheritance to extend the functionality of an object.

    Facade Structural Pattern

    The Facade design pattern is a structural pattern that provides a simplified interface to a complex subsystem of classes, making it easier to use and understand. A Facade class can then be used by clients to access the subsystem without having to know about the complexity of the lower-level classes.

    UML describing the Facade Structural Pattern

    C++ Example of the Facade Structural Pattern

    In C++, this pattern can be used to create a high-level interface that hides the complexity of the lower-level subsystem. To implement this pattern in C++, we can define a Facade class that provides a simplified interface to the subsystem classes. Here’s an example:

    class SubsystemA {
    public:
      void operationA() {
        // Perform some operation
      }
    };
    
    class SubsystemB {
    public:
      void operationB() {
        // Perform some operation
      }
    };
    
    class Facade {
    public:
      Facade() : subsystemA_(), subsystemB_() {}
    
      void operation() {
        subsystemA_.operationA();
        subsystemB_.operationB();
      }
    
    private:
      SubsystemA subsystemA_;
      SubsystemB subsystemB_;
    };
    
    Code language: C++ (cpp)

    In this example, the SubsystemA and SubsystemB classes represent the complex subsystem that the Facade class will simplify. The Facade class provides a simplified interface to the subsystem by hiding the complexity of the lower-level classes. The Facade class also holds instances of the subsystem classes and calls their methods to perform the operation.

    This pattern allows us to simplify the interface to a complex subsystem, making it easier to use and understand. But, my favorite use of it is to isolate clients from changes to the subsystems by abstraction.

    Behavioral Design Patterns

    Behavioral design patterns are used in software development to address problems related to object communication and behavior. These patterns provide solutions for managing the interactions between objects and for coordinating their behavior.

    Behavioral design patterns provide a variety of techniques for object communication, such as using message passing, delegation, or collaboration to manage the interactions between objects. They also provide ways to manage the behavior of objects by defining how they respond to events or changes in the system.

    Some examples of behavioral design patterns include the Observer pattern, which allows objects to be notified when a change occurs in another object, and the Command pattern, which encapsulates a request as an object, allowing it to be parameterized and queued.

    Overall, behavioral design patterns help to improve the flexibility, modularity, and extensibility of software systems by providing a more structured and standardized way to manage object communication and behavior. They are particularly useful in systems that involve complex interactions between objects, such as user interfaces, network protocols, or event-driven systems.

    Mediator Behavioral Pattern

    The Mediator design pattern is a behavioral pattern that promotes loose coupling between objects by encapsulating their communication through a mediator object. In C++, this pattern can be used to reduce dependencies between objects that communicate with each other.

    UML describing the Mediator Behavioral Pattern

    C++ Example of the Mediator Behavioral Pattern

    To implement the Mediator pattern in C++, we can define a Mediator class that knows about all the objects that need to communicate with each other. The Mediator class then provides a centralized interface for these objects to communicate through. Here’s an example:

    class Component;
    
    class Mediator {
    public:
      virtual void sendMessage(Component* sender, const std::string& message) = 0;
    };
    
    class Component{
    public:
      Component(Mediator* mediator) : mediator_(mediator) {}
    
      virtual void receiveMessage(const std::string& message) = 0;
    
    protected:
      Mediator* mediator_;
    };
    
    class ConcreteComponentA : public Component{
    public:
      ConcreteComponentA(Mediator* mediator) : Component(mediator) {}
    
      void send(const std::string& message) {
        mediator_->sendMessage(this, message);
      }
    
      void receiveMessage(const std::string& message) override {
        // Process the message
      }
    };
    
    class ConcreteComponentB : public Component{
    public:
      ConcreteComponentB(Mediator* mediator) : Component(mediator) {}
    
      void send(const std::string& message) {
        mediator_->sendMessage(this, message);
      }
    
      void receiveMessage(const std::string& message) override {
        // Process the message
      }
    };
    
    class ConcreteMediator : public Mediator {
    public:
      void addComponent(Component* component) {
        components_.push_back(component);
      }
    
      void sendMessage(Component* sender, const std::string& message) override {
        for (auto component: components_) {
          if (component != sender) {
            component->receiveMessage(message);
          }
        }
      }
    
    private:
      std::vector<Component *> components_;
    };
    Code language: C++ (cpp)

    In this example, the Component classes represent objects that need to communicate with each other. The Mediator class provides a centralized interface for the Component classes to communicate through. The ConcreteMediator class knows about all the Component objects and provides the sendMessage() method to send messages between them.

    I like to use this pattern in a system where I have multiple components, such as interfaces to external hardware modules or subsystems. Those interface classes will inherit the Component base class and communicate one with another via the Mediator. In this manner, if one Component changes its interface, then I don’t need to go change all other Component classes — just the changing Component and the necessary portions of the Mediator.

    Using the Mediator pattern allows us to reduce dependencies between objects that communicate with each other, making our code more maintainable and easier to understand. It also promotes loose coupling between objects, which makes it easier to change the way objects communicate without affecting other parts of the system.

    Strategy Behavioral Pattern

    The Strategy design pattern is a behavioral pattern that defines a family of algorithms, encapsulates each algorithm, and makes them interchangeable at runtime. This pattern allows the algorithms to vary independently from clients that use them.

    I have found this pattern extremely useful when I have an application that needs to support many protocols to various clients. Using the strategy pattern, I can easily swap which protocol is in use at any given time based on the client connection.

    UML describing the Strategy Behavioral Pattern

    C++ Example of the Strategy Behavioral Pattern

    To implement this pattern, we define an abstract Strategy class that represents the interface for all algorithms. Then, we can define concrete implementations of the Strategy class for each algorithm. In my protocol case, the Strategy class was my base Protocol class. Then I had concrete protocol classes derived from the base Protocol.

    Here’s an example:

    class Strategy {
    public:
      virtual void execute() = 0;
    };
    
    class ConcreteStrategyA : public Strategy {
    public:
      void execute() override {
        // Algorithm A implementation
      }
    };
    
    class ConcreteStrategyB : public Strategy {
    public:
      void execute() override {
        // Algorithm B implementation
      }
    };
    
    class Context {
    public:
      Context(Strategy* strategy) : strategy_(strategy) {}
    
      void setStrategy(Strategy* strategy) {
        strategy_ = strategy;
      }
    
      void executeStrategy() {
        strategy_->execute();
      }
    
    private:
      Strategy* strategy_;
    };
    Code language: C++ (cpp)

    In this example, the Strategy class represents the interface for all algorithms. The ConcreteStrategyA and ConcreteStrategyB classes represent concrete implementations of the Strategy class for two different algorithms.

    The Context class represents the client that uses the algorithms. It has a setStrategy() method to set the current algorithm and an executeStrategy() method to execute the current algorithm.

    Using the Strategy pattern allows us to change the behavior of a system at runtime by simply changing the current algorithm in the Context object. Conscious use of this pattern promotes code reuse, flexibility, and maintainability.

    State Behavioral Pattern

    I saved my favorite for last!

    The State design pattern is a behavioral pattern that allows an object to alter its behavior when its internal state changes. This pattern is useful when an object’s behavior depends on its state, and that behavior must change dynamically at runtime depending on the state.

    Essentially this boils down to defining a State base class that defines the basic structure for state information and defines the common interface, such as entry(), do(), and exit() methods.

    UML describing the State Behavioral Pattern

    C++ Example of the State Behavioral Pattern

    In C++, we can implement the State pattern using inheritance and polymorphism. We create a State base class that represents the interface for all states. Then, we create concrete implementations of the State class for each possible state of the object. Finally, we define a Context class that acts as the context in which the state machine operates. Clients utilize the interface in the Context class to manipulate the state machine.

    Here’s an example:

    class State {
    public:
        virtual void entry() = 0;
        virtual void do() = 0;
        virtual void exit() = 0;
    };
    
    class ConcreteStateA : public State {
    public:
        void entry() override {
            // Entry behavior for state A
        }
        void do() override {
            // State behavior for state A
        }
        void exit() override {
            // Exit behavior for state A
        }
    };
    
    class ConcreteStateB : public State {
    public:
        void entry() override {
            // Entry behavior for state B
        }
        void do() override {
            // State behavior for state B
        }
        void exit() override {
            // Exit behavior for state B
        }
    };
    
    class Context {
    public:
        Context(State* state) : state_(state) {}
    
        void transitionTo(State* state) {
            state_->exit();
            state_ = state;
            state_->entry();
        }
    
        void request() {
            state_->do();
        }
    
    private:
        State* state_;
    };
    Code language: C++ (cpp)

    In this example, the State class represents the interface for all states. The ConcreteStateA and ConcreteStateB classes represent concrete implementations of the State class for two different states.

    The Context class represents the object whose behavior depends on its internal state. It has a transitionTo() method to set the current state and a request() method to trigger a behavior that depends on the current state.

    You can also make use of templates in the Context class to define an addState() function. In this manner, you can enforce transitions to only a specific set of State classes and utilize custom lambda functions for the entry(), do(), and exit() functions for each state.

    Using the State pattern allows us to change the behavior of an object at runtime by simply changing its internal state. This makes our code more flexible and easier to maintain. It also promotes code reuse, as we can easily add new states by implementing new State classes.

    Conclusion

    In conclusion, design patterns are a powerful tool in software development that can help us solve common problems and improve our code’s flexibility, maintainability, and scalability. In this post, we have explored several design patterns in C++, including the Singleton, Abstract Factory, Adapter, Decorator, Facade, Mediator, Strategy, and State patterns.

    While each pattern has its unique characteristics and use cases, they all share the same goal: to provide a well-structured, reusable, and extensible solution to common software development problems. By understanding and using these patterns, we can write more efficient, robust, and maintainable code that can be easily adapted to changing requirements.

    As a conscious software developer, it’s essential to keep learning and improving our skills by exploring new ideas and concepts. Design patterns are an excellent place to start, as they can provide us with a deeper understanding of software architecture and design principles. By mastering design patterns, we can become more efficient and effective developers who can deliver high-quality, scalable, and maintainable software solutions.

    Additional Resources

    Here are a few additional resources for diving deeper into design patterns for software development.

    1. “Design Patterns: Elements of Reusable Object-Oriented Software” by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. This book is considered the definitive guide to design patterns and is a must-read for anyone interested in the subject.
    2. “Head First Design Patterns” by Eric Freeman, Elisabeth Robson, Bert Bates, and Kathy Sierra. This book offers a more accessible and engaging approach to learning design patterns and is ideal for beginners.
    3. Design Patterns in Modern C++ – This Udemy course covers the basics of design patterns and shows how to apply them using modern C++ programming techniques.
    4. C++ Design Patterns – This GitHub repository contains a collection of code examples for various design patterns in C++.
    5. Refactoring Guru – This website provides an extensive catalog of design patterns with code examples in multiple programming languages, including C++.
    6. Software Engineering Design Patterns – This Coursera course covers the principles and applications of design patterns in software engineering, including C++ examples.
  • 9 Tips for Writing Clean and Effective C/C++ Code

    Writing clean and effective code is essential for software developers. Not only does it make the code easier to maintain and update, but it also ensures that the code runs efficiently and without bugs. As a programming language, C/C++ is widely used in many applications, from system programming to game development. To help you write better C/C++ code, I’ve compiled a list of 10 tips from my laundry list of what makes good, clean, and effective C/C++ code. I hope these will guide you in making conscious decisions when coding, since many of these tips can be applied to other languages as well! So, whether you are an experienced C/C++ developer or just starting out, these tips will help you write cleaner, more efficient, and effective code.

    Tip #1: Variable Scope Awareness

    In C/C++, variables can have three different scopes: global scope, local scope, and member scope. Each of them have their place in software development and each have their own pros and cons.

    My rule of thumb is this. Make everything a local variable. If I need access to it in other object methods, I promote it to a member variable. If that still doesn’t work (which is extremely rare), then I make it a static global variable. With proper software design, I have found I never need to declare a true global variable, even if I protect it with appropriate locks.

    One last comment when dealing with global variables — you really should always make them const. The guidelines also state that you should always prefer scoped objects, rather than ones on the heap.

    Tip #2: Use Standard Types When Available

    Using standard type definitions in your C/C++ code has several benefits that can make your code more readable, portable, and maintainable. Here are some reasons why you should consider using standard type definitions in your code:

    1. Readability: Standard type definitions like size_t, int32_t, uint64_t, etc. are self-documenting and convey a clear meaning to the reader of your code. For example, using size_t instead of int to represent the size of a container makes it clear that the variable can only hold non-negative integers, which can help prevent bugs.
    2. Portability: Different platforms may have different data types with different sizes and behaviors. By using standard type definitions, you can ensure that your code is portable and will work consistently across different platforms.
    3. Type safety: Using standard type definitions can help prevent bugs caused by type mismatches, such as assigning a signed int to an unsigned int variable, or passing the wrong type of parameter as a function argument.
    4. Code maintenance: Standard type definitions can make your code easier to maintain by reducing the need for manual conversions and ensuring that the types of your variables are consistent throughout your codebase.

    Overall, using standard type definitions can help make your code more readable, portable, and maintainable, and following these recommendations can help you make conscious decisions about which type definitions to use in your code.

    Tip #3: Organize Related Data Into Objects

    When working with complex systems, it is often worthwhile to organize sets of data into objects for three primary reasons: encapsulation, abstraction, and modularity. Each of these are powerful principles that can help improve your code.

    Encapsulation

    Encapsulation is a fundamental principle of object-oriented programming and can help make your code more modular and maintainable.

    By organizing related data into an object, you can encapsulate the data and the operations that can be performed on it. This allows you to control access to the data and ensure that it is only modified in a safe and consistent way. In addition, you can make changes to the underlying data representation without changing the interface, which means that users of your object don’t have to change as well.

    Abstraction

    Objects allow you to abstract away the details of the data and provide a simplified interface for interacting with it. This can make your code easier to read and understand, as well as more resistant to changes in the underlying data representation.

    Modularity

    Organizing related data into an object can help you break down a large, complex problem into smaller, more manageable pieces. Each object can represent a distinct component of the system, with its own data and behavior, that can be developed and tested independently of the other components.

    Finally, once you have objects that you are manipulating, you can start returning those objects from your functions. Even cooler than that, you can return tuples containing your object and status information from your methods!

    Tip #4: Be Consistent in the Organization of Your Objects

    When you organize your data into objects and start defining member variables and methods, be consistent in the organization of your objects. For example, declare all public interface information up front, and keep all protected and private information at the end of the class.

    class BadExample
    {
    private:
      double m_data{73.0};
    
    public:
      BadExample();
      BadExample(const double &data) : m_data(data) {}
      ~BadExample();
    
      void SetFlag(const bool flag) { m_flag = flag; }
      void SetBytes(const std::size_t bytes) { m_nBytes = bytes; }
    
    private:
      bool m_flag{false};
      std::size_t m_nBytes{0};
    };
    
    class GoodExample
    {
    public:
      BadExample();
      BadExample(const double &data) : m_data(data) {}
      ~BadExample();
    
      void SetFlag(const bool flag) { m_flag = flag; }
      void SetBytes(const std::size_t bytes) { m_nBytes = bytes; }
    
    private:
      double m_data{73.0};
      bool m_flag{false};
      std::size_t m_nBytes{0};
    
      void SetData(const double data) { m_data = data; }
    };Code language: PHP (php)

    By declaring all private member variables and methods in a single private section, it makes the class definition much easier to read and follow. I know that when I read the GoodExample class definition that when I see the private keyword that everything coming after that keyword will be private and not accessible to me as a normal user.

    Tip #5: Place All Documentation in Header Files

    When you document your functions and variables, document them in the header file for one primary reason: keep the interface and implementation separate.

    Keeping the interface definition of your object separate from the implementation is a solid object-oriented design principle. The header file is where you define the interface for your users. That is where your users are going to look to understand what the purpose of a function is, how it should be used, what the arguments mean, and what the return value will contain. Many times the user of your object will not have access to the source code, so placing documentation there is pointless, from an interface perspective.

    Tip #6: Enforce a Coding Style

    Enforcing a code style can bring several benefits to your development process, including:

    1. Consistency: By enforcing a code style, you can ensure that your codebase looks consistent across different files and modules. This can make your code easier to read and understand, and can help reduce the amount of time developers spend trying to figure out how different parts of the codebase work.
    2. Maintainability: A consistent code style can also make your code easier to maintain, as it can help you identify patterns and common practices that are used throughout the codebase. This can make it easier to update and refactor the code, as you can more easily find and update all instances of a particular pattern.
    3. Collaboration: Enforcing a code style can also make it easier to collaborate with other developers, especially if they are working remotely or in different time zones. By using a consistent code style, developers can more easily understand each other’s code and can quickly identify where changes need to be made.
    4. Automation: Enforcing a code style with clang-format can also help automate the code review process, as it can automatically format code to the desired style. This can save time and effort in the code review process, and can ensure that all code is formatted consistently, even if developers have different preferences or habits.
    5. Industry standards: Many organizations and open-source projects have established code style guidelines that are enforced using tools like clang-format. By following these standards, you can ensure that your codebase adheres to best practices and can more easily integrate with other projects.

    Tip #7: Be const-Correct in All Your Definitions

    A major goal of mine when working in C and C++ is to make as many potential pitfalls and runtime bugs compiler errors rather than runtime errors. Striving to be const-correct in everything accomplishes a few things for the conscious coder:

    1. It conveys intent about what the method or variable should do or be. A const method cannot modify an object’s state, and a const variable cannot change its value post-declaration. This can make your code safer and reduce the risk of bugs and unexpected behavior.
    2. It makes your code more readable, as it can signal to other developers that the value of the object is not meant to be changed. This can make it easier for other developers to understand your code and can reduce confusion and errors.
    3. It allows the compiler to make certain optimizations that can improve the performance of your code. For example, the compiler can cache the value of a const object, which can save time in certain situations.
    4. It promotes a consistent coding style, making it easier for other developers to work with your code and reduce the risk of errors and confusion.
    5. It makes your code more compatible with other libraries and frameworks. Many third-party libraries require const-correctness in order to work correctly, so adhering to this standard can make it easier to integrate your code with other systems.

    Here are a couple of examples:

    class MyConstCorrectClass
    {
    public:
      MyConstCorrectClass() = default;
    
      void SetFlag(const bool flag) { m_flag = flag; } // Method not marked const because it modifies the state
                                                       // The argument is immutable though, and is thus marked const
      bool GetFlag() const { return m_flag' } // Marked as const because it does not modify state
    
    private:
      bool m_flag{false};
    };
    
    void function1(void)
    {
      MyConstCorrectClass A;
      A.SetFlag(true);
      std::cout << "A: " << A.GetFlag() << std::endl;
    
      const MyConstCorrectClass B;
      B.SetFlag(true);   // !! Compiler error because B is constant
      std::cout << "B: " << B.GetFlag() << std::endl;
    }Code language: PHP (php)

    Tip #8: Wrap Single-line Blocks With Braces

    Single-line blocks, such as those commonly found in if/else statements, should always be wrapped in braces. Beyond the arguments that it increases readability, maintainability, and consistency, for me this is a matter of safety. Consider this code:

    if (isSafe())
      setLED(LED::OFF);Code language: C++ (cpp)

    What happens when I need to take additional action when the function returns true? Sleeping developers would simply add the new action right after the setLED(LED::OFF) statement like this:

    if (isSafe())
      setLED(LED::OFF);
      controlLaser(LASER::ON, LASER::HIGH_POWER);
    Code language: C++ (cpp)

    Now consider the implications of such an action. The controlLaser(LASER::ON, LASER::HIGH_POWER); statement gets run every single time, not just if the function isSafe() returns true. This has serious consequences, which is exactly why you should always wrap your single-line blocks with braces!

    if (isSafe())
    {
      setLED(LED::OFF);
      controlLaser(LASER::ON, LASER::HIGH_POWER);
    }
    Code language: C++ (cpp)

    Tip #9: Keep Your Code Linear — Return from One Spot

    This is also known as the “single exit point” principle, but the core of it is that you want your code to be linear. Linear code is easier to read, to maintain, and debug. When you return from a function in multiple places, this can lead to hard to follow logic that obscures what the developer is really trying to accomplish. Consider this example:

    std::string Transaction::GetUUID(void) const
    {
      std::string uuid = xg::Guid();  // empty ctor for xg::Guid gives a nil UUID
      if (m_library->isActionInProgress())
      {
        return m_library->getActionIdInProgress();
      }
      return uuid;
    }
    Code language: C++ (cpp)

    This seems fairly simple to follow and understand, but it doesn’t follow the single exit point principle — the flow of the method is non-linear. If the logic in this function ever gets more complex, this can quickly get harder to debug. This simple change here ensures that the flow is linear and that future modifications follow suit.

    std::string Transaction::GetUUID(void) const
    {
      std::string uuid = xg::Guid();  // empty ctor for xg::Guid gives a nil UUID
      if (m_library->isActionInProgress())
      {
        uuid = m_library->getActionIdInProgress();
      }
      return uuid;
    }
    Code language: C++ (cpp)

    You may argue that the first function is slightly more efficient because you save the extra copy to the temporary variable uuid. But most any modern compiler worth using will optimize that copy out, and you’re left with the same performance in both.

    A quick bit of wisdom — simple code, even if it has more lines, more assignments, etc. is more often than not going to result in better performance than complex code. Why? The optimizer can more readily recognize simple constructs and optimize them than it can with complex algorithms that perform the same function!


    Conclusion

    In this post, we covered a variety of topics related to C++ programming best practices. We discussed the benefits of using standard type definitions, the importance of organizing related data into objects, the placement of function documentation comments, the use of clang-format to enforce code style, the significance of being const-correct in all your definitions, and the reasons why it is important to wrap single-line blocks with braces and to return from only a single spot in your function.

    By adhering to these best practices, C++ programmers can create code that is more readable, maintainable, and easy to debug. These principles help ensure that code is consistent and that common sources of errors, such as memory leaks or incorrect program behavior, are avoided.

    Overall, by following these best practices, C++ programmers can create high-quality, efficient, and robust code that can be easily understood and modified, even as the codebase grows in size and complexity.

  • Mastering Variable Scopes in C/C++: Best Practices for Clean and Effective Code

    As software developers, we rely on variables to store and manipulate data in our programs. However, it is crucial to understand the scope of a variable and how it affects its accessibility and lifetime. In C and C++, the scope of a variable determines where in the program it can be used and for how long it will exist. In this blog post, we will be exploring the different types of scopes in C/C++ and the best practices for handling them to write clean, maintainable, and effective code.

    We will look at global, local, and member scopes and how they affect the lifetime of variables. We will also discuss how to properly handle pointers, which have their own unique set of considerations when it comes to scope. By understanding the different types of scopes and how to handle them, you will be equipped to make conscious decisions about how you use variables in your code, leading to more reliable, efficient, and maintainable programs.

    Variable Scope Awareness

    Awareness of variable lifetimes and scopes, particularly when it comes to pointers, is critical to writing clean and effective C/C++ code. The lifetime of a variable is the period of time during which it is allocated memory and exists in the program. In C/C++, variables can have three different scopes: global scope, local scope, and member scope.

    Global Scope Variables

    Global scope variables are declared outside of all functions and are accessible throughout the entire program. They have a longer lifetime and persist throughout the execution of the program, but using too many global scope variables can lead to cluttered code and potential naming conflicts. However, in my mind, the more serious implications of improper use of a global variable is race conditions.

    A race condition occurs when two or more threads access a shared resource, such as a global variable, simultaneously and the final result depends on the timing of the access. In a safety critical environment, where errors in the system can have severe consequences, race conditions can cause significant harm.

    // Example of a global variable, including a potential race condition
    int32_t g_temperature_C = 0;
    
    void thread1(void)
    {
      // Read the temperature from the sensor
      g_temperature_C = ReadTemperatureFromSensor();
    }
    
    void thread2(void)
    {
      if ((g_temperature_C > 0) && (g_temperature_C < 70)) // !! Simple race condition
      {
        // Do some safety critical work
      }
      else
      {
        // Manage temperature out of bounds (i.e., cool down or heat up)
      }
    }
    Code language: C++ (cpp)

    In the example above, thread2 is doing some safety critical work, but only when g_temperature_C is within a certain range, which is updated in thread1. If the temperature is out of bounds, then the system needs to take a different action. The issue here is that the wrong action can lead to serious consequences, either for the safety of the system, or in the case where humans are involved, the safety of the user.

    In this case, a global variable is a poor choice of scope for g_temperature_C.

    If you find you do have to use global variables, you can still limit their scope to the specific compilation unit where they are defined (i.e., the file where the variable is declared). You can do this by adding the static keyword to the variable declaration. The advantage to this is that it limits the scope of the variable to just the specific module, rather than the entire program.

    // Limit scope of global variable to the specific compilation unit (i.e., this file)
    static int32_t g_temperature_C = 0;
    Code language: C++ (cpp)

    Local Scope Variables

    Local scope variables, on the other hand, are declared within a function or block and are only accessible within that specific scope. They have a shorter lifetime, are allocated on the stack, and are automatically deallocated from memory once the function or block has finished execution. Using local scope variables is recommended over global variables as they limit the potential for naming conflicts, allow for cleaner code, and also eliminate race conditions.

    // Example of a local variable, resolving the race condition above
    void thread2(void)
    {
      int32_t l_temperature_C = ReadTemperatureFromSensor();
      if ((l_temperature_C > 0) && (l_temperature_C < 70)) // !! NO race condition
      {
        // Do some safety critical work
      }
      else
      {
        // Manage temperature out of bounds (i.e., cool down or heat up)
      }
    }
    Code language: C++ (cpp)

    As you can see, the race condition from using a global variable is avoided here because the variable is local and cannot be changed outside of this function.

    Member Scope Variables

    Member scope variables, also known as class member variables, are declared within a class and are accessible by all member functions of that class. Their scope is tied to the lifetime of the object they are a member of.

    #include <iostream>
    
    class TemperatureSensor
    {
    public:
      TemperatureSensor() = default;
    
      void GetTemperature()
      {
        m_temp_C = ReadTemperatureFromSensor();
        return m_temp_C;
      }
    
    private:
      int32_t m_temp_C{0};
    };
    
    int main()
    {
      TemperatureSensor sensor;
      std::cout << "Temperature: " << sensor.GetTemperature() << std::endl;
      std::cout << "Temperature: " << sensor.GetTemperature() << std::endl;
    
      return 0;
    }Code language: PHP (php)

    You can think of the scope of member variables to be similar to that of static global variables. Instead of being limited to the compilation unit where they are declared, they are limited to the scope of the class that they are part of. Race conditions on member variables are a real possibility. Precautions must be taken to ensure you avoid them, such as proper locking or an improved architecture to avoid the race altogether.

    Properly Scoping Pointers

    Pointers are a powerful tool in C and C++, allowing you to efficiently work with data objects in your programs. However, naive usage of pointers can lead to significant problems, including hard to find bugs and difficult to maintain code.

    In C and C++, pointers have their own lifetime, separate from the objects they point to. When a pointer goes out of scope, the object it referenced remains in memory but is no longer accessible. When dynamically allocating memory, this leads to memory leaks where the memory is not properly deallocated, leading to a buildup of memory usage over time.

    Smart Pointers

    To prevent memory leaks and ensure that your programs are efficient and reliable, it is important to handle pointers with care. Modern C++ provides smart pointers types, which automatically manage the lifetime of objects and deallocate them when they are no longer needed. Using smart pointer types of std::shared_ptr and std::unique_ptr, you can be assured that when you create (and allocate) a pointer to an object, that object is constructed (and initialized if following RAII principles) and the pointer is valid. Then, when that pointer goes out of scope, the object is destructed and the memory is deallocated.

    #include <memory>
    #include <iostream>
    
    void PrintTemperature()
    {
      // Create a unique pointer to a TemperatureSensor object
      std::unique_ptr<TemperatureSensor> pTS = std::make_unique<TemperatureSensor>();
      
      // Use the unique pointer within the scope of the current function
      std::cout << "Temperature: " << pTS->GetTemperature() << std::endl;
      
      // The unique pointer goes out of scope at the end of the main function
      // and its dynamically allocated memory is automatically deallocated
    
    }Code language: PHP (php)

    When working with raw pointers, it’s critical to be aware of the lifetime of the objects being pointed to. For example, if the lifetime of the object ends before the pointer is deallocated, the pointer becomes a “dangling pointer”. This can cause undefined behavior, such as crashing the program or returning incorrect results. Smart pointers are typically a better choice and avoid this risk by managing the lifetime of the object themselves.


    In conclusion, understanding and properly handling the scope of variables in C/C++ is a crucial aspect of writing clean, maintainable, and effective code. By becoming familiar with global, local, and member scopes, and considering the lifetime and accessibility of variables, you can make informed decisions about how to use variables in your programs.

    Additionally, pointers require their own set of considerations when it comes to scope, and it is essential to handle them with care to prevent memory leaks and other issues.

    By following best practices and being aware of the potential pitfalls, you can ensure that your programs are reliable, efficient, and easy to maintain.

  • Mastering CPM: 6 Tips for Effectively Managing Dependencies with CMake’s Package Manager

    Include the right version right in your repo; provide a custom command to update CPM.cmake — example

    Setup cmake_modules path — example CMake

    Best Practices

    • Avoid short hand
    • Keep package dependencies with their targets

    CMake is a cross-platform build system that can be used to build, test, and package software projects. One of the most powerful features of CMake is the ability to manage dependencies, and using CMake Package Manager (CPM) make that a breeze. CPM is a package manager for CMake that allows you to easily download and use libraries and other dependencies in your project without having to manually manage them.

    Using CPM effectively can greatly simplify the process of managing dependencies in your project. Here are a few tips to help you get the most out of CPM:

    1. Start by adding CPM to your project. This can be done by adding the following line to the top of your CMakeLists.txt file (note that you will need to have a cmake/CPM.cmake in that path relative to your CMakeLists.txt file). You can find up-to-date versions of CPM.cmake here (documentation is here).
    include(cmake/CPM.cmake)Code language: PHP (php)
    1. Next, specify the dependencies you need for your project by adding CPMAddPackage commands. For example, to add the msgpack-c library, you would add the following stanza:
    CPMAddPackage(
      NAME msgpack
      GIT_TAG cpp-4.1.1
      GITHUB_REPOSITORY "msgpack/msgpack-c"
      OPTIONS "MSGPACK_BUILD_DOCS OFF" "MSGPACK_CXX20 ON" "MSGPACK_USE_BOOST OFF"
    )
    Code language: JavaScript (javascript)
    1. Once you have added all the dependencies you require, you can use them in your project by including the appropriate headers and linking against the appropriate libraries. Note that CPM will pull the dependency from the repository specified and then run CMake (if the project uses CMake to build). Because CMake is run for the project, the CMake targets are available for you to use. For example, to use the msgpack-c library in your project, you would add the following lines to your CMakeLists.txt file:
    target_link_libraries(your_target msgpackc-cxx)
    1. CPM also allows you to specify options for modules, such as disabling tests or building in a specific configuration. To specify options for a module, you can use the OPTIONS argument, as shown above.
    2. When the dependency does not have a CMakeLists.txt file, CPM will still checkout the repository, but will not configure it. In that case, you are required to write your own CMake to perform the build as required. For example, the embedded web server called Mongoose from Cesanta does not provide a CMakeLists.txt to build it, but we can still pull it in like this (note the use of the CPM generated variable mongoose_SOURCE_DIR):
    CPMAddPackage(
      NAME mongoose
      GIT_TAG 7.8
      GITHUB_REPOSITORY "cesanta/mongoose"
    )
    if (mongoose_ADDED)
      add_library(mongoose SHARED ${mongoose_SOURCE_DIR}/mongoose.c)
      target_include_directories(mongoose SYSTEM PUBLIC ${mongoose_SOURCE_DIR})
      target_compile_definitions(mongoose PUBLIC MG_ENABLE_OPENSSL)
      target_link_libraries(mongoose PUBLIC ssl crypto)
      install(TARGETS mongoose)
    endif(mongoose_ADDED)Code language: PHP (php)
    1. Add dependencies with CPM in the same CMakeLists.txt as the target that uses the dependency. If multiple targets use the same dependency, CPM will not pull multiple copies, rather it will use the copy already downloaded. By doing this, you ensure that if you ever refactor your CMake, or pull a CMakeLists.txt for a module, you get all the dependencies and don’t miss anything.

    CPM is a powerful tool that can help simplify the process of managing dependencies in your project. By following the tips outlined in this blog, you can effectively use CPM to manage your dependencies, ensuring that your project is always up-to-date and easing the burden of keeping your dependencies up-to-date as well. With the right approach, CPM can help you save time and effort when managing your project dependencies, allowing you to focus on building and delivering your project.

  • Getting Started with CMake: A Beginner’s Guide to Building Your Project

    CMake is an open-source, cross-platform build system that helps developers to manage their projects and build them on different platforms. It is widely used in the software development community, especially for C and C++ projects. In this blog post, we will explore how to use CMake effectively to manage your projects and improve your workflow as a software developer.

    An Example CMakeLists.txt

    First, let’s start with the basics of CMake. CMake uses a simple, human-readable language called CMakeLists.txt to describe the build process of a project. This file contains instructions on how to find and configure dependencies, set compiler flags, and create the final executable or library. Here is an example of how I typically define my CMake from my open-source ZeroMQ-based RPC library.

    ###############################################################################
    # CMakeLists.txt for zRPC library
    #  - Creates a CMake target library named 'zRPC'
    ###############################################################################
    cmake_minimum_required(VERSION 3.14 FATAL_ERROR)
    
    # Define the project, including its name, version, and a brief description
    project(zRPC
            VERSION "0.0.1"
            DESCRIPTION "0MQ-based RPC client/server library with MessagePack support"
           )
    
    # Define CMake options to control what targets are generated and made available to build
    option(ZRPC_BUILD_TESTS "Enable build of unit test applications" ON)
    
    # Setup default compiler flags
    set(CMAKE_C_STANDARD 11)
    set(CMAKE_C_STANDARD_REQUIRED ON)
    set(CMAKE_CXX_STANDARD 20)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)
    set(compile_options -pedantic-errors
                        -pedantic
                        -Wall
                        -Wextra
                        -Wconversion
                        -Wsign-conversion
                        -Wno-psabi
                        -Werror
        CACHE INTERNAL "Compiler Options"
       )
    
    ###############################################################################
    # Bring in CPM
    ###############################################################################
    include(cmake/CPM.cmake)
    
    ###############################################################################
    # Bring in CPPZMQ header-only API
    ###############################################################################
    CPMAddPackage(
      NAME cppzmq
      VERSION 4.8.1
      GITHUB_REPOSITORY "zeromq/cppzmq"
      OPTIONS "CPPZMQ_BUILD_TESTS OFF"
    )
    
    ###############################################################################
    # Bring in MSGPACK-C header-only API
    ###############################################################################
    CPMAddPackage(
      NAME msgpack
      GIT_TAG cpp-4.1.1
      GITHUB_REPOSITORY "msgpack/msgpack-c"
      OPTIONS "MSGPACK_BUILD_DOCS OFF" "MSGPACK_CXX20 ON" "MSGPACK_USE_BOOST OFF"
    )
    
    ###############################################################################
    # Bring in C++ CRC header-only API
    ###############################################################################
    CPMAddPackage(
      NAME CRCpp
      GIT_TAG release-1.1.0.0
      GITHUB_REPOSITORY "d-bahr/CRCpp"
    )
    if(CRCpp_ADDED)
      add_library(CRCpp INTERFACE)
      target_include_directories(CRCpp SYSTEM INTERFACE ${CRCpp_SOURCE_DIR}/inc)
    endif(CRCpp_ADDED)
    
    ###############################################################################
    # zRPC library
    ###############################################################################
    add_library(${PROJECT_NAME} SHARED)
    target_include_directories(${PROJECT_NAME} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
    target_link_libraries(${PROJECT_NAME} PUBLIC cppzmq msgpackc-cxx CRCpp pthread)
    target_sources(${PROJECT_NAME} PRIVATE  src/zRPCClient.cpp
                                            src/zRPCServer.cpp
                                            src/zRPCPublisher.cpp
                                            src/zRPCSubscriber.cpp
                                   PUBLIC   include/zRPC.hpp
                  )
    target_compile_options(${PROJECT_NAME} PUBLIC ${compile_options})
    
    
    ###############################################################################
    # Test applications
    ###############################################################################
    if (ZRPC_BUILD_TESTS)
      add_executable(client tests/client.cpp)
      target_link_libraries(client zRPC)
      target_compile_options(client PUBLIC ${compile_options})
    
      add_executable(server tests/server.cpp)
      target_link_libraries(server zRPC)
      target_compile_options(server PUBLIC ${compile_options})
    
      add_executable(publisher tests/publisher.cpp)
      target_link_libraries(publisher zRPC)
      target_compile_options(publisher PUBLIC ${compile_options})
    
      add_executable(subscriber tests/subscriber.cpp)
      target_link_libraries(subscriber zRPC)
      target_compile_options(subscriber PUBLIC ${compile_options})
    
      include(cmake/CodeCoverage.cmake)
      append_coverage_compiler_flags()
      add_executable(unittest tests/unit.cpp)
      target_link_libraries(unittest zRPC)
      target_compile_options(unittest PUBLIC ${compile_options})
    
      setup_target_for_coverage_gcovr_xml(NAME ${PROJECT_NAME}_coverage
                                          EXECUTABLE unittest
                                          DEPENDENCIES unittest
                                          BASE_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
                                          EXCLUDE "tests"
    				                             )
    endif(ZRPC_BUILD_TESTS)
    Code language: CMake (cmake)

    Once you have your CMakeLists.txt file created, you can use the CMake command-line tool to generate the build files for a specific platform, such as Makefiles or Visual Studio project files. It is considered best practice to keep your build files separated from your source files, so I am in the habit of creating a “_bld” directory for that purpose.

    mkdir _bld; cd _bld
    cmake ..

    CMake Targets

    Targets are the basic building blocks of a CMake project. They represent the executable or library that is built as part of the project. Each target has a unique name and is associated with a set of source files, include directories, and libraries that are used to build it.

    CMake also supports creating custom targets, which can be used to run arbitrary commands as part of the build process, such as running tests or generating documentation. You can specify properties for the target, like include directories, libraries, or compile options. You can also specify dependencies between the targets, so that when one target is built, it will automatically build any targets it depends on.

    This is a really powerful feature that CMake provides because when I define my library target, I define what it needs to build such as the source files, includes, and external libraries. Then, when I define my executable, I only need to specify the library that it depends on — the requisite includes and other libraries that need to be linked in come automatically!

    add_library(${PROJECT_NAME} SHARED)
    target_include_directories(${PROJECT_NAME} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
    target_link_libraries(${PROJECT_NAME} PUBLIC cppzmq msgpackc-cxx CRCpp pthread)
    target_sources(${PROJECT_NAME} PRIVATE  src/zRPCClient.cpp
                                            src/zRPCServer.cpp
                                            src/zRPCPublisher.cpp
                                            src/zRPCSubscriber.cpp
                                   PUBLIC   include/zRPC.hpp
                  )
    target_compile_options(${PROJECT_NAME} PUBLIC ${compile_options})
    
    # The executable only needs to depend on zRPC now,
    # not all the other dependencies and include directories
    add_executable(client tests/client.cpp)
    target_link_libraries(client zRPC)Code language: PHP (php)

    Dependency Management

    One of the most important aspects of CMake is its ability to help you find and use dependencies. CMake provides a number of built-in commands that can be used to find and configure dependencies, such as find_package and find_library. These commands can be used to locate and configure external libraries, such as Boost or OpenCV, and make them available to your project. This can save you a lot of time and effort compared to manually configuring dependencies for each platform, which is how it was done with plain Makefiles in the past.

    In my example above, I use a tool called CPM, or the CMake Package Manager. This is an abstraction of the find_package and find_library methods available in the CMake language. One huge advantage of this tool is that it can not only be used to find and use local packages, but it can be used to pull packages at a specific version or tag from remote git repositories. You can see how I used this to pull in the cppzmq, msgpack, and CRCpp packages that my library depends on.

    ###############################################################################
    # Bring in CPM
    ###############################################################################
    include(cmake/CPM.cmake)
    
    ###############################################################################
    # Bring in CPPZMQ header-only API
    ###############################################################################
    CPMAddPackage(
      NAME cppzmq
      VERSION 4.8.1
      GITHUB_REPOSITORY "zeromq/cppzmq"
      OPTIONS "CPPZMQ_BUILD_TESTS OFF"
    )
    
    ###############################################################################
    # Bring in MSGPACK-C header-only API
    ###############################################################################
    CPMAddPackage(
      NAME msgpack
      GIT_TAG cpp-4.1.1
      GITHUB_REPOSITORY "msgpack/msgpack-c"
      OPTIONS "MSGPACK_BUILD_DOCS OFF" "MSGPACK_CXX20 ON" "MSGPACK_USE_BOOST OFF"
    )
    
    ###############################################################################
    # Bring in C++ CRC header-only API
    ###############################################################################
    CPMAddPackage(
      NAME CRCpp
      GIT_TAG release-1.1.0.0
      GITHUB_REPOSITORY "d-bahr/CRCpp"
    )
    if(CRCpp_ADDED)
      add_library(CRCpp INTERFACE)
      target_include_directories(CRCpp SYSTEM INTERFACE ${CRCpp_SOURCE_DIR}/inc)
    endif(CRCpp_ADDED)
    Code language: CMake (cmake)

    Cross-Platform Build Support

    Another powerful feature of CMake is its ability to generate build files for multiple platforms. For example, you can use CMake to generate Makefiles for Linux, Visual Studio project files for Windows, or Xcode project files for macOS. This allows you to easily build and test your project on different platforms, without having to manually configure the build process for each one.

    # Basic command to generate Makefiles (Linux/MacOS)
    cmake -G "Unix Makefiles" ..
    
    # Basic command to generate Visual Studio build files
    cmake -G "Visual Studio 16" -A x64 ..
    
    # More complex command from the VS Code CMake extension performing cross-compilation for ARM
    /usr/bin/cmake --no-warn-unused-cli -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE \
        -DCMAKE_BUILD_TYPE:STRING=Release 
        -DCMAKE_C_COMPILER:FILEPATH=/usr/bin/arm-linux-gnueabihf-gcc-10 
        -DCMAKE_CXX_COMPILER:FILEPATH=/usr/bin/arm-linux-gnueabihf-g++-10 
        -DARCH:STRING=armv7 -DENABLE_TESTS:STRING=ON 
        -S/workspaces/zRPC -B/workspaces/zRPC/_bld/ARM/Release 
        -G "Unix Makefiles"Code language: PHP (php)

    Best Practices

    To improve your workflow with CMake, there are a few best practices that you should follow:

    • Keep your CMakeLists.txt files small and organized. The build process of a project can become complex, so it’s important to keep your CMakeLists.txt files well-organized and easy to understand.
    • Use variables to define common build options, such as compiler flags or library paths. This makes it easy to change these options globally, without having to modify multiple parts of your CMakeLists.txt files.
    • Use include() and add_subdirectory() commands to split your project into smaller, more manageable parts. This makes it easier to understand the build process, and also makes it easy to reuse parts of your project in other projects. I have found that many, small CMake files are easier to manage and maintain than fewer, large CMake files.
    • Use the install() command to specify where the final executable or library should be installed. This makes it easy to distribute your project to other users.
    • Use the add_custom_command() and add_custom_target() commands to add custom build steps to your project. For example, you can use these commands to run a script that generates source code files or to run a test suite after building.

    By following these best practices, you can effectively use CMake to manage your projects and improve your workflow as a software developer. CMake is a powerful tool that can save you a lot of time and effort, and by mastering its features, you can build and distribute your projects with ease.

  • 4 Must-Have C++17 Features for Embedded Developers

    C++17 is a version of the C++ programming language that was standardized in 2017, and adds additonal new features and improvements to what is considered “modern C++”. Some major new features that I have really loved in C++17 include:

    • Structured bindings: Allows you to easily extract multiple variables from a single tuple or struct, and to use them in a more readable way.
    • Inline variables: Allows for the definition of variables with the “inline” keyword in classes and elsewhere.
    • New attributes: Allows for more readable code by marking certain areas with specific attributes that the compiler understands.
    • std::shared_mutex: Allows for multiple “shared” locks and a single “exclusive” lock. This is basically a standard read/write mutex!

    In embedded systems, support for C++17 will depend on the compiler and platform you are using. Some more popular compilers, such as GCC and Clang, have already added support for C++17. However, support for C++17 for your project may be limited due to the lack of resources and the need to maintain backwards compatibility with older systems.

    A good summary of the new features and improvements to the language can be found on cppreference.com. They also provide a nice table showing compiler support for various compilers.

    Structured Bindings

    I already wrote about my love for structured bindings in another post (and another as well), but this article would be incomplete without listing this feature!

    My main use of structured bindings is to extract named variables from a std::tuple, whether that be a tuple of my own creation or one returned to me by a function.

    But I just realized there is another use for them that makes things so much better when iterating over std::maps:

    // What I used to write
    for (const auto &val : myMap)
    {
        // Print out map data, key:value
        std::cout << val.first << ':' << val.second << std::endl;
    }
    
    // What I now write
    for (const auto &[key, value] : myMap)
    {
        // Print out map data, key:value
        std::cout << key << ':' << value << std::endl;
    }Code language: PHP (php)

    The second for loop is much more readable and much easier for a maintainer to understand what is going on. Conscious coding at its best!

    Inline Variables

    An inline variable is a variable that is defined in the header file and is guaranteed to be the same across all translation units that include the header file. This means that each compilation unit will have its own copy of the variable, as opposed to a single shared copy like a normal global variable.

    One of the main benefits of inline variables is that they allow for better control over the storage duration and linkage of variables, which can be useful for creating more flexible and maintainable code.

    I find this new feature most useful when declaring static variables inside my class. Now I can simply declare a static class variable like this:

    class MyClass
    {
      public:
        static const inline std::string MyName = "MyPiClass";
        static constexpr inline double MYPI = 3.14159265359;
        static constexpr double TWOPI = 2*MYPI; // note that inline is not required here because constexpr implies inline
    };Code language: PHP (php)

    This becomes especially useful when you would like to keep a library to a single header, and you can avoid hacks that complicate the code and make it less readable.

    New C++ Attributes

    C++17 introduces a few standard attributes that make annotating your code for the compiler much nicer. These attributes are as follows.

    [[fallthrough]]

    This attribute is used to allow case body to fallthrough to the next without compiler warnings.

    In the example given by C++ Core Guidelines ES.78, having a case body fallthrough to the next case leads to hard to find bugs, and it just is plain hard to read. However, there are certain instances where this is absolutely appropriate. In those cases, you simply add the [[fallthough]]; attribute where the break statement would normally go.

    switch (eventType) {
    case Information:
        update_status_bar();
        break;
    case Warning:
        write_event_log();
        [[fallthough]];
    case Error:
        display_error_window();
        break;
    }Code language: JavaScript (javascript)

    [[maybe_unused]]

    This attribute is used to mark entities that might be unused, to prevent compiler warnings.

    Normally, when writing functions that take arguments that the body does not use, the approach has been to cast those arguments to void to eliminate the warning. Many static analyzers actually suggest this as the suggestion. The Core Guidelines suggest to simply not provide a name for those arguments, which is the preferred approach.

    However, in the cases where the argument is conditionally used, you can mark the argument with the [[maybe_unused]] attribute. This communicates to the maintainer that the argument is not used in all cases, but is still required.

    RPC::Status RPCServer::HandleStatusRequest(
        const RPC::StatusRequest &r [[maybe_unused]])
    {
      if (!m_ready)
      {
        return RPC::Status();
      }
      return ProcessStatus(r);
    }Code language: PHP (php)

    This attribute can also be used to mark a static function as possible unused, such as if it is conditionally used based on whether DEBUG builds are enabled.

    [[maybe_unused]] static std::string toString(const ProcessState state);Code language: PHP (php)

    [[nodiscard]]

    This attribute is extremely useful when writing robust and reliable library code. When marking a function or method with this attribute, the compiler will generate errors if a return value is not used.

    Many times, developers will discard return values by casting the function to void, like this.

    (void)printf("Testing...");Code language: JavaScript (javascript)

    This is against the Core Guidelines ES-48, but how do you get the compiler to generate errors for your functions in a portable, standard way? With [[nodiscard]]. When a developer fails to check the return value of your function (i.e., they don’t store the result in some variable or use it in a conditional), the compiler will tell them there is a problem.

    // This code will generate a compiler error
    [[nodiscard]] bool checkError(void)
    {
      return true;
    }
    
    int main(void)
    {
      checkError();
      return 0;
    }
    
    // Error generated
    scratch.cpp:36:13: error: ignoring return value of ‘bool checkError()’, declared with attribute ‘nodiscard’ [-Werror=unused-result]
       36 |   checkError();
    
    
    // However, this takes care of the issue because we utilize the return value
    if (checkError())
    {
      std::cout << "An error occurred!" << std::endl;
    }Code language: JavaScript (javascript)

    I love how this can be used to make your users think about what they are doing!

    Shared (Read/Write) Mutex

    A shared lock allows multiple threads to simultaneously read a shared resource, while an exclusive lock allows a single thread to modify the resource.

    In many instances, it is desirable to have a lock that protects readers from reading stale data. That is the whole purpose of a mutex. However, with a standard mutex, if one reader holds the lock, then additional readers have to wait for the lock to be released. When you have many readers trying to acquire the same lock, this can result in unnecessarily long wait times.

    With a shared lock (or a read/write mutex), the concept of a read lock and a write lock are introduced. A reader must wait for the write lock to be released, but will simply increment the read lock counter when taking the lock. A writer, on the other hand, must wait for all read and write locks to be released before it can acquire a write lock. Essentially, readers acquire and hold a shared lock, while writers acquire and hold an exclusive lock.

    Here is an example of how to use a shared_mutex:

    #include <iostream>
    #include <thread>
    #include <mutex>
    
    std::shared_mutex mtx;
    int sharedCount = 0;
    
    void writevalue()
    {
        for (int i = 0; i < 10000; ++i)
        {
            // Get an exclusive (write) lock on the shared_mutex
            std::unique_lock<std::shared_mutex> lock(mtx);
            ++sharedCount ;
        }
    }
    
    void read()
    {
        for (int i = 0; i < 10000; ++i)
        {
            // Get a shared (read) lock on the shared_mutex
            std::shared_lock<std::shared_mutex> lock(mtx);
            std::cout << sharedCount << std::endl;
        }
    }
    
    int main()
    {
        std::thread t1(writevalue);
        std::thread t2(read);
        std::thread t3(read);
    
        t1.join();
        t2.join();
        t3.join();
    
        std::cout << sharedCount << std::endl;
        return 0;
    }
    
    Code language: C++ (cpp)

    Here you can see that std::shared_mutex is used to protect the shared resource sharedCount. Thread t1 increments the counter using an exclusive lock, while threads t2 and t3 read the counter using shared locks. This allows for concurrent read operations and exclusive write operation. It greatly improves performance where you have a high number of read operations versus write operations.

    With C++17, this type of lock is standardized and part of the language. I have found this to be extremely useful and makes my code that much more portable when I need to use this type of mutex!


    C++17 offers a wide range of new features that provide significant improvements to the C++ programming language. The addition of structured bindings, inline variables, and the new attributes make code more readable and easier to maintain, and the introduction of the “std::shared_mutex” type provides performance improvements in the situations where that type of lock makes sense. Overall, C++17 provides an even more modern and efficient programming experience for developers. I encourage you to start exploring and utilizing these new features in your own projects!

  • Optimizing Boot Time on an iMX6ULL Processor

    I recently had the opportunity, or the critical need rather, to optimize the boot time of some applications on an iMX6ULL processor. For anyone unfamiliar with this processor, it is a single core ARM processor running a Cortex-A7 core at up to 900 MHz. It provides a number of common interfaces for a microprocessor, including 16-bit DDR, NAND/NOR flash, eMMC, Quad SPI, UART, I2C, SPI, etc.

    This particular implementation is running a recent Linux kernel with multiple Docker applications to execute the business logic. The problem was that the time from power on to the system being ready and available was over 6 minutes! Obviously, this was a huge issue and not acceptable performance from a user perspective, so I was tasked with reduce that boot time. I was not given any acceptable performance numbers, so I started by just doing the best that I could with the hardware.

    TL;DR

    By optimizing the kernel configuration and boot time of my applications, I was able to cut the boot time in half. Through this process, I was also able to find some issues in the hardware design of our board that we were able to address to improve read/write speeds to our eMMC.

    In total, I was able to shave over three minutes off of our boot time. It still is not optimal, but I am waiting on the hardware rework before continuing to see how much more is required.

    <insert a table showing gross estimates of time saved in kernel (drivers/modules), systemd (removing services not required), docker apps (upgrading docker to go version from python/rewriting plugin in rust)>

    Boot/Initialization PeriodOriginal TimeOptimized Time
    Bootloader and Kernel Load30 seconds22 seconds
    systemd Service Initialization5 minutes 30 seconds2 minutes 30 seconds
    Optimization Summary

    Approach

    My typical approach to optimizing a Linux system always starts by working to optimize the kernel. This can usually save only a few seconds of boot time, but it was what I was most familiar with, so that is where I began.

    The Linux kernel is designed to allow various drivers and features to be enabled or disabled at runtime via kernel modules. However, depending on your system and requirements, have everything available in the kernel by way of modules can significantly slow down your kernel load times. My approach when working in embedded systems with custom hardware is to optimize the kernel configuration in such a way that all the necessary drivers are built into the kernel and all critical features are also built into the kernel. Everything else that may be needed later in runtime by applications can be left as a kernel module. All features that are definitely not required are completely removed.

    Please note that there is a fine line here when optimizing your kernel. If you make everything built in, that bloats the size of your kernel, which means it will take longer to read from disk. So leaving some things as modules is okay to keep your kernel size smaller, this optimizing load time. But if you have too many modules, those have to be loaded before the rest of the system can be loaded that depends on them, so there is a balance to be struck.

    After I focused on the kernel, I turned my eye toward optimizing the run time and boot applications. Our system made use of systemd to manage system initialization, so it was clear that the right tool to use was going to be systemd-analyze. This is an extremely useful tool to use when you need to see how all the services interact one with another during initialization. You can get a list of how long each service took during the initialization process, view a graph of how those are all related, and even see the critical chain of services through initialization. The typical process to optimize your initialization is two-fold: a) identify what services are being run that you do not require and can turn off, and b) identify what services are taking an abnormal amount of time, so you can optimize those specific services.

    Finally, after the kernel was optimized, and I had removed all the unnecessary services, I was able to focus on the few services and applications that were simply taking forever to start up.

    Kernel Optimization

    In order to optimize the kernel, I had to make use of a few commands in the Yocto build system to manipulate the kernel configuration. In Yocto, most systems make use of kernel configuration fragments, which are a really easy and clean way to manage certain features and drivers in a hardware and kernel agnostic fashion.

    The way this works is that your primary kernel recipe in Yocto will manage a default kernel configuration that defines typical features and drivers for that version of the kernel. This primary layer typically will come from the microprocessor vendor, in this case from NXP. Your hardware layer will provide configuration fragments that override those features. Finally, you can have additional meta layers that provide additional fragments if necessary. You can read more about kernel configuration in the Yocto Project documentation here.

    bitbake linux-lmp-fslc-imx-rt -c menuconfig
    Using menuconfig from bitbake.

    With dealing with optimizations, however, you are typically overriding most everything in the default configuration. Rather than editing the defconfig file itself, or providing an entirely new one, I stuck with the fragment approach.

    # Create a kernel configuration from the default configuration
    # (i.e., build the kernel recipe through the configure step)
    bitbake linux-yocto -c kernel_configme -f
    
    # Make the necessary configuration changes desired for the fragment
    bitbake linux-yocto -c menuconfig
    # Exit menuconfig, saving the configuration
    
    # Create the fragment
    bitbake linux-yocto -c diffconfig
    
    # Copy the fragment to your repository and add to your recipe(s)Code language: PHP (php)

    You can read more about creating kernel configuration fragments in this quick tip.

    When editing the kernel configuration via menuconfig, I made all my desired changes and then generated the configuration fragment and built the kernel. This resulted in many QA warnings about certain features being defined in the configuration chain, but did not end up in the final configuration. Those warnings simply mean that the build system is having a hard time verifying what the correct configuration should be. This is usually because your defconfig file contains one CONFIG_*, but that is not part of the final configuration because you implicitly removed it (i.e., you removed a feature that the specific configuration depends on). To address this, simply take the CONFIG_* option mentioned in the QA warning and drop that into your optimization fragment with a “=n” at the end to explicitly disable it.

    Initialization Optimization via systemd-analyze

    systemd is a popular choice for managing system components in Linux systems. Most all modern Linux systems make use of it in one way or another. It provides a wide range of tools to help administrators manage their systems as well.

    In my case, since my system was using systemd to manage all the run time services and initialization, I was able to make use of the systemd-analyze tool. Here is a short snippet from the man page:

    systemd-analyze may be used to determine system boot-up performance statistics and retrieve other state and tracing information from the system and service manager, and to verify the correctness of unit files. It is also used to access special functions useful for advanced system manager debugging.

    For this exercise, I made use of the commands blame, plot, dot, and critical-chain.

    To start my debug process, I wanted to know at a high level what services were taking the longest to initialize. To do this, I made use of the blame and plot commands.

    systemd-analyze blame will look at all the services that systemd manages and provide a sorted list and the amount of time they took to initialize. This was exactly the kind of information I was after and gave me a starting point in my search for what to optimize. However, when looking at this data you have to be a little careful, because services are many times interdependent. The initialization time of one service could be really long because it cannot finish initializing until another service is also complete with its own initialization.

    user@localhost:~$ systemd-analyze blame
    22.179s NetworkManager-wait-online.service
    21.986s docker-vxcan.service
    19.405s docker.service
    14.119s dev-disk-by\x2dlabel-otaroot.device
    12.161s systemd-resolved.service
    10.973s systemd-logind.service
    10.673s containerd.service
     9.702s systemd-networkd.service
     9.443s systemd-networkd-wait-online.service
     6.789s ModemManager.service
     5.690s fio-docker-fsck.service
     5.676s systemd-udev-trigger.service
     5.552s systemd-modules-load.service
     5.127s btattach.service
     5.062s user@1001.service
     4.670s sshdgenkeys.service
     3.793s NetworkManager.service
     3.780s systemd-journald.service
     2.945s systemd-timesyncd.service
     2.837s bluetooth.service
     2.409s systemd-udevd.service
     2.084s zram-swap.service
     1.677s systemd-userdbd.service
     1.621s avahi-daemon.service
     1.284s systemd-remount-fs.service
     1.080s dev-mqueue.mount
     1.025s sys-kernel-debug.mount
     1.015s modprobe@fuse.service
     1.010s sys-kernel-tracing.mount
     1.004s modprobe@configfs.service
     1.004s modprobe@drm.service
      997ms kmod-static-nodes.service
      871ms systemd-rfkill.service
      832ms systemd-journal-catalog-update.service
      732ms systemd-tmpfiles-setup.service
      668ms systemd-sysusers.service
      592ms systemd-user-sessions.service
      562ms systemd-tmpfiles-setup-dev.service
      533ms ip6tables.service
      507ms iptables.service
      464ms systemd-sysctl.service
      390ms systemd-journal-flush.service
      364ms systemd-random-seed.service
      359ms systemd-update-utmp-runlevel.service
      346ms systemd-update-utmp.service
      318ms user-runtime-dir@1001.service
      232ms tmp.mount
      220ms var.mount
      219ms sys-fs-fuse-connections.mount
      210ms var-rootdirs-mnt-boot.mount
      207ms systemd-update-done.service
      201ms var-volatile.mount
      165ms sys-kernel-config.mount
      147ms docker.socket
      140ms sshd.socket
      134ms ostree-remount.service
      123ms dev-zram0.swapCode language: JavaScript (javascript)

    Because blame doesn’t show any dependency information, systemd-analyze plot > file.svg was the next tool in my quiver to help. This command will generate the same information as blame but will place it all in a nice plot form, so you can see what services started first, how long each took, and also pick out some dependencies between services.

    Section of plot generated with systemd-analyze plot command.

    My main use of the blame and plot commands was to identify services that were obviously taking a lot of time, but to also identify services that were simply not required. systemctl –type=service also helped with this process by simply listing all the services that systemd had enabled.

    Dependencies can still be hard to definitively spot when just looking at the plot, however. Because of that, the systemd-analyze dot ‘<pattern>’ command is really handy. When optimizing my boot time, I would use blame and plot to identify potential culprits and then run them through dot to see how they were related. For example, I found that my network configuration was taking an abnormal amount of time, so I took a look at systemd-analyze dot ‘*network*.*’ to see how the systemd-networkd and related services were interacting one with another. This led me to some information that helped me understand that half of the services that were being started in support of the network were not actually required (such as NetworkManager and the IPv6 support services). By disabling those, I was able to save over 30 seconds of boot time, simply by removing a couple of services that were not required.

    Finally, I was able to make use of the systemd-analyze critical-chain command to view just the critical chain of services. This command simply prints the time-critical chain of services that lead to a fully-initialized system. Only the services taking the most amount of time are shown in this chain.

    user@localhost:~$ systemd-analyze critical-chain
    The time when unit became active or started is printed after the "@" character.
    The time the unit took to start is printed after the "+" character.
    
    multi-user.target @1min 21.457s
    `-docker.service @1min 3.755s +17.683s
      `-fio-docker-fsck.service @26.784s +36.939s
        `-basic.target @26.517s
          `-sockets.target @26.493s
            `-sshd.socket @26.198s +258ms
              `-sysinit.target @25.799s
                `-systemd-timesyncd.service @23.827s +1.922s
                  `-systemd-tmpfiles-setup.service @23.224s +518ms
                    `-local-fs.target @22.887s
                      `-var-volatile.mount @22.596s +222ms
                        `-swap.target @22.393s
                          `-dev-zram0.swap @22.208s +113ms
                            `-dev-zram0.device @22.136sCode language: JavaScript (javascript)

    This information is also useful because it is more of a quick and dirty way of getting to the same information as using blame, plot, and dot separately. However, because it doesn’t show all services, it can only help you optimize the worst offenders.

    Application Optimization

    Finally, after all the kernel and service optimizations, I still was seeing that a couple application services were taking the majority of the time to start up. Specifically, these applications were Docker and a Docker plugin for managing CAN networks.

    These services were the last of the service chain to start up, so there were no other dependent services they were waiting on. Once the network was up and configured, these services would start up. Because they were not dependent on any other services, I was able to laser focus in on what was causing those applications to take so long to start and optimize them directly.

    First, Docker Compose was taking well over two minutes to start and load containers. Second, my Docker plugin for CAN networks was taking well over 20 seconds to startup as well.

    When I checked my version of Docker Compose, I found that it was still running a Python-based version of Compose, rather than the newer and faster Golang-based version. By upgrading my version of Compose, I was able to reduce my startup time from well over two minutes to about 1 minute 40 seconds.

    I also found that the CAN network plugin was written in Python as well. So, rather than continue using it, I rewrote that plugin in Rust, which also gave me the opportunity to fix a couple of shortcomings I found in the plugin itself. This reduced the initialization time of the plugin from over 30 seconds to under a second — a huge savings!

    488ms docker-vxcan.serviceCode language: CSS (css)

    Conclusion

    Overall, this was a great exercise for me in the steps to optimize the boot process of a Linux system. While I certainly could optimize the system some more, I believe the gains to be had will be minimal — at least until I have some new hardware with the flash memory interfaces running at full speed. Then I can revisit this and see if I can get things any faster!

    What is your experience with speeding up Linux boot and initialization on embedded systems? Any tips and tricks you would like to share? Comment them below!