Building and Packaging

Introduction

Building and packaging are the main development operations for Fuze and lead to generation of a FuzeID. For convenicne, we will use the term “build” for any and all operations that result in new FuzeID.

What is a Build?

A build is one type of Fuze operation that generates a FuzeID.

  • A build is, at its core, a set of any commands defined in a build map and executed within the context of a CBE. However, for a build to make any sense for this tool, it should produce an output to be tagged with the FuzeID.

  • A build can be the compile and linking of source code to produce a binary artifact output.

  • A build can be a complex operation of building source with external dependencies and then packaging those artifacts with other packages, artifacts, and files to produce a singular artifact.

  • A build can be a packaging operation of files that pre-exist.

Common Build Environment (CBE)

A CBE is required for build operations. It is a container that is a 100% configuration managed, traceable, and reproducible environment. CBE’s are one of the technologies that allow Fuze to execute from any platform. For example, you could use a linux CBE to build an ARM target from a Windows host machine.

More detailed information about CBE’s can be found in the CBE Reference.

The role of the CBE in the build operations is detailed below.

The Build Map File

A Build Map file is the main user-generated input for the Fuze build operation. It contains all of the information for Fuze to perform a build operation, including packaging.

The build map file is required for all build operations and contains all of the commands, directions and operations that direct the build and packaging process. Detailed reference information on the build map file can be found in the Build Map Reference.

Workspace Context for Build and Package

The following shows the workspace contexts in which the 2 main sections of the build map execute.

_images/fuze-build-workspace-context.png

  • All build commands, operations, and associated actions from the “build” section execute in the CBE workspace.

  • All package commands, operations, and associated actions from the “packages” section execute in the local host workspace.

Types of Builds

Several types of builds can be generated from Fuze, and each have specific meanings and functionality. The FuzeID formats for each can be found on the FuzeID page.

Universal (or Regular)

The most common, and the only type of build that has meaning outside of one’s own local system, is the universal (sometimes referred to as “regular”) Fuze build. A universal build occurs for all scenarios not defined in the other specific build types below.

Universal builds, and the packages that are created from them are available to all users. A universal build is required for the following:

  • Releasing/Deployment

  • Recreating the workspace and the build

  • Sharing a FuzeID/build with a colleague

  • Making packages available for others to use

  • Searching for a FuzeID

Universal FuzeID’s include a “-” character between the date and minutes field.

Example of Universal FuzeID:

123456-123-12345

Local

Local builds, and the packages that are created, have most of the characteristics of a universal build, but are ONLY available to you on your local machine.

Here are some local build characteristics and limitations:

  • You may use a local build as a dependency, on your own system.

  • A local build has traceability locally.

  • A local build CANNOT be used as a dependency on anybody else’s system.

  • A local build CANNOT be released

  • A local build CANNOT share packages with another user or system.

Go to the Local Builds documentation for a detailed explanation of why and how to use local builds.

Local FuzeID’s include a “L” character between the date and the minutes field.

Example of Local FuzeID:

123456L123-12345

Meaningless

Meaningless builds, and the packages that are created, have none of the traceability, config management, or sharing characteristics of a universal build. Meaningless builds are generated when Fuze cannot guarantee any configuration management of the impending build due to one or more untraceable elements, such as when you have uncommitted local changes, or when using the --localhost-build option.

Here are some meaningless build characteristics and limitations:

  • A meaningless build generates artifacts and packages from the build map just as a universal or local build does.

  • A meaningless build CANNOT be used as a dependency on the local or any other system.

  • A meaningless build CANNOT be released

  • A meaningless build CANNOT share packages with another user or system.

Meaningless FuzeID’s include a “X” character between the date and the minutes field.

Example of Meaningless FuzeID:

123456X123-12345

What is a Package?

A package is a collection of specified files that Fuze puts together into a single zip file. Typically, a package comprises some or all of the build artifacts and may include any other files from the workspace, including other files created during the build process.

Packages can be used as dependencies for other builds, and provide Fuze the ability configuration manage those dependencies.

Like with build, the packaging process is defined in the build map. The Build Map reference guide shows the details of the packaging section.

Workspace Context for Packaging

As shown in the figure above, the packaging section of the build map executes in the context of the host machine. Packaging syntax must be compatible with the HostOS, but do NOT need to be compatible with the build and CBE syntax. For example, the build commands can be developed for a Windows CBE and the packaging commands for a Linux CBE.

Package Targets and Format

The build map contains a section for defining the package targets. The number of package targets, like the build targets, is unbounded.

Fuze creates zip files for packages. Each zip file is a single package. The zip file name is the following format:

<package_filename_pefix>-<package_target>.<release_version>.v<FuzeID>.zip

See build map refernce for detailed understanding of these fields.

For example:

internal-test-cs7-kepler-slc38.v7.11.230130-4db-7047.db

In this case:

  • internal is the “package filename prefix” field under the package target

  • test-cs7-kepler-slc38 is the “package target” name

  • 7.11 is the “release_version”

  • 230130-4db-7047 is the FuzeID generated during the build

Dependencies

A main feature of Fuze is its ability to configuration manage all of the dependencies that comprise any particular build. This includes object code dependencies, such as pre-built libraries, as well as runtime build dependencies, such as from other source code.

With a FuzeID, you can track all of the depencies that were used in building that particular target. To do so, you create packages from builds that other builds may utilize in the "dependecies" section. For example, one build map may build and package a library, and another build pulls in that library as a dependency during its build.

With a package dependency, you have the option of specifying a particular version (i.e. FuzeID) of that package, or you can allow Fuze to pull the latest from the HEAD of the package repo by leaving the field blank.

Public Versus Private Availability

To utilize a package as a dependency for another build, you must mark that package as publicly available through setting the "is_dependency" field to true. The Build Map reference provides more information.

Where are My Packages?

Depending upon the Fuze packaging argument used, packages will be either:

  • Available on the local host only (--package)

  • Available through a specific FuzeID (--add-packages)

  • Available as the HEAD of the package repository and through a specific FuzeID (--update-packages)

Warning

Packages used as dependencies must be in the FuzeDB

You must use --update-packages or --add-packages (during current build, or to add to a pre-existent build) to make a package available as a dependency

The following shows the difference.

_images/fuze-packaging.png

Fuze contains a package repository for each “product” it encounters. The “product” name is unique for a build map and each time Fuze encounters a build and --update-packages with that “product” name, it updates the HEAD of that product’s package repository.

Making packages available for both downloading and dependency import is a basic and important operation with Fuze.

You may either use the --add-packages concurrent with your build, and in that case, the packages will be tied immediately to the FuzeID in the FuzeDB. Or, you may invoke --add-packages on a pre-existent FuzeID.

Versioning

Each Fuze execution through a build map generates a unique version. All build and package targets created with that build receive that unique version. The version is appended to package filenames.

A version is made up of two parts:

  • The human readable part, which comes from the "release_version" section of the build map

  • The autogenerated and unique FuzeID

v<release_version>.<FuzeID>

The FuzeID is the only part required for full configuration management, but the "release_version" provides a helpful tool for searching and is required for advancing a build to a release.

If you use a <MAJOR>.<MINOR> scheme for the "release_version", then you have access to two convenient macro variables (${MAJOR} and ${MINOR}) in the build map during execution time.

How to Build and Package

All build functions utilize the fuze option --build-map and the build actions are governed by the build map file.

A build operation is invoked as follows:

fuze --build-map <build_map_file>
    [--target [clean | TARGET...]]
    [--local]
    [--localhost-build]
    [--package [PACKAGE...]]
    [--add-packages]
    [--update-packages]
    [--draw-graph]
    [--out <filename>]
    [--no-out]
    [--no-shared-volumes]

Arguments and Options

--build-map

  • Required: Yes

  • Purpose: Specifies the build map file that governs the build

  • Arg Format: <path>/<build_map_file>.json

  • Comments: Typically, the build map file resides at the root of the workspace, but this is NOT required. It must, however, be inside the workspace. If your build map file is part of your source repository, it can be referenced from its location within the repo. The build operation will save the build map in its entirety as part of the FuzeID metadata. Therefore, you may create or modify the build map dynamically as part of your process.

--target

  • Required: No

  • Purpose: Specifies 1 or more targets. If omitted, Fuze will automatically build all targets contained in the build map.

  • Arg Format: TARGET0 TARGET1 ... TARGETn

  • Comments: A single target or a list of targets can be specified with a space-separated list

  • Special Case: the target keyword clean is used to instruct Fuze to perform a cleaning of the workspace’s specified output directories. See the clean target explanation below.

--local

  • Required: No

  • Purpose: Perform all operations locally only (See the Working Locally page for more information).

  • Comments: When working locally, the generated FuzeIDs are NOT meaningful outside of the local host machine. Local FuzeIDs cannot be released/deployed.

--localhost-build

  • Required: No

  • Purpose: Use the localhost for all build operations (ie, do NOT use a CBE).

  • Comments: This option allows Fuze to utilize the user’s localhost system to perform the build command operations. The user’s system MUST be able to support those commands, including the syntax. Meaningless FuzeIDs cannot be released/deployed.

    Warning

    Any builds using this option will generate a meaningless FuzeID. That FuzeID is not stored, nor can it be used in any other operations.

--package

  • Required: No

  • Purpose: Specifies 1 or more packages to be created and available in you local file system. If a list of package names is omitted, Fuze will automatically generate all package targets contained in the build map.

  • Arg Format: [optional] PACKAGE0 PACKAGE1 ... PACKAGEn

  • Comments: Without the --package option being specified, the packaging phase of the build is skipped. All build artifacts that are created by the “build commands” are available in the workspace, but none of the additional packages are generated. This option ONLY creates the packages and makes them available to you in your local file system. If you wish to make the packages available publicly, you must either use the --update-packages option, or the --add-packages procedure defined below. You must use one of options in conjunction with --package to specficy a subset of package targets.

--add-packages

  • Required: No

  • Purpose:

    1. For current build: Generate specified packages and add them to FuzeDB for general access. If a list of package names is omitted, Fuze will automatically generate and add all package targets contained in the build map.

    2. For pre-existent FuzeID: Add the specified packages to the specified FuzeID.

  • Arg Format:

    1. For current build: None

    2. For pre-existent FuzeID: <package_target>:<path_to_pkg> <FuzeID> [--set-dependency]

  • Comments: This operation makes the generated packages available with the --get-packages command, but does NOT update the HEAD of the package repository. For the current build, use in conjunction with --package if specifying a subset of package targets.

    Note

    Downloading Previously Built Packages

    Whenever --add-packages is used, all created packages will be available for download by anybody via the --get-packages command. See here for more info.

    Only packages also marked with --set-dependency can be used as dependencies.

--update-packages

  • Required: No

  • Purpose: Generate specified packages, and update the HEAD of the package repository with the generated packages. If a list of package names is omitted, Fuze will automatically generate and update all package targets contained in the build map.

  • Arg Format: None

  • Comments: This operation will update the HEAD of the package repository with the packages generated. Use in conjunction with --package if specifying a subset of package targets.

    Warning

    --update-packages updates the HEAD of the Package Repo

    Any future build map that does NOT specify a particular FuzeID for this package will be pulling the HEAD. The HEAD is updated by the use of this option.

    Note

    Downloading Previously Built Packages

    Whenever --update-packages is used, all created packages will be available for download by anybody via the --get-packages command. See here for more info.

--draw-graph

  • Required: No

  • Purpose: Generates a PNG image file that shows a block diagram of the dependencies.

  • Comments: This operation does NOT generate a FuzeID

--out

  • Required: No

  • Purpose: Change the output filename from fuze.out (the default value), to a user specified value.

  • Comments: This option does not affect whether the output file is generated (it is by default).

--no-out

  • Required: No

  • Purpose: If included, it suppresses the creation of the fuze.out output file.

  • Comments: Inclusion overrides the --out <filename> option.

--no-shared-volumes

  • Required: No for Linux, MacOS, and cloud Windows, but Yes for local Windows when using msbuild.exe as the build tool

  • Purpose: If included, the docker container will not mount a shared volume between the host and itself. The workspace files will be copied into and out of the container. You might use this option if you are using msbuild.exe as the build tool (see Comments below) or if you are generating artifacts and output files that you do NOT wish to end up in your local host’s workspace.

  • Comments: The msbuild.exe build tool cannot utilize a shared volume when operating on a physical machine. Therefore, this option is required if performing an msbuild.exe operation on a physical Windows machine in a Windows container.

    Note

    • Copying files into and out of the container adds significant time to a build operation.

    • This is a good option to use when your build will dynamically change files that are part of your repository

Special Target Case: clean Target

Including the target keyword clean in the build target list invokes a special Fuze cleaning operation. When included, Fuze will read the build map file, and delete all local workspace directories and files that are specified in the “build” => “out” section.

Note

Fuze will only clean when it finds that target keyword

The cleaning operation cannot be batched in the target list with other target builds, due to ambiguity. Therefore, if you specify the clean keyword anywhere in the target list, ONLY the cleaning operation will occur.

Warning

The build map “out” section directs the workspace cleaning

A Fuze build execution is a standalone operation. No special knowledge is passed between executions. The target cleaning can only be governed by the given build map’s “out” section. If you have previously executed a build, and then execute a --target clean with a different build map that contains different entries in the “out” section, only what is specified in the current build map will be deleted.

Build and Package Artifact Output

You may be asking (rightfully), “where do I get my output?”. Well, here you go:

Fuze Output File

Fuze creates a text output file in the directory from which fuze was invoked. The output file contains the important information from the current build execution.

The default filename is fuze.out, but can be changed with the --out <filename> option, or it can be suppressed entirely with the --no-out option.

The purpose of this file is mainly for creating automation that uses Fuze (ie, CI tools, personal scripting, etc), although the information contained in the file is easily readable.

The specification for the output file is:

fuze_version=<fuze_version>
fuzeid=<FuzeID>
ftl=<path_to_ftl>
build_errors=<int_count>
package_errors=<int_count>
total_errors=<int_count>
untracked_files=<#_untracked_repo_files_for_build>
<package_name1>=<path_plus_filename>
<package_name2>=<path_plus_filename>
...
<package_nameN>=<path_plus_filename>

Fuze Temp Location (FTL)

The Fuze Temp Location (FTL) is a temporary directory in your local host filesystem that is utilized by Fuze to perform operations in your local host system. For example, all packaging operations are performed in your local host system. These occur in the FTL.

More information about the details of the FTL can be found here.

Build Artifacts

At the conclusion of the build operations, all specified artifacts (files and directories) are copied from the CBE back to the user local workspace.

The resulting build artifact locations are defined by the “out” section in the build map file. This means you are free to copy, or omit, whichever build artifacts you wish. You are also free to copy them to any place in the user local workspace.

However, the recommendation is that you copy and locate your build artifacts as if you were building in your local OS, or in a location amenable to the debugger.

Packages

Output package zip file final location will be in the FTL, inside a directory named by the “output” value defined in the “packages” section of the build map file. Underneath the “output” directory, the packages will be inside separate directories named with the “package target” name defined in the build map.

Log Files

Log files are located in “logs” directory of the FTL. More information about the log files can be found here.

Output Example

The following is a basic example of where the output can be found for a build with the following assumptions:

  • The workspace is under the “code” directory and it contains a source code repo called “src_repo”

  • The build map file is called “build_src_repo.json”

  • The build was executed with a target called “target”, and it produces several output files

  • The build map file “out” section directs Fuze to place the set of build artifacts into a directory called “out” underneath the “src_repo” directory

  • The FTL is the “tmp” directory

  • The packages “output” value is “my_packages”

  • Two packages are created: “internal” and “internal-test”

Here is the file system before the build is executed:

/code
    - workspace.ini
    - build_src_repo.json
    /src_repo
        /src
        /inc

Here is the local workspace and file system after the build has executed:

/code
    - workspace.ini
    - build_src_repo.json
    /src_repo
        /src
        /inc
        # The out directory is created by the build operations
        # in the CBE, and exported to local workspace
        /out
            - target.hex
            - target.elf
            - target.lst
            - target.o
            - target.log

# The tmp directory is the FTL
/tmp
    /my_packages
        /internal
            - internal.zip
        /internal-test
            - internal-test.zip
    /logs
        - target-log.txt
        - cbe-test.log
        - fuze.log

Adding Packages to the Current Building FuzeID

Making packages available for both downloading and dependency import is a basic and important operation with Fuze.

The --update-packages does this automatically, but also updates the HEAD of the packag repo. If you don’t want that update, you may either use the --add-packages concurrent with your build, or, you may invoke --add-packages on a pre-existent FuzeID.

Adding Packages to Current Build

Using --add-packages with the current build attaches the created packages to current build FuzeID. The main use case for this is when you are building from a branch, you want the packages attached and available to the public, but you do NOT want to update the HEAD of the package repo (i.e., as with --update-packages).

To do so, simply call your build with the --add-packages [<pkg0> .. <pkgN>]. Note that using --add-packages supplants the need to use --package.

Adding Packages of Files to Pre-Existing FuzeID

You can also add a single file or a package to a pre-existent FuzeID. A typical use case for this is when you need to manually modify the built packages or add some files to a release.

See Modifying Pre-Existent FuzeID Objects for more detailed information.

How Target Definitions Affect FuzeIDs and Builds

A single FuzeID is generated per Fuze build execution. Therefore, all targets requested to be built with a single Fuze build execution will be equated with the same FuzeID.

To generate unique FuzeIDs for variants of your target images, create separate targets to each of those variants. Then call Fuze build separately for each identified target.

See the Build Map Reference guide for detailed information on designing your build.