Integrating a New Autocoder

This guide will walk the user through integrating a new autocoder into the F´ build system. In the most basic form, an autocoder consists of several elements: input models, output generated code, steps to generate output from the given input, and a trigger to start the process. As a byproduct, this process also produces "module dependencies" and "file dependencies" used to tie the autocoder into the make system itself.

Autocoder CMake Structure

An autocoder needs to implement a CMake file that will be used to run the autocoder itself. Within this file the developer is required to implement several functions and set several variables to tell the system how to run the autocoder and set up the build trigger. Each function needed will be expanded on below, with examples.

All functions are prefixed with the autocoder's name to prevent collisions with other autocoder functions. The name is the name of the CMake file without the .cmake extension. For example, the FPP autocoder is implemented in a file called fpp.cmake and the name is thus fpp.

There are three primary parts of an autocoder: 1. Call one of autocoder_setup_for_individual_sources() or autocoder_setup_for_multiple_sources() from file scope 2. Implement <autocoder name>_is_supported(AC_POSSIBLE_INPUT_FILE) returning true the autocoder processes given source 3. Implement <autocoder name>_setup_autocode AC_INPUT_FILE) to run the autocoder on files filter by item 2.

Once the autocoder file is constructed, the autocoder needs to be registered to modules. This is done through a custom target. See: target-integration.

Select Individual or Multiple Sources

When processing a single module, the autocoder system can call an autocoder multiple times passing an individual autocoder source each time or it can call the autocoder one time passing in all autocoder sources in the given module. The autocoder itself needs to configure which style the autocoder supports. This is done by calling one of two possible functions: autocoder_setup_for_individual_sources() or autocoder_setup_for_multiple_sources(). These calls must be made in the file scope of the autocoder file (i.e. not within a function/macro).

For example, the FPP autocoder calls autocoder_setup_for_multiple_sources() as it processes all .fpp files of a module in a single call.

...
include(autocoder/helpers)

autocoder_setup_for_multiple_sources()

On the other hand, the AI XML autocoder can only handle one Ai.xml file at a time and thus it calls autocoder_setup_for_individual_sources().

...
include(autocoder/helpers)
include(autocoder/ai-shared)

autocoder_setup_for_individual_sources()

Implement `<autocoder name>_is_supported(AC_POSSIBLE_INPUT_FILE)`

Autocoders typically only process a subset of the sources of a given module. These are typically the model files like .fpp files, but could be any of the sources in the module. If an autocoder need to process sources generated by another autocoder, see Autocoding Using Autocoded Inputs below.

<autocoder name>_is_supported takes one argument: a source file path to a possible autocoder input. If the autocoder needs to process this file, <autocoder name>_is_supported must set IS_SUPPORTED to TRUE in PARENT_SCOPE otherwise <autocoder name>_is_supported must set IS_SUPPORTED to FALSE in PARENT_SCOPE. Since most autocoders determine support via suffix, autocoder/helpers.cmake supplies a helper macro to do that easily. Note: The autocoder must include autocoder/helpers to access that helper.

For example, the FPP autocoder supports all files ending with the extension .fpp and it uses the suffix helper as shown below. Since the helper is a macro it will automatically set IS_SUPPORTED in PARENT_SCOPE.

...
include(autocoder/helpers)
...
function(fpp_is_supported AC_INPUT_FILE)
    autocoder_support_by_suffix(".fpp" "${AC_INPUT_FILE}")
endfunction(fpp_is_supported)

A more generalized pattern can be seen below for users who need more than suffix-based processing.

function(<autocoder name>_is_supported AC_INPUT_FILE)
    # First set the default: not supported
    set(IS_SUPPORTED FALSE PARENT_SCOPE) 
    # Second check some condition and set IS_SUPPORTED to TRUE in that case
    if (<some condition>)
        set(IS_SUPPORTED TRUE PARENT_SCOPE)
    endif()
endfunction(<autocoder name>_is_supported )

Should an autocoder need to indicate that a given input file will cause inter-target dependency changes (i.e. CMake re-generation must be performed) then the function requires_regeneration(<INPUT_FILE>) must be called. This can be done directly or by providing TRUE as an optional third argument to autocoder_support_by_suffix. Both ways are shown below.

...
include(autocoder/helpers)
...
function(<autocoder name>_is_supported AC_INPUT_FILE)
    requires_regeneration("${AC_INPUT_FILE}")
endfunction(<autocoder name>_is_supported )

...
include(autocoder/helpers)
...
function(<autocoder name>_is_supported AC_INPUT_FILE)
    autocoder_support_by_suffix(".fpp" "${AC_INPUT_FILE}" TRUE)
endfunction(<autocoder name>_is_supported )

Implement `<autocoder name>_setup_autocode AC_INPUT_FILES)`

Implementing <autocoder name>_setup_autocode AC_INPUT_FILES) should setup the autocoder for generating files. It will be called once with a list of supported inputs when the autocoder handles multiple inputs and will be called multiple times with an individual supported input when the autocoder handles individual inputs. The argument to the function will therefore be a list or a single item depending on the original setup.

Every autocoder must do two things in this function: 1. Name the generated files using set(AUTOCODER_GENERATED <autocoder outputs> PARENT_SCOPE) 2. Generate the files using add_custom_command

These autocoders may optionally set additional variables to help define the system. These are discussed below.

There are two models for generating files. The simple model is to let the autocoder system call add_custom_command and simply set variables to control that call, and the general model where the implementor calls add_custom_command themselves.

Cross Platform Utilities and CMake

CMake is designed to run on many different hosts. As such it provides common implementations of many operating system provided utilities. These are invoked through cmake -E. To touch a file for instance, use cmake -E touch. A full description of available commands are found in CMake's command line tool documentation.

To run the current CMake executable (the one invoked by the user) reference it with ${CMAKE_COMMAND}. Thus, to make a directory in the autocoder the following would be used:

add_custom_command(COMMAND ${CMAKE_COMMAND} -E make_directory ...)

Simple Model

This model works well when the autocoder fits the basic pattern of passing any and all inputs to a script, and produces outputs. Optionally, these autocoders can set inter-module dependencies. Simple-model autocoders are expected to operate with a command-line invocation following the pattern:

executable [--arg [[--arg]] input1 [input2...]

That is, the autocoder has a single executable step, the executable and arguments come first, then the input files(s). Anything that breaks this pattern must use the General Model.

This model can easily be accomplished by simply setting variables in the parent scope and the autocoder system will set up the add_custom_command CMake call automatically based on those variables. All these variables need to be set in PARENT_SCOPE. These variables are:

AUTOCODER_SCRIPT: executable command line argument(s) run to generate files. Use cmake -E to run cross-platform safe commands, set up the environment, etc.
AUTOCODER_GENERATED: list of files generated by the autocoder run. Required of all autocoders.
AUTOCODER_INPUTS: list of files to supply to the autocoder script as command line arguments. These files are monitored and the autocoder script will rerun when these files change.
(optional) AUTOCODER_DEPENDENCIES: set to other modules this autocoder requires to have built first.

The following is an implementation that uses touch to generate files using this pattern. This version assumes that sources are handled individually.

function(touch_autocode_setup AC_INPUT_FILE)
    # Get the name without the directory
    get_filename_component(BASENAME "${AC_INPUT_FILE}" NAME)
    # Generate <name>.touched file in the build cache 
    set(AUTOCODER_GENERATED "${CMAKE_CURRENT_BINARY_DIR}/${BASENAME}.touched" PARENT_SCOPE)
    # File generated using cmake -E touch <filename>.touched. Note: this is a list of arguments: cmake, -E, touch, ...
    set(AUTOCODER_SCRIPT ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/${BASENAME}.touched PARENT_SCOPE)
    # Only need to supply and monitor the single input file, no configuration, or auxiliary files
    set(AUTOCODER_INPUTS "${AC_INPUT_FILE}" PARENT_SCOPE)
    # Depends on the Fw_Time and Fw_Comp modules just as an example
    set(AUTOCODER_DEPENDENCIES Fw_Time Fw_Comp PARENT_SCOPE)
endfunction()

This invokes the following command line to generate files any time the AC_INPUT_FILE is changed. Note: since inputs are supplied to the autocoder, we will touch both the generated file and the input file.

cmake -E touch ${CMAKE_CURRENT_BINARY_DIR}/${BASENAME}.touched ${AC_INPUT_FILE}

The add_custom_command setup for this autocoder by the autocoding system is equivalent to the following:

add_custom_command(
    OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${BASENAME}.touched"
    COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/${BASENAME}.touched
    DEPENDS ${AC_INPUT_FILE} Fw_Time Fw_Comp
)

Although straightforward and avoiding some of the nuances of the add_custom_command this model may be overly restrictive for some use cases. Users who cannot fit the steps of the autocoder into this paradigm may use the general model.

General Model

The general model for autocoders is to ensure that AUTOCODER_GENERATED is set in the parent scope and that files are generated using custom commands. Any implementation of the <autocoder name>_setup_autocode function that does this should work. This model can be used for things that do not fit in the standard model and may include any of the following:

Running multiple COMMAND steps for a multi-part autocoder
Creating complex command lines interleaving flags and inputs
Using features of add_custom_command outside of OUTPUT, COMMAND, and DEPENDS

The general model calls add_custom_command directly and thus users should start by consulting the CMake documentation of that call. This documentation will touch briefly on the key options for this call.

Setting Outputs

An autocoder generates outputs and the autocoder needs to configure this. For the autocoder system, we set the variable AUTOCODER_GENERATED. For specifying to cmake we must use the OUTPUT portion of add_custom_command() as shown below. Note: AUTOCODER_GENERATED is needed in both local scope for the call into add_custom_command, and set in PARENT_SCOPE for the larger system hence the appearance of two sets.

...
set(AUTOCODER_GENERATED ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.1
                        ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.2
)
add_custom_command(OUTPUT ${AUTOCODER_GENERATED} ...)
set(AUTOCODER_GENERATED "${AUTOCODER_GENERATED}" PARENT_SCOPE)

Running Commands

An autocoder runs commands to generates files. This is accomplished with one or more in-order COMMAND portions of the CMake add_custom_command() calls. This can be seen below where a directory is created before touching two files.

add_custom_command(...
    COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/directory
    COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.1
    COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.2
    ...
)

Setting Triggers and Dependencies

An autocoder should regenerate files under certain conditions. Namely, if a file or module dependency changes the autocoder needs to rerun. Both are specified with the DEPENDS portion of the command. Files should be contained within the module, and cross-module dependencies should be as _ delimited names (e.g. Fw_Time).

add_custom_command(...
    DEPENDS ${AC_INPUT_FILE} Fw_Time Fw_Comp
)

Putting It All Together

The full implementation of this function can be seen below. It should be noted that it has several improvements on the simple model while being essentially the same autocoder. These improvements include:

It can run multiple command steps, including making a directory and touching an additional file.
The autocoder input isn't forcibly included in the input to the touch command preventing the superfluous touch.

This comes at the expense of careful handling of the add_custom_command call.

function(touch_autocode_setup AC_INPUT_FILE)
    # Get the name without the directory
    get_filename_component(BASENAME "${AC_INPUT_FILE}" NAME)
    # Setup local-scope variable with generated files
    set(AUTOCODER_GENERATED ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.1
                            ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.2
    )
    # Setup to make a directory, touch two files, and rerun on changes to the input file/two modules
    add_custom_command(OUTPUT ${AUTOCODER_GENERATED}
        COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/directory
        COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.1
        COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/directory/${BASENAME}.touched.2
        DEPENDS ${AC_INPUT_FILE} Fw_Time Fw_Comp
    )
    # Copy local scope into parent scope as required
    set(AUTOCODER_GENERATED "${AUTOCODER_GENERATED}" PARENT_SCOPE)
endfunction()

Autocoding Using Autocoded Input

Some autocoders need to run based on the output of another autocoder. For example, heritage AI XML autocoders run on the output of the FPP autocoder, which generates the AI XML files. The process to do this is the same as described above with the following caveats.

1. Always Run The Source Autocoder First

This is done in the target definition when it runs autocoders. The source autocoder must be run first. This is easy to do as the run_ac_set() takes a set of autocoders to run. Remember, this is done in the target definition that calls this autocoder.

Example: FPP Sources AI XML in the Build Target

function(build_add_module_target MODULE TARGET SOURCES DEPENDENCIES)
    ...
    run_ac_set("${SOURCES}" autocoder/fpp autocoder/ai_xml)
    ...
endfunction(build_add_module_target)

Do not worry, autocoders can be run using run_ac_set multiple times, the actual invocation will happen once and be shared.

2. Do Not Read the Autocoder Input Outside of `add_custom_command`

Many autocoders read and/or preprocess autocoder input files to set up dependencies, scripts, etc. However, it is essential to remember that in this case that file only exists during the build of the project, not during the setup of the project. Thus, these autocoders should make all decisions on the filename only and not the contents of the file.

Usually, it is a good idea to have the source autocoder provide all dependencies, and have the secondary autocoder set no additional dependencies. The secondary autocoder is simple input to output.