From source to .hex

1. Compiling, assembling, and linking

When you check the “Show verbose output during: compilation” in the Arduino IDE preferences, what do all of those messages mean?

We will be looking at the compilation process beginning with the example from day12 with the avrgcc-asm-functions.ino sketch with the extra assembly file asmOne.S. That uses the ATTinyCore package with settings to build for an ATtiny85.

It is not necessary to use variables in the command lines, as is done in the following breakdown example. In this real example there are so many items that using variables helps to group the options and file paths by their purpose.

Definitions of variables are not necessarily given in each block. Scroll up to find where they are assigned values.

1.1. arduino-builder

As step 1, the IDE calls the program arduino-builder twice, first with the option -dump-prefs and second with -compile, each with the same huge set of options. That program is what actually calls the subsequent avr-g++ and avr-gcc programs.

# use some (Bash) shell variables to shorten some long names
IDE_INSTALL=/home/dan/foss/arduino-1.8.13
IDE_USER=/home/dan/.arduino15
SKETCH_DIR=/home/dan/ed/ece422/avr/avrgcc-asm-functions

# the trailing "\" continues the command on the next line

${IDE_INSTALL}/arduino-builder \
    -dump-prefs \  # then -compile
    -logger=machine \
    \
    \ # where to find various tools and such (compiler, avrdude, etc)
    -hardware ${IDE_INSTALL}/hardware \
    -hardware ${IDE_USER}/packages \
    -tools ${IDE_INSTALL}/tools-builder \
    -tools ${IDE_INSTALL}/hardware/tools/avr \
    -tools ${IDE_USER}/packages \
    \
    \ # where do we find libraries?
    -built-in-libraries ${IDE_INSTALL}/libraries \
    -libraries /home/dan/Arduino/libraries \
    \
    \ # all of the options selected under the Tools menu
    -fqbn=ATTinyCore:avr:attinyx5:LTO=enable,TimerClockSource=default,chip=85,clock=1internal,eesave=aenable,bod=disable,millis=disabled \
    \
    \ # USB identifier thing
    -vid-pid=10C4_EA60 \
    -ide-version=10813 \
    \
    \ # temporary folder to hold outputs
    -build-path /tmp/arduino_build_183258 \
    \
    \ # Preferences -> Compiler warnings is set to "All"
    -warnings=all \
    \
    \ # Cache for things (core libraries) that don't often change
    -build-cache /tmp/arduino_cache_234816 \
    \
    \ # option value
    -prefs=build.warn_data_percentage=75 \
    \
    \ # where to find the compiler and programmer tools
    -prefs=runtime.tools.avr-gcc.path=${IDE_USER}/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7 \
    -prefs=runtime.tools.avr-gcc-7.3.0-atmel3.6.1-arduino7.path=${IDE_USER}/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7 \
    -prefs=runtime.tools.avrdude.path=${IDE_USER}/packages/arduino/tools/avrdude/6.3.0-arduino18 \
    -prefs=runtime.tools.avrdude-6.3.0-arduino18.path=${IDE_USER}/packages/arduino/tools/avrdude/6.3.0-arduino18 \
    -prefs=runtime.tools.micronucleus.path=${IDE_USER}/packages/ATTinyCore/tools/micronucleus/2.5-azd1b \
    -prefs=runtime.tools.micronucleus-2.5-azd1b.path=${IDE_USER}/packages/ATTinyCore/tools/micronucleus/2.5-azd1b \
    \
    \ # Added by checkbox: Preferences -> Show verbose output during: compilation
    -verbose \
    \
    \ # finally, the sketch itself
    ${SKETCH_DIR}/avrgcc-asm-functions.ino
The above phase also generates the file avrgcc-asm-functions.ino.cpp from the original sketch.



Next up is a phase to extract all of the libraries that are used (explicitly and implicitly) in the sketch and other source files.


AVR_GCC=/home/dan/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin
ATTinyCore_DIR=/home/dan/.arduino15/packages/ATTinyCore/hardware/avr/1.5.2

ATTinyCoreIncludes=\  # include paths for ATTinyCore
    -I${ATTinyCore_DIR}/cores/tiny \
    -I${ATTinyCore_DIR}/variants/tinyX5

TN85_OPTS=\  # ATtiny85-specific configuration
    -mmcu=attiny85 \
    -DF_CPU=1000000L \
    -DCLOCK_SOURCE=0 \
    -DARDUINO=10813 \
    -DARDUINO_AVR_ATTINYX5 \
    -DARDUINO_ARCH_AVR \
    -DDISABLEMILLIS \
    -DNEOPIXELPORT=PORTB

# Lots of options, common to later runs.  So collect them here into a shell variable.
AVR_GCC_OPTIONS=\
    -c \  # compile
    -g \  # with debugging symbols
    -Os \ # optimize for small code size
    -w \  # no warning messages
    -std=gnu++11 \  # compliant with 2011 C++ standard with GNU extensions
    \               # see https://gcc.gnu.org/projects/cxx-status.html
    \ # some options for C++
    -fpermissive \
    -fno-exceptions \
    -ffunction-sections \
    -fdata-sections \
    -fno-threadsafe-statics \
    -flto \  # link-time optimization
    -w \
    -x c++ \  # force C++ language parser
    -E \
    -CC \
    ${TN85_OPTS} \ # CPU configuration
    ${ATTinyCoreIncludes}  # Library locations


${AVR_GCC}/avr-g++ \
    ${AVR_GCC_OPTIONS} \  # Lots!  See above.
    \
    \ # This file is generated from the original .ino sketch.
    \ # It properly adds function prototypes *before* the functions are used.
    \ # Go look at an example to see this happening.
    ${TEMP_DIR}/sketch/avrgcc-asm-functions.ino.cpp \
    \
    -o /dev/null \  # no output file
    \ # because we are analyzing the sketch to detect the libraries we used
    -DARDUINO_LIB_DISCOVERY_PHASE


${AVR_GCC}/avr-g++ \
    ${AVR_GCC_OPTIONS} \  # Lots!  See above.
    \
    \ # The extra assembly file.  No pre filtering since it isn't a .ino file.
    \ # A ".c" or ".cpp" file likewise is not pre-filtered.
    ${TEMP_DIR}/sketch/asmone.S
    \
    -o /dev/null \  # no output file
    \ # because we are analyzing the sketch to detect the libraries we used
    -DARDUINO_LIB_DISCOVERY_PHASE



The next phase is “Generating function prototypes…​”.

We’ll skip that for now. It has to do with more detecting and morphing the code into proper C++.

1.2. Compiling to object files

Compiling sketch…​

Finally we begin to compile the source code into object files. In some ways, this compling phase is the only place you actually need to start, ignoring the previous phases. Those previous phases are to possibly later avoid compiling things again that could be cached instead to save time rebuilding the project.

1.2.1. Assembly code .S

AVR_GCC=/home/dan/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin
ATTinyCoreInclude=/home/dan/.arduino15/packages/ATTinyCore/hardware/avr/1.5.2

#
# Compile (assemble) the assembly file
#
AVR_GCC_ASM_OPTS=\
    -c \        # compile but don't link
    -g \        # with debug info
    -x assembler-with-cpp \  # language is assembly with added cpp features (#define, etc)
    -flto

${AVR_GCC}/avr-gcc \
    ${AVR_GCC_ASM_OPTS} \
    ${TN85_OPTS} \  # ATtiny85-specific configuration
    ${ATTinyCoreIncludes} \
    ${TEMP_DIR}/sketch/asmOne.S \       # input assembly
    -o ${TEMP_DIR}/sketch/asmOne.S.o    # output object file

1.2.2. C code

#
# Compile the C source file
#
AVR_GCC_C_OPTS=\
    -c \        # compile but don't link
    -g \        # with debug info
    -Os \       # opt for small size
    -Wall \     # turn on "all" warnings
    -Wextra \   # and some more warnings
    -std=gnu11 \    # 2011 C (not C++ !) standard with GNU extensions
    -ffunction-sections \
    -fdata-sections \
    -MMD \
    -flto \
    -fno-fat-lto-objects

${AVR_GCC}/avr-gcc \
    ${AVR_GCC_C_OPTS} \
    ${TN85_OPTS} \  # ATtiny85-specific configuration
    ${ATTinyCoreIncludes} \
    ${TEMP_DIR}/sketch/foo.c \      # input C source
    -o ${TEMP_DIR}/sketch/foo.c.o   # output object file

1.2.3. C++ code

We are not really using C for this course. Note, however, that Arduino and `.ino` *is* C. You can see this in the way that you interact with some libraries — cf Serial.begin() and Serial.println().

#
# Compile the C++ source file (derived from the main .ino sketch file)
#
AVR_GCC_CPLUSPLUS_OPTS=\
    -c \        # compile but don't link
    -g \        # with debug info
    -Os \       # opt for small size
    -Wall \     # turn on "all" warnings
    -Wextra \   # and some more warnings
    -std=gnu++11 \  # 2011 C++ standard with GNU extensions
    -fpermissive \
    -fno-exceptions \
    -ffunction-sections \
    -fdata-sections \
    -fno-threadsafe-statics \
    -MMD \
    -flto \

${AVR_GCC}/avr-g++ \
    ${TN85_OPTS} \  # ATtiny85-specific configuration
    ${ATTinyCoreIncludes} \
    ${TEMP_DIR}/sketch/avrgcc-asm-functions.ino.cpp \     # input C++
    -o ${TEMP_DIR}/sketch/avrgcc-asm-functions.ino.cpp.o  # ouput object file


That was intense.

The theme is:

  • There are a bunch of options. Those are decisions and then fed to the tools as needed.

  • The compiler needs to know the specific CPU.

  • Some options are passed to the code via #define (-D…​ arguments).

  • Library search locations need to be defined.

  • Each type of source file has a slightly different invocation, specific to assembly, C, or C++ source files.

  • Every individual source file is compiled to an individual .o object file.

The -flto options enable GCC WiKi: Link Time Optimization. See the link for more specific information.

compilation unit

a single source file after pre-processing (#include, etc.).

Without LTO, the compiler can only optimize what is within a single compilation unit, which is generally a single source file. Therefore, without LTO, it can’t, for example, remove functions in a library that are never actually called from the main code.

Any libraries that are used in the sketch were discovered by the arduino-builder tool. These also need to be compiled to object files. In this example, there are no other libraries, so this phase is empty.

Compiling libraries…​

Compiling core…​

The “core” is the code that forms the Arduino-ness. It also is where the main() function is defined, which looks essentially like:

int main(void) {
    core_setup();
    setup();   // from your sketch

    while (1) {
        loop();   // from your sketch
    }
}

Using precompiled core: /tmp/arduino_cache_296676/core/core_ae4204c0d6efd8d62f842a80ff5a750f.a

^ The “core” had already been compiled this session, so it was cached. You can learn about the differences between .o objects and .a archives on your own.

1.3. Linking objects into a single binary

Linking everything together…​

The final phase is called “linking”, which is the process of finally resolving all symbolic address labels to actual numbers. The link-time optimizer also operates in this phase.

${AVR_GCC}/avr-gcc \
    -Wall \
    -Wextra \
    -Os \
    -g \
    -flto \
    -fuse-linker-plugin \
    -Wl,--gc-sections \
    -mmcu=attiny85 \
    -o ${TEMP_DIR}/avrgcc-asm-functions.ino.elf \ # output file
    \
    \ # list of the object files to link together
    \ # created in the compilation phase
    ${TEMP_DIR}/sketch/asmOne.S.o \
    ${TEMP_DIR}/sketch/foo.c.o \
    ${TEMP_DIR}/sketch/avrgcc-asm-functions.ino.cpp.o \
    \ # this was compiled earlier and cached
    ${TEMP_DIR}/../arduino_cache_296676/core/core_ae4204c0d6efd8d62f842a80ff5a750f.a \
    \
    -L${TEMP_DIR} \  # add the source director to the include search path
    \
    -lm  # the "m" library "libm" which implements <math.h>

The output file of this linking is an ELF format binary file:

  • avrgcc-asm-functions.ino.elf

Executable and Linkable Format - Wikipedia

elf101 64
Figure 1. Via Angea Albertini - Basics of Computing binary formats

1.4. Output IntelHEX for programming

avr-objcopy is a utility to extract things from the ELF file.

# Get the EEPROM contents to be programmed
${AVR_GCC}/avr-objcopy \
    -O ihex \  # output Intel HEX format
    -j .eeprom \  # just this section
    --set-section-flags=.eeprom=alloc,load \
    --no-change-warnings \
    --change-section-lma \
    .eeprom=0 \
    /tmp/arduino_build_882897/avrgcc-asm-functions.ino.elf \  # input file
    /tmp/arduino_build_882897/avrgcc-asm-functions.ino.eep    # output file


# The Flash contents to be programmed
${AVR_GCC}/avr-objcopy \
    -O ihex \  # output Intel HEX format
    -R .eeprom \  # remove this section from the output (see above!)
    /tmp/arduino_build_882897/avrgcc-asm-functions.ino.elf \  # input file
    /tmp/arduino_build_882897/avrgcc-asm-functions.ino.hex    # output file
${AVR_GCC}/avr-size -A \
    ${TEMP_DIR}/avrgcc-asm-functions.ino.elf

# This outputs information about resource usage,
# which arduino-build then uses to report the following:

Sketch uses 314 bytes (3%) of program storage space. Maximum is 8192 bytes.
Global variables use 5 bytes (0%) of dynamic memory, leaving 507 bytes for local variables. Maximum is 512 bytes.

1.5. Listing file: binary back to assembly

To output the .lst file, the Sketch → Export compiled Binary menu item does the following command:

${AVR_GCC}/avr-objdump \
    --disassemble \
    --source \
    --line-numbers \
    --demangle \
    --section=.text \
    ${TEMP_DIR}/avrgcc-asm-functions.ino.elf \
    > ${TEMP_DIR}/avrgcc-asm-functions.ino.lst

2. ShortSquawker

It’s one thing to take a tour. Another thing to be the tour guide.

ShortSquawker is Prof. White’s extended-version of the ECE 322 final project

It’s one thing to take a tour. Another thing to be the tour guide.

ShortSquawker is Prof. White’s extended version of the ECE 322 final project. Your task will be to take this non-trival application, clean up the code, remove any dependency on the Arduino platform, and write one or more critical functions in assembly.

https://agnd.net/valpo/322/shortsquawker-1.html

https://github.com/etihwnad/short-squawker