E.P.A. Finds More VW Cheating Software, Including in Porsches
Background: I used to be an automotive software engineer. I speak for
myself only here.Whenever the topic of automotive software comes up on HN there are
comments alongs the lines of "global variables bad", but not much
construtive feedback.I want to explain some of the tradeoffs that lead to the architecture
used in automotive software to get a better discussion going with HN
readers about that architecture.tl;dr Given the hardware restrictions, real-time requirements and
measurement capabilities required in automotive software, shared global
variables without locks is a fast and safe way to share state between
different software components as long as each variable is only written
in one place in the program.The microprocessor has to run for 10+ years in a wide range of
temperatures and be dirt cheap, so you end up with specs like 180 MHz, 4
MB of flash and 128 KB of RAM.The program must run deterministicly with respect to memory. There is no
malloc/new in the code. All variables are statically allocated.Because the physical world doesn't pause, no code is allowed to block
while waiting for resources, especially synchronization primitives like
mutexes.The software architecture is in 2 main parts: basic software
containing the real-time OS and hardware drivers, and the
application layer which has the domain-specific code for controlling the
engine, brakes, etc.The basic software is implemented using usual C programming techniques.
It has an API provided by function calls and structs to hide
implementation details of each microcontroller.The application software is where the programming model is different.To understand why, you need to know where automotive software comes
from and what it is trying to acheive.Originally all controllers were mechanical: a valve opens proportionally
to the vacuum in a part of the system. Then some controllers were
implemented in analog electronics: take multiple voltages, feed them
through an op-amp and use the output to control a valve.So automotive software reproduces this: get some inputs, compute the
same physical equations at a regular rate and generate outputs.This is dataflow programming. Blocks of code have inputs and outputs.
They are executed at a fixed rate that depends on the physical phenomena
(air flow changes fast, temperature changes slowly). Different blocks
are conneceted together in a hierachical way to form subsystems.
Encapsulation is acheived by viewing these blocks as black boxes: you
don't need to care how the block works if you are only interested in
knowing which inputs it uses and outputs it produces.Here's an example component to control a gizmo.It might be implemented in a visual environment like Simulink by
MathWorks, or it implemented by hand from a spec.  #include "GizmoController_data.h"
  
  void GizmoController_100ms() {
    Gizmo_Gain = interpolate2d(Gizmo_Gain_MAP, EngineSpeed, CoolantTemp);
  }
  
  void GizmoController_10ms() {
    Gizmo_Error = Gizmo_PositionDesired - Gizmo_Position;
    Gizmo_DutyCycle = limit(Gizmo_Gain * Gizmo_Error + Gizmo_Offset_VAL, 0, 100);
  }

It takes some inputs (EngineSpeed, CoolantTemp, Gizmo_PositionDesired,
Gizmo_Position), has some intermediate values (Gizmo_Error), and
outputs (Gizmo_DutyCycle). Those are implemented as global variables. It
also uses some constants (Gizmo_Gain_MAP, Gizmo_Offset_VAL). It has 2
processes, running every 100ms and 10ms. All this information would be
specified in an XML file.The header GizmoController_data.h is auto-generated at compile time by a
tool from the XML file mentioned above. It will contain global variable
definitions for the inputs, intermediates and outputs with the
appropriate volatile, const, static and extern storage classes/type
qualifiers. This ensures that the compiler will enforce that inputs
can't be written to, intermediate values are private to the component
and outputs can be read by other modules.Note that no explicit synchronization is needed to access inter-process
variables like Gizmo_Gain or inter-component variables like
Gizmo_Position.  It's shared memory between 2 processes scheduled in OS
tasks that can potentially interrupt each other, but since the write is
atomic and happens only in one place, there is no data race. This is
huge! Concurrent programming, without locks, with the best efficiency
possible, using a simple technique anybody can understand: only one
place in the program is allowed to write to any global memory location.Calibration is another aspect of automotive software. In most software
the constants either never change or can be set in some kind of
configuration file. For an automotive controller, the value of constants
(gains, offsets, limits, etc) depend on the vehicle so they must
be configurable at run time during development. This is implemented in
the C code by putting all constants in a memory area that is ROM in
production units, but RAM in development units. The compiler enforces
that application software cannot change constants, but the basic
software includes code so that constants can be changed from the outside
in development. This process is called calibration and is done by
calibration engineers who are usually not the ones who wrote the
software. Note that calibration can drastically affect the behavior of
the software. What would happen if Gizmo_Gain_MAP is set to all zeros?Measurement of variables is essential to understanding what's going on
inside the embedded controller. Having all that state available in
global variables makes it possible for the calibration tool request the
value of any variable in the software at a fixed rate and display it in a
virtual oscilloscope.The measurement and calibration tool needs to know how to access the
variables and constants. It uses a file that maps from names to
addresses for a particular version of software. That file can easily be
generated a compile time since all allocations are static.Going back to the architecture of the application software, let's look
at where our gizmo controller fits.  It is not the only component needed
to make the gizmo work. You also need components to calculate the gizmo
position from some external signal (let's say an analog voltage), to
route the output signal to the powerstage driver on the PCB, to
determine which position the gizmo should currently occupy. These would
form the gizmo subsystem package.When the supplier releases gizmo 2.0 (TM) they upgrade the input signal
to be a PWM input instead of an analog input. Modularity in the software
allows the software team to simply replace the gizmo position component
with one that reads a PWM instead of an analog voltage and keep the rest
of the gizmo subsystem the same.  In the future, projects that use gizmo
1.0 use one version of the gizmo subsystem and projects that use 2.0 use
another.This is true at any level in the hierarchy: as long as the inputs and
outputs are the same, a component or subystem package can be replaced by
another.Version control in automotive software reflects this. Instead of having
one tree of versions and releases like a typical software project, each
component, subsystem package and software project has its own tree of
versions. Each software project will reference the subsystem packages
required for their engine type, vehicle platform, sensors and actuators,
etc. This is how code reuse is acheived.Testing is a mix of simulation (the sensor/actuator is simulated in
Simulink and connected to Simulink block diagram of the software
component), hardware-in-the-loop (a computer simulates the vehicle, but
the real electronic control unit is used) and vehicle testing.Thanks for reading. I hope this improves your understanding of how
automotive software is structured.I'm hoping the discussion will bring examples from other fields like
robotics, drones and aeronautics that have similar real-time
requirements on how they architect their software.