With multicore processors becoming the norm, a combination of isocpus, some low latency kernel options, and some virtualization are all that's needed imo.
I get sub-microsecond response times on my servers which, afaik, is a few orders of magnitude better than what's needed for control systems.
Basically, on a modern ARM chipset, the requirements of a linux distro suited for cars aren't particularly exotic these days. All the logical bits are there, it just needs firmware/drivers.
Real time has nothing to do with response times or performance. It has to do with predictability and ability to reason about how much time an operation will take (most of the time it is enough to be able to have some kind of hard limit on how much time something will take).
This means that you need all your operations that you call from critical realtime code to not suddenly start doing some housekeeping.
On Linux, even as simple operation as dereferencing a pointer can cause context to be switched to kernel to suddenly start doing its memory management stuff (not really Linux fault, that's how x86 architecture is built).
So while a machine might be able to respond "orders of magnitude" better than needed for control systems it is the one response in X thousand or million that is not meeting the limit that is deal breaker here.
I have implemented an embedded controller for coin acceptor. If the driver does not respond within X milliseconds the coin passes the gate and there is one more angry customer.
I have implemented a framework for algorithmic trading for one of largest brokerage houses on Warsaw Stock Exchange. You have XX microseconds to respond to market on event and if you don't there could be financial consequences.
Both apps were implemented on Linux. In both cases I had to work my ass off to track down multitude of potential causes of unpredictability and I think I did pretty good job but I cannot give guarantee that it will always work within time limit. If there was somebody's life at stake I would refuse to do it.
Did you know that x86 CPUs stop from time to time to do their housekeeping, like checking temperature to decide whether to throttle or not? You can turn it off (at the cost of the server potentially overheating) but how many more potential sources of problems are there?
Yes, I am extremely familiar with high frequency trading, I've been doing it for about 9 years now on most of the highest volume exchanges in the world.
Satisfying hard real time requirements is hard, but not impossible. When testing, set a fault/trace point on long response times, debug, rinse, repeat.
The point is that Linux (or QNX or whatever other RTOS) have way more resources, libraries, and commoditization than a dedicated microcontroller/FPGA and are more widely tested than some proprietary OS.
And those Intel hardware faults you mention, aside from being irrelevant to the kind of architectures a car would use, are on the order of microseconds, well within the operating requirements of a car (or coin system).
He meant algorithmic trading which is close (but not exactly the same) as high frequency trading. HFT is obsessed with latency. Algorithmic trading is mostly about automating wishes of a trader because if he can execute his strategy faster he gets a bit of an additional edge.
The coin I mentioned is completely different area. This was for a fixed odds betting terminal. The company ported their software from RTOS to Linux but they had issue with occasionally missing events generated by their custom hardware board controlling the device. This being betting terminal caused grief to customers and unhappy customers == no income.
I was contracted to solve this issue for them which I figured out would be to write a proper kernel driver for their board, restructuring of their existing code and to tune their Linux kernel/configuration.
The device they used as coin acceptor was very cheap (one figures for a device that returned its cost in couple days of operation...) and it had no controller, just analog output. The output would show the current value of something related to the induced current in the coin while the coin is falling through a piece of tubing inside an energized coil. The software has to probe the value in regular intervals while the coin is passing the sensor, and then classify as one of the possible coins or an unknown object based on duration, strength and profile of the signal. This relates to the dimensions and materials used to build the coin.
The precision at which you perform the measurements greatly influences precision of detection of various types of coins and it allows tighter tolerances which helps prevents fraud.
Once the coin passes the detector it has couple centimeters before it arrives at a gate which sorts the coins to their possible destinations.
The software has to periodically (multiple times a millisecond) monitor the value of the analog value, and then has couple milliseconds to decide what the coin is and send command to the gate.
There is not much to it unless you have some other interrupt completely disrupting the process and this is the part they could not figure out how to solve. The calculations are not complex and there is plenty of time to perform them.
I suggested they can easily implement it on their board and return buffer of all values to their Linux but they told me they have huge amount of these machines in the field and they don't want new design for their controller.
> With multicore processors becoming the norm, a combination of isocpus, some low latency kernel options, and some virtualization are all that's needed imo.
Have you worked in vehicle embedded systems or are you just pulling requirements (or lack thereof) out of your hat?
I'm not an expert, but I would guess that they do require deterministic behaviour, which, AFAIK, Linux doesn't offer. What happens if for some weird reason the scheduler starves the critical process for 40 ms in some uncommon situation that only happens 0.00001% of the time? How can you prove that such a thing won't ever happen?
His point is that there are ways to keep the OS from ever fooling with certain cores, after startup. At the extreme, you put an exo-, para-, iso-, whatsit-kernel on them, but just starting a process that does all its kernel activity at startup, then never does another system call after, is more common.
Usually that involves mapping device registers into the process address space via a custom ioctl, and then just polling in a tight loop.
That's not really enough. As an example ANY operation involving filesystem or memory access has no defined limit on how long it can take.
Your filesystem can decide to do some housekeeping while you do write and as simple thing as dereferencing a pointer can cause cause context switch to Linux kernel to do the TLB thing. You might want to allocate some memory and then Linux might decide to move a bunch of memory around before it gives the pointer back to you.
What I mean is, to be able to guarantee execution of a piece of code is not a simple task for the faint hearted. You need to forget IO. You need to forget switching context to Linux (basically forget any syscalls). You need to forget allocating any memory after program startup. You need to use really huge pages. Forget being green, because changing CPU frequency or sleeping can cause unexpected latency in the middle of some operation.
If you are not doing any system calls, it means you are perforce not doing file system operations.
Memory is normally pre-allocated and mlocked into place at startup. TLB misses can happen, but those are filled in hardware, just like regular cache misses, not by kernel activity (and often from cache). The "TLB shootdowns" you read about happen when memory is unmapped, so you don't. The kernel can block you if you write to memory mapping a file backed by a physical disk, so you don't.
And, yes, the core is locked in C0 state, drawing maximum power full time.
It is certainly possible to do I/O, by operating directly on the device registers that I noted were mapped into process memory. And, in fact, this is a routine method of conducting such operations. You can read about it by searching "kernel bypass".
There is usually no need or desire to do it in a VM, but kernel-bypass libraries are quite capable of mapping their registers into a VM's memory, so somebody must be doing it.
Commonly, input from network interfaces is dumped into one or more big-ass ring buffers mapped onto /dev/hugepages; and other, regular processes, running on other cores, map those read-only and pick over it for interesting bits, and maybe do file system operations of their own.
I disagree, it's fine for automotive use, but it will be a stripped down fork optimized for it. Just like ucLinux and other special built distros.