Migrating From VxWorks To Embedded Linux
Migrating From VxWorks To Embedded Linux
Linux
Introduction
The UBM Tech 2013 Embedded Market Study shows that nearly half of the embedded developers that
responded are using a version of Linux as a current platform with 27% using Embedded Linux and more
than half considering a move to Embedded Linux.
The top reasons for this interest are low cost, adaptability, available networking and driver libraries,
performance, and control of features including access to source code.
The major concerns about migration from VxWorks to Embedded Linux are compatibility with the
current development infrastructure, performance and real time capability, and memory usage.
This paper will:
VxWorks 5.x
VxWorks 6.x
Embedded
Linux
API
Task based
C/C++, Java,
C/C++, Java,
ADA
ADA
Architecture
NA
From US$28000
for a single seat
up to ~$150,000
to 199,000 or
more (estimated)
for a large
1
installations
See notes below
Free (Obviously
with costs for
support contracts
and other
services)
Cost of Updates
Varies
Varies Several
thousand dollars
2
per year
Free
Linux, Solaris,
Windows
Linux, Solaris,
Windows
Windows, Linux
IDE
Tornado
Wind River
Workbench
Eclipse,
NetBeans, Anjuta,
Aptana Studio,
Bloodshed Dev,
Code::Blocks,
Geany, Qt
Creator, IntelliJ
IDEA
(Eclipse based)
No source
No source
C source code
available
available
4/13/2021
Minimum 6580KB
Minimum 350500KB
Typical 500KB
Typical 3 1.1
1.2MB
ARM, Intel,
MIPS, PowerPC
Processor
ColdFire, ARM,
SuperH
Real-Time support
Yes
Yes
Preemptive
Scheduling
available
5 MB image size
can be achieved
for the simplest
applications, but
typical size for
x86 is 256MB,
X86, ARM,
PowerPC,
AVR32, ETRAX
CRIS, FR-V,
H8300, IP7000,
m68k, MIPS,
mn10300,
SuperH, and
Xtensa
Real-time support
option available in
the kernel
256 Priority
Levels
Runtime Royalties
Yes
Yes
None
Build Tools
Tornado, Yocto
Project
Workbench,
Yocto Project
crosstool-ng,
PTXdist,
Buildroot, Yocto
Project
Cost Balancing
A VxWorks 6.9 ARM kernel that includes standard components: GUI, Web server, network stack, file system,
standard I/O
As the above table shows VxWorks and Embedded Linux are very similar in most ways with the
exceptions of cost, footprint, and access to kernel and driver source code. When deciding the platform
to build an embedded product with the following costs have to be considered:
Development
o VxWorks and Linux are equally well supported by a full range of developer tools.
Developer tools for Linux are available for free as open source or can be purchased from
commercial vendors. The entire Linux source code is freely available
Hardware
o VxWorks has a footprint advantage over Linux for less complex devices that do not
require UI or networking - potentially saving on hardware expenses.
Licensing
o Linux is royalty-free.
Maintenance
o : Linux is community-maintained saving regular maintenance expenses.
Performance and Reliability
o VxWorks and Linux have substantial numbers of embedded deployments. Both systems
are used in mission-critical systems with strict reliability and uptime requirements.
Quality Assurance
o Open source creates transparency enabling anyone to examine the code which in turn
creates confidence in the quality. Projects such as the Yocto Project with broad backing
from commercial entities provide regular release cycles and quality assurance.
A real time operating system (RTOS) is generally categorized in two flavors hard and soft.
Hard versus soft real time performance is quite often a matter of interpretation. Generally, if system
time deadlines cannot be met all of the time the system cannot be considered to be hard real time. Hard
real time systems need to be deterministic with known maximum execution times for all system
operations under any condition and load. RTOS systems meeting this standard are often termed
guaranteed real time. It is important to remember that no matter how able an RTOS is ultimately it is
the design and implementation of the application that determines if a system does meet real time
needs.
Soft real time systems are guaranteed to eventually complete the task but there may be a significant
time delay although it does not affect the systems usability.
Firm real time is between the two. Sometimes referred to as conditional real time a majority of the tasks
are completed before the deadline but some may not. A firm definition of what performance constitutes
a firm real time system is too contentious to discuss here.
Latency and Jitter
Latency is simply the time between an event that requires a system response and the time in which it
takes the system to run that response. The components making up latency are not that simple. Latency
is comprised of task context switching overheads, IRQ handling interrupt request hardware and
software service routines, memory management issues, thread priority issues, and scheduler
methodologies.
Jitter is the differences in latencies of events. Jitter is a big factor in a platforms suitability as an RTOS.
High jitter means a system behavior is unpredictable and unsuitable for hard real time use low
determinism. An RTOS needs to be deterministic.
Evaluating Embedded Linux Real Time Performance
To determine if Embedded Linux meets the RTOS needs of a project is a matter of the development
team testing it. Searching the Internet provides conflicting opinions and much of the data is outdated.
The Linux kernel with the PREEMPT_RT configuration option has worst case latency dropping to single
digit milliseconds 4.
PREEMPT_RT Linux will handle most real time situations. Hard real time requirements may require a
hard real time microkernel solution. If life-safety hard real time is required then custom assembler
and/or hardware is required no RTOS can be relied upon to meet an absolute standard.
Embedded Linux Real Time Alternatives
Patched Linux - PREEMPT_RT
Starting with the 2.4 kernel the Open Source Automation Development Lab (OSADL) PREEMPT_RT patch
added real time attributes including preemptive scheduling (in kernel 2.6) to Linux with the ability to
interrupt a most tasks executing the kernel code. PREEMPT_RT also added to support full preemption of
critical sections and interrupt handlers. These RTOS enhancements have become mainstream Linux and
are now in the common kernel and do not require specialized code additions.
Linux Kernel Real-Time Technology Enhancements include:
Non-Pre-emptible Kernel (SMP) Locks (what was the Big Kernel Lock) converted to Protection of
Kernel Critical Sections by Pre-emptible Priority Inheritance (PI) Mutex
Deterministic Scheduler
Full real time preemption support (when real time preempt patches are also applied)
High Resolution Timers
IRQ-Disable Virtualization for Drivers
o IRQ threads disabled without masking hardware (masks can be slow)
IRQ handlers run in preemptible threads context allowing preemption and prioritization
o both hard and soft IRQs can run in thread context
o The real hardware interrupt context is still available
o All hard and soft IRQs execute as high priority kernel threads
Preemption Support including preemptive Read-Copy Update
R/W Semaphore Cleanup
Raw Spinlock Annotation
Read-Write Locks converted to PI Mutex
Spin-Locks converted to PI Mutex
User-Space Mutex
Migration Strategy
Attempting to develop a new project on a new platform using new tools could overwhelm any team.
Porting a known product with known performance from VxWorks to Embedded Linux gives the team a
baseline for performance and behavior comparison as well as familiarity with the code.
Porting need not even start with a complete recoding. Alternatives are available that require no or
minimal recoding to run a VxWorks application in Embedded Linux.
Virtual Hosting
Virtualization can support migration by using virtual machine management (VMM) software that allows
a VxWorks application and VxWorks itself to run mostly intact as a virtual machine with Linux executing
in its own partition. This solution is an alternative when legacy code has dependencies on VxWorks APIs
and on performance characteristics.
Embedded virtualization can be a quick and easy migration strategy but it still has VxWorks run-time
royalties and there may be need of a license from the VMM supplier. Virtualization software, including
embedded-specific paravirtualization solutions are available from many commercial developers and
open source projects like Adeo and Xenomai.
Another method uses a separate kernel that runs with the Linux kernel. These kernels are designed to
take over all real time processes and are designed to operate as true real time systems. The Linux kernel
itself is run as a lowest priority task and used to continue non real time operations.
This is adding real time under Linux rather than into it. An example of this is Xenomai. Xenomai adds a
real-time subsystem micro kernel that is used through the Linux kernel. This micro-kernel operates
between the hardware and Linux implementing a real-time scheduler and is responsible for executing all
hard real-time tasks. The Xenomai micro-kernel intercepts interrupts preventing the Linux kernel from
preempting the hard real-time microkernel. Linux kernel executes in the background as a low-priority
task that remains in control of non-real-time tasks.
Xenomai uses the ADEOS real-time nano-kernel to handle real-time interrupt dispatching to provide the
hard real-time development framework with real-time support for user-space applications. It also
provides different APIs for creating RT tasks, timers, synchronization objects. It also compatibility layer
APIs called skins including VxWorks, POSIX, RTAI, and other allowing easier porting to Linux.
NOTE: There are many other commercial and open source Embedded Linux RTOS solutions available.
Compatibility Layers
Legacy VxWorks applications can be run on Linux using hosted runtime to service RTOS system calls and
other APIs. Although many VxWorks RTOS entry points and stand-alone compiler library routines have
exact analogs in Linux and in the glibc run-time library, many do not. Developers can choose to build and
maintain emulation libraries themselves or use commercial offerings or an open source project like v2lin
that emulates several dozen commonly used VxWorks APIs.
Migration Procedure
Careful planning is key to a successful migration. Embedded Linux provides a rich infrastructure through
organizations like The Linux Foundation that will provide the information and training required.
Transparent function name, prototype, parameters and types are identical although
developers have to be careful that the semantics and context are the same.
Similar minor differences in parameters or types easily handles through #define in headers.
Emulation parameters can be typecast or directed through a stub or wrapper
Complex no equivalent function is available or one-to-many mapping is not possible
If the port involves more than mapping calls and replacement of libraries a redesign of the application
may have needed.
Tune performance:
o Run dynamic performance tests
o Compare the legacy application bottlenecks to the migrated code performance
o Determine comparative ratios execution time in user code, libraries, and kernel code
o Examine interrupt latency, preemption, scheduling latency, boot-time
o Incrementally recode and redesign to take advantage Embedded Linux models
Generally, Linux processes, having a greater overhead, are created at startup. Process code needs to be
running from startup and through the application run time. VxWorks tasks created on the fly are
mapped to Linux threads that are created by the Linux process. Linux threads share the environment of
the process that created it so when mapping VxWorks tasks those that are interdependent are created
by the same Linux process. This approach avoids the overhead of Linux process creation at the expense
of higher system load.
NOTE: Porting is an opportunity to audit use of tasks in legacy VxWorks code and attempt optimize
any other resource use.
Non MMU Supporting Projects
Embedded Linux can be used for hardware without a memory management unit. Embedded Linux
supports both MMU and MMU-less architectures, though the latter have some limitations, such as no
memory protection and the risk of having the address space extremely fragmented.
Modern systems-on-chip (SoC) now commonly include an MMU on chip letting system architects take
full advantage of memory protection and virtual memory in their embedded designs.
Top-down: Using a prebuilt embedded Linux distribution such as OpenWRT or even Android and
customizing it to the project requirements.
Both methods have approach is significant challenges. Bottom-up requires an understanding of the
components of a Linux operating system stack as well as the tools to build it. Ready-to go embedded
Linux distributions offer a good starting point and commonly include toolchains to build them but
customization beyond the original design may be complex. Both approaches carry the burden of
maintenance but tools like Yocto make this easier.
The Yocto Project builds upon and extends a series of upstream open-source projects that form its
toolkit. With each Yocto release, updated versions of these projects with new features and bug fixes are
included. Every new release of Yocto also provides updated recipes and templates to track changes,
features and bug fixes of the source packages that make up a Linux distribution such as the Linux kernel.
Ability to create workgroups and collaborate to solve pressing Linux or open source issues in a
vendor neutral setting
Access to the Linux Foundation media network, including Linux.com. The Linux Foundation
reaches 2 million users and developers a month through its online channels and newsletters and
promotes members directly to these audiences. Get the maximum impact with your products by
reaching both consumers and developers.
Directly collaborate with the technical leaders of Linux
Discounts on Linux training and Linux Foundation events
Event priority
Exclusive member content, such as the Briefing Book, that keeps you up to date on the Linux
market to make the most of your investment in Linux
Logo listing on the Linux Foundation site and the ability to add a member logo to your site or
marketing materials
Networking opportunities and a unique introductions service. Meet other Linux Foundation
members and Linux users in small settings or get introduced to companies in a one-on-one
fashion by Linux Foundation staff
Open Source legal best practices at our members-only legal summits
Participate in Linux Foundation member-only activities like the Linux Foundation Collaboration
Summit and learn, influence and participate with the Linux Foundation workgroups
Right to participate in Linux Foundation member councils
Right to vote and run for Linux Foundation board seats and influence the direction of the
organization
As with Linux, The Linux Foundation relies on the community to create and drive the content and
conversation and future of development.
Conclusion
The embedded hardware and software landscape is changing. Until a few years ago, the majority of
embedded systems were comprised of specialized hardware running a software stack providing a
limited set of functionality. Today's embedded systems have evolved into devices commonly requiring
networking capabilities, sophisticated user interface, remote management among many other functions.
Traditional embedded operating systems, like VxWorks, are keeping up with the trend by integrating
more and more functionality typically found in general purpose operating systems (GPOS). GPOS, like
Linux, on the other hand are increasingly adopting features previously limited to real-time operating
systems. The two ends of the spectrum are moving together bridging the technological gap.
This convergence is not only affecting technological equivalence but carries through to other enters of
interest to the engineering mangers and team leads.
Project costs are mitigated using Linux both at the front and back ends of the project as well as during.
Initial startup is free and Linux is royalty free. Updates and fixes during the project are also no charge.
Any Linux support required is open to competition by many firms or can be handled internally. VxWorks
requires Wind River support.
Team skillsets are not as large a factor as before. VxWorks has developed from a bare metal OS to
become POSIX compliant and use common development tools meaning VxWorks shops already have the
skills to adopt Embedded Linux.
Time to market using Linux was a factor in the past. VxWorks had an advantage in hardware support,
libraries, documentation, and training in the past. Linux now rivals (or surpasses) VxWorks in all of these
aspects. Linux is well-suited to meet the challenges of future embedded devices comprising the Internet
of Things. By utilizing the resources of the Linux Foundation organizations seeking to adapt Linux can
significantly accelerate their embedded product development based on Linux and open source software.