Cits2002 Compressed PDF

CITS2002 Systems Programming
CITS2002 CITS2002 schedule
Welcome to CITS2002 Systems Programming

This unit, first presented in 2012 as CITS1002, is one of the core units in each of UWA's Computer Science, Data Science, and Engineering Science (SE) majors.
The unit explores the role of contemporary operating systems and their support for high-level programming languages, how they manage efficient access to computer hardware, and how a computer's resources may be accessed and controlled by the C programming language.
The unit will be presented by Chris McDonald - which is pronounced 'Chris', not 'Dr', not 'Professor'.
Our UWA Handbook entry
Understanding the relationship between a programming language and the contemporary operating systems on which it executes is central to developing many skills in Computer Science. This unit introduces the standard C programming language, on which many
other programming languages and systems are based, through a study of core operating system services including input and output, memory management and file systems.
The C language is introduced through discussions on basic topics like data types, variables, expressions, control structures, scoping rules, functions and parameter passing. More advanced topics like C's run-time environment, system calls, dynamic memory
allocation, and pointers are presented in the context of operating system services related to process execution, memory management and file systems. The importance of process scheduling, memory management and interprocess communication in modern
operating systems is discussed in the context of operating system support for multiprogramming. Laboratory and tutorial work place a strong focus on the practical application of fundamental programming concepts, with examples designed to compare and contrast
many key features of contemporary operating systems.
The UWA Handbook entry for this unit strongly recommends that you take one of three units of Advisable Prior Study before taking this unit - CITS1001, CITS1401, or CITS2401. Students who took this unit in recent years, and had chosen to not take one of the units of
Advisable Prior Study, found the material in this unit difficult. This unit is not suitable for first-time programmers.
CITS2002 Systems Programming, Lecture 1, p1, 29th July 2019.
Topics to be covered in CITS2002 Systems Programming

It's important to know where we're heading, so here's a list of topics that we'll be covering:
An introduction to the ISO-C99 programming language

The structure of a C program, basic datatypes and variables, compiling and linking.
An introduction to Operating Systems

A brief history of operating systems, the role of contemporary operating systems, the relationship between programming languages, programs, and operating systems.
An overview of computer hardware components

The processor and its registers, the memory hierarchy, input and output (I/O) and storage components.
C programs in greater detail

Arrays and character strings, user-defined types and structures, how the computer hardware represents data, functions, parameter passing and return values.
Operating system services

Creating and terminating processes, a program's runtime environment, command-line arguments, accessing operating system services from C.
Managing memory
Allocating physical memory to processes, sharing memory between multiple processes, allocating and managing memory in C programs.
Files and their use in programs

The file management system, file allocation methods, file and directory operations and attributes, file input and output (I/O), raw and formatted I/O, unbuffered and buffered I/O functions.
By the end of this unit you'll have this knowledge - it just won't all be presented strictly in this order.
Here is our unit's schedule.

Why teach C?
Since its beginnings in the early 1970s, the C programming language has evolved to become one of the world's most popular, and widely deployed programming languages. The language has undergone extensive formal standardization to produce the ANSI-C standard in
1989, the ISO-C99 standard in 1999, ISO-C11 (revision) in Dec 2011, and ISO-C18 ("bugfix release", no new features) in June 2018.
C is the programming language of choice for most systems-level, engineering, and scientific programming:
most of the world's popular operating systems, Linux, Windows and macOS, their interfaces and file-systems, are written in C,
the infrastructure of the Internet, including most of its networking protocols, web servers, and email systems, are written in C,
software libraries providing graphical interfaces and tools, and efficient numerical, statistical, encryption, and compression algorithms, are written in C,
the software for most embedded devices, including those in cars, aircraft, robots, smart appliances, sensors, mobile phones, and game consoles, is written in C,
the software on the Mars Phoenix Lander is written in C,
much of the safety-critical software on the F-35 joint strike fighter, is written in C, but
C was not used on the Apollo-11 mission!
Is C still relevant? [ref: Tiobe survey]

(The Tiobe survey is based on search-engine queries - is not about the best programming language or the language in which most lines of code have been written.)
Though, of course, popularity is a poor measure of quality - otherwise, McDonald's Restaurants would receive Michelin stars.
Other interesting surveys:
Stackoverflow's Developer Survey Results 2019

Jetbrains' The State of Developer Ecosystem in 2019
HackerRank's 2019 Developer Skills Report
So what is C?
“
A programming language that doesn't affect the way you think about
programming isn't worth knowing.
— Alan Perlis, 1st Turing Award winner
In one breath, C is often described as a good general purpose language, an excellent systems programming language, and just a glorified assembly language. So how can it be all three?
C can be correctly described as a general purpose programming language -

a description also given to Java, Python, Visual-Basic, C++, and C#.
C is a procedural programming language, not an object-oriented language like Java, (parts of) Python, Objective-C, or C#.
C programs can be "good" programs, if they are:
well designed,
clearly written,
written for portability,
well documented,
use high level programming practices, and
well tested.
Of course, the above properties are independent of C, and are offered by many high level languages.
C has programming features provided by most procedural programming languages - strongly typed variables, constants, standard (or base) datatypes, enumerated types, user-defined types, aggregate structures, standard control flow, recursion, and program
modularization.
C does not offer tuples or sets, Java's concept of classes or objects, nested functions, subrange types, and has only recently added a Boolean datatype.
C does have, however, separate compilation, conditional compilation, bitwise operators, pointer arithmetic, and language independent input and output.
A Systems Programming Language

C is frequently, and correctly, described as an excellent systems programming language.
C also provides an excellent operating system interface through its well defined, hardware and operating system independent, standard library.
The C language began its development in 1972, as a programming language in which to re-write significant portions on the Unix operating system:
Unix was first written in assembly languages for PDP-7 and PDP-11 computers.
In 1973 Dennis Ritchie was working on a programming language for operating system development. Basing his ideas upon BCPL, he developed B and finally created one called C.
(Yes, there is a language named 'D', but it's not a descendant of C)
By the end of 1973, the UNIX kernel was 85% written in C which enabled it to be ported to other machines for which a C compiler could be fashioned.
This was a great step because it no longer tied the operating system to the PDP-7 as it would have been if it remained in assembly language. In 1976 Dennis Ritchie and Stephen Johnston ported Unix to an Interdata 8/32
machine. Since then, Unix and Linux have been ported to over 260 different processor architectures.
Today, well in excess of 95% of the Unix, Linux, macOS, and Windows operating system kernels and their standard library routines are all written in the C programming language - it's extremely difficult to find an operating system not written in either C or its descendants C++ or
Objective-C.
Portability on different architectures

C compilers have been both developed and ported to a large number and type of computer architectures:
from 4-bit and 8-bit microcontrollers,

through traditional 16-, 32-, and 64-bit virtual memory architectures in most PCs and workstations,
to larger 64- and 128-bit supercomputers.
Compilers have been developed for:
traditional large instruction set architectures, such as Intel x86, AMD, ARM, Motorola 680x0, Sun SPARCs, and DEC-Alpha,
newer reduced instruction set architectures (RISC), such as SGI MIPS, IBM/Motorola PowerPC,
mobile phones, home theatre equipment, routers and access-points, and
parallel and pipelined architectures.
All it requires is a ported C compiler

Once a C compiler has been developed for a new architecture, the terabytes of C programs and libraries available on other C-based platforms can also be ported to the new architecture.
What about assembly languages?

It is often quoted that a compiled C program will run only 1-2% slower than the same program hand-coded in the native assembly language for the machine.
But the obvious advantage of having the program coded in a readable, high level language, provides the overwhelming advantages of maintainability and portability.
Very little of an operating system, such as Windows, macOS, or Linux, is written in an assembly language - in most cases the majority is written in C.
Even an operating system's device drivers, often considered the most time-critical code in an operating system kernel, today contain assembly language numbered in only the hundreds of lines.
The unreadability of C programs

C is described as nothing more than a glorified assembly language, meaning that C programs can be written in such an unreadable fashion that they look like your monitor is set at the wrong speed.
(in fact there's a humorous contest held each year, The International Obfuscated C Code Contest to design fully working but indecipherable code,
and the Underhanded C Contest whose goal is to write code that is as readable, clear, innocent and straightforward as possible, and yet it must fail to perform at its apparent function).
Perhaps C's biggest problem is that the language was designed by programmers who, folklore says, were not very proficient typists.
C makes extensive use of punctuation characters in the syntax of its operators and control flow. In fact, only the punctuation characters
@ ` and $
are not used in C's syntax! (and DEC-C once used the $ character, and Objective-C now uses the @).
It is not surprising, then, that if C programs are not formatted both consistently and with sufficient white space between operators, and if very short identifier names are used, a C program will be very difficult to read.
To partially address these problems, a number of text-editors, integrated development environments (IDEs), and beautification programs (such as indent) can automatically reformat our C code according to consistent specifications.
Criticisms of C's execution model

C is criticized for being too forgiving in its type-checking at compile time.
It is possible to cast an instance of some types into other types, even if the two instances have considerably different types.
A pointer to an instance of one type may be coerced into a pointer to an instance of another type, thereby permitting the item's contents to be interpreted differently.
Badly written C programs make incorrect assumptions about the size of items they are managing. Integers of 8-, 16-, and 32-bits can hold different ranges of values. Poor choices, or underspecification can easily lead to errors.
C provides no runtime protection against arithmetic errors.
There is no exception handling mechanism, and errors such as division-by-zero and arithmetic overflow and underflow, are not caught and reported at run-time.
C offers no runtime checking of popular and powerful constructs like pointer variables and array indices.
Subject to constraints imposed by the operating system's memory management routines, a pointer may point almost anywhere in a process' address space and seemingly random addresses may be read or written to.
Although all array indices in C begin at 0, it is possible to access an array's elements with negative indices or indices beyond the declared end of the array.
There are occasions when each of these operations make sense, but they are rare.
C does not hold the hand of lazy programmers.
We avoid all of these potential problems by learning the language well, and employing safe programming practices.
What is the best programming language?

The question, even arguments, of whether C, Java, Visual-Basic, C++, or C# is the best general purpose programming language is pointless.
The important question is:

"which language is most suited for the task at hand?"
This unit will answer the questions:
"when is C the best language to use?" and

"how do we best use C's features for systems programming?"
Through a sequence of units offered by Computer Science & Software Engineering you can become proficient in a wide variety of programming languages - procedural, object-oriented, functional, logic, set-based, and formal - and know the most appropriate one to select for
any project.
The Standardization of C - K&R C

Despite C's long history, being first designed in the early 1970s, it underwent considerably little change until the late 1980s.
This is a very lengthy period of time when talking about a programming language's evolution.
The original C language was mostly designed by Dennis Ritchie and then described by Brian Kernighan and Dennis Ritchie in their imaginatively titled book The C Programming Language.
The language described in this seminal book, described as the "K&R" book, is now described as "K&R" or "old" C.
228 pages.
The Standardization of C - ANSI-C (K&R-2)
In the late 1980s, a number of standards forming bodies, and in particular the American National Standards Association X3J11 Committee, commenced work on rigorously defining both the C language and the commonly provided standard C
library routines. The results of their lengthy meetings are termed the ANSI-X3J11 standard, or informally as ANSI-C, C89, or C90.
The formal definition of ANSI-C introduced surprisingly few modifications to the old "K&R" language and only a few additions.
Most of the additions were the result of similar enhancements that were typically provided by different vendors of C compilers, and these had generally been considered as essential extensions to old C. The ANSI-C language is extremely similar to
old C. The committee only introduced a new base datatype, modified the syntax of function prototypes, added functionality to the preprocessor, and formalized the addition of constructs such as constants and enumerated types.
272 pages.
The Standardization of C - ANSI/ISO-C99 and ISO/IEC 9899:2011 (C11)
A new revision of the C language, named ANSI/ISO-C99 (known as C99), was completed in 1999.
Many features were "cleaned up", including the addition of Boolean and complex datatypes, single line comments, and variable length arrays, and the removal of many unsafe features, and ill-defined constructs.
753 pages.
A revision of C99, ISO/IEC 9899:2011 (known as C11), was completed in December 2011.
In this unit we will focus exclusively on ANSI/ISO-C99,

and only mention other versions of C when the differences are significant.
What Standardization Provides

These quite formal standards specify the form and establishes the interpretation of programs written in the C programming language. They specify:
the representation of C programs;

the syntax and constraints of the C language;
the semantic rules for interpreting C programs;
the representation of input data to be processed by C programs;
the representation of output data produced by C programs;
the restrictions and limits imposed by a conforming implementation of C.
They do not specify:
the mechanism by which C programs are transformed for use by a data-processing system;
the mechanism by which C programs are invoked for use by a data-processing system;
the mechanism by which input data are transformed for use by a C program;
the mechanism by which output data are transformed after being produced by a C program;
the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor;
all minimal requirements of a data-processing system that is capable of supporting a conforming implementation.
What's (deliberately) missing from the C language?

At first glance, the C language appears to be missing some commonly required features that other languages, such as Java, provide in their standards.
For example, C does not provide features for graphics, networking, cryptography, or multimedia.
Instead, C permits, enables, and encourages additional 3rd-party libraries (both open-source and commercial) to provide these facilities. The reason for these "omissions" is that C rigorously defines what it does provide, and rigorously defines how C must interact with external
libraries.
Here are some well-respected 3rd-party libraries, frequently employed in large C programs:
Function domain 3rd-party libraries

operating system services
OS-specific libraries, e.g. glibc, Win32, Carbon
(files, directories, processes, inter-process communication)
web-based programming libcgi, libxml, libcurl
data structures and algorithms the generic data structures library (GDSL)
GUI and graphics development OpenGL, GTK, Qt, UIKit, Win32, Tcl/Tk
image processing (GIFs, JPGs, etc) gd
networking Berkeley sockets, AT&T's TLI
security, cryptography openssl, libmp
scientific computing NAG, Blas3, GNU scientific library (gsl)
concurrency, parallel and GPU programming pthreads, OpenMPI, openLinda, CUDA, OpenCL

The structure of C programs

Let's looks at the high-level structure of a short C program, rotate.c (using ellipsis to omit some statements for now).
At this stage it's not important what the program is supposed to do.
Of note in this example:

#include <stdio.h>
Characters such as a space, tab, or newline, may appear almost anywhere - they are stripped out and ignored by the C compiler.
#include <stdlib.h>
#include <string.h>
#include <ctype.h> We use such whitespace characters to provide a layout to our programs. While the exact layout is not important, using a consistent layout is very good practice.
/* Compile this program with: Keywords, in bold, mean very specific things to the C compiler.
cc -std=c99 -Wall -Werror -pedantic -o rotate rotate.c
*/ Lines commencing with a '#' in blue are processed by a separate program, named the C preprocessor.
#define ROT 13 In practice, our program is provided as input to the preprocessor, and the preprocessor's output is given to the C compiler.
static char rotate(char c) Lines in green are comments. They are ignored by the C compiler, and may contain (almost) any characters.
{
..... C99 provides two types of comments -
return c;
} /* block comments */ and
// comments to the end of a line
int main(int argcount, char *argvalue[])
{
// check the number of arguments
if(argcount != 2) {
....
exit(EXIT_FAILURE);
}
else {
....
exit(EXIT_SUCCESS);
}
return 0;
}
CITS2002 Systems Programming, Lecture 2, p1, 2nd August 2019.
The structure of C programs, continued
Same program, but more to note:

#include <stdio.h>
A variety of brackets are employed, in pairs, to group together items to be considered in the same way. Here:
#include <stdlib.h>
#include <string.h>
angle brackets enclose a filename in a #include directive,
#include <ctype.h>
round brackets group items in arithmetic expressions and function calls,
/* Compile this program with: square brackets enclose the index when access arrays (vectors and matrices...) of data, and
cc -std=c99 -Wall -Werror -pedantic -o rotate rotate.c
*/ curly brackets group together sequences of one or more statements in C. We term a group of statements a block of statements.
#define ROT 13 Functions in C, may be thought of as a block of statements to which we give a name. In our example, we have two functions - rotate() and main().
static char rotate(char c) When our programs are run by the operating system, the operating system always starts our program from main(). Thus, every complete C program requires a main() function.
{
..... The operating system passes some special information to our main() function, command-line arguments, and main() needs a special syntax to receive these.
return c;
} Most C programs you read will name main()'s parameters as argc and argv.
int main(int argcount, char *argvalue[]) When our program finishes its execution, it returns some information to the operating system. Our example here exits by announcing either its failure or success.
{
// check the number of arguments
if(argcount != 2) {
....
}
else {
....
}
return 0;
}
Compiling and linking our C programs

C programs are human-readable text files, that we term source-code files.
This makes them very easy to copy, read, and edit on different computers and different operating systems.
C is often described as being portable at the source-code level.
Before we can run (execute) our C programs, we must translate, or compile, their source-code files to files that the operating system can better manage.
A program known as a compiler translates (compiles) source-code files into object-code files.
Finally, we translate or link one or more object-code files to produce an executable program, often termed a 'binary', an 'executable', or an 'exe' file.
A program known as a linker performs this translation, also linking our object-code file(s) with standard libraries and (optionally) 3rd-party libraries.
Depending on how we invoke the compiler, sometimes we can 'move' straight from the source-code files to the executable program, all in one step.
In reality the compiler is 'silently' executing the linker program for us, and then removing any unwanted object-files.
Variables
Variables are locations in a computer's memory. A typical desktop or laptop computer will have 4-16GB of memory, or four-sixteen billion addressable memory locations,
A typical C program will use 4 bytes to hold a single integer value, or 8 bytes to hold a single floating-point value.
Any variable can only hold a single value at any time - they do not maintain a history of past values they once had.
Naming our variables

To make programs more readable, we provide variables with simple names. We should carefully choose names to reflect the role of the variable in our programs.
While variable names can be almost anything (but not the same as the keywords in C) there's a simple restriction on the permitted characters in a name -
they must commence with an alphabetic or the underscore character (_ A-Z a-z), and
be followed by zero or more alphabetic, underscore or digit characters (_ A-Z a-z 0-9).
C variable names are case sensitive, thus:
MYLIMIT, mylimit, Mylimit and MyLimit
are four different variable names.
While not required, it's preferred that variable names do not consist entirely of uppercase characters.
We'll consistently use uppercase-only names for constants provided by the C preprocessor, or user-defined type names:
MAXLENGTH, AVATAR, BUFSIZ, and ROT
Older C compilers limited variable names to, say, 8 unique characters. Thus, for them,
turn_nuclear_reactor_coolant_on and turn_nuclear_reactor_coolant_off
are the same variable! Keep this in mind if ever developing portable code for old environments.
Basic datatypes
Variables are declared to be of a certain datatype, or just type.
We use different types to represent the permissible values that a program's variable has.
For example, if we're using a variable to just count things, we'll use an integer variable to hold the count; if performing trigonometry on angles expressed in radians, we'll use floating-point variables to hold values with both an integral and a fractional part.
C provides a number of standard, or base types to hold commonly required values, and later we'll see how we can also define our own user-defined types to meet our needs.
Let's look quickly at some of C's base datatypes:
typename description, and an example of variable initialization

Boolean (truth values), which may only hold the values of either true or false
bool
e.g. bool finished = false;
character values, to each hold a single values such as an alphabetic character, a digit character, a space, a tab...
char
e.g. char initial = 'C';
integer values, negative, positive, and zero
int
e.g. int year = 2006;
floating point values, with a typical precision of 10 decimal digits (on our lab machines)
float
e.g. float inflation = 4.1;
"bigger" floating point values, with a typical precision of 17 decimal digits (on our lab machines)
double
e.g. double pi = 3.1415926535897932;
Some textbooks will (too quickly) focus on the actual storage size of these basic types, and emphasise the ranges of permissible values. When writing truly portable programs - that can execute consistently across different hardware architectures and operating systems - it's
important to be aware of, and avoid, their differences. We'll examine this issue later, but for now we'll focus on using these basic types in their most obvious ways.
From where does the bool datatype get its name? - the 19th century mathematician and philosopher, George Boole.
The Significance of Integers in C

Throughout the 1950s, 60s, and 70s, there were many more computer hardware manufacturers than there are today. Each company needed to promote its own products by distinguishing them from their competitors.
At a low level, different manufacturers employed different memory sizes for a basic character - some just 6 bits, some 8 bits, 9, and 10. The unfortunate outcome was the incompatability of computer programs and data storage formats.
The C programming language, developed in the early 1970s, addressed this issue by not defining the required size of its datatypes. Thus, C programs are portable at the level of their source code - porting a program to a different computer architecture is possible, provided that
the programs are compiled on (or for) each architecture. The only requirement was that:
sizeof(char) ≤ sizeof(short) ≤ sizeof(int) ≤ sizeof(long)
Since the 1980s, fortunately, the industry has agreed on 8-bit characters or bytes. But (compiling and) running the C program on different architectures:
may produce different (though still correct) results:

#include <stdio.h>
char 1
int main(void) short 2
{ int 4
printf("char %lu\n", sizeof(char));
long 8
printf("short %lu\n", sizeof(short));
printf("int %lu\n", sizeof(int)); It's permissible for different C compilers on different architectures to employ different sized integers.
printf("long %lu\n", sizeof(long));
return 0;
}
Why does this matter? Different sized integers can store different maximum values - the above datatypes are signed (supporting positive and negative values) so a 4-byte integer can only represent the values -2,147,483,648 to 2,147,483,647.
If employing integers for 'simple' counting, or looping over a known range of values, there's rarely a problem. But if using integers to count many (small) values, such as milli- or micro-seconds, it matters:
Fun fact: GPS uses 10 bits to store the week. That means it runs out... oh heck ? April 6, 2019
Airlines Have To Reboot Their Airbus A350 Planes After Every 149 Hours
To keep a Boeing Dreamliner flying, reboot once every 248 days
The scope of variables

The scope of a variable describes the range of lines in which the variable may be used. Some textbooks may also term this the visibility or lexical range of a variable.
C has only 2 primary types of scope:
global scope (sometimes termed file scope) in which variables are declared outside of all functions and statement blocks, and
block scope in which variables are declared within a function or statement block.
The variable count has global scope.
01 #include <stdio.h> It is defined on line 06, and may be used anywhere from line 06 until the end of the file (line 26).
02 #include <stdlib.h>
03 #include <string.h> The variable count is also preceded by the keyword static, which prevents it from being 'seen' (read or written) from outside of this file rot.c
04 #include <ctype.h>
05 The variable nfound has block scope.
06 static int count = 0;
07 It is defined on line 10, and may be used anywhere from line 10 until the end of the block in which it was defined (until line 26).
08 int main(int argcount, char *argvalue[])
09 { The variable nerrors has block scope.
10 int nfound = 0;
11 It is defined on line 14, and may be used anywhere from line 14 until line 18.
12 // check the number of arguments
13 if(argcount != 2) { The variable ntimes has block scope.
14 int nerrors = 1;
15 It is defined on line 20, and may be used anywhere from line 20 until line 24.
16 ....
17 exit(EXIT_FAILURE);
18 }
We could define a different variable named nerrors in the block of lines 20-24 - without problems.
19 else {
20 int ntimes = 100;
21 We could define a different variable named nfound in the block of lines 20-24 - but this would be a very bad practice!
22 ....
23 exit(EXIT_SUCCESS);
24 }
25 return 0;
26 }

Flow of control in a C program

A program's control flow describes how sequences of statements are executed.
Flow control in a C program is very similar to most other imperative and object-oriented languages.
C programs commence their execution at their main() function, execute their statements, and exit (return the flow of control) to the operating system.
It's fairly obvious that statements need to be executed in a well-defined order, as we expect programs to always behave the same way (unless some random data directs the execution path, as in computer games, simulations, and heuristic algorithms).
Default flow of control executes each statement in order, top-to-bottom.
Programs that only execute from top-to-bottom are pretty boring, and we need to control their flow with a variety of conditional statements, loops, and function-calls.
CITS2002 Systems Programming, Lecture 3, p1, 5th August 2019.
Conditional execution
Conditional statements first evaluate a Boolean condition and then, based on whether it's true or false, execute other statements.
The most common form is: Sometimes, the else clause is omitted: Often, the else clause provides further if statements:
if(condition) if(condition) if(condition1)

{ { {
// more statements; // more statements; // more statements;
..... ..... .....
} } }
else else if(condition2)
{ {
// more statements; // more statements;
..... .....
} }
else
{
// more statements;
.....
}
Note that in the examples, above, each block of statements to be executed has been written within curly-brackets.
The curly-brackets are not required (we could just write a single statement for either if or else clause). However, adding curly-brackets is considered a good practice. They provide a safeguard for when additional statements are added later.
Boolean values
Of significance, and a very common cause of errors in C programs, is that C standards, prior to ISO-C99, had no Boolean datatype.
Historically, an integer value of zero evaluated equivalent to a Boolean value of false; any non-zero integer value evaluated as true.
You may read some older C code: which may be badly and accidently coded as:
int initialised = 0; // evaluates to false int initialised = 0; // evaluates to false

.... ....
if(! initialised) if(initialised = 0)

{ {
// initialisation statements; // initialisation statements;
..... .....
initialised = 1; initialised = 1;
} }
In the second example, the conditional test always evaluates to false, as the single equals character requests an assignment, not a comparison.
It is possible (and occassionally reasonable) to perform an assignment as part of a Boolean condition -

you'll often see:
while( (nextch = getc(file) ) != EOF ) {....
Whenever requiring the true and false constants (introduced in C99), we need to provide the line:
#include <stdbool.h>
Switch statements
When the same (integer) expression is compared against a number of distinct values, it's preferred to evaluate the expression once, and compare it with possible values:
Cascading if..else..if.. statements: The equivalent switch statement: Less-common features of the switch statement:
if(expression == value1) switch(expression) switch(expression)

{ { {
// more statements; case value1 : case value1 :
..... { case value2 :
} // more statements; {
else if(expression == value2) . . . . . // handle either value1 or value2
{ break; . . . . .
// more statements; } break;
..... case value2 : }
} { case value3 :
else // more statements; {
{ . . . . . // more statements;
// more statements; break; . . . . .
..... } // no 'break' statement, drop through
} default : }
{ default :
// more statements; {
. . . . . // more statements;
break; . . . . .
} break;
} }
}
Typically the 'expression' is simply an identifier, but it may be arbitrarily complex - such as an arithmetic expression, or a function call.
The datatype of the 'expression' must be an integer (which includes characters, Booleans, and enumerated types), but it cannot be a real or floating-point datatype.
The break statement at the end of each case indicates that we have finished with the 'current' value, and control-flow leaves the switch statement.
Without a break statement, control-flow continues "downwards", flowing into the next case branch (even though the expression does not have that case's value!).
switch statements with 'dense' values (none, or few integer missing) provide good opportunities for optimised code.
Flow of control in a C program - bounded loops

One of the most powerful features of computers, in general, is to perform thousands, millions, of repetitive tasks quickly
(in fact, one of the motivating first uses of computers in the 1940s was to calculate trigonometric tables for the firing of artillery shells).
C provides its for control statement to loop through a sequence of statements, a block of statements, a known number of times:
The most common form appears below, in which we introduce a loop control variable, i, to count The loop control variable does not always have to be an integer:
how many times we go through the loop:
// here, variable i holds the values 1,2,...10 // here, variable ch holds each lowercase value
for(int i = 1 ; i <= 10 ; i = i+1) for(char ch = 'a' ; ch <= 'z' ; ch = ch+1)

{ {
// the above introduced a loop-control variable, i .....
..... printf("loop using character '%c'\n", ch);
printf("loop number %i\n", i); .....
..... }
// variable i is available down to here
}
// but variable i is not available from here
Notice that in both cases, above, we have introduced new variables, here i and ch, to specifically control the loop.
The variables may be used inside each loop, in the statement block, but then "disappear" once the block is finished (after its bottom curly bracket).
It's also possible to use any other variable as the loop control variable, even if defined outside of the for loop. In general, we'll try to avoid this practice - unless the value of the variable is required outside of the loop.
Flow of control in a C program - unbounded loops

The for loops that we've just seen should be used when we know, ahead of time, how many times we need to loop (i.e. 10 times, or over the range 'a'..'z').
Such loops are termed bounded loops and, unless we've made an unseen coding error, always terminate after a fixed number of iterations.
There are also many occasions when we don't know, ahead of time, how many iterations may be required. Such occasions require unbounded loops.
C provides two types of unbounded loop:

The most common is the while loop, where zero or more iterations are made through the Less common is the do....while loop, where at least one iteration is made through the loop:
loop:
#define NLOOPS 20 #define NLOOPS 20
int i = 1; int i = 1;
int n = 0; int n = 0;
..... .....
while(i <= NLOOPS) do

{ {
printf("iteration number %i\n", i); printf("iteration number %i\n", i);
..... .....
..... .....
i = some_calculation_setting_i; i = some_calculation_setting_i;
n = n + 1; n = n + 1;
} } while(i <= NLOOPS);
printf("loop was traversed %i times\n", n); printf("loop was traversed %i times\n", n);
Notice that in both cases we still use a variable, i, to control the number of iterations of each loop, and that the changing value of the variable is used to determine if the loop should "keep going".
However, the statements used to modify the control variable may appear almost anywhere in the loops. They provide flexibility, but can also be confusing when loops become severals tens or hundreds of lines long.
Notice also that while, and do....while loops cannot introduce new variables to control their iterations, and so we have to use existing variables from an outer lexical scope.
Writing loops within loops

There's a number of occassions when we wish to loop a number of times (and so we use a for loop) and within that loop we wish to perform another loop. While a little confusing, this construct is often quite common. It is termed a nested loop.
#define NROWS 6
#define NCOLS 4
for(int row = 1 ; row <= NROWS ; row = row+1) // the 'outer' loop
{
for(int col = 1 ; col <= NCOLS ; col = col+1) // the 'inner' loop
{
printf("(%i,%i) ", row, col); // print row and col as if "coordinates"
}
printf("\n"); // finish printing on this line
}
The resulting output will be:

(1,1) (1,2) (1,3) (1,4)
(2,1) (2,2) (2,3) (2,4)
(3,1) (3,2) (3,3) (3,4)
(4,1) (4,2) (4,3) (4,4)
(5,1) (5,2) (5,3) (5,4)
(6,1) (6,2) (6,3) (6,4)
Notice that we have two distinct loop-control variables, row and col.
Each time that the inner loop (col's loop) starts, col's value is initialized to 1, and advances to 4 (NCOLS).
As programs become more complex, we will see the need for, and write, all combinations of:
for loops within for loops,

while loops within while loops,
for loops within while loops,
and so on....
Changing the regular flow of control within loops

There are many occasions when the default flow of control in loops needs to be modified.
Sometimes we need to leave a loop early, using the break statement, possibly skipping some iterations and some Sometimes we need to start the next iteration of a loop, even before executing all statements in the loop:
statements:
for(int i = 1 ; i <= 10 ; i = i+1) for(char ch = 'a' ; ch <= 'z' ; ch = ch+1)

{ {
// Read an input character from the keyboard if(ch == 'm') // skip over the character 'm'
..... continue;
if(input_char == 'Q') // Should we quit? .....
break; .....
..... statements that will never see ch == 'm'
..... .....
} .....
// Come here after the 'break'. i is unavailable }
In the first example, we iterate through the loop at most 10 times, each time reading a line of input from the keyboard. If the user indicates they wish to quit, we break out of the bounded loop.
In the second example, we wish to perform some work for all lowercase characters, except 'm'.
We use continue to ignore the following statements, and to start the next loop (with ch == 'n').
The equivalence of bounded and unbounded loops

We should now be able to see that the for, while, and do ... while control flow statements are each closely related.
To fully understand this, however, we need to accept (for now), that the three "pieces" of the for construct, are not always initialization, condition, modification.
More generally, the three pieces may be C expressions - for the moment we'll consider these as C statements which, if they produce a value, the value is often ignored.
The following loops are actually equivalent:
for( expression1 ; expression2 ; expression3 ) expression1;

{ while(expression2)
statement1; {
.... statement1;
} ....
expression3;
}
In both cases, we're expecting expression2 to produce a Boolean value, either true or false, as we need that truth value to determine if our loops should "keep going".
You should think about these carefully, perhaps perform some experiments, to determine where control flow really goes when we introduce break and continue statements.
Some unusual loops you will encounter

As you read more C programs written by others, you'll see some statements that look like for or while loops, but appear to have something missing.
In fact, any (or all!) of the 3 "parts" of a for loop may be omitted.
For example, the following loop initially sets i to 1, and increments it each iteration, but it doesn't have a "middle" conditional test to see if the loop has finished. The missing condition constantly evaluates to true:
for(int i = 1 ; /* condition is missing */ ; i = i+1)

{
.....
.....
}
Some loops don't even have a loop-control variable, and don't test for their termination. This loop will run forever, until we interrupt or terminate the operating system process running the C program.
We term these infinite loops :
// cryptic - avoid this mechanism // clearer - use this mechanism

for( ; ; ) while( true )
{ {
..... .....
..... .....
} }
While we often see and write such loops, we don't usually want them to run forever!
We will typically use an enclosed condition and a break statement to terminate the loop, either based on some user input, or the state of some calculation.
An Introduction to Operating Systems

What is an operating system?
A piece of systems software that provides a convenient, efficient environment for the execution of user programs.
It's probably the largest and most complex program you'll ever run!
Why do we need an operating system?
The user's viewpoint:

to provide the user interface, command interpreter, and directory structure, and to execute application programs (word processor, email client, web browser, MP3 player).
The programming environment viewpoint:

to enhance the bare machine, to provide utility programs (such as compilers, editors, filters), to provide high-level input and output (I/O), to structure information into files, and to improve access to memory (size, protection, sharing).
The efficiency viewpoint:

to replace the (long departed) human operator, to schedule tasks, to efficiently store and retrieve data, and to invoke and share programs.
The economic viewpoint:

to allow simultaneous use and scheduling of resources, including disk-bound data and expensive peripherals.
Traditionally, we would summarize an operating system's goals as making "the system" convenient to use and scheduling its resources efficiently and fairly.
In addition, it must support hardware and software not yet developed.

Operating System ≠ User/Computer Interface

An operating system is often simply seen and described as the user/computer interface.
We often (mistakenly) claim to understand, and like or dislike, an "operating system" based on its interface.
Such an interface provides us with:
program creation (editors, compilers, debuggers, linkers)

program execution (character and graphical)
access to I/O devices (both fixed and removable)
constrained access to files of media
constrained access to "internal" resources
error detection, response, reporting, and
accounting and monitoring.
Whether or not a certain interface runs on a particular hardware or operating system platform is usually dictated by economics, marketing, and politics - not technology.
Operating System ≡ Resource Manager

An operating system is better considered as being in control of its hardware and software resources.
Better still, because the "controls" are often temporal or external to the operating system itself, let's consider the operating system as a resource manager.
An operating system is just another program running on the available hardware.
Most of the time, the operating system relinquishes control to the user processes until the hardware again dispatches the control.
Operating Systems Must Be Extensible

Of importance is an operating system's ability to evolve to meet new hardware and software demands:
New hardware is constantly introduced - adding more memory presents little difficulty; new types of disks, video cards, etc, are more problematic.
New application programs, tools, and system services are added.
Fixes and patches are released to correct operating system deficiencies.
All of this suggests that the operating system, as a program, needs to be extensible - a modular design seems essential. Consider the following figure, taken from Stallings.
Layers and Views of a Computer System
Of course, the above diagram provides a very simplified representation of an operating system. In practice, the relationships between the modular components become very complex - Linux and Windows-10.
Traditional Operating System Services

CPU scheduling:
distribute or apportion computing time among several processes (or tasks) which appear to execute simultaneously.
Memory management:
divide and share physical memory among several processes.
Swapping:
move processes and their data between main memory and disk to present the illusion of a bigger machine.
I/O device support:

provide specialized code to optimally support device requirements.
File system:
organize mass storage (on disk) into files and directories.
Traditional Operating System Services, continued

Utility programs:
accounting, setting/constraining system resource access, manipulating the file system.
A command interface:
textual or graphical, to enable interactive interrogation and manipulation of operating system features.
System calls:
allow constrained access to the interior of the running operating system (as a program).
Protection:
keep processes from interfering with each other, their data, and "the system", whilst permitting sharing when requested.
Communication:
allow users and processes to communicate within a single machine (inter-process communication), and across networks.
A Whirlwind History of Operating Systems

To understand the way modern operating systems are the way they are, it is useful to examine their evolution over the last sixty years or so.
Advances in operating systems often accompanied advances in hardware, falling prices, and "exploding" capacities.
The first true digital computer was designed by English mathematician Charles Babbage (1792-1871).
Although Babbage spent most of his working life and fortune building his "analytical engine", its mechanical design and the wooden technology of the day could not provide the required precision.
Needless to say, the analytical engine did not have an operating system.
“
Everything that can be invented has been invented.
— Charles H. Duell, Commissioner, U.S. Office of Patents, 1899.
1945-55: Vacuum Tubes and Plugboards

Until World War II, little progress was made in constructing digital computers. Six significant groups can reasonably claim the first electrical computers:
Tommy Flowers and Max Newman, Bletchley Park, England,

Howard Aitken, Harvard,
John von Neumann, Institute of Advance Studies, Princeton,
Tom Kilburn and Freddie Williams, Manchester,
J.P. Eckert and W. Mauchley, University of Pennsylvania, and
Konran Zuse, Germany.
A single group of people designed, built, programmed, operated and maintained each machine. Although filling small warehouses, with tens of thousands of vacuum tubes, they were no match for today's cheapest home computers.
“
I think there is a world market for maybe five computers.
— Thomas Watson (1874-1956), Chairman of IBM, 1943.
Programs were loaded manually using console switches, or more likely by direct reconfiguration of wiring; indication of a program's execution and debugging returned through console lights.
Advantages:
Interactive, and user received immediate response.
Disadvantages:
Expensive machine was idle most of the time, because people's reactions (and thinking) were slow.
Programming and debugging were tedious; hardware was very unreliable.
Each program was self contained, including its own code for mathematical functions and I/O device support.
1955-65: Transistors and Batch Systems

Programming languages and operating systems (as we know them today) still unheard of. Collections of subroutines (procedures) to drive peripherals and to evaluate trigonometric functions were the first examples of operating systems services. The mid-1950s saw the user
(still as the programmer) submitting a deck of punched Hollerith cards describing a job to be executed.
Given the high cost of computers, ways to increase their utility were quickly sought. The general solution was the batch system.
Similar/related programs, perhaps each requiring the FORTRAN (FORmula TRANslation) compiler, or a set of mathematical routines, were batched together so that the required routines need only be
physically loaded once.
Programs were first written on paper tape, in the emerging FORTRAN language or in assembly language, and then copied to punched cards. Such decks of cards included job control cards, the program itself, and often the program's data.
Jobs submitted by different users were sequenced automatically by the operating system's resident monitor. Early peripherals, such as large magnetic tape drives, were used to batch input (jobs and data) and spool (from Simultaneous Peripheral Operation OnLine) output.
1955-65: Transistors and Batch Systems, continued

Generally, an inexpensive computer, such as an IBM 1401, was used for reading cards and printing from output tapes. The expensive machine, such as an IBM 7094, was used solely for the mathematical computations.
Advantages:
The (true, computational) computer was kept busier.
Disadvantages:
The computer was no longer interactive. Jobs experienced a longer turnaround time.
The CPU was still idle much of the time for jobs. Other jobs remained queued for execution.
The significant operating system innovation at this time was the introduction of a command interpreter (a job control language - JCL) to describe, order, and commence execution of jobs.
The resident monitor was also protected from the user programs, and managed the automated loading of the programs after the monitor.
1965-1980: Integrated Circuits and Multiprogramming

In the early 1960s, computer manufacturers typically made two types of computers - word-oriented, large scale scientific computers (such as the IBM-7094), and character-oriented commercial computers (such as the IBM-1401), which were really better suited for I/O.
Incompatibility and a lack of an upgrade path were the problems of the day.
IBM attempted to address both problems with the release of their System/360, a family of software compatible machines differing only in capacity, price and performance. The machines had the same architecture and instruction set.
With heavy CPU-bound scientific calculations, I/O is infrequent, so the time spent (wasted) waiting was not significant. However, commercial processing programs in the emerging COBOL (Computer Oriented Business Organizational Language) often spent 80-90% of its time
waiting for I/O to complete.
The advent of separate I/O processors made simultaneous I/O and CPU execution possible.
The CPU was multiplexed (shared), or employed multiprogramming, amongst a number of jobs - while one job was waiting for I/O from comparatively slow I/O devices (such as a keyboard or tape), another job could use the CPU.
Jobs would run until their completion or until they made an I/O request.
Advantages:
Interactivity was restored.

The CPU was kept busy if enough jobs were ready to run.
Disadvantages:
The computer hardware and the operating system software became significantly more complex (and there has been no looking back since!).
1965-1980: Integrated Circuits and Multiprogramming, continued

Still, the desire for quicker response times inspired a variant of multiprogramming in which each user communicated directly with one of a multitude of I/O devices.
The introduction of timesharing introduced pre-emptive scheduling. Jobs would execute for at most pre-defined time interval, after which it was another job's turn to use the CPU.
The first serious timesharing system (CTSS, from MIT 1962) lacked adequate memory protection.
Most (modern) operating system complexity was first introduced with the support of multiprogramming - scheduling algorithms, deadlock prevention, memory protection, and memory management.
The world's first commercially available time-sharing computer, the DEC PDP-6, was installed in UWA's Physics Building in 1965 - cf.
Cyberhistory, by Keith Falloon, UWA MSc thesis, 2001, and pdp6-serials.
Early Operating System Security

Very early operating systems, on one-user-at-a-time computer systems, assisted the user to load their programs and commence their execution.
There was little to protect and, if an errant program modified the executive program, then only the current user was affected. As more of a courtesy, the executive might clear memory segments and check itself before accepting the
next program.
As multi-tasking operating systems emerged, accountability of resource use became necessary, and operating system monitors oversaw the execution of programs.
As with modern operating systems, there was the need to:
protect the operating system from the program,

protect programs from themselves,
protect programs from each other, and
constrain data access to the correct program(s).
Until system resources became more plentiful (and cheaper) attempts were made to maximize resource sharing - security was a consequent, not an initial, goal.
e.g. process scheduling policies were dominated by already running processes requesting resources (such as libraries and tapes) that were already in use.
At this level, computer security is more concerned with reliability and correctness, than about deliberate attacks on programs, data, and the system.
1970s: Minicomputers and Microcomputers
“
There are only two things to come out of Berkeley, Unix and LSD, and I
don't think this is a coincidence.
— Jeremy S. Anderson.
Another major development occurring in parallel was the phenomenal growth in minicomputers, starting with the DEC (Digital Equipment Corporation) PDP-1 (Programmed Data Processor) in 1961. The PDP-1, with 4K of 18-bit words cost only US$120,000 - 5% of the IBM
7094.
The trend was towards many small mid-range personal computers, rather than a single mainframe.
Early minicomputers and microcomputers were simple in their hardware architectures, and so there was some regression to earlier operating system ideas (single user, no pre-emption, no multiprogramming).
For example, MS-DOS on an IBM-PC (circa. 1975) was essentially introduced as a batch system, similar to those of the 1960s, with a few modern additions, such as a hierarchical file system.
With some notable exceptions, the trend quickly moved towards support of all modern operating system facilities on microcomputers.
“
There is no reason anyone would want a computer in their home.
— Ken Olsen, DEC Founder and Chairman, 1977.
Perhaps most significant has been the evolution, and importance, of operating systems' user interfaces.
In particular, the graphical desktop metaphor has remained for some time.
1980-90: Personal Computers and Networking
“
640K ought to be enough for anybody.
— Bill Gates (1955-), in 1981.
The decentralization of computing resources, now data and not the hardware, required more support for inter-operating system communication - both physical support and application program support.
As minicomputers shrunk in size, but exploded in capacity, the powerful computer workstation was born. Companies such as Sun Microsystems (SUN) and Silicon Graphics (SGI) rode this wave of success.
Local-area networks (primarily Ethernet and token-ring) connected workstations, while wide-area networks connected minicomputers.
Operating system developments included the development of fast and efficient network communication protocols, data encryption (of networks and file systems), security, reliability, and consistency of distributed data.
1990s and Beyond

Hardware prices drop dramatically, and capacities explode.
High speed, long distance communication links encourage graphical and audio communication (surpassing text).
The desktop metaphor for computer interfaces becomes only an instance of the wider web metaphor.
Computers become portable and increasingly ubiquitous.
Computers again become idle; artificial uses (such as complex screensavers) keep them busy.
“
For years, we thought that a million monkeys sitting at a million
keyboards would produce the complete works of Shakespeare. Today,
thanks to the Internet, we know that's not true.
— Anon.

Introducing functions
C is a procedural programming language, meaning that its primary synchronous control flow mechanism is the procedure call.
C names its procedures functions (in contrast, Java has a different mechanism - methods).
In Mathematics, we apply a function, such as the trigonometric function cos, to one or more values.
The function performs an evaluation, and returns a result.
In many programming languages, including C, we call or invoke a function.

We evaluate zero or more expressions, the result of each expression is copied to a memory location where the function can receive them as arguments, the function's statements are executed (often involving the arguments), and a result is returned (unless the function
is stuck in an infinite-loop or exits the process!)
We've already seen the example of main() - the function that all C programs must have, which we might write in different ways:
#include <stdio.h> #include <stdio.h>

#include <stdlib.h> #include <stdlib.h>
int main(int argcount, char *argvalue[]) int main(int argcount, char *argvalue[])
{ {
// check the number of arguments int result;
if(argcount != 2)
{ // check the number of arguments
.... if(argcount != 2)
exit(EXIT_FAILURE); {
} ....
else result = EXIT_FAILURE;
{ }
.... else
exit(EXIT_SUCCESS); {
} ....
return 0; result = EXIT_SUCCESS;
} }
return result;
}
The operating system calls main(), passing to it some (command-line) arguments, main() executes some statements, and returns to the operating system a result - usually EXIT_SUCCESS or EXIT_FAILURE.
Why do we require functions?

The need for, and use of, main() should be clear. However, there's 4 other primary motivations for using functions:
1. Functions allow us to group together statements that have a strongly related purpose - statements, in combination, performing a single task.
We prefer to keep such statements together, providing them with a name (as for variables), so that we may refer to the statements, and call them, collectively.
This provides both convenience and readability.
2. We often have sequences of statements that appear several times throughout larger programs.
The repeated sequences may be identical, or very similar (differing only in a very few statements). We group together these similar statement sequences, into a named function, so that we may call the function more than once and have it perform similarly for each call.
Historically, we'd identify and group similar statements into functions to minimize the total memory required to hold the (repeated) statements.
Today, we use functions not just to save memory, but to enhance the robustness and readability of our code (both good Software Engineering techniques).
3. From the Systems Programming perspective, the operating system kernel employs functions as well-defined entry points from user-written code into the kernel.
Such functions are named system calls.
4. Functions provide a convenient mechanism to package and distribute code. We can distribute code that may be called by other people's code, without providing them with a complete program.
We frequently use libraries for this purpose.

Where do we find functions?

1. We'll initially write our own functions, in the same file as main(), to simplify our code and to make it easier to read.
2. Soon, we'll write our own functions in other, multiple files, and call them from our main file.
3. Collections of related functions are termed libraries of functions.
The most prominent example, that we've already seen, is C's standard library - a collection of frequently required functions that must be provided by a standards' conforming C compiler.
In our programming, so far, we've already called library functions such as:
printf(), atoi(), and exit().
4. Similarly, there are many task-specific 3rd-party libraries. They are not required to come with your C compiler, but may be downloaded or purchased - from Lecture 1 -

The role of function main()

In general, small programs, even if just written in a single file, will have several functions.
We will no longer place all of our statements in the main() function.
main() should be constrained to:
receive and check the program's command-line arguments,
report errors detected with command-line arguments, and then call exit(EXIT_FAILURE),
call functions from main(), typically passing information requested and provided by the command-line arguments, and
finally call exit(EXIT_SUCCESS) if all went well.
And, in a forthcoming lecture, the following will make more sense:
All error messages printed to the stderr stream.
All 'normal' output printed to the stdout stream (if not to a requested file).
The datatype of a function

There are two distinct categories of functions in C:
1. functions whose role is to just perform a task, and to then return control to the statement (pedantically, the expression) that called it.
Such functions often have side-effects, such as performing some output, or modifying a global variable so that other statements may access that modified value.
These functions don't return a specific value to the caller, are termed void functions, and we casually say that they "return void".
2. functions whose role is to calculate a value, and to return that value for use in the expressions that called them. The single value returned will have a type, such as int, char, bool, or float.
These functions may also have side-effects.

#include <math.h>
void output(char ch, int n)
{ float square(float x)
for(int i=1 ; i<=n ; i=i+1) {
{ return x * x;
printf("%c", ch); }
}
} int main(int argcount, char *argvalue[])
{
int main(int argcount, char *argvalue[]) if(argcount > 2)
{ {
output(' ', 19); float a, b, sum;
output('*', 1);
output('\n', 1); a = atof(argvalue[1]);
b = atof(argvalue[2]);
return 0;
} sum = square(a) + square(b);
printf("hypotenuse = %f\n", sqrt(sum) );
}
return 0;
}
Note that this 2nd example uses the sqrt() function from the standard maths library.
We should thus compile it with: cc [EXTRAOPTIONS] -o program program.c -lm
Passing parameters to functions

The examples we've already seen show how parameters are passed to functions:
a sequence of expressions are separated by commas, as in:
a = average3( 12 * 45, 238, x - 981 );
each of these expressions has a datatype. In the above example, each of the expressions in an int.
when the function is called, the expressions are evaluated, and the value of each expression is assigned to the parameters of the function:
float average3( int x, int y, int z )

{
return (x + y + z) / 3.0;
}
during the execution of the function, the parameters are local variables of the function.
They have been initialized with the calling values (x = 12 * 45 ...), and the variables exist while the function is executing.
They "disappear" when the function returns.
Quite often, functions require no parameters to execute correctly. We declare such functions with:
void backup_files( void )

{
.....
}
and we just call the functions without any parameters: backup_files();

Very common mistakes with parameter passing

Some common misunderstandings about how parameters work often result in incorrect code
(even a number of textbooks make these mistakes!):
The order of evaluation of parameters is not defined in C. For example, in the code:
int square( int a )

{
printf("calculating the square of %i\n", a);
return a * a;
}
void sum( int x, int y )

{
printf("sum = %i\n", x + y );
}
....
....
sum( square(3), square(4) );
are we hoping the output to be:
calculating the square of 3 // the output on PowerPC Macs

calculating the square of 4
sum = 25
or
calculating the square of 4 // the output on Intel Macs

calculating the square of 3
sum = 25
Do not assume that function parameters are evaluated left-to-right. The compiler will probably choose the order of evaluation which produces the most efficient code, and this will vary on different processor architectures.
(A common mistake is to place auto-incrementing of variables in parameters.)

Very common mistakes with parameter passing, continued

Another common mistake is to assume that function arguments and parameters must have the same names to work correctly.
Some novice programmers think that the matching of names is how the arguments are evaluated, and how arguments are bound to parameters.
For example, consider the code:
int sum3( int a, int b, int c )

{
return a + b + c;
}
....
int a, b, c;
a = 1;
b = 4;
c = 9;
printf("%i\n", sum3(c, a, b) );
Here, the arguments are not "shuffled" until the names match.
It is not the case that arguments must have the same names as the parameters they are bound to.
Similarly, the names of variables passed as arguments are not used to "match" arguments to parameters.
If you ever get confused by this, remember that arithmetic expressions, such as 2*3 + 1, do not have names, and yet they are still valid arguments to functions.
Very common mistakes with parameter passing, continued

While not an example of an error, you will sometimes see code such as:
return x;
and sometimes:
return(x);
While both are correct, the parentheses in the 2nd example are unnecessary.
return is not a function call, it is a statement, and so does not need parentheses around the returned value.
However - at any point when writing code, use extra parentheses if they enhance the readability of your code.
Functions receiving a variable number of arguments

To conclude our introduction to functions and parameter passing,
we consider functions such as printf() which may receive a variable number of arguments!
We've carefully introduced the concepts that functions receive strongly typed parameters, that a fixed number of function arguments in the call are bound to the parameters, and that parameters are then considered as local variables.
But, consider the perfectly legal code:
#include <stdio.h>
int i = 238;
float x = 1.6;
printf("i is %i, x is %f\n", i, x);

....
printf("this function call only has a single argument\n");
....
printf("x is %f, i is %i, and x is still %f\n", x, i, x);
In these cases, the first argument is always a string, but the number and datatype of the provided arguments keeps changing.
printf() is one of a small set of standard functions that permits this apparent inconsistency. It should be clear that the format specifiers of the first argument direct the expected type and number of the following arguments.
Fortunately, within the ISO-C99 specification, our cc compiler is permitted to check our format strings, and warn us (at compile time) if the specifiers and arguments don't "match".
prompt> cc -o try try.c

try.c:9:20: warning: format specifies type 'int' but the argument has type 'char *'
[-Wformat]
printf("%i\n", "hello");
~~ ^~~~~~~
%s
1 warning generated.

An Overview of Computer Hardware

Any study of operating systems requires a basic understanding of the components of a computer system.
Although the variety of computer system configurations is forever changing, as (new) component types employ different standards for their interconnection, it is still feasible to discuss a simple computer model, and to discuss components' roles in operating systems.
Traditionally, we consider four main structural components:
The Central Processing Unit, or CPU, undertakes arithmetic and logical computation, and directs most input and output services from memory and peripherals. There may be multiple processors in a system, each
executing the (same, single) operating system or user/application programs.
Main Memory, or RAM (Random Access Memory) is used to store both instructions and data. Processors read and write items of memory both at the direction of programs (for data), and as an artifact of running programs
(for instructions).
Secondary Storage and Peripheral Devices, (or input/output modules) and their I/O controllers, move data to and from the other components usually to provide longer-term, persistent storage of data (disks, tapes),
A communications bus, or system bus, connects the processor(s), main memory, and I/O devices together, providing a "highway" on which data may travel between the components. Typically only one component may
control the bus at once, and bus arbitration decides which that will be.
Excellent, albeit expensive, computer organisation texts

Computer Organization and Design, Fifth Edition: The Hardware/Software Interface, by David A. Patterson and John L. Hennessy. Computer Organization: Basic Processor Structure, James Gil de Lamadrid, Chapman and Hall/CRC,
Morgan Kaufmann Publishers, 5th edition, October 2013. Published February 23, 2018, 372pp, ISBN 9781498799515.
Basic Computer Components

Many OS textbooks (often in their 1st or 2nd chapters) outline a traditional computer model, in which the CPU, main memory, and I/O devices are all interconnected by a single system bus (figures are taken from Stallings' website).
Instruction and data fetching

The CPU fetches a copy of the contents of uniquely-addressed memory locations, by identifying the required location in its MAR (Memory Address Register).
Depending on why the CPU requested the memory's value, it executes the contents as an instruction, or operates on the contents as data.
Similarly, the CPU locates data from, or for, distinct locations in the I/O devices using its I/O-AR (Address Register).
Role of operating systems

The role of the OS in managing the flow of data to and from its CPU and I/O devices, made very challenging by the wide variety of devices.
The OS attempts to attain maximum throughput of its computation and data transfer.
Processor scheduling attempts to keep the (expensive) processor busy at all times, by interleaving computation and communication.
While for a slow device to complete its I/O transfer, the CPU may be able to undertake other activities, such as performing some computation or managing faster I/O.
The Range of I/O Device Data Rates
See also Wikipedia's List of interface bit rates.

Processor Registers
As well as special logic to perform arithmetic and logic functions, the processor houses a small number of very fast memory locations, termed processor registers.
Data in registers can be read and written very rapidly (with a typical access time of 0.5-3ns). If the required data is available in registers, rather than main memory, program execution may proceed 10-500X faster.
Different types of processors have varying number of registers, For example, some processors have very few (3-16), some have many (32-100s).
The number of general-purpose CPU registers, and the width of each register (measured in bits, e.g. 64-bit registers), contribute to the power and speed of a CPU.
Processors place constraints on how some registers are used. Some processors expect certain types of data to reside in specific registers. For example, some registers may be expected to hold integer operands for integer arithmetic instructions, whereas some
registers may be reserved for holding floating-point data.
The Role of Processor Registers

All data to be processed by the CPU must first be copied into registers - the CPU cannot, for example, add together two integers residing in RAM.
Data must first be copied into registers; the operation (e.g. addition) is then performed on the registers and the result left in a register, and that result (possibly) copied back to RAM.
Registers are also often used to hold a memory address, and the register's contents used to indicate which item from RAM to fetch.
The transfer of data to and from registers is completely transparent to users (even to programmers).
Generally, we only employ assembly language programs to manipulate registers directly. In compiled high-level languages, such as C, the compiler translates high-level operations into low-level operations that access registers.
Register types
Registers are generally of two types:
User-accessible registers -
are accessible to programs, and may usually be read from and written to under program control. Programs written in an assembly language, either by a human programmer or generated by a compiler for a high-level language, are able to read and write these registers
using specific instructions understood by the processor which usually accept the names of the registers as operands.
The user-accessible registers are further of two types:
Data registers hold values before the execution of certain instructions, and hold the results after executing certain instructions.
Address registers hold the addresses (not contents) of memory locations used in the execution of a program, e.g.
the memory address register (MAR) holds the address of memory to be read or written;
the memory buffer register (MBR) holds the memory's data just read, or just about to be written;
index registers hold an integer offset from which of memory references are made; and
a stack pointer (SP) holds the address of a dedicated portion of memory holding temporary data and other memory addresses.
Control and status registers -

hold data that the processor itself maintains in order to execute programs, e.g. the instruction register(IR) holds the current instruction being executed, and the program counter (PC) holds the memory address of the next instruction to be executed.
Special registers reflect the status of the processor. The processor status word (PSW) reflects whether or not the processor may be interrupted by I/O devices and whether privileged instructions may be executed, and it uses condition bits to reflect the status of recently
executed operations.
In order evaluate results, and to determine if branching should occur, the PSW may record -
whether an arithmetic operation overflowed,

whether an arithmetic operation performed a carry,
whether a division by zero was attempted,
whether the last comparison instruction succeeded or failed.
The Memory Hierarchy

The role of memory is to hold instructions and data until they are requested by the processor (or, some devices). While it is easy to make a case for as much memory as possible, having too much can be wasteful (financially) if it is not all required.
We also expect memory to be able to provide the necessary data, as quickly as possible, when called upon. Unfortunately, there is a traditional trade-off between cost, capacity, and access time:
the faster the access time, the greater the cost per bit,
the greater the capacity, the smaller the cost per bit and, the greater the capacity, the slower the access time.
The Memory Hierarchy, continued

The solution taken is not to rely on a single, consistent form of memory, but instead to have a memory hierarchy, constrained by requirements and cost.
Memory Access-time Capacity Technology Managed by

Registers 0.5-3ns 1-4KB custom CMOS compiler
Level-1 cache (on-chip) 2-8ns 8KB-128KB SRAM hardware
Level-2 cache (on-chip) 5-12ns 0.5MB-8MB SRAM hardware
Main memory (RAM) 10-60ns 64MB-64GB DRAM operating system
hard disk 3-10M ns 200MB-16000GB magnetic operating system
solid-state disk (SSD) 0.5-1M ns 16GB-8000GB DRAM/SRAM operating system
For example, a contemporary laptop or home computer system may include:
a modest amount of cache memory (1MB) to deliver data as quickly as possible to the processor,
a larger main memory (8GB) to store entire programs and less-frequently required data, and
long term, persistent storage in the form of a hard disk (1TB), or SSD (256GB).
Units of data: bits, bytes, and words

The basic building block is the bit (binary digit), which can contain a single piece of binary data (true/false, zero/one, high/low, etc.).
Although processors provide instructions to set and compare single bits, it is rarely the most efficient method of manipulating data.
Bits are organised into larger groupings to store values encoded in binary bits. The most basic grouping is the byte: the smallest normally addressable quantum of main memory (which can be different from the minimum amount of memory fetched at one time).
In modern computers, a byte is almost always an 8-bit byte, but history has seen computers with 7-, 8-, 9-, 12-, and 16-bit bytes.
A word is the default data size for a processor. The word size is chosen by the processor's designer and reflects some basic hardware issues (such as the width of internal or external buses).
The most common word sizes are 16 and 32 bits; historically words have ranged from 16 to 60 bits.
It is very common to speak of a processor's wordsize, such as a 32-bit or 64-bit processor.

However, different sources will confuse whether this means the size of a single addressable memory location, or the default unit of integer arithmetic.
Some processors require that data be aligned, that is, 2-byte quantities must start on byte addresses that are multiples of two; 4-byte quantities must start on byte addresses that are multiples of four; etc.
Some processors allow data to be unaligned, but this can result in slower execution as the processor may have to align the data itself.
On the interpretation of data

We have seen that computer systems store their data as bits, and group bits together as bytes and words.
However, it is important to realise that the processor can interpret a sequence of bits only in context: on its own, a sequence of bits means nothing.
A single 32-bit pattern could refer to:
4 ASCII characters,
a 32-bit integer,
2 x 16-bit integers,
1 floating point value,
the address of a memory location, or
an instruction to be executed.
No meaning is stored along with each bit pattern: it is up to the processor to apply some context to the sequence to ascribe it some meaning.
For example, a sequence of integers may form a sequence of valid processor instructions that could be meaningfully executed; a sequence of processor instructions can always be interpreted as a vector of, say, integers and can thus be added together.
Critical errors occur when a bit sequence is interpreted in the wrong context. If a processor attempts to execute a meaningless sequence of instructions, a processor fault will generally result: Linux announces this as a "bus error". Similar faults occur when instructions expect
data on aligned data boundaries, but are presented with unaligned addresses.
On the interpretation of data, continued

As an example of how bytes may be interpreted in different ways, consider the first few hundred bytes of the disk file /bin/ls. We know this to be a program, and we expect the operating system to interpret its contents to be a program, and request the processor to execute its
contents (a mixture of instructions and data).
However, another program could read the bytes from /bin/ls and interpret them in other ways, e.g. as 32-bit integers:
prompt> od -i /bin/ls
0000000 1179403647 65793 0 0
0000020 196610 1 134518416 52
0000040 66628 0 2097204 2621447
0000060 1638426 6 52 134512692
0000100 134512692 224 224 5
0000120 4 3 276 134512916
0000140 134512916 19 19 4
0000160 1 1 0 134512640
.......
or as octal (8-bit) bytes:
prompt> od -b /bin/ls
0000000 177 105 114 106 001 001 001 000 000 000 000 000 000 000 000 000
0000020 002 000 003 000 001 000 000 000 220 226 004 010 064 000 000 000
0000040 104 004 001 000 000 000 000 000 064 000 040 000 007 000 050 000
0000060 032 000 031 000 006 000 000 000 064 000 000 000 064 200 004 010
0000100 064 200 004 010 340 000 000 000 340 000 000 000 005 000 000 000
0000120 004 000 000 000 003 000 000 000 024 001 000 000 024 201 004 010
0000140 024 201 004 010 023 000 000 000 023 000 000 000 004 000 000 000
0000160 001 000 000 000 001 000 000 000 000 000 000 000 000 200 004 010
.......
or as ASCII characters:
prompt> od -c /bin/ls
0000000 177 E L F 001 001 001 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000020 002 \0 003 \0 001 \0 \0 \0 220 226 004 \b 4 \0 \0 \0
0000040 D 004 001 \0 \0 \0 \0 \0 4 \0 \0 \a \0 ( \0
0000060 032 \0 031 \0 006 \0 \0 \0 4 \0 \0 \0 4 200 004 \b
0000100 4 200 004 \b 340 \0 \0 \0 340 \0 \0 \0 005 \0 \0 \0
0000120 004 \0 \0 \0 003 \0 \0 \0 024 001 \0 \0 024 201 004 \b
0000140 024 201 004 \b 023 \0 \0 \0 023 \0 \0 \0 004 \0 \0 \0
0000160 001 \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 \0 200 004 \b
.......
And each interpretation could be correct, depending on context.

Introducing arrays
As programs become more complex, we notice that they require more variables, and thus more variable names to hold all necessary values.
We could define:
int x1, x2, x3, x4, x5..... ;
but referring to them in our programs will quickly become unwieldy, and their actual names may be trying to tell us something.
In particular, our variables are often related to one another - they hold data having a physical significance, and the data value held in one variable is related to the data in another variable.
For example, consider a 2-dimensional field, where each square metre of the field may be identified by its rows and column coordinates. We may record each square's altitude, or temperature, or its number of ants:
(0,0) (1,0) (2,0) (3,0)
(0,1) (1,1) (2,1) (3,1)
(0,2) (1,2) (2,2) (3,2)
Like most languages, C provides a simple data structure, termed an array, to store and access data where the data items themselves are closely related.
Depending on the context, different problem domains will describe different kinds of arrays with different names:
1-dimensional arrays are often termed vectors,

2-dimensional arrays are often termed matrices (as in our example, above),
3-dimensional arrays are often termed volumes, and so on.
We'll start with the simple 1-dimensional arrays.

1-dimensional arrays
C provides support for 1-dimensional arrays by allowing us to identify the required data using a single index into the array.
Syntactically, we use square-brackets to identify that the variable is an array, and use an integer expression inside the array to identify which "part" of it we're requiring.
In all cases, an array is only a single variable, with one or more elements.
Consider the following code: What do we learn from this example?
We declare our 1-dimensional arrays with square brackets, and indicate the maximum number of elements within those brackets.
#define N 20
A fixed, known value (here N, with the value 20) is used to specify the number of elements of the array.
int myarray[ N ]; We access elements of the array by providing the array's name, and an integer index into the array.
int evensum;
Elements of an array may be used in the same contexts as basic (scalar) variables. Here myarray is used on both the left-hand and right-hand sides of assignment statements.
evensum = 0;
for(int i=0 ; i < N ; ++i) We may also pass array elements as arguments to functions, and return their values from functions.
{
myarray[ i ] = i * 2; Array indicies start "counting" from 0 (not from 1).
evensum = evensum + myarray[ i ];
} Because our array consists of N integers, and indicies begin at zero, the highest valid index is actually N-1.
Initializing 1-dimensional arrays

Like all variables, arrays should be initialized before we try to access their elements. We can:
initialize the elements at run-time, by executing statements to assign values to the elements:
#define N 5
int myarray[ N ];
....
for(int i=0 ; i < N ; ++i) {

myarray[ i ] = i;
}
we may initialize the values at compile-time, by telling the compiler what values to initially store in the memory represented by the array. We use curly-brackets (braces) to provide the initial values:
#define N 5
int myarray[ N ] = { 0, 1, 2, 3, 4 };
we may initialize the values at compile-time, by telling the compiler what values to initially store in the memory represented by the array, and having the compiler determine the number of elements in the array(!).
int myarray[ ] = { 0, 1, 2, 3, 4 };
#define N (sizeof(myarray) / sizeof(myarray[0]))
or, we may initialize just the first few values at compile-time, and have the compiler initialize the rest with zeroes:
#define HUGE 10000
int myarray[ HUGE ] = { 4, 5 };
Variable-length arrays
All of the examples of 1-dimensional arrays we've seen have the array's size defined at compile time:
#define N 5
int global_array[ N ];
....
for(int i=0 ; i < N ; ++i) {

int array_in_block[ 100 ];
....
}
As the compiler knows the exact amount of memory required, it may generate more efficient code (in both space and time) and more secure code.
More generally, an array's size may not be known until run-time. These arrays are termed variable-length arrays, or variable-sized arrays. However, once defined, their size cannot be changed.
In all cases, variable-length arrays may be defined in a function and passed to another. However, because the size is not known until run-time, the array's size must be passed as well. It is not possible to determine an array's size from its name.
void function2(int array_size, char vla[ ])

{
for(int i=0 ; i < array_size ; ++i) {
// access vla[i] ...
....
}
}
void function1(void)
{
int size = read an integer from keyboard or a file;
char vla[ size ];
function2(size, vla);
}
Variable-length arrays were first defined in the C99 standard, but then made optional in C11 - primarily because of their inefficient implementation on embedded devices. Modern Linux operating system kernels are now free of variable-length arrays.
Strings are 1-dimensional arrays of characters

In contrast to some other programming languages, C does not have a basic datatype for strings.
However, C compilers provide some basic support for strings by considering strings to simply be arrays of characters.
We've already seen this support when calling the printf() function:
printf("I'm afraid I can't do that Dave\n");
The double quotation characters simply envelope the characters to be treated as a sequence of characters.
In addition, a standards' conforming C compiler is required to also provide a large number of string-handling functions in its standard C library. Examples include:
#include <string.h>
// which declares many functions, including:
int strlen( char string[] ); // to determine the length of a string
int strcmp( char str1[], char str2[] ); // to determine if two strings are equal
char *strcpy( char destination[], char source[] ); // to make a copy of a string
In reality these functions are not "really" managing strings as a basic datatype, but are just managing arrays of characters.
Initializing character arrays

As we've just seen with 1-dimensional arrays of integers, C also provides facility to initialize character arrays.
All of the following examples are valid:
char greeting[5] = { 'h', 'e', 'l', 'l', 'o' };
char today[6] = "Monday";
char month[] = "August";
The 3rd of these is the most interesting.

We have not specified the size of the array month ourselves, but have permitted the compiler to count and allocate the required size.
Strings are terminated by a special character

Unlike other arrays in C, the support for character arrays is extended by treating one character, the null byte, as having special significance.
We may specify the null byte, as in the example:
array[3] = '\0';
The null byte is used to indicate the end of a character sequence, and it exists at the end of all strings that are defined within double-quotes.
Inside the computer's memory we have: Of note, when dealing with strings:
the string requires 6 bytes of memory to be stored correctly, but

h e l l o \0 functions such as strlen(), which calculate the string's length, will report it as 5.
There is no inconsistency here - just something to watch out for.
Because the null byte has special significance, and because we may think of strings and character arrays as the same thing, we can manipulate the contents of strings by changing the array elements. Consider:
h e l l o w o r l d \0
If we execute the statement:
array[5] = '\0';
the space between the two words is replaced by the null byte.
The result is that the array still occupies 12 bytes of storage, but if we tried to print it out, we would only get hello.
Copying strings
As strings are so important, the standard C library provides many functions to examine and manipulate strings.
However, C provides no basic string datatype, so we often need to treat strings as array of characters.
Consider these implementations of functions to copy one string into another:
// DETERMINE THE STRING LENGTH, THEN USE A BOUNDED LOOP // DO NOT WRITE STRING-PROCESSING LOOPS THIS WAY
void my_strcpy(char destination[], char source[]) void my_strcpy(char destination[], char source[])
{ {
int length = strlen(source); int i;
for(int i = 0 ; i < length ; ++i) for(i = 0 ; i < strlen(source) ; ++i)

{ {
destination[i] = source[i]; destination[i] = source[i];
} }
destination[length] = '\0'; destination[i] = '\0';
} }
// USE AN UNBOUNDED LOOP, COPYING UNTIL THE NULL-BYTE // USE AN UNBOUNDED LOOP, COPYING UNTIL THE NULL-BYTE
void my_strcpy(char destination[], char source[]) void my_strcpy(char destination[], char source[])
{ {
int i = 0; int i = 0;
while(source[i] != '\0') do
{ {
destination[i] = source[i]; destination[i] = source[i];
i = i+1; i = i+1;
} } while(source[i-1] != '\0');
destination[i] = '\0'; }
}
Formatting our results into character arrays

There are many occasions when we wish our "output" to be written to a character array, rather than to the screen.
Fortunately, we need to learn very little - we now call standard function sprintf, rather than printf, to perform our formatting.
#include <stdio.h>
char chess_outcome[64];
if(winner == WHITE)
{
sprintf(chess_outcome, "WHITE with %i", nwhite_pieces);
}
else
{
sprintf(chess_outcome, "BLACK with %i", nblack_pieces);
}
printf("The winner: %s\n", chess_outcome);
We must be careful, now, not to exceed the maximum length of the array receiving the formatted printing.
Thus, we prefer functions which ensure that not too many characters are copied:
char chess_outcome[64];
// FORMAT, AT MOST, A KNOWN NUMBER OF CHARACTERS

if(winner == WHITE)
{
snprintf(chess_outcome, 64, "WHITE with %i", nwhite_pieces);
}
// OR, GREATLY PREFERRED:

if(winner == WHITE)
{
snprintf(chess_outcome, sizeof(chess_outcome), "WHITE with %i", nwhite_pieces);
}
Pre- and post-, increment and decrement

These last two slides should have appeared in Lecture-3.
To date, we've always employed the "traditional" mechanism of incrementing integer values, in both assignment statements and in for loops:
int value = 0;
....
value = value + 1;
....
for(int i=0 ; i < MAXVALUE ; ++i)
{
....
}
C provides a shorthand notation for incrementing and decrementing scalar variables, by one:
int value = 0;
char ch = 'z';
++value; // value is now 1

--ch; // ch is now 'y'
....
for(int i=0 ; i < MAXVALUE ; ++i)
{
....
}
for(char letter='a' ; letter <= 'z' ; ++letter)

{
....
}
The notation used above is always used to increment or decrement by one, and the 2 statements:
++value ; // pre-increment value

value++ ; // post-increment value
produce the exact same result.

Pre- and post-, increment and decrement, continued

While pre- and post- incrementing (and decrementing) initially appears simple, we must be careful when using modified variables in expressions.
Consider these results (all statements executed top-to-bottom):
int x = 0;
int y = 0;
int what = 0;
// ------------------- what --- X --- Y -
what = ++x; // 1 1 0
what = y++; // 0 1 1
what = y++; // 1 1 2
what = ++y; // 3 1 3
Shorthand arithmetic
A similar notation may be used to perform any standard arithmetic operation on a variable. For example, assuming the correct declarations:
value += 2; // equivalent to value = value + 2;

value -= y; // equivalent to value = value - y;
total *= x; // equivalent to total = total * x;
half /= 2; // equivalent to half = half / 2;
poly += x*1; // equivalent to poly = poly + (x*1);

Processes
The fundamental activity of an operating system is the creation, management, and termination of processes.
What is a process? Naively:
a program under execution,

the "animated" existence of a program,
an identifiable entity executed on a processor by the operating system.
More particularly, we consider how the operating system itself views a process:
as an executable instance of a program,

as the associated data operated upon by the program (variables, temporary results, external (file) storage, ...), and
as the program's execution context.
It is a clear requirement of modern operating systems that they enable many processes to execute efficiently, by maximising their use of the processor, by supporting inter-process communication, and by maintaining reasonable response time.
This is an ongoing challenge: as hardware improves, it is "consumed" by larger, "hungrier" pieces of interlinked software.
Process States
We can view the process from two points of view: that of the processor, and that of the process itself.
The processor's view is simple: the processor's only role is to execute machine instructions from main memory. Over time, the processor continually executes the sequence of instructions indicated by the program counter (PC).
The processor is unaware that different sequences of instructions have a logical existence, that different sequences under execution are referred to as processes or tasks.
From the process's point of view, it is either being executed by the processor, or it is waiting to be executed (for simplicity here, we consider that a terminated process no longer exists).
We've identified two possible process states that a process may be in at any one time: Running and Ready.
Question: Can a process determine in what state it is?

Process Transitions
The operating system's role is to manage the execution of existing and newly created processes by moving them between the two states until they finish.
So we have a simple model consisting of two recurring steps:
1. Newly created processes are created and marked as Ready, and are queued to run.
2. There is only ever a single process in the Running state. It will either:
complete its execution and terminate (exit), or
be suspended (by itself or by the operating system), be marked as Ready, and be again queued to run.
One of the other Ready processes is then commenced (or resumed).
Here the operating system has the role of a dispatcher - dispatching work for the processor according to some defined policy addressing a combination of fairness, priority, apparent "interactivity", ...
___________
For simplicity (of both understanding and implementation) modern operating systems support the idle process which is always ready to run, and never terminates.
The Simple 2-state Process Model

As we generally have more than two processes available, the Ready state is implemented as a queue of available processes:
When scheduling is discussed, we will introduce process priorities when deciding which Ready process should be the next to execute.
Process Creation
In supporting the creation of a new process, the operating system must allocate resources both for the process and the operating system itself.
The process (program under execution) will require a portion of the available memory to contain its (typically, read-only) instructions and initial data requirements. As the process executes, it will demand additional memory for its execution stack and its heap.
The operating system (as dispatcher) will need to maintain some internal control structures to support the migration of the process between states.
Where do new processes come from?
an "under-burdened" operating system may take new process requests from a batch queue.
a user logging on at a terminal usually creates an interactive control or encapsulating process (shell or command interpreter).
an existing process may request a new process,
the operating system itself may create a process after an indirect request for service (to support networking, printing, ...)
Different operating systems support process creation in different ways.
by requesting that an existing process be duplicated (ala fork() call in macOS and Linux),
by instantiating a process's image from a named location, typically the program's image from a disk file (ala the spawn() call in (old)DEC-VMS and the CreateProcess() call in Windows).
Process Termination
Stallings summarises typical reasons why a process will terminate:
normal termination,
execution time-limit exceeded,
a resource requested is unavailable,
an arithmetic error (division by zero),
a memory access violation,
an invalid request of memory or a held resource,
an operating system or parent process request, or
its parent process has terminated.
These and many other events may either terminate the process, or simply return an error indication to the running process. In all cases, the operating system will provide a default action which may or may not be process termination.
It is clear that process termination may be requested (or occur) when a process is either Running or Ready. The operating system (dispatcher) must handle both cases.
If a process is considered as a (mathematical) function, its return result, considered as a Boolean or integral result, is generally made available to (some) other processes.
Timer Interrupts
Why does a process move from Running to Ready?
The operating system must meet the two goals of fairness amongst processes and maximal use of resources (here, the processor and, soon, memory).
The first is easily met: enable each process to execute for a predetermined period before moving the Running process to the Ready queue.
A hardware timer will periodically generate an interrupt (say, every 10 milliseconds). Between the execution of any two instructions, the processor will "look for" interrupts. When the timer interrupt occurs, the processor will begin execution of the interrupt handler.
The handler will increment and examine the accumulated time of the currently executing process, and eventually move it from Running to Ready.
The maximum time a process is permitted to run is often termed the time quantum.
The Blocking of Processes

The above scenario is simple and fair if all Ready processes are always truly ready to execute.
However, the existence of processes which continually execute to the end of their time quanta, often termed compute-bound processes, is rare.
More generally, a process will request some input or output (I/O) from a comparatively slow source (such as a disk drive, tape, keyboard, mouse, or clock). Even if the "reply" to the request is available immediately, a synchronous check of this will often exceed the remainder of
the process's time quantum. In general the process will have to wait for the request to be satisfied.
The process should no longer be Running, but it is not Ready either: at least not until its I/O request can be satisfied.
We now introduce a new state for the operating system to support, Blocked, to describe processes waiting for I/O requests to be satisfied. A process requesting I/O will, of course, request the operating system to undertake the I/O, but the operating system supports this as three
activities:
1. requesting I/O to or from the device,

2. moving the process from Running to Blocked,
3. preparing to accept an interrupt when I/O completes.
(Very simply) when the I/O completion interrupt occurs, the requesting process is moved from Blocked to Ready.
A degenerate case of blocking occurs when a process simply wishes to sleep for a given period. We consider such a request as "blocking until a timer interrupt", and have the operating system handle it the same way.
The 5-State Model of Process Execution

At this point, Stallings introduces his 5-state model:
This includes two new states:
New for newly created processes which haven't yet been admitted to the Ready queue for resourcing reasons;
Exit for terminated processes whose return result or resources may be required by other processes, e.g. for post-process accounting.
Each of these states, except for Running, is likely to 'hold' more than one process (i.e. there may be more than one process in one of these states).
If a queue of processes is not always ordered in a first-come-first-served (FCFS) manner, then a priority or scheduling mechanism is necessary.
Supporting Multiple Blocked States

When notification of an I/O or timer completion occurs, the simplest queuing model requires the operating system to scan its Blocked queue to determine which process(es) requested, or are interested in, the event:
Supporting Multiple Blocked States, continued

A better scheme is to maintain a queue for each possible event type. When an event occurs, its (shorter) queue is scanned more quickly:
A typical example would have one queue for processes blocked on disk-drive-1, another blocked on disk-drive-2, another blocked on the keyboard ......
Then, when an interrupt occurs, it's quick to determine which process(es) should be unblocked, and moved to Ready.
The Dispatching Role of Operating Systems

As should now be clear, this view of the operating system as a dispatcher involves the moving of processes from one execution state to another.
The process's state is reflected by where it resides, although its state will also record much other information.
The possible state transitions that we have now discussed are:
Null → New a new process is requested.
New → Ready resources are allocated for the new process.
Ready → Running a process is given a time quantum.
Running → Ready a process's execution time quantum expires.
Running → Blocked a process requests slow I/O.
Blocked → Ready an I/O interrupt signals that I/O is ready.
Running → Exit normal or abnormal process termination.
Ready or Blocked → Exit external process termination requested.
Suspension of Processes
Recall that the processor is much faster than I/O. As a consequence, it is quite possible for all processes to be blocked on I/O requests, when the processor will be idle most of the time while waiting for I/O interrupts.
Question: How to get more executing processes, given that resources such as memory are finite?
To enable more true work to be performed by the processor, we could provide more memory to support the requirements of more processes.
But aside from the expense, providing more memory tends to encourage larger processes, not (in general) better support for more processes.
Swapping of Processes
Another solution is swapping: moving (part of) a process's memory to disk when it is not needed.
When none of the processes in main memory is Ready, the operating system swaps the memory of some of the Blocked processes to disk to recover some memory space. Such processes are moved to a new state: the Suspend state, a queue of processes that have been
"kicked out" of main memory:
If desperate for even more memory, the operating system can similarly reclaim memory from Ready processes.
When memory becomes available, the operating system may now resume execution of a process from Suspend, or admit a process from New to Ready.
Raw input and output

We've recently seen how C99 employs arrays of characters to represent strings, treating the NULL-byte with special significance.
At the lowest level, an operating system will only communicate using bytes, not with higher-level integers or floating-point values. C99 employs arrays of characters to hold the bytes in requests for raw input and output.
File descriptors - reading from a file

Unix-based operating systems provide file descriptors, simple integer values, to identify 'communication channels' - such as files, interprocess-communication pipes, (some) devices, and network connections (sockets).
In combination, our C99 programs will use integer file descriptors and arrays of characters to request that the operating system performs input and output on behalf of the process - see man 2 open.
#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#define MYSIZE 10000
void read_using_descriptor(char filename[])

{
// ATTEMPT TO OPEN THE FILE FOR READ-ONLY ACCESS
int fd = open(filename, O_RDONLY);
// CHECK TO SEE IF FILE COULD BE OPENED

if(fd == -1) {
printf("cannot open '%s'\n", filename);
}
// DEFINE A CHARACTER ARRAY TO HOLD THE FILE'S CONTENTS

char buffer[MYSIZE];
size_t got;
// PERFORM MULTIPLE READs OF FILE UNTIL END-OF-FILE REACHED

while((got = read(fd, buffer, sizeof buffer)) > 0) {
.....
}
// INDICATE THAT THE PROCESS WILL NO LONGER ACCESS FILE

close(fd);
}
Note that the functions open, read, and close are not C99 functions but operating system system-calls, providing the interface between our user-level program and the operating system's implementation.
File descriptors - writing to a file

Similarly, we use integer file descriptors and arrays of characters to write data to a file. We require a different file descriptor for each file - the descriptor identifies the file to use and the operating system (internally) remembers the requested (permitted) form of access.
Copying a file using file descriptors
#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#define MYSIZE 10000
int copy_file(char destination[], char source[])

{
// ATTEMPT TO OPEN source FOR READ-ONLY ACCESS
int fd0 = open(source, O_RDONLY);
// ENSURE THE FILE COULD BE OPENED
if(fd0 == -1) {
return -1;
}
// ATTEMPT TO OPEN destination FOR WRITE-ONLY ACCESS

int fd1 = open(destination, O_WRONLY);
// ENSURE THE FILE COULD BE OPENED
if(fd1 == -1) {
close(fd0);
return -1;
}
// DEFINE A CHARACTER ARRAY TO HOLD THE FILE'S CONTENTS

char buffer[MYSIZE];
size_t got;
// PERFORM MULTIPLE READs OF FILE UNTIL END-OF-FILE REACHED

while((got = read(fd0, buffer, sizeof buffer)) > 0) {
if(write(fd1, buffer, got)) != got) {
close(fd0); close(fd1);
return -1;
}
}
close(fd0); close(fd1);
return 0;
}
Reading and writing text files

We'll next focus on reading and writing from human-readable text files. C99 provides additional support above the operating systems's system-calls to provide more efficient buffering of I/O operations, and treating text files as a sequence of lines (as strings).
We open a text file using C's fopen() function.

To this function we pass the name of the file we wish to open (as a character array), and describe how we wish to open, and later access, the file.
The returned value is a FILE pointer, that we use in all subsequent operations with that file.
#include <stdio.h>
#define DICTIONARY "/usr/share/dict/words"
....
// ATTEMPT TO OPEN THE FILE FOR READ-ACCESS
FILE *dict = fopen(DICTIONARY, "r");
// CHECK IF ANYTHING WENT WRONG

if(dict == NULL) {
printf( "cannot open dictionary '%s'\n", DICTIONARY);
}
// READ AND PROCESS THE CONTENTS OF THE FILE

....
// WHEN WE'RE FINISHED, CLOSE THE FILE

fclose(dict);
If fopen() returns the special value NULL, it indicates that the file may not exist, or that the operating system is not giving us permission to access it as requested.
Declaring how the file will be used

In different applications, we may need to open the file in different ways.
We pass different strings as the second parameter to fopen() to declare how we'll use the file:
"r" open for reading

"r+" open for reading and writing
"w" create or truncate file, then open for writing
"w+" create or truncate file, then open for reading and writing
"a" create if necessary, then open for appending (at the end of the file)
"a+" create if necessary, then open for reading and appending
All future operations to read and write an open file are checked against the initial file access mode. Future operations will fail if they do not match the initial declaration.
NOTE - File access mode flag "b" can optionally be specified to open a file in binary mode (described later). This flag has effect only on Windows systems, and is ignored on macOS and Linux. This flag has no effect when reading and writing text files.
Reading a text file, one line at a time

Having opened the file (for read access), we now wish to read it in - one line at a time.
We generally don't need to store each line of text; we just check or use it as we traverse the file:
The text data in the file is (unsurprisingly) also stored as a sequence of characters. We can use C's character arrays to store each line as we read it:
Of note in this code:
#include <stdio.h> we pass to fgets() our character array, into which each line will be read.
.... we indicate the maximum size of our array, so that fgets() doesn't try to read in too much.
FILE *dict;
char line[BUFSIZ]; We use the sizeof operator to indicate the maximum size of our character array.
Using sizeof here, rather than just repeating the value BUFSIZ, means that we can change the size of line at any time, by only changing the size at its definition.
dict = fopen( ..... );
.... We pass our file pointer, dict, to our file-based functions to indicate which file to read or write.
It's possible to have many files open at once.
// READ EACH LINE FROM THE FILE,
// CHECKING FOR END-OF-FILE OR AN ERROR
The fgets() functions returns the constant NULL when it "fails". When reading files, this indicates that the end-of-file has been reached, or some error detected (e.g. USB key removed!).
while( fgets(line, sizeof line, dict) != NULL )
{
....
Assuming that we've reached end-of-file, and that we only need to read the file once, we close the file pointer (and just assume that the closing has succeeded).
.... // process this line
....
}
// AT END-OF-FILE (OR AN ERROR), CLOSE THE FILE
fclose(dict);
What did we just read from the file?

Our call to the fgets() function will have read in all characters on each line of the dictionary but, if we're interested in processing the characters as simple strings, we find that we've got "too much".
Each line read will actually have:
a a r d v a r k \n \0
The character '\n', the familiar newline character often used in print(), is silently added to text lines by our text editors.
In fact on Windows' machines, text files also include a carriage-return character before the newline character.
W i n d o w s \r \n \0
As we know, we can simply turn this into a more manageable string by replacing the newline or carriage-return character by the null-byte.
Trimming end-of-line characters from a line

To make future examples easy to read, we'll write a function, named trim_line(), that receives a line (a character array) as a parameter, and "removes" the first carriage-return or newline character that it finds.
It's very similar to functions like my_strlen() that we've written in laboratory work:
// REMOVE ANY TRAILING end-of-line CHARACTERS FROM THE LINE

void trim_line(char line[])
{
int i = 0;
// LOOP UNTIL WE REACH THE END OF line

while(line[i] != '\0') {
// CHECK FOR CARRIAGE-RETURN OR NEWLINE

if( line[i] == '\r' || line[i] == '\n' ) {
line[i] = '\0'; // overwrite with null-byte
break; // leave the loop early
}
i = i+1; // iterate through character array
}
}
We note:
we simply overwrite the unwanted character with the null-byte.

the function will actually modify the caller's copy of the variable.
we do not return any value.
Writing text output to a file

We've used fgets() to 'get' a line of text (a string) from a file;
we similarly use fputs() to 'put' (write) a line of text.
The file pointer passed to fputs() must previously have been opened for writing or appending.
Copying a text file using file pointers

We now have all the functions necessary to copy one text file to another, one line line at a time:
#include <stdio.h>
#include <stdlib.h>
void copy_text_file(char destination[], char source[])

{
FILE *fp_in = fopen(source, "r");
FILE *fp_out = fopen(destination, "w");
// ENSURE THAT OPENING BOTH FILES HAS BEEN SUCCESSFUL

if(fp_in != NULL && fp_out != NULL) {
char line[BUFSIZ];
while( fgets(line, sizeof line, fp_in) != NULL) {

if(fputs(line, fp_out) == EOF) {
printf("error copying file\n");
}
}
}
// ENSURE THAT WE ONLY CLOSE FILES THAT ARE OPEN
if(fp_in != NULL)
fclose(fp_in);
if(fp_out != NULL)
fclose(fp_out);
}
More text file reading - the game of Scrabble

Let's quickly consider another example employing reading a text file.
In the game of Scrabble, each letter tile has a certain value - the rarer a letter is in the English language, the higher the value of that letter ('E' is common and has the value
1, 'Q' is rare and has the value 10).
Can we write C functions to determine:
the value of a word in Scrabble?

the word in a dictionary with the highest value?
the best word to choose from a set of tiles?
In writing these functions we won't consider all of the rules of Scrabble.
Another consideration (which we'll ignore) is that there are fixed tile frequencies - for example, there are
12 'E's but only 1 'Q'.
Refer to Wikipedia for the actual distributions.
The value of a word in Scrabble

To answer our Scrabble questions, we'll develop two simple helper functions:
Letter Value
#include <stdbool.h> A 1
#include <string.h>
B 3
#include <ctype.h>
C 3
// ENSURE THAT A WORD CONTAINS ONLY LOWERCASE CHARACTERS D 2
bool valid_word(char word[])
{ E 1
int i = 0; F 4
// IF NOT A LOWERCASE CHARACTER, THE FUNCTION RETURNS false G 2
while(word[i] != '\0') { H 4
if( ! islower( word[i] )) { // if not islower ...
return false; I 1
} J 8
i = i+1;
} K 5
// WE'VE REACHED THE END OF THE WORD - IT'S ALL LOWERCASE L 1
return true;
} M 3
N 1
// CALCULATE THE SCRABBLE VALUE OF ANY WORD
int calc_value(char word[])
O 1
{ P 3
// AN ARRAY TO PROVIDE THE VALUE OF EACH LETTER, FROM 'a' TO 'z'
int values[] = { 1, 3, 3, 2, 1, 4, 2, 4, 1, 8, 5, 1, 3, 1,
Q 10
1, 3, 10, 1, 1, 1, 1, 4, 4, 8, 4, 10 }; R 1
int total = 0;
S 1
int i = 0;
T 1
// TRAVERSE THE WORD DETERMINING THE VALUE OF EACH LETTER
U 1
while(word[i] != '\0') {
total = total + values[ word[i] - 'a' ]; V 4
i = i+1; W 4
}
return total; X 8
} Y 4
Z 10
The word with the highest value

Can we find which valid word from a dictionary has the highest value?
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#define DICTIONARY "/usr/share/dict/words"

#define LONGEST_WORD 100
// FIND THE WORD WITH THE BEST VALUE

void findbest( char filename[] )
{
FILE *fp = fopen(filename, "r");
// ENSURE THAT WE CAN OPEN (WITH READ-ACCESS) THE FILE

if(fp != NULL)
{
char bestword[LONGEST_WORD];
int bestvalue = 0;
char thisword[LONGEST_WORD];
int thisvalue = 0;
// READ EACH LINE OF THE FILE

while( fgets(thisword, sizeof thisword, fp) != NULL ) {
// REPLACE THE NEWLINE CHARACTER WITH A NULL-BYTE
trim_line( thisword );
// ENSURE THAT THIS WORD IS VALID (previously defined)

if( valid_word(thisword) ) {
thisvalue = calc_value( thisword );
// IS THIS WORD BETTER THAN THE PREVIOUSLY BEST?

if(bestvalue < thisvalue) {
bestvalue = thisvalue; // save current details
strcpy(bestword, thisword);
}
}
}
fclose(fp);
printf("best word is %s = %i\n", bestword, bestvalue);
}
}
int main(int argc, char *argv[])

{
findbest( DICTIONARY );
return 0;
}
Reading and writing files of binary data

To date, our use of files has dealt exclusively with lines of text, using fgets() and fputs() to perform our I/O.
This has provided a good introduction to file input/outpout (I/O) as textual data is easy to "see", and printing it to the screen helps us to verify our functions:
the standard fgets function manages the differing lengths of input lines by reading until the '\n' or '\r' character is found,
fgets terminates input lines by appending a null-byte to them, 'turning' them into C strings, and
the null-byte is significant when later managing (copying, printing, ...) strings.
Reading and writing files of binary data, continued

However, when managing files of arbitrary data, possibly including null-bytes as well, we must use different functions to handle binary data:
#include <stdio.h>
#include <stdlib.h>
void copyfile(char destination[], char source[])

{
FILE *fp_in = fopen(source, "rb");
FILE *fp_out = fopen(destination, "wb");
// ENSURE THAT OPENING BOTH FILES HAS BEEN SUCCESSFUL

if(fp_in != NULL && fp_out != NULL) {
char buffer[BUFSIZ];
size_t got, wrote;
while( (got = fread(buffer, 1, sizeof buffer, fp_in)) > 0) {

wrote = fwrite(buffer, 1, got, fp_out);
if(wrote != got) {
printf("error copying files\n");
}
}
// ENSURE THAT WE ONLY CLOSE FILES THAT ARE OPEN

if(fp_in != NULL)
fclose(fp_in);
if(fp_out != NULL)
fclose(fp_out);
}
NOTE - The access mode flag "b" has been used in both calls to fopen() as we're anticipating opening binary files. This flag has effect only on Windows systems, and is ignored on macOS and Linux. This flag has no effect when reading and writing text files.
While we might request that fread() fetches a known number of bytes, fread() might not provide them all!
we might be reading the last "part" of a file, or

the data may be arriving (slowly) over a network connection, or
the operating system may be too busy to provide them all right now.
Reading and writing binary data structures

The fread() function reads an indicated number of elements, each of which is the same size:
size_t fread(void *ptr, size_t eachsize, size_t nelem, FILE *stream);
This mechanism enables our programs to read arbitrary sized data, by setting eachsize to one (a single byte), or to read a known number of data items each of the same size:
#include <stdio.h>
int intarray[ N_ELEMENTS ];

int got, wrote;
// OPEN THE BINARY FILE FOR READING AND WRITING

FILE *fp = fopen(filename, "rb+");
....
got = fread( intarray, sizeof int, N_ELEMENTS, fp);

printf("just read in %i ints\n", got);
// MODIFY THE BINARY DATA IN THE ARRAY

....
// REWIND THE FILE TO ITS BEGINNING

rewind(fp);
// AND NOW OVER-WRITE THE BEGINNING DATA

wrote = fwrite( intarray, sizeof int, N_ELEMENTS, fp);
....
fclose(fp);
Reading and writing binary data structures, continued

When reading and writing arbitrary binary data, there is an important consideration - different hardware architectures (the computer's CPU and data circuitry) store and manipulate their data in different formats.
The result is that when binary data written via one architecture (such as an Intel Pentium) is read back on a different architecture (such as an IBM PowerPC), the "result" will be different!
Writing on a 32-bit Intel Pentium: Reading on a 32-bit PowerPC:
#define N 10 #define N 10
int array[N]; int array[N];
for(int n=0 ; n < N ; ++n) fread(array, N, sizeof int, fp_in);

{
array[n] = n; for(int n=0 ; n < N ; ++n)
} {
printf("%i ", array[n]);
fwrite(array, N, sizeof int, fp_out); }
printf("\n");
Prints the output:

0 16777216 33554432 50331648 67108864 83886080 100663296 117440512 134217728 150994944
The problems of reading and writing of binary data, to and from different architectures and across networks, are discussed and solved in later units, such as CITS3002 Networks and Security.
Jonathan Swift's Gulliver's Travels, published in 1726, provided the earliest literary reference to computers, in which a machine would write books. This early attempt at artificial intelligence was characteristically marked by its inventor's call for public funding and the employment of student operators. Gulliver's diagram of the machine actually contained errors, these being either an attempt to protect his
invention or the first computer hardware glitch.
The term endian is used because of an analogy with the story Gulliver's Travels, in which Swift imagined a never-ending fight between the kingdoms of the Big-Endians and the Little-Endians (whether you were Lilliputian or Brobdignagian), whose only difference is in where they crack open a hard-boiled egg.

Creating a new process using fork()

fork() is very unusual because it returns different values in the (existing) parent process, and the (new) child process:
the value returned by fork() in the parent process will be the process-indentifier, of process-ID, of the child process;
the value returned by fork() in the child process will be 0, indicating that it is the child, because 0 is not a valid process-ID.
Each successful invocation of fork() returns a new monotonically increasing process-ID (the kernel 'wraps' the value back to the first unused positive value when it reaches 100,000).
#include <stdio.h>
#include <unistd.h>
void function(void)
{
int pid; // some systems define a pid_t
switch (pid = fork()) {

case -1 :
printf("fork() failed\n"); // process creation failed
break;
case 0: // new child process

printf("c: value of pid=%i\n", pid);
printf("c: child's pid=%i\n", getpid());
printf("c: child's parent pid=%i\n", getppid());
break;
default: // original parent process

sleep(1);
printf("p: value of pid=%i\n", pid);
printf("p: parent's pid=%i\n", getpid());
printf("p: parent's parent pid=%i\n", getppid());
break;
}
fflush(stdout);
}
produces:
c: child's value of pid=0

c: child's pid=5642
c: child's parent pid=5641
p: parent's value of pid=5642
p: parent's pid=5641
p: parent's parent pid=3244
Of note, calling sleep(1) may help to separate the outputs, and we fflush() in each process to force its output to appear.
Where does the first process come from?

The last internal action of booting a Unix-based operating system results in the first single 'true' process, named init.
init has the process-ID of 1. It is the ancestor process of all subsequent processes.
In addition, because the operating system strives to maintain a hierarchical relationship amongst all processes, a process whose parent terminates is 'adopted' by the init process.
The general calling sequence of system calls

If a single program has two distinct execution paths/sequences, then the parent and child may run different parts of the same program. Typically the parent will want to know when the child terminates.
The typical sequence of events is:
the parent process fork()s a new child process.
the parent waits for the child's termination, calling the blocking function wait( &status ).
[optionally] the child process replaces details of its program (code) and data (variables) by calling the execve() function.
the child calls exit(value), with an integer value to represent its success or failure. By convention, zero (= EXIT_SUCCESS) indicates successful execution, non-zero otherwise.
the child's value given to exit() is written by the operating system to the parent's status.
Waiting for a Process to Terminate

The parent process typically lets the child process execute, but wants to know when the child has terminated, and whether the child terminated successfully or otherwise.
A parent process calls the wait() system call to suspend its own execution, and to wait for any of its child processes to terminate.
The (new?) syntax &status permits the wait() system call (in the operating system kernel) to modify the calling function's variable. In this way, the parent process is able to receive information about how the child process terminated.
#include <stdio.h>
#include <unistd.h>
#include <sys/wait.h>
void function(void)
{
switch ( fork() ) {
case -1 :
printf("fork() failed\n"); // process creation failed
break;
case 0: // new child process

printf("child is pid=%i\n", getpid() );
for(int t=0 ; t<3 ; ++t) {

printf(" tick\n");
sleep(1);
}
break;
default: { // original parent process

int child, status;
printf("parent waiting\n");
child = wait( &status );
printf("process pid=%i terminated with exit status=%i\n",

child, WEXITSTATUS(status) );
break;
}
}
}
Memory in Parent and Child Processes

The (existing) parent process and the (new) child process continue their own execution.
Importantly, both the parent and child have their own copy of their program's memory (variables, stack, heap).
The parent naturally uses the memory that it had before it called fork(); the child receives its own copy of the same memory. The copy is made at the time of the fork().
As execution proceeds, each process may update its own memory without affecting the other process.
[ OK, I lied - on contemporary operating systems, the child process does not receive a full copy of its parent's memory at the time of the fork():
the child can share any read-only memory with its parent, as neither process can modify it.
the child's memory is only copied from the parent's memory if either the parent modies its (original) copy, or if the child attempts to write to its copy (that it hasn't yet received!)
this sequence is termed copy-on-write.
]
Running a New Program

Of course, we do not expect a single program to meet all our computing requirements, or for both parent and child to conveniently execute different paths through the same code, and so we need the ability to commence the execution of new programs after a fork().
Under Unix/Linux, a new program may replace the currently running program. The new program runs as the same process (it has the same pid, confusing!), by overwriting the current process's memory (instructions and data) with the instructions and data of the new program.
The single system call execv() requests the execution of a new program as the current process:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
char *program_arguments[] = {
"ls",
"-l",
"-F",
NULL
};
....
execv( "/bin/ls", program_arguments );
// A SUCCESSFUL CALL TO exec() DOES NOT RETURN
exit(EXIT_FAILURE); // IF WE GET HERE, THEN exec() HAS FAILED
On success, execv() does not return (to where would it return?)

On error, -1 is returned, and errno is set appropriately (EACCES, ENOENT, ENOEXEC, ENOMEM, ....).
The single system call is supported by a number of library functions (see man execl) which simplify the calling sequence.
Typically, the call to execv() (via one of its library interfaces) will be made in a child process, while the parent process continues its execution, and eventually waits for the child to terminate.
Why the exit status of a program is important

To date, we've always used exit(EXIT_FAILURE) when a problem has been detected, or exit(EXIT_SUCCESS) when all has gone well. Why?
The operating system is able to use the exit status of a program to determine if it was successful.
Consider the following program which exits with the integer status provided as a command-line argument:
#include <stdio.h>
#include <stdlib.h>

{
int status = EXIT_SUCCESS; // DEFAULT STATUS IS SUCCESS (=0)
if(argc > 1) {
status = atoi(argv[1]);
}
printf("exiting(%i)\n", status);
exit(status);
}
Why the exit status of a program is important, continued

Most operating system shells are, themselves, programming languages, and they may use a program's exit status to direct control-flow within the shells - thus, the programming language that is the shell, is treating your programs as if they are external functions.
Shells are typically programmed using files of commands named shellscripts or command files and these will often have conditional constructs, such as if and while, just like C. It's thus important for our programs to work with the shells that invoke them.
We now compile our program, and invoke it with combinations of zero, and non-zero arguments:
prompt> mycc -o status status.c
prompt> ./status 0 && ./status 1

exiting(0)
exiting(1)
prompt> ./status 1 && ./status 0

exiting(1)
prompt> ./status 0 || ./status 1

exiting(0)
prompt> ./status 1 || ./status 0

exiting(1)
exiting(0)
Example1 - consider the sequence prompt> cd mydirectory && rm -f *
Example2 - consider the actions in a Makefile (discussed in a later lecture).

If a target has more than one action, then the make program executes each until one of them fails (or until all succeed).
Operating System Services

All operating systems provide service points through which a general application program may request services of the operating system kernel. These points are variously termed system calls, system traps, or syscalls.
It is considered desirable to minimise the number of system calls that an operating system provides, in the belief that doing so simplifies design and improves correctness. Unix 6th Edition (circa 1975) had 52 system calls, whereas the modern macOS system boasts 521 (see
/usr/include/sys/syscall.h).
We've recently seen an "explosion" in the number of system calls provided, as systems attempt to support legacy and current 32-bit system calls, while introducing new 64-bit and multi-processor calls.
Does the complexity of an OS's implementation matter? Linux, Windows (both about 2011).
Some material in this lecture is from two historic texts, both available in the Science Library:
Marc J. Rochkind,
Advanced Unix Programming, Prentice-Hall, 1985.
W. Richard Stevens,
Advanced Programming in the Unix Environment, Addison-Wesley, 1992.
CITS2002 Systems Programming, Lecture 11, p1, 2nd September 2019.
Interfaces to C
The system call interfaces of modern operating systems are presented as an API of C-language prototypes, regardless of the programmer's choice of application language (C++, Java, Visual-Basic). This is a clear improvement over earlier interfaces in assembly languages.
The technique used in most modern operating systems is to provide an identically-named interface function in a standard C library or system's library (for example /usr/lib/system/libsystem_kernel.dylib on macOS and /lib/libc.so.6 on Linux).
An application program, written in any programming language, may invoke the system calls provided that the language's run-time mechanisms support the operating system's standard calling convention.
In the case of a programming language employing a different calling convention, or requiring strong controls over programs (e.g. running them in a sandbox environment, as does Java), direct access to system calls may be limited.
As the context switch between application process and the kernel is relatively expensive, most error checking of arguments is performed within the library, avoiding a call of the kernel with incorrect parameters:
#include <syscall.h>
#include <unistd.h>
.....
int write(int fd, void *buf, size_t len)

{
if (any_errors_in_arguments) {
errno = EINVAL;
return (-1);
}
return syscall(SYS_write, fd, buf, len);
}
But also, system calls need to be ''paranoid'' to protect the kernel from memory access violations! They will check their arguments, too.
Status Values Returned from System Calls

To provide a consistent interface between application processes and the operating system kernel, a minimal return-value interface is supported by a language's run-time library.
The kernel will use a consistent mechanism, such as using a processor register or the top of the run-time stack, to return a status indicator to a process. As this mechanism is usually of a fixed size, such as 32 bits, the value returned is nearly always an integer, occasionally a
pointer (an integral value interpreted as a memory address).
For this reason, globally accessible values such as errno, convey additional state, and values 'returned' via larger structures are passed to the kernel by reference (cf. getrusage() - discussed later).
The status interface employed by Unix/Linux and its C interface involves the globally accessible integer variable errno. From /usr/include/sys/errno.h:
#define EPERM 1 /* Operation not permitted */

#define ENOENT 2 /* No such file or directory */
#define ESRCH 3 /* No such process */
#define EINTR 4 /* Interrupted system call */
#define EIO 5 /* I/O error */
#define ENXIO 6 /* No such device or address */
#define E2BIG 7 /* Arg list too long */
#define ENOEXEC 8 /* Exec format error */
#define EBADF 9 /* Bad file number */
#define ECHILD 10 /* No child processes */
(Most) system calls consistently return an integer value:
with a value of zero on success, or

with a non-zero value on failure, and further description of the error is provided by errno.
Obvious exceptions are those system calls needing to return many possible correct values - such as open() and read(). Here we often see -1 as the return value indicating failure.
Using errno and perror()

On success, system calls return with a value of zero; on failure, their return value will often be -1, with further characterisation of the error appearing in the integer variable errno.
ISO-C99 standard library functions employ the same practice.
As a convenience (not strictly part of the kernel interface), the array of strings sys_errlist[] may be indexed by errno to provide a better diagnostic:
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
...
if(chdir("/Users/someone") != 0)
{
printf("cannot change directory, why: %s\n", sys_errlist[errno]);
}
...
or, alternatively, we may call the standard function perror() to provide consistent error reporting:
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
...

{
...
if (chdir("/Users/someone") != 0)
{
perror(argv[0]);
}
...
Note that a successful system call or function call will not set the value of errno to zero. The value will be unchanged.
Library Interface to System Calls

System calls accept a small, bounded number of arguments; the single syscall entry point loads the system call's number, and puts all arguments into a fixed location, typically in registers, or on the argument stack.
Ideally, all system call parameters are of the same length, such a 32-bit integers and 32-bit addresses.
It is very uncommon for an operating system to use floating point values, or accept them as arguments to system calls.
Depending on the architecture, the syscall() entry point will eventually invoke a TRAP or INT machine instruction - an 'illegal' instruction, or software interrupt, causing the hardware to jump to code which examines the required system call number and retrieves its arguments.
Such code is often written in assembly language (see <sys/syscall.h>):
#define SYSCALL3(x) \
.globl NAME(x) ; \
NAME(x): \
push %ebx; \
mov 8(%esp), %ebx; \
mov 12(%esp), %ecx; \
mov 16(%esp), %edx; \
lea SYS_##x, %eax; \
int $0x80; \
pop %ebx; \
ret; \
END(x)
There is a clear separation of duties between system calls and their calling functions. For example, the memory allocation function malloc() calls sbrk() to extend a process's memory space by increasing the process's heap. malloc() and free() later manage this space.
The Execution Environment of a Process

Although C programs appear to begin at main() or its equivalent on some embedded platforms), standard libraries must first prepare the process's execution environment.
An additional function, linked at a known address, is often provided by the standard run-time libraries to initialise that environment.
For example, the C run-time library provides functions (such as) _init() to initialise (among other things) buffer space for the buffered standard I/O functions. (For example, /usr/include/sys/syslimits.h limits a process's arguments and environment to 256KB).
The execution environment of a process
In particular, command-line arguments and environment variables are located at the beginning of each process's stack, and addresses to these are passed to main() and assigned to the global variable environ.
Environment variables
As with command-line arguments, each process is invoked with a vector of environment variables (NULL-terminated character strings):
The environment variables of a process
These are typically maintained by application programs, such as a command-interpreter (or shell), with calls to standard library functions such as putenv() and getenv().
#include <stdio.h>
#include <stdlib.h>
...
// A POINTER TO A VECTOR OF POINTERS TO CHARACTERS - OUCH, LATER!

// (LET'S CALL IT AN ARRAY OF STRINGS, FOR NOW)
extern char **environ;

{
putenv("ANIMAL=budgie");
for(int i=0 ; environ[i] != NULL ; ++i)

{
printf("%s\n", environ[i]);
}
return 0;
}
A process's environment (along with many other attributes) is inherited by its child processes.
Interestingly, the user's environment variables are never used by the kernel itself.
The runtime library and environment variables

However, a programming language's run-time library may use environment variables to vary its default actions.
For example, the C library function execlp() may be called to commence execution of a new program:
Steps in the invocation of a process
execlp() - receives the name of the new program, and the arguments to provide to the program, however it does not know how to find the program.
execlp() - locates the value of the environment variable PATH, assuming it to be a colon-separated list of directory names to search,
e.g. PATH="/bin:/usr/bin:.:/usr/local/bin", and appends the program's name to each directory component.
execlp() - makes successive calls to the system call execve() until one of them succeeds in beginning execution of the required program.
Initializing and exiting a process

Similarly, a process is quickly terminated by the system call exit(), but the library function exit() is usually called to flush buffered I/O, and call any functions requested via on_exit() and atexit().
We can consider _init() to include:
int _init(int argc, char *argv[], char **envp)

{
// ... set up the library's run-time state ...
exit( main( argc, argv, environ = envp ) );

}
Functions called to commence and terminate a process
This shows how main() may either call exit(), call return, or simply 'fall past its bottom curly bracket'.
What is cc really doing - the condensed version

We understand how cc works in its simplest form:
we invoke cc on a single C source file,

we know the C-processor is invoked to include system-wide header files, and to define our own preprocessor and definitions macros,
the output of the preprocessor becomes the input of the "true" compiler,
the output of the compiler (for correct programs!) is an executable program (and we may use the -o option to provide a specific executable name).
What is cc really doing - the long version

Not surprisingly, there's much more going on!
cc is really a front-end program to a number of passes or phases of the whole activity of "converting" our C source files to executable programs:
1. foreach C source file we're compiling:
i. the C source code is given to the C preprocessor,

ii. the C preprocessor's output is given to the C parser,
iii. the parser's output is given to a code generator,
iv. the code generator's output is given to a code optimizer,
v. the code optimizer's output, termed object code, is written to a disk file termed an object file,
2. all necessary object files (there may be more than one, and some may be standard C libraries, operating system-specific, or provided by a third-party), are presented to a program named the linker, to be "combined" together, and
3. the linker's output is written to disk as an executable file.

CITS2002 Systems Programming, Lecture 12, p1, 6th September 2019.
What is cc really doing - in a picture

Additional details:
cc determines which compilation phases to perform based on the command-line options and the file name extensions provided.
The compiler passes object files (with the filename suffix .o) and any unrecognized file names to the linker.
The linker then determines whether files are object files or library files (often with the filename suffix .a).
The linker combines all required symbols (e.g. your main() function from your .o file and the printf() function from C's standard library) to form the single executable program file.
Developing larger C programs in multiple files

Just as C programs should be divided into a number of functions (we often say the program is modularized), larger C programs should be divided into multiple source files.
The motivations for using multiple source files are:
each file (often containing multiple related functions) may perform (roughly) a single role,
the number of unnecessary global variables can be significantly reduced,
we may easily edit the multiple files in separate windows,
large projects may be undertaken by multiple people each working on a subset of the files,
each file may be separately compiled into a distinct object file,
small changes to one source file do not require all other source files to be recompiled.
All object files are then linked to form a single executable program.
A simple multi-file program

For this lecture we'll develop a simple project to calculate the correlation of some student marks, partitioned into multiple files. The input data file contans two columns of marks - from a project marked out of 40, and an exam marked out of 60.
calcmarks.h - contains globally visible declarations of types, functions, and variables
calcmarks.c - contains main(), checks arguments, calls functions
globals.c - defines global variables required by all files
readmarks.c - performs all datafile reading
correlation.c - performs calculations
Each C file depends on a common header file, which we will name calcmarks.h.
Providing declarations in header files

We employ the shared header file, calcmarks.h, to declare the program's:
C preprocessor constants and macros,

globally visible functions (may be called from other files), and
globally visible variables (may be accessed/modified from all files).
The header file is used to announce their existence using the extern keyword.
The header file does not actually provide function implementations (code) or allocate any memory space for the variables.
#include <stdio.h>
#include <math.h>
// DECLARE GLOBAL PREPROCESSOR CONSTANTS

#define MAXMARKS 200
// DECLARE GLOBAL FUNCTIONS

extern int readmarks(FILE *); // parameter is not named
extern void correlation(int); // parameter is not named
// DECLARE GLOBAL VARIABLES

extern double projmarks[]; // array size is not provided
extern double exammarks[]; // array size is not provided
extern bool verbose; // declarations do not provide initializations
Notice that, although we have indicated that function readmarks() accepts one FILE * parameter, we have not needed to give it a name.
Similarly, we have declared the existence of arrays, but have not indicated/provided their sizes.
Providing our variable definitions

In the C file globals.c we finally define the global variables.
It is here that the compiler allocates memory space for them.
In particular, we now define the size of the projmarks and exammarks arrays, in a manner dependent on the preprocessor constants from calcmarks.h
This allows us to provide all configuration information in one (or more) header files. Other people modifying your programs, in years to come, will know to look in the header file(s) to adjust the constraints of your program.
#include "calcmarks.h" // we use double-quotes
double projmarks[ MAXMARKS ]; // array's size is defined

double exammarks[ MAXMARKS ]; // array's size is defined
bool verbose = false; // global is initialized
Global variables are automatically 'cleared'

By default, global variables are initialized by filling them with zero-byte patterns.
This is convenient (of course, it's by design) because the zero-byte pattern sets the variables (scalars and arrays) to:
0 (for ints),
'\0' (for chars),
0.0 (for floats and doubles),
false (for bools), and
zeroes (for pointers).
Note that we could have omitted the initialisation of verbose to false, but providing an explicit initialisation is much clearer.
The main() function

All of our C source files now include our local header file. Remembering that file inclusion simply "pulls in" the textual content of the file, our C files are now provided with the declarations of all global functions and global variables.
Thus, our code may now call global functions, and access global variables, without (again) declaring their existence:
#include "calcmarks.h" // local header file provides declarations

{
int nmarks = 0;
// IF WE RECEIVED NO COMMAND-LINE ARGUMENTS, READ THE MARKS FROM stdin

if(argc == 1)
{
nmarks += readmarks(stdin);
}
// OTHERWISE WE ASSUME THAT EACH COMMAND-LINE ARGUMENT IS A FILE NAME
else
{
for(int a=1 ; a<argc ; ++a)
{
FILE *fp = fopen(argv[a], "r");
if(fp == NULL)
{
printf("Cannot open %s\n", argv[a]);
}
nmarks += readmarks(fp);
// CLOSE THE FILE THAT WE OPENED
fclose(fp);
}
}
// IF WE RECEIVED SOME MARKS, REPORT THEIR CORRELATION
if(nmarks > 0)
{
correlation(nmarks);
}
return 0;
}
In the above function, we have used to a local variable, nmarks, to maintain a value (both receiving it from function calls, and passing it to other functions).
nmarks could have been another global variable but, generally, we strive to minimize the number of globals.
Reading the marks from a file

Nothing remarkable in this file:
int readmarks(FILE *fp)

{
char line[BUFSIZ];
int nmarks = 0;
double thisproj;
double thisexam;
....
// READ A LINE FROM THE FILE, CHECKING FOR END-OF-FILE OR AN ERROR
while( fgets(line, sizeof line, fp) != NULL )
{
// WE'RE ASSUMING THAT WE LINE PROVIDES TWO MARKS

.... // get 2 marks from this line
projmarks[ nmarks ] = thisproj; // update global array

exammarks[ nmarks ] = thisexam;
++nmarks;
if(verbose) // access global variable

{
printf("read student %i\n", nmarks);
}
}
return nmarks;
}
Calculate the correlation coefficient (the least exciting part)
void correlation(int nmarks)

{
// MANY LOCAL VARIABLES REQUIRED TO CALCULATE THE CORRELATION
double sumx = 0.0;
double sumy = 0.0;
double sumxx = 0.0;
double sumyy = 0.0;
double sumxy = 0.0;
double ssxx, ssyy, ssxy;

double r, m, b;
// ITERATE OVER EACH MARK

for(int n=0 ; n < nmarks ; ++n)
{
sumx += projmarks[n];
sumy += exammarks[n];
sumxx += (projmarks[n] * projmarks[n]);
sumyy += (exammarks[n] * exammarks[n]);
sumxy += (projmarks[n] * exammarks[n]);
}
ssxx = sumxx - (sumx*sumx) / nmarks;

ssyy = sumyy - (sumy*sumy) / nmarks;
ssxy = sumxy - (sumx*sumy) / nmarks;
// CALCULATE THE CORRELATION COEFFICIENT, IF POSSIBLE

if((ssxx * ssyy) == 0.0)
{
r = 1.0;
}
else
{
r = ssxy / sqrt(ssxx * ssyy);
}
printf("correlation is %.4f\n", r);
// DETERMINE THE LINE OF BEST FIT, IT ONE EXISTS

if(ssxx != 0.0)
{
m = ssxy / ssxx;
b = (sumy / nmarks) - (m*(sumx / nmarks));
printf("line of best fit is y = %.4fx + %.4f\n", m, b);
}
}
Maintaining multi-file projects

As large projects grow to involve many, tens, even hundreds, of source files, it becomes a burden to remember which ones have been recently changed and, hence, need recompiling.
This is particularly difficult to manage if multiple people are contributing to the same project, each editing different files.
As an easy way out, we could (expensively) just compile everything!
cc -std=c99 -Wall -pedantic -Werror -o calcmarks calcmarks.c globals.c readmarks.c correlation.c
Introducing make
The program make maintains up-to-date versions of programs that result from a sequence of actions on a set of files.
make reads specifications from a file typically named Makefile or makefile and performs the actions associated with rules if indicated files are "out of date".
Basically, in pseudo-code (not in C) :
if (files on which a certain file depends)

i) do not exist, or
ii) are not up-to-date
then
create an up-to-date version;
make operates over rules and actions recursively and will abort its execution if it cannot create an up-to-date file on which another file depends.
Note that make can be used for many tasks other than just compiling C - such as compiling other code from programming languages, reformatting text and web documents, making backup copies of files that have recently changed, etc.
Dependencies between files

From our pseudo-code:
if (files on which a certain file depends)

i) do not exist, or
ii) are not up-to-date
then
create an up-to-date version;
we are particularly interested in the dependencies between various files - certain files depend on others and, if one changes, it triggers the "rebuilding" of others:
The executable program prog is dependent on one or more object files (source1.o and source2.o).
Each object file is (typically) dependent on one C source file (suffix .c) and, often, on one or more header files (suffix .h).
So:
If a header file or a C source file are modified (edited),

then an object file needs rebuilding (by cc).
If one or more object files are rebuilt or modified (by cc),

then the executable program need rebuilding (by cc).
NOTE that the source code files (suffix .c) are not dependent on the header files (suffix .h).
A simple Makefile for our program

For the case of our multi-file program, calcmarks, we can develop a very verbose Makefile which fully describes the actions required to compile and link our project files.
# A Makefile to build our 'calcmarks' project
calcmarks : calcmarks.o globals.o readmarks.o correlation.o

—— tab —→cc -std=c99 -Wall -pedantic -Werror -o calcmarks \
calcmarks.o globals.o readmarks.o correlation.o -lm
calcmarks.o : calcmarks.c calcmarks.h

—— tab —→cc -std=c99 -Wall -pedantic -Werror -c calcmarks.c
globals.o : globals.c calcmarks.h

—— tab —→cc -std=c99 -Wall -pedantic -Werror -c globals.c
readmarks.o : readmarks.c calcmarks.h

—— tab —→cc -std=c99 -Wall -pedantic -Werror -c readmarks.c
correlation.o : correlation.c calcmarks.h

—— tab —→cc -std=c99 -Wall -pedantic -Werror -c correlation.c
download this Makefile.
Of note:
each target, at the beginning of lines, is followed by the dependencies (typically other files) on which it depends,
each target may also have one or more actions that are performed/executed if the target is out-of-date with respect to its dependencies,
actions must commence with the tab character, and
each (line) is passed verbatim to a shell for execution - just as if you would type it by hand.
Very long lines may be split using the backslash character.
Variable substitutions in make

As we see from the previous example, Makefiles can themselves become long, detailed files, and we'd like to "factor out" a lot of the common information.
It's similar to setting constants in C, with #define
Although not a full programming language, make supports simple variable definitions and variable substitutions (and even functions!).
C99 = cc -std=c99
CFLAGS = -Wall -pedantic -Werror
calcmarks : calcmarks.o globals.o readmarks.o correlation.o

$(C99) $(CFLAGS) -o calcmarks \
calcmarks.o globals.o readmarks.o correlation.o -lm
calcmarks.o : calcmarks.c calcmarks.h

$(C99) $(CFLAGS) -c calcmarks.c
globals.o : globals.c calcmarks.h

$(C99) $(CFLAGS) -c globals.c
readmarks.o : readmarks.c calcmarks.h

$(C99) $(CFLAGS) -c readmarks.c
correlation.o : correlation.c calcmarks.h

$(C99) $(CFLAGS) -c correlation.c
Of note:
variables are usually defined near the top of the Makefile.
the variables are simply expanded in-line with $(VARNAME).
warning - the syntax of make's variable substitutions is slightly different to those of our standard shells.
Variable substitutions in make, continued

As our projects grow, we add more C source files to the project. We should refactor our Makefiles when we notice common patterns:
PROJECT = calcmarks
HEADERS = $(PROJECT).h
OBJ = calcmarks.o globals.o readmarks.o correlation.o
C99 = cc -std=c99
$(PROJECT) : $(OBJ)
$(C99) $(CFLAGS) -o $(PROJECT) $(OBJ) -lm
calcmarks.o : calcmarks.c $(HEADERS)

$(C99) $(CFLAGS) -c calcmarks.c
globals.o : globals.c $(HEADERS)

$(C99) $(CFLAGS) -c globals.c
readmarks.o : readmarks.c $(HEADERS)

$(C99) $(CFLAGS) -c readmarks.c
correlation.o : correlation.c $(HEADERS)

$(C99) $(CFLAGS) -c correlation.c
clean:
rm -f $(PROJECT) $(OBJ)
Of note:
we have introduced a new variable, PROJECT, to name our project,
the value of the new variable, HEADERS is defined by accessing the value of $(PROJECT),
we have introduced a new variable, OBJ, to collate all of our object files,
our project specifically depends on our object files,
we have a new target, named clean, to remove all unnecessary files. clean has no dependencies, and so will always be executed if requested.
Employing automatic variables in a Makefile

We further note that each of our object files depends on its C source file, and that it would be handy to reduce these very common lines.
make provides a (wide) variety of filename patterns and automatic variables to considerably simplify our actions:
PROJECT = calcmarks
HEADERS = $(PROJECT).h
OBJ = calcmarks.o globals.o readmarks.o correlation.o
C99 = cc -std=c99
$(PROJECT) : $(OBJ)
$(C99) $(CFLAGS) -o $(PROJECT) $(OBJ) -lm
%.o : %.c $(HEADERS)

$(C99) $(CFLAGS) -c $<
clean:
rm -f $(PROJECT) $(OBJ)
Of note:
the pattern %.o matches, in turn, each of the 4 object filenames to be considered,
the pattern %.c is "built" from the C file corresponding to the %.o file,
the automatic variable $< is "the reason we're here", and
the linker option -lm indicates that our project requires something from C's standard maths library (sqrt() ).
make supports many automatic variables, which it "keeps up to date" as its execution proceeds:
$@ This will always expand to the current target.

$< The name of the first dependency. This is the first item listed after the colon.
$? The names of all the dependencies that are newer than the target.
Fortunately, we rarely need to remember all of these patterns and variables, and generally just copy and modify existing Makefiles.
Allocating primary memory to processes

The important task of allocating memory to processes, and efficiently ensuring that processes have their instructions and data in main memory when needed, is termed memory management.
We'll need to consider the role that memory plays from two (conflicting?) perspectives:
the operating system's perspective of how to allocate and manage all memory fairly and efficiently, and
the process's perspective of how to access the memory allocated to it, and how to obtain more.
An important goal of the operating system is to keep as many processes executable as possible, and it will achieve this only when processes have available the memory they require.
Processes requiring specific memory (holding either instructions or data) cannot execute, and are blocked until that memory is available (to them).
For simplicity, we'll initially assume that a process's image occupies a contiguous region of main memory.
Requirements of Memory Management

Logical Organisation:
Although processes in memory often occupy linear sequences of addresses, programs are seldom written this way.
Structured programming and, more recently, object-oriented techniques, encourage/enforce programming using modules which are developed and compiled independently.
Ideally, all references from one module to another are resolved at run-time, maintaining their independence (termed late binding).
Physical Organisation:
The relationship between primary memory (RAM) and secondary memory (disk) is straightforward, but not one that programmers wish to manage. Old techniques termed overlays permitted reuse of a process's memory, but (today) are unnecessarily complex.
Moreover, in a multi-programmed system, the programmer cannot predict the size nor location of a process's memory. The task of moving information between main and secondary memory is clearly the responsibility of the operating system.
Requirements of Memory Management, continued

Sharing:
Even with protection, there is also a need to allow processes to share memory. For example, multiple processes running the same program can share the (read+execute only) instructions for the program, and co-operating processes may wish to share and communicate via
memory containing data structures.
Relocation:
In a multi-programming system, the execution of a single process is often unrelated to others. When a process is first created, it is difficult (if not impossible) to know where its image will be placed when initially loaded into memory.
Similarly, when a process is swapped-out (Suspended), it is unlikely that the process will be swapped-in back to exactly the same memory location.
Memory management determines where both instructions and data are located, i.e. how a process's memory references (requests) translate into actual physical memory addresses.
Protection:
Each process must be protected from either accidental or deliberate "interference" from other processes. Although compilers for high-level programming languages offer some support (e.g. constrained control flow, static array bound references), most data references are
dynamic (array access and pointer following).
Memory references made by a process must be checked (at run-time) to ensure that they lie within the bounds of memory allocated by the operating system.
Checks are performed by hardware at run-time, and invalid references generate an access violation interrupt, trap, or exception, for the operating system software to handle.
The memory protection must be performed by the processor (hardware) rather than the operating system (software), because the operating system cannot anticipate all the memory references that a program will make. Even if this were possible, it would be prohibitively
time-consuming to screen each program in advance for possible memory violations.
Initial Memory Allocation Using Partitioning

In modern operating systems offering memory management, the operating system itself occupies a (fixed?) portion of main memory. The remainder is
available for multiple user/application processes.
The simplest technique is to consider main memory being in fixed-sized partitions, with two clear choices:
equal sized partitions, or
unequal sized partitions.
Any new process whose size is less than or equal to a partition's size may be loaded into that partition.
Example of Fixed Partitioning of a 64-Mbyte Memory
Initial Memory Allocation Using Partitioning, continued

Equal sized partitions introduce two problems:
1. a process's requirements may exceed the partition size, and
2. a small process still occupies a full partition. Such wastage of memory is termed internal memory
fragmentation.
The initial choice of partition - the placement algorithm - is, of course, trivial with equal-sized partitions.
Unequal sized partitions offer obvious advantages with respect to these problems, but they complicate the
placement algorithm. Either:
1. a process is placed in the largest (large-enough) partition, to minimise internal memory fragmentation, or
2. a process is placed in the smallest (large-enough) available partition.
The initial placement algorithm is again simple, but also introduces excessive internal memory fragmentation.
Example of Fixed Partitioning of a 64-Mbyte Memory
Dynamic Memory Partitioning

Dynamic partitioning overcomes some shortcomings of fixed partitioning: partitions are of variable length and number.
When a process commences, it occupies a partition of exactly the required size, and no more.
The Effect of Dynamic Partitioning
As the above figure shows, dynamic partitioning introduces the problem of external memory fragmentation, where there is insufficient contiguous free memory to hold a new process, even though sufficient free memory exists in the system.
Dynamic Partitioning Placement Algorithms

One obvious question suggested by dynamic partitioning is "Where do we place a new process" Three simple algorithms exist:
Memory Configuration Before and After Allocation of 16 Mbyte Block
First-fit:
find the first unused block of memory that can contain the process, searching from Address 0,
Best-fit:
find the smallest unused block that can contain the process, or
Next-fit:
remember where the last process's memory was allocated (say Address k), and find the first unused block that can contain the process, searching from Address k.
The Need for Address Relocation

Simple memory management schemes share one significant assumption: that, when a process is swapped-out, it will always be swapped back into memory, having access to the same memory locations as before.
This assumption actually complicates the memory management task, and contributes to memory fragmentation.
We need to define three terms:
A logical address
is a reference to a memory location independent of any current assignment of data to main memory.
A relative address
is a logical address expressed relative to a fixed (logical) location, such as the beginning of the process's image.
A physical address, or absolute address

is an actual location in main (physical) memory.
We've previously (implicitly) assumed that when a process is initially loaded (from disk), its relative addresses are replaced by absolute addresses.
More realistically, we enable processes to be swapped-in to any feasible range of physical memory: and this location is unlikely to be the same as before.
Hardware Address Translation

If a process may be swapped-in (back) to a different range of physical addresses, we need to update its relative addressing. We could have software modify all addresses found (slow), or have hardware translate all addresses, on-the-fly, as/if they are required.
While a process is executing, we employ a hardware base register to indicate the beginning of the process's partition, and a hardware bounds register to indicate the partition's extent.
Hardware Support for Relocation
Each process requires a pair of (hardware) base and bound registers, and the pair must be saved and restored as each process is swapped-out, and later swapped back in.
Simple Paging of Memory

We have just seen that fixed-sized partitions introduce internal fragmentation, and variable-sized partitions introduce external fragmentation.
However, internal fragmentation is bounded by the maximum size of a partition, and so if we allocate to a process several small fixed-sized fragments, we'll see minimal internal fragmentation only within the last fragment, and no external fragmentation.
Assignment of Process Pages to Free Page Frames
We term the small, equal-sized 'chunks' of a process's image pages, and place them in equal-sized 'chunks' of main memory, variously termed frames, or page frames.
We can now also remove the restriction (the assumption) that a process's sections of memory must be contiguous. Clearly a single base register is insufficient - we need a large number of base registers, one for each page, to identify the starting address of that page. (We do
not need multiple bounds registers; Why not?)
Data Structures for the previous figure at Time Epoch (f)
Page Registers and Page Tables

So, the operating system now maintains a set of page registers, or a page table. The page table holds the (physical, absolute) frame location for each page of the Running process. Within each process, a logical address now consists of a page number and an offset within that
page's frame.
In the following figure, 6 bits from each 16-bit logical address indicate which page table entry to use. The remaining 10 bits of the logical address are appended to the contents of the page table entry to provide the actual physical address.
Logical-to-Physical Address Translation using Paging
The Logical-to-Physical mapping can still be performed in hardware, provided that hardware knows how to access the page table of the current process.
The Principle of Referential Locality

Numerous studies of the memory accesses of processes have observed that memory references cluster in certain parts of the program: over long periods, the centres of the clusters move, but over shorter periods, they are fairly static.
For most types of programs, it is clear that:
Except for infrequent branches and function/procedure invocation, program execution is sequential. The next instruction to be fetched usually follows the last one executed.
Programs generally operate at the same "depth" of function-invocation. References to instructions cluster within (and between) a small collection of functions.
Most iterative control flow (looping) is over short instruction sequences. Instructions from the same memory locations are fetched several times in succession.
Access to memory locations holding data is, too, constrained to a few frequently required data structures, or sequential steps through memory (e.g. when traversing arrays).
With reference to paging schemes, this locality of reference suggests that, within a process, the next memory reference will very likely be from the same page as the last memory reference.
This will impact heavily on our next enhancement to memory management: the use of virtual memory.
Paging vs Partitioning
When we compare paging with the much simpler technique of partitioning, we see two clear benefits:
As processes are swapped-out and then back in, they may occupy different regions of physical memory.
This is possible because hardware efficiently translates each logical address to a physical address, at run-time.
The operating system's memory management software manipulates the hardware (page table registers) to facilitate the translation.
A process is broken into pages and these need not be contiguous in physical memory.
In combination with the principle of program locality, we now have a significant breakthrough:
If the above two characteristics are present, then it is not necessary for all pages of a process to be in memory at any one time during its execution.
Advantages of Paging
Execution of any process can continue provided that the instruction it next wants to execute, or the data location it next wants to access, is in physical memory.
If not, the operating system must load the required memory from the swapping (or paging) space before execution can continue.
However, the swapping space is generally on a slow device (a disk), so the paging I/O request forces the process to be Blocked until the required page of memory is available. In the interim, another process may be able to execute.
Before we consider how we can achieve this, and introduce additional efficiency, consider what advantages are now introduced:
More (pieces of) processes may be maintained in main physical memory (either Ready or Running).
Most processes do not require all of their memory before they can execute: memory may be loaded on demand.
If the swapping space is larger than the physical memory, any single process may now demand more memory than the amount of physical memory installed.
This last aspect gives the technique its name: virtual memory.
Virtual Memory and Resident Working Sets

The principle of program locality again tells us that at any time, only a small subset of a process's instructions and data will be required.
We define a process's set of pages, or segments, in physical memory, as its resident (or working) memory set.
prompt> ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 1372 432 ? S Sep12 0:04 init
root 4 0.0 0.0 0 0 ? SW Sep12 0:04 [kswapd]
root 692 0.0 0.2 1576 604 ? S Sep12 0:00 crond
xfs 742 0.0 0.8 5212 2228 ? S Sep12 0:23 xfs -droppriv -da
root 749 0.0 0.1 1344 340 tty1 S Sep12 0:00 /sbin/mingetty tt
...
chris 3865 0.0 0.6 2924 1644 pts/1 S Sep15 0:01 -zsh
chris 25366 0.0 6.0 23816 15428 ? S 14:34 0:06 /usr/bin/firefox
chris 25388 0.0 1.4 17216 3660 ? S 14:34 0:00 (dns helper)
chris 26233 0.0 0.2 2604 688 pts/1 R 19:11 0:00 ps aux
In the steady state, the memory will be fully occupied by the working sets of the Ready and Running processes, but:
If the processes' working sets are permitted to be too large, fewer processes can ever be Ready.
If the processes' working sets are forced to be too small, then additional requests must be made of the swapping space to retrieve required pages or segments.
All modern operating systems employ virtual memory based on paging (Q: can you determine the macOS or Linux page size?). Windows systems also employ virtual memory based on segmentation.
Virtual Memory Hardware using Page Tables

We saw that with simple paging, each process has its own page table entries. When a process's (complete) set of pages were loaded into memory, the current (hardware) page tables were saved and restored by the operating system.
Using virtual memory, the same approach is taken, but the contents of the page tables becomes more complex. Page table entries must include additional control information, indicating at least:
if the page is present in physical memory (a P bit), and
if the page has been modified since it was brought into physical memory (an M bit).
Address Translation in a Paging System
Virtual Memory Hardware using Page Tables, continued

The total size of the page table entries also becomes an issue, because the number of pages that a process may access greatly exceeds the number of actual frames.
This is addressed using a two-level addressing scheme:
Address Translation in a Two-Level Paging System
Virtual Memory Page Replacement

When the Running process requests a page that is not in memory, a page fault results, and (if the memory is currently 'full') one of the frames currently in memory must be replaced by the required page.
To make room for the required page, one or more existing pages must be "evicted" (to the swap space). Clearly, the working set of some process must be reduced.
However, if a page is evicted just before it is required (again), it'll just need to be paged back in! If this continues, the activity of page thrashing is observed.
We hope that the operating system can avoid thrashing with an intelligent choice of the page to discard.
Virtual Memory Implementation Considerations

The many different implementations of virtual memory differ in their treatment of some common considerations:
1. When should a process's pages be fetched?
A process initially requires the first page containing its starting address (and some initial data structures), but thereafter when should each page be allocated?
A VM system can employ demand paging in which a page is allocated only when a reference to it is made, or predictive pre-paging where pages are "intelligently" allocated before they are required.
2. Where in physical memory should pages be allocated?
Should we use policies such as first-fit, best-fit, or next-fit (which we saw when discussing basic memory partitioning)?
Does it matter?
Virtual Memory Implementation Considerations, continued

3. Which existing blocks should be replaced?
i.e. what is the replacement policy?
To avoid thrashing, we wish to replace only pages unlikely to be required soon, but this must be balanced against how many frames can be allocated to a process, and if the Running process's pages should be displaced (a local policy) or if other processes' pages can
be displaced (a global policy).
A number of replacement algorithms exist (seemingly a preoccupation of 1970's OS research) which select pages to be replaced.
Fundamental algorithms include first-in, first-out (obvious, but disadvantages long-running programs with high locality) and least-recently-used (almost ideal, but requires time-expensive hardware to maintain time-stamps on page usage).
4. How many processes to admit to the Ready and Running states?
The degree of multi-programming permitted must balance processor utilisation (minimising idle time due to I/O blocking) against utility (many processes executing with small resident set sizes and possible thrashing).
No-one, yet, claims memory management is easy.

A slow introduction to pointers in C

The ISO-C99 programming language has a very powerful feature for addressing and modifying memory which, for now, we'll casually term "pointers in C".
If used correctly pointers can enable very fast, efficient, execution of C programs.
If misunderstood or used incorrectly, pointers can make your programs do very strange, incorrect things, and result in very hard to diagnose and debug programs.
The primary role of pointers - to allow a program (at run-time) to access its own memory - sounds like a useful feature, but is often described as a very dangerous feature.
There is much written about the power and expressiveness of C's pointers, with general agreement that C's pointers are a threshold concept in Computer Science.
(Recently much has been written about Java's lack of pointers. More precisely, Java does have pointers, termed references, but the references to Java's objects are so consistently and carefully constrained at both compile- and run-
times, that little information about the run-time environment is exposed to the running program, and little can go wrong). Pointers on C, Understanding
Kenneth Reek, and Using C
Addison-Wesley, Pointers,
636pp, 1998. Richard Reese,
O'Reilly Media,
226pp, 2013.
What are pointers?

We know that C has both ''standard'' variables, holding integers, characters, and floating-point values, also described as scalar variables.
In addition, we've seen arrays of these variables.
Let's follow this simplified explanation:
We understand that variables occupy memory locations (1 or more bytes) of a computer's memory.
Each variable requires enough bytes to store the values the variable will (ever) need to hold. For example, on our CSSE labs' computers, a simple C integer will require 4 bytes of memory.
Similarly, an array of 100 integers, will require 400 bytes of contiguous memory - there is no padding between elements.
Computers have a large amount of memory, e.g. our lab computers have 8 gigabytes of memory (8GB), or nearly 8.6 billion bytes.
Each of a computer's memory bytes is uniquely numbered, from 0 to some large value. Each such number is termed the byte's memory address.
We often refer to memory locations as just addresses and the action of identifying an address as addressing.
With these points in mind, we can make 3 simple statements:
1. Pointers are variables that hold the address of a memory location.

2. Pointers are variables that point to memory locations.
3. Pointers (usually) point to memory locations being used to hold variables' values/contents.
The & operator, the address-of operator, the ampersand operator

The punctuation character &, often pronounced as the address-of operator, is used to find a variable's address.
For example, we'd pronounce this as:
int total;
.... &total ....
"the address of total", and if the integer variable total was located at memory address 10,000 then the value of &total would be 10,000.
We can now introduce a variable named p, which is a pointer to an integer

(pedantically, p is a variable used to store the address of a memory location that we expect to hold an integer value).
int total;
int *p ;
p = &total ;
If the integer variable total was located at memory address 10,000 then the value of p would be 10,000.
If necessary (though rarely), we can print out the address of a variable, or the value of a pointer, by first casting it to something we can print, such as an unsigned integer, or to an "generic" pointer:
int total;
int *p = &total ;
printf("address of variable is: %lu\n", (unsigned long)&total );

printf(" value of pointer p is: %lu\n", (unsigned long)p );
printf(" value of pointer p is: %p\n", (void *)p );
Dereferencing a pointer
We now know that a pointer may point to memory locations holding variables' values.
It should also be obvious that if the variable's value (contents) changes, then the pointer will keep pointing to the same variable, (which now holds the new value).
We can use C's concept of dereferencing a pointer to determine the value the pointer points to:
int total;
int *p = &total ;
total = 3;
printf("value of variable total is: %i\n", total );

printf("value pointed to by pointer p is: %i\n", *p );
++total; // increment the value that p points to

Even though the variable's value has changed (from 3 to 4), the pointer still points at the variable's location.
The pointer first pointed at an address containing 3, and the pointer kept pointing at the (same) address which now contains 4.
Dereferencing a pointer, continued

We now know that changing the value that a pointer points to does not change the pointer (good!).
Now we'd like to change the value held in the address that the pointer points to.
Similarly, this will not change the pointer itself.
int total;
int *p = &total ;
int bigger;
total = 8;

printf("value pointed to by pointer p is: %i\n\n", *p );
*p = *p + 2 ; // increment, by 2, the value that p points to

printf("value pointed to by pointer p is: %i\n\n", *p );
bigger = *p + 2 ; // just fetch the value that p points to

printf("value of variable bigger is: %i\n", bigger );
An array's name is an address

When finding the address of a scalar variable (or a structure), we precede the name by the address of operator, the ampersand:
int total;
int *p = &total ;
However, when requiring the address of an array, we're really asking for the address of the first element of that array:
#define N 5
int totals[N];
int *first = &totals[0]; // the first element of the array

int *second = &totals[1]; // the second element of the array
int *third = &totals[2]; // the third element of the array
As we frequently use a pointer to traverse the elements of an array (see the following slides on pointer arithmetic), we observe the following equivalence:
int *p = &totals[0] ;
// and:
int *p = totals ;
That is: "an array's name is synonymous with the address of the first element of that array".
Pointer Arithmetic
Another facility in C is the use of pointer arithmetic with which we may change a pointer's value so that it points to successive memory locations.
We specify pointer arithmetic in the same way that we specify numeric arithmetic, using the symbols ++ and - - to request pre- and post- increment and decrement operators.
We generally use pointer arithmetic when accessing successive elements of arrays.
Consider the following example, which initializes all elements of an integer array:
#define N 5
int totals[N];
int *p = totals; // p points to the first/leftmost element of totals
for(int i=0 ; i<N ; ++i)

{
*p = 0; // set what p points to to zero
++p; // advance/move pointer p "right" to point to the next integer
}

{
printf("address of totals[%i] is: %p\n", i, (void *)&totals[i] );
printf(" value of totals[%i] is: %i\n", i, totals[i] );
}
How far does the pointer move?

It would make little sense to be able to ''point anywhere'' into memory, and so C automatically 'adjusts' pointers' movement (forwards and backwards) by values that are multiples of the size of the base types to which the pointer points(!).
In our example:

{
*p = 0; // set what p points to to zero
++p; // advance/move pointer p "right" to point to the next integer
}
p will initially point to the location of the variable:
totals[0], then to
totals[1], then to
totals[2] ...
Similarly, we can say that p has the values:
&totals[0], then
&totals[1], then
&totals[2] ...
Combining pointer arithmetic and dereferencing

With great care (because it's confusing), we can also combine pointer arithmetic with dereferencing:
#define N 5
int totals[N];
int *p = totals ;

{
*p++ = 0; // set what p points to to zero, and then
// advance p to point to the "next" integer
}

{
printf("value of totals[%i] is: %i\n", i, totals[i] );
}
In English, we read this as:
"set the contents of the location that the pointer p currently points to the value zero, and then increment the value of pointer p by the size of the variable that it points to"
Similarly we can employ pointer arithmetic in the control of for loops. Consider this excellent use of the preprocessor:
int array[N];
int n;
int *a;
#define FOREACH_ARRAY_ELEMENT for(n=0, a=array ; n<N ; ++n, ++a)
FOREACH_ARRAY_ELEMENT
{
if(*a == 0)
{
.....
}
}
Functions with pointer parameters

We know that pointers are simply variables.
We now use this fact to implement functions that receive pointers as parameters.
A pointer parameter will be initialized with an address when the function is called.
Consider two equivalent implementations of C's standard strlen function - the traditional approach is to employ a parameter that "looks like" an array; new approaches employ a pointer parameter:
int strlen_array( char array[] ) int strlen_pointer( char *strp ) int strlen_pointer( char *strp )
{ { {
int len = 0; int len = 0; char *s = strp;
while( array[len] != '\0' ) { while( *strp != '\0' ) { while( *s != '\0' ) {

++len; ++len; ++s;
} ++strp; }
} return (s - strp);
return len; return len; }
} }
During the execution of the function, any changes to the pointer will simply change what it points to.
In this example, strp traverses the null-byte terminated character array (a string) that was passed as an argument to the function.
We are not modifying the string that the pointer points to, we are simply accessing adjacent, contiguous, memory locations until we find the null-byte.
Returning a pointer from a function

Let's consider another example. Here we provide two equivalent implementations of C's standard strcpy function, which copies a string from its source, src, to a new destination, dest.
The C99 standards state that the strcpy function function must return a copy of its (original) destination parameter.
In both cases, we are returning a copy of the (original) dest parameter - that is, we are returning a pointer as the function's value.
We say that "the function's return-type is a pointer".

Returning a pointer from a function, continued

Consider the following functions to copy one string into another:
char *strcpy_array( char dest[], char src[] ) // returns a pointer

{
int i = 0;
while( src[i] != '\0' ) {

dest[ i ] = src[ i ];
+ + i ;
}
dest[ i ] = '\0';
return dest; // returns the original destination parameter

}
char *strcpy_pointer( char *dest, char *src ) // two pointer parameters

{
char *origdest = dest; // take a copy of the dest parameter
while( *src != '\0' ) {

*dest = *src; // copy one character from src to dest
++src;
++dest;
}
*dest = '\0';
return origdest; // returns a copy of the original destination parameter

}
Note:
in the array version, the function returns a pointer as its value. This further demonstrates the equivalence between array names (here, dest) and a pointer to the first element of that array.
in the pointer version, we move the dest parameter after we have copied each character, and thus we must first save and then return a copy of the parameter's original value.
if very careful, we could reduce the whole loop to the statement while((*dest++ = *src++));
But don't.
Passing pointers to functions

Consider a very simple function, whose role is to swap two integer values:
#include <stdio.h>
void swap(int i, int j)

{
int temp;
temp = i;
i = j;
j = temp;
}

{
int a=3, b=5; // MULTIPLE DEFINITIONS AND INITIALIZATIONS
printf("before a=%i, b=%i\n", a, b);
swap(a, b); // ATTEMPT TO SWAP THE 2 INTEGERS
printf("after a=%i, b=%i\n", a, b);

return 0;
}
before a=3, b=5

after a=3, b=5
Doh! What went wrong?
The "problem" occurs because we are not actually swapping the values contained in our variables a and b, but are (successfully) swapping copies of those values.
Passing pointers to functions, continued

Instead, we need to pass a 'reference' to the two integers to be interchanged.
We need to give the swap() function "access" to the variables a and b, so that swap() may modify those variables:
#include <stdio.h>
void swap(int *ip, int *jp)

{
int temp;
temp = *ip; // swap's temp is now 3

*ip = *jp; // main's variable a is now 5
*jp = temp; // main's variable b is now 3
}

{
int a=3, b=5;
printf("before a=%i, b=%i\n", a, b);
swap(&a, &b); // pass pointers to our local variables
printf("after a=%i, b=%i\n", a, b);
return 0;
}
before a=3, b=5

after a=5, b=3
Much better! Of note:
The function swap() is now dealing with the original variables, rather than new copies of their values.
A function may permit another function to modify its variables, by passing pointers to those variables.
The receiving function now modifies what those pointers point to.
Duplicating a string
We know that:
C considers null-byte terminated character arrays as strings, and

that the length of such strings is not determined by the array size, but by where the null-byte is.
So how could we take a duplicate copy, a clone, of a string? We could try:
#include <string.h>
char *my_strdup(char *str)

{
char bigarray[SOME_HUGE_SIZE];
strcpy(bigarray, str); // WILL ENSURE THAT bigarray IS NULL-BYTE TERMINATED
return bigarray; // RETURN THE ADDRESS OF bigarray

}
But we'd instantly have two problems:
1. we'd never be able to know the largest array size required to copy the arbitrary string argument, and
2. we can't return the address of any local variable. Once function my_strdup() returns, variable bigarray no longer exists, and so we can't provide the caller with its address.
Allocating new memory

Let's first address the first of these problems - we do not know, until the function is called, how big the array should be.
It is often the case that we do not know, until we execute our programs, how much memory we'll really need!
Instead of using a fixed sized array whose size may sometimes be too small, we must dynamically request some new memory at runtime to hold our desired result.
This is a fundamental (and initially confusing) concept of most programming languages - the ability to request from the operating system additional memory for our programs.
C99 provides a small collection of functions to support memory allocation.

The primary function we'll see is named malloc(), which is declared in the standard <stdlib.h> header file:
#include <stdlib.h>
extern void *malloc( size_t nbytes );
malloc() is a function (external to our programs) that returns a pointer.

However, malloc() doesn't really know what it's returning a pointer to - it doesn't know if it's a pointer to an integer, or a pointer to a character, or even to one our own user-defined types.
For this reason, we use the generic pointer, pronounced "void star" or "void pointer".
It's a pointer to "something", and we only "know" what that is when we place an interpretation on the pointer.
malloc() needs to be informed of the amount of memory that it should allocate - the number of bytes we require.
We use the standard datatype size_t to hold an integer value that may be 0 or positive (we obviously can't request a negative amount of memory!).
We have used, but skipped over, the use of size_t before - it's the datatype of values returned by the sizeof operator, and the pedantically-correct type returned by the strlen() function.
Checking memory allocations

Of course, the memory in our computers is finite (even if it has several physical gigabytes, or is using virtual memory), and if we keep calling malloc() in our programs, we'll eventually exhaust available memory.
Note that a machine's operating system will probably not allocate all memory to a single program, anyway. There's a lot going on on a standard computer, and those other activities all require memory, too.
For programs that perform more than a few allocations, or even some potentially large allocations, we need to check the value returned by malloc() to determined if it succeeded:
#include <stdlib.h>
int bytes_wanted = 1000000 * sizeof(int);
int *huge_array = malloc( bytes_wanted );
if(huge_array == NULL) { // DID malloc FAIL?

printf("Cannot allocate %i bytes of memory\n", bytes_wanted);
exit( EXIT_FAILURE );
}
Strictly speaking, we should check all allocation requests to both malloc() and calloc().
Duplicating a string revisited

We'll now use malloc() to dynamically allocate, at runtime, exactly the correct amount of memory that we need.
When duplicating a string, we need enough new bytes to hold every character of the string, including a null-byte to terminate the string.
This is 1 more than the value returned by strlen:
#include <stdlib.h>
#include <string.h>
char *my_strdup2(char *str)

{
char *new = malloc( strlen(str) + 1 );
if(new != NULL) {
strcpy(new, str); // ENSURES THAT DUPLICATE WILL BE NUL-TERMINATED
}
return new;
}
Of note:
we are not returning the address of a local variable from our function - we've solved both of our problems!
we're returning a pointer to some additional memory given to us by the operating system.
this memory does not "disappear" when the function returns, and so it's safe to provide this value (a pointer) to whoever called my_strdup2.
the new memory provided by the operating system resides in a reserved (large) memory region termed the heap. We never access the heap directly (we leave that to malloc()) and just use (correctly) the space returned by malloc().
Allocating an array of integers

Let's quickly visit another example of malloc(). We'll allocate enough memory to hold an array of integers:
#include <stdlib.h>
int *randomints(int wanted)

{
int *array = malloc( wanted * sizeof(int) );
if(array != NULL) {
for(int i=0 ; i<wanted ; ++i) {
array[i] = rand() % 100;
}
}
return array;
}
Of note:
malloc() is used here to allocate memory that we'll be treating as integers.

malloc() does not know about our eventual use for the memory it returns.
how much memory did we need?
We know how many integers we want, wanted, and we know the space occupied by each of them, sizeof(int).
We thus just multiply these two to determine how many bytes we ask malloc() for.
Requesting that new memory be cleared

In many situations we want our allocated memory to have a known value. The C99 standard library provides a single function to provide the most common case - clearing allocated memory:
#include <stdlib.h>
extern void *calloc( size_t nitems, size_t itemsize );

....
int *intarray = calloc(N, sizeof(int));
It's lost in C's history why malloc() and calloc() have different calling sequences.
To explain what is happening, here, we can even write our own version, if we are careful:
#include <stdlib.h>
#include <string.h>
void *my_calloc( size_t nitems, size_t itemsize )

{
size_t nbytes = nitems * itemsize;
void *result = malloc( nbytes );
if(result != NULL) {
memset( result, 0, nbytes ); // SETS ALL BYTES IN result TO THE VALUE 0
}
return result;
}
....
int *intarray = my_calloc(N, sizeof(int));
Deallocating memory with free

In programs that:
run for a long time

(perhaps long-running server programs such as web-servers), or
temporarily require a lot of memory, and then no longer require it,
we should deallocate the memory provided to us by malloc() and calloc().
The C99 standard library provides an obvious function to perform this:
extern void free( void *pointer );
Any pointer successfully returned by malloc() or calloc() may be freed.
Think of it as requesting that some of the allocated heap memory be given back to the operating system for re-use.
#include <stdlib.h>
int *vector = randomints( 1000 );
if(vector != NULL) {
// USE THE vector
......
}
free( vector );
Note, there is no need for your programs to completely deallocate all of their allocated memory before they exit - the operating system will do that for you.
Reallocating previously allocated memory

We'd already seen that it's often the case that we don't know our program's memory requirements until we run the program.
Even then, depending on the input given to our program, or the execution of our program, we often need to allocate more than our initial "guess".
The C99 standard library provides a function, named realloc() to grow (or rarely shrink) our previously allocate memory:
extern void *realloc( void *oldpointer, size_t newsize );
We pass to realloc() a pointer than has previously been allocated by malloc(), calloc(), or (now) realloc().
Most programs wish to extend the initially allocated memory:
int original; int nitems = 0;

int newsize; int *items = NULL;
int *array;
int *newarray; ....
while( fgets(line ....) != NULL ) {
.... items = realloc( items, (nitems+1) * sizeof(items[0]) );
array = malloc( original * sizeof(int) ); if(items == NULL) {
if(array == NULL) { // HANDLE THE ALLOCATION FAILURE
// HANDLE THE ALLOCATION FAILURE }
} .... // COPY OR PROCESS THE LINE JUST READ
.... ++nitems;
}
newarray = realloc( array, newsize * sizeof(int) ); ....
if(newarray == NULL) { if(items != NULL) {
// HANDLE THE ALLOCATION FAILURE free( items );
} }
....
Of note:
if realloc() fails to allocate the revised size, it returns the NULL pointer.
if successful, realloc() copies any old data into the newly allocated memory, and then deallocates the old memory.
if the new request does not actually require new memory to be allocated, realloc() will usually return the same value of oldpointer.
a request to realloc() with an "initial" address of NULL, is identical to just calling malloc().
Sorting an array of values

A frequently required operation is to sort an array of, say, integers or characters. In fact you'll often read that (a busy) computer spends more time sorting things than almost anything else.
Writing a simple sorting function is easy (you may have developed one in our Labsheet 3). However, writing a very efficient sort function is difficult, and it's just the kind of thing we want standard libraries to provide for us.
The standard C library provides a generic function named qsort().

Sorting an array of values, contined

Because qsort() is a very general sorting function, which can sort almost anything, it has a very confusing prototype:
#include <stdlib.h>
void qsort(void *array,

size_t number_of_elements,
size_t sizeof_each_element,
int (*comparison_function)(const void *p1, const void *p2) );
Let's explain each of these "pieces":
1. qsort() is a function returning no value (a function of type void).

2. qsort() is a function receiving 4 parameters.
3. the 1st parameter is a pointer (think of it as an array), but qsort() doesn't care if it's an array of integers, an array of characters, or even an array of our own user-defined types.
For this reason, we use the generic pointer, pronounced "void star" or "void pointer". This means that qsort() receives a pointer (the beginning of an array), but qsort() doesn't care what type it is.
4. the 2nd parameter indicates how many elements are in the array (how many are to be sorted). Standard C's type size_t is used (think of it as an integer whose value must be 0 or positive).
5. the 3rd parameter indicates the size of each array element. As qsort() doesn't know the datatype it's sorting, it needs to know how "big" each element is.
6. the 4th parameter... Oh boy! Where do we start?
i. comparison_function is a function (that we must provide) that returns an integer.

ii. it receives 2 parameters, p1 and p2, each of which is a pointer.
iii. a pointer to what? We don't care (yet), so we declare p1 and p2 as "void pointers".
iv. comparison_function finally promises to not change what its parameters p1 and p2 point to, and so p1 and p2 are preceded by the keyword const, making p1 and p2 each a constant pointer.
Sorting an array of integers

Let's put this knowledge to work by sorting an array of integers (a common task).
We'll first focus on the main() function, which fills an array with some random integers, prints the array, sorts the array, and again prints the array.
We'll write our comparison function, compareints(), soon:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <time.h>
#define N 10

{
int values[N];
// FILL OUR ARRAY WITH N INTEGERS BETWEEN 0 and 99

srand( time(NULL) );
for(int i=0 ; i<N ; i++) {
values[i] = rand() % 100;
}
// PRINT THE ARRAY BEFORE SORTING
for(int i=0 ; i<N ; i++) {
printf("%i ", values[i]);
}
printf("\n\n");
// SORT THE ARRAY USING qsort

qsort(values, N, sizeof(values[0]), compareints);
// PRINT THE ARRAY AFTER SORTING

for(int i=0 ; i<N ; i++) {
printf("%i ", values[i]);
}
printf("\n");
return 0;
}
Sorting an array of integers, continued

Finally let's write, and explain, our comparison function. Our comparison function, compareints(), will be called many times by C's qsort() function.
Each time, qsort() passes the address of 2 elements in the array being sorted. As qsort() doesn't know the datatype of the array it's sorting, it can only pass (to us) void pointers.
int compareints(const void *p1, const void *p2)

{
int *ip1 = (int *)p1;
int value1 = *ip1;

int value2 = *ip2;
return (value1 - value2);

}
Our function's first task is "to get" the values to be compared:
1. We first convert the void pointers to integer pointers, with:

2. We next extract each integer value pointed to by our integer pointers.

int value1 = *p1;
int value2 = *p2;
3. We finally report (return) to qsort() the ordering of these two values. qsort() expects:
i. a negative value if the 1st value is less-than the 2nd,
ii. zero if the 1st value and 2nd values are equal, and
iii. a positive value if the 1st value is greater-than the 2nd.
We can support this simply by subtracting one value from the other.
Notice that compareints() does not modify the array's values being sorted, and so the use of const pointers is correct.
Identifying related data

Let's consider the 2012 1st project for CITS1002.
The goal of the project was to manage the statistics of AFL teams throughout the season, calculating their positions on the premiership ladder at the end of each week.
Let's consider the significant global variables in its sample solution:
// DEFINE THE LIMITS ON PROGRAM'S DATA-STRUCTURES

#define MAX_TEAMS 24
#define MAX_TEAMNAME_LEN 30
....
// DEFINE A 2-DIMENSIONAL ARRAY HOLDING OUR UNIQUE TEAMNAMES

char teamname[MAX_TEAMS][MAX_TEAMNAME_LEN+1]; // +1 for null-byte
// STATISTICS FOR EACH TEAM, INDEXED BY EACH TEAM'S 'TEAM NUMBER'

int played [MAX_TEAMS];
int won [MAX_TEAMS];
int lost [MAX_TEAMS];
int drawn [MAX_TEAMS];
int bfor [MAX_TEAMS];
int bagainst[MAX_TEAMS];
int points [MAX_TEAMS];
....
// PRINT EACH TEAM'S RESULTS, ONE-PER-LINE, IN NO SPECIFIC ORDER

for(int t=0 ; t<nteams ; ++t) {
printf("%s %i %i %i %i %i %i %.2f %i\n", // %age to 2 decimal-places
teamname[t],
played[t], won[t], lost[t], drawn[t],
bfor[t], bagainst[t],
(100.0 * bfor[t] / bagainst[t]), // calculate percentage
points[t]);
}
It's clear that the variables are all strongly related, but that we're naming and accessing them as if they are independent.
CITS2002 Systems Programming, Lecture 17, p1, 23rd September 2019.
Defining structures
Instead of storing and identifying related data as independent variables, we prefer to "collect" it all into a single structure.
C provides a mechanism to bring related data together, structures, using the struct keyword.
We can now define and gather together our related data with:
// DEFINE AND INITIALIZE ONE VARIABLE THAT IS A STRUCTURE

struct {
char *name; // a pointer to a sequence of characters
int red; // in the range 0..255
int green;
int blue;
} rgb_colour = {
"DodgerBlue",
30,
144,
255
};
We now have a single variable (named rgb_colour) that is a structure, and at its point of definition we have initialised each of its 4 fields.
Defining and array of structures

Returning to our AFL project example, we can now define and gather together its related data with:

....
struct {
char teamname[MAX_TEAMNAME_LEN+1]; // +1 for null-byte
// STATISTICS FOR THIS TEAM, INDEXED BY EACH TEAM'S 'TEAM NUMBER'

int played;
int won;
int lost;
int drawn;
int bfor;
int bagainst;
int points;
} team[MAX_TEAMS]; // DEFINE A 1-DIMENSIONAL ARRAY NAMED team
We now have a single (1-dimensional) array, each element of which is a structure.

We often term this an array of structures.
Each element of the array has a number of fields, such as its teamname (a whole array of characters) and an integer number of points.
Accessing the fields of a structure

Now, when referring to individual items of data, we need to first specify which team we're interested in, and then which field of that team's structure.
We use a single dot ('.' or fullstop) to separate the variable name from the field name.
The old way, with independent variables:

teamname[t],
played[t], won[t], lost[t], drawn[t],
bfor[t], bagainst[t],
(100.0 * bfor[t] / bagainst[t]), // calculate percentage
points[t]);
}
The new way, accessing fields within each structure:

team[t].teamname,
team[t].played, team[t].won, team[t].lost, team[t].drawn,
team[t].bfor, team[t].bagainst,
(100.0 * team[t].bfor / team[t].bagainst), // calculate percentage
team[t].points);
}
While it requires more typing(!), it's clear that the fields all belong to the same structure (and thus team).
Moreover, the names teamname, played, .... may now be used as "other" variables elsewhere.
Accessing system information using structures

Operating systems (naturally) maintain a lot of (related) information, and keep that information in structures.
So that the information about the structures (the datatypes and names of the structure's fields) can be known by both the operating system and users' programs, these structures are defined in system-wide header files - typically in /usr/include and /usr/include/sys.
For example, consider how an operating system may represent time on a computer:
#include <stdio.h>
#include <sys/time.h>
// A value accurate to the nearest microsecond but also has a range of years
struct timeval {
int tv_sec; // Seconds
int tv_usec; // Microseconds
};
Note that the structure has now been given a name, and we can now define multiple variables having this named datatype (in our previous example, the structure would be described as anonymous).
We can now request information from the operating system, with the information returned to us in structures:
#include <stdio.h>
#include <sys/time.h>
struct timeval start_time;

struct timeval stop_time;
gettimeofday( &start_time, NULL );

printf("program started at %i.06%i\n",
(int)start_time.tv_sec, (int)start_time.tv_usec);
....
perform_work();
....
gettimeofday( &stop_time, NULL );

printf("program stopped at %i.06%i\n",
(int)stop_time.tv_sec, (int)stop_time.tv_usec);
Here we are passing the structure by address, with the & operator, so that the gettimeofday() function can modify the fields of our structure.
(we're not passing a meaningful pointer as the second parameter to gettimeofday(), as we're not interested in timezone information)
Accessing structures using a pointer

We've seen that we can access fields of a structure using a single dot ('.' or fullstop).
What if, instead of accessing the structure directly, we only have a pointer to a structure?
We've seen "one side" of this situation, already - when we passed the address of a structure to a function:
struct timeval start_time;
gettimeofday( &start_time, NULL );
The function gettimeofday(), must have been declared to receive a pointer:
extern int gettimeofday( struct timeval *time, ......);
Consider the following example, in which a pointer to a structure is returned from a function.
We now use the → operator (pronounced the 'arrow', or 'points-to' operator) to access the fields via the pointer:
#include <stdio.h>
#include <time.h>
void greeting(void)
{
time_t NOW = time(NULL);
struct tm *tm = localtime(&NOW);
printf("Today's date is %i/%i/%i\n",

tm->tm_mday, tm->tm_mon + 1, tm->tm_year + 1900);
if(tm->tm_hour < 12) {

printf("Good morning\n");
}
else if(tm->tm_hour < 17) {
printf("Good afternoon\n");
}
else {
printf("Good evening\n");
}
}
Defining our own datatypes

We can further simplify our code, and more clearly identify related data by defining our own datatypes.
We use the typedef keyword to define our new datatype in terms of an old (existing) datatype.

....
typedef struct {
char teamname[MAX_TEAMNAME_LEN+1]; // +1 for null-byte
....
int played;
....
} TEAM;
TEAM team[MAX_TEAMS];
As a convention (but not a C99 requirement), we'll define our user-defined types using uppercase names.

TEAM *tp = &team[t];

tp->teamname,
tp->played, tp->won, tp->lost, tp->drawn,
tp->bfor, tp->bagainst,
(100.0 * tp->bfor / tp->bagainst), // calculate percentage
tp->points);
}
Another example - using a pointer to our own datatype

Let's consider another example - the starting (home) and ending (destination) bustops from the CITS2002 1st project of 2015.
We starting with some of its definitions:
// GLOBAL CONSTANTS, BEST DEFINED ONCE NEAR THE TOP OF FILE

#define MAX_FIELD_LEN 100
#define MAX_STOPS_NEAR_ANYWHERE 200 // in Transperth: 184
....
// 2-D ARRAY OF VIABLE STOPS FOR COMMENCEMENT OF JOURNEY

char viable_home_stopid [MAX_STOPS_NEAR_ANYWHERE][MAX_FIELD_LEN];
char viable_home_name [MAX_STOPS_NEAR_ANYWHERE][MAX_FIELD_LEN];
int viable_home_metres [MAX_STOPS_NEAR_ANYWHERE];
int n_viable_homes = 0;
// 2-D ARRAY OF VIABLE STOPS FOR END OF JOURNEY

char viable_dest_stopid [MAX_STOPS_NEAR_ANYWHERE][MAX_FIELD_LEN];
char viable_dest_name [MAX_STOPS_NEAR_ANYWHERE][MAX_FIELD_LEN];
int viable_dest_metres [MAX_STOPS_NEAR_ANYWHERE];
int n_viable_dests = 0;
(After a post-project workshop) we later modified the 2-dimensional arrays to use dynamically-allocated memory:
// 2-D ARRAY OF VIABLE STOPS FOR COMMENCEMENT OF JOURNEY

char **viable_home_stopid = NULL;
char **viable_home_name = NULL;
int *viable_home_metres = NULL;
int n_viable_homes = 0;
// 2-D ARRAY OF VIABLE STOPS FOR END OF JOURNEY

char **viable_dest_stopid = NULL;
char **viable_dest_name = NULL;
int *viable_dest_metres = NULL;
int n_viable_dests = 0;
and we can now use typedef to define our own datatype:
// A NEW DATATYPE TO STORE 1 VIABLE STOP

typedef struct {
char *stopid;
char *name;
int metres;
} VIABLE;
// A VECTOR FOR EACH OF THE VIABLE home AND dest STOPS

VIABLE *home_stops = NULL;
VIABLE *dest_stops = NULL;
int n_home_stops = 0;
int n_dest_stops = 0;
Finding the attributes of a file

The operating system manages its data in a file system, in particular maintaining its files in a hierarchical directory structure - directories contain files and other (sub)directories.
As we saw with time-based information, we can ask the operating system for information about files and directories, by calling some system-provided functions.
We employ another POSIX† function, stat(), and the system-provided structure struct stat, to determine the attributes of each file:
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
char *progname;
void file_attributes(char *filename)

{
struct stat stat_buffer;
if(stat(filename, &stat_buffer) != 0) // can we 'stat' the file's attributes? {

perror( progname );
}
else if( S_ISREG( stat_buffer.st_mode ) ) {
printf( "%s is a regular file\n", filename );
printf( "is %i bytes long\n", (int)stat_buffer.st_size );
printf( "and was last modified on %i\n", (int)stat_buffer.st_mtime);
printf( "which was %s", ctime( &stat_buffer.st_mtime) );

}
}
†POSIX is an acronym for "Portable Operating System Interface", a family of standards specified by the IEEE for maintaining compatibility between operating systems. POSIX defines the application programming interface (API), along with command line shells and utility
interfaces, for software compatibility with variants of Unix (such as macOS and Linux) and other operating systems (e.g. Windows has a POSIX emulation layer).
Reading the contents of a directory

Most modern operating systems store their data in hierarchical file systems, consisting of directories which hold items that, themselves, may either be files or directories.
The formats used to store information in directories in different file-systems are different(!), and so when writing portable C programs, we prefer to use functions that work portably.
Consider the strong similarities between opening and reading a (text) file, and opening and reading a directory:

#include <dirent.h>
void print_file(char *filename) void list_directory(char *dirname)

{ {
FILE *fp; DIR *dirp;
char line[BUFSIZ]; struct dirent *dp;
fp = fopen(filename, "r"); dirp = opendir(dirname);

if(fp == NULL) { if(dirp == NULL) {
perror( progname ); perror( progname );
exit(EXIT_FAILURE); exit(EXIT_FAILURE);
} }
while(fgets(line, sizeof(buf), fp) != NULL) { while((dp = readdir(dirp)) != NULL) {

printf( "%s", line); printf( "%s\n", dp->d_name );
} }
fclose(fp); closedir(dirp);
} }
With directories, we're again discussing functions that are not part of the C99 standard, but are defined by POSIX standards.
Investigating the contents of a directory

We now know how to open a directory for reading, and to determine the names of all items in that directory.
What is each "thing" found in the directory - is it a directory, is it a file...?
To answer those questions, we need to employ the POSIX function, stat(), to determine the attributes of the items we find in directories:

#include <sys/types.h> #include <sys/types.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <sys/param.h> #include <sys/param.h>
#include <dirent.h> #include <dirent.h>
#include <unistd.h> #include <unistd.h>
static void list_directory(char *dirname) static void list_directory(char *dirname)

{ {
char fullpath[MAXPATHLEN]; char fullpath[MAXPATHLEN];
..... .....
while((dp = readdir(dirp)) != NULL) { while((dp = readdir(dirp)) != NULL) {
struct stat stat_buffer; struct stat stat_buffer;
struct stat *pointer = &stat_buffer;
sprintf(fullpath, "%s/%s", dirname, dp->d_name ); sprintf(fullpath, "%s/%s", dirname, dp->d_name );
if(stat(fullpath, &stat_buffer) != 0) { if(stat(fullpath, pointer) != 0) {

} }
else if( S_ISDIR( stat_buffer.st_mode )) { else if( S_ISDIR( pointer->st_mode )) {
printf( "%s is a directory\n", fullpath ); printf( "%s is a directory\n", fullpath );
} }
else if( S_ISREG( stat_buffer.st_mode )) { else if( S_ISREG( pointer->st_mode )) {
printf( "%s is a regular file\n", fullpath ); printf( "%s is a regular file\n", fullpath );
} }
else { else {
printf( "%s is unknown!\n", fullpath ); printf( "%s is unknown!\n", fullpath );
} }
} }
closedir(dirp); closedir(dirp);
} }

Organisation of File Systems

A clear and obvious requirement of an operating system is the provision of a convenient, efficient, and robust filing system.
The file system is used not only to store users' programs and data, but also to support and represent significant portions of the operating system itself.
Stallings introduces the traditional file system concepts of:
fields
represent the smallest logical item of data "understood" by a file-system: examples including a single integer or string. Fields may be of a fixed length (recorded by the file-system as "part of" each file), or be of variable length, where either the length is stored with the field or
a sentinel byte signifies the extent.
records
consist of one or more fields and are treated as a single unit by application programs needing to read or write the record. Again, records may be of fixed or variable length.
files
are familiar collections of identically represented records, and are accessed by unique names. Deleting a file (by name) similarly affects its component records. Access control restricts the forms of operations that may be performed on files by various classes of users and
programs.
databases
consist of one or more files whose contents are strongly (logically) related. The on-disk representation of the data is considered optimal with respect to its datatypes and its expected methods of access.
Organisation of File Systems, continued

Today, nearly all modern operating systems provide relatively simple file systems consisting of byte-addressable, sequential files.
The basic unit of access, the record from Stallings's taxonomy, is the byte, and access to these is simply keyed using the byte's numeric offset from the beginning of each file (the size of the variable holding this offset will clearly limit a file's extent).
The file-systems of modern operating systems, such as Linux's ext2, ext3, and ext4 systems and the Windows-NT File System (NTFS), support volumes with sizes up to 1 exbibyte (EiB) and files with sizes up to 16 tebibytes (TiB).
Operating Systems and Databases

More complex arrangements of data and its access modes, such as using search keys to locate individual fixed-length records, have been "relegated" to being supported by run-time libraries and database packages (see Linux's gdbm information).
In commercial systems, such as for a large eCommerce system, a database management system (DBMS) many store and provide access to database information independent of an operating system's representation of files.
The DBMS manages a whole (physical) disk-drive, or a dedicated partition, and effectively assumes much of the operating system's role in managing access, security, and backup facilities.
The File Management System

The increased simplicity of the file systems provided by modern operating systems has enabled a concentration on efficiency, security and constrained access, and robustness.
Operating systems provide a layer of system-level software, using system-calls, to provide services relating to the provision of files.
The consistent use of system-calls for this task, obviates the need for each application program to manage its own disk (space) allocation and access.
Moreover, using system-calls to enter the kernel permits the OS to use file-system access as an opportunity to schedule processes. As a kernel-level responsibility, the file management system again provides a central role in resource (buffer) allocation and process scheduling.
File-based activity System-calls
File creation, deletion open(), creat(), close(), truncate(), unlink()

Accessing a file's contents read(), write(), lseek()
Accessing a file's attributes chmod(), chown(), stat()
(Some) directory operations mkdir(), rmdir()
From the "viewpoint" of the operating system kernel itself, the file management system has a number of goals:
to support the storage, searching, and modification of user data,

to guarantee the correctness and integrity of the data,
to optimise both the overall throughput (from the operating system's global view) and response time (from the user's view).
to provide "transparent" access to many different device types such as hard disks, CD-ROMs, and portable devices (accessed by their file-system),
to provide a standardised set of I/O interface routines, perhaps ameliorating the concepts of file, device and network access.
User Requirements
A further critical requirement in a multi-tasking, multi-user system is that the file management system meet users' requirements in the presence of different users.
Subject to appropriate permissions, important requirements include:
the recording of a primary owner of each file (for access controls and accounting),
each file should be accessed by (at least) one symbolic name,
each user should be able to create, delete, read, and modify files,
users should have constrained access to the files of others,
a file's owner should be able to modify the types of access that may be performed by others,
a facility to copy/backup files to identical or different forms of media,
a secure logging and notification system if inappropriate access is attempted (or achieved).
Components of the File Management System

Stallings provides a generic depiction of the structure of the file management system:
File System Software Architecture
Its components, from the bottom up, include:
The device drivers communicate directly with I/O hardware. Device drivers commence I/O requests and receive asynchronous notification of their completion (via DMA).
The basic file system exchanges fixed-sized pieces (blocks) of data with the device drivers, buffers those blocks in main memory, and maps blocks to their physical location.
The I/O supervisor manages the choice of device, its scheduling and status, and allocates I/O buffers to processes.
The logical I/O layer maps I/O requests to blocks, and maintains ownership and access information about files.
The Role of Directory Structures

All modern operating systems have adopted the hierarchical directory model to represent collections of files.
A directory itself is typically a special type of file storing information about the files (and hence directories) it contains.
Although a directory is "owned" by a user, the directory is truly owned by the operating system itself. The operating system must constrain access to important, and often hidden, information in the directory itself.
Although the "owner" of a directory may examine and (attempt to) modify it, true modification is only permitted through OS system-calls affecting the internal directory structure.
For example, deleting a file involves both deallocating any disk blocks used by that file, and removing that file's information from its container directory.
If direct modification of the directory were possible, the file may become "unlinked" in the hierarchical structure, and thereafter be inaccessible (as it could not be named and found by name).
In general, each directory is stored as a simple file, containing the names of files within it. The directory's "contents" may be read with simple user-level routines (introduced in Lecture 17).
File Information Structures - inodes

Early operating systems maintained information about each file in the directory entries themselves. However, the use of the hierarchical directory model encourages a separate information structure for each file.
As an example, Unix refers to these information structures as inodes, accessing them by their integral inode number on each device (themselves referred to by two integral device numbers).
Each information structure is accessed by a unique key (unique to the I/O device), and the role of the directory is now simply to provide <filename, inode> pairs.
A distinct file-entry is identified by a <major-device-number, minor-device-number, inode> triple.
Typical information contained within each inode includes:
the file's type (plain file, directory, etc),

the file's size, in bytes and/or blocks,
any limits set on the file's size (implied?),
the primary owner of the file,
information about other potential users of this file,
access constraints on the owner and other users,
dates and times of creation, last access and last modification,
pointers to the file's actual data blocks.
File Information Structures - inodes, continued

As a direct consequence, multiple file names (even from different directories) may refer to the same file. These are termed file-system links.
Individual processes access open files though their file descriptor table;
this table indexes the kernel's global file table; and
that table indexes the device's inode table.
File Allocation Methods - Contiguous

Of particular interest is the method that the basic file system uses to place the logical file blocks onto the physical medium (for example, into the disk's tracks and sectors).
Contiguous File Allocation
The simplest policy (akin to memory partitioning in process management) is the use of a fixed or contiguous allocation. This requires that a file's maximum (ever) size be determined when the file is created, and that the file cannot grow beyond this limit (as in the above figure).
The file allocation table stores, for each file, its starting block and length.
Like simple memory allocation schemes, this method suffers from both internal fragmentation (if the initial allocation is too large) and external fragmentation (as files are deleted over time).
Again, as with memory allocation, a compaction scheme is required to reduce fragmentation ("defragging" your disk).
File Allocation Methods - Chained

The opposite extreme to contiguous allocation is chained allocation.
Here, the blocks allocated to a file form a linked list (or chain) and, as a file's length is extended (by appending to the file), a new block is allocated and linked to the last block in the file:
Chained File Allocation
A small "pointer" of typically 32 or 64 bits is allocated within each file data block to indicate the next block in the chain. Thus seeking within a file requires a read of each block to follow the pointers.
New blocks may be allocated from any free block on the disk. In particular, a file's blocks need no longer be contiguous.
File Allocation Methods - Indexed

The file allocation method of choice in both Unix and Windows is the indexed allocation method.
This method was championed by the Multics operating system in 1966. The file-allocation table contains a multi-level index for each file - just as we have seen in the use of inodes, which contain direct pointers to data blocks, and pointers to indirection blocks (which point to
more data blocks).
Indexed Allocation with Block Portions
Indirection blocks are introduced each time the total number of blocks "overflows" the previous index allocation.
Typically, the indices are neither stored with the file-allocation table nor with the file, and are retained in memory when the file is opened.
Further Reading on File Systems

Analysis of the Ext2fs structure, by Louis-Dominique Dubeau.
The FAT - File Allocation Table, by O'Reilly & Associates.
The two standard output streams - stdout and stderr

To date, most of our output has been produced using the function printf() which, by default, sends its output to the terminal screen.
Pedantically, printf() actually sends its output to the stdout (pronounced standard-output) stream which, by default, is connected to the screen, but may be redirected to a file or even to another program (using operating system features, not C features).
For example, on macOS and Linux:
prompt> mycc -o myprogram myprogram.c

prompt> myprogram # appears on the screen
prompt> myprogram > output.dat
prompt> myprogram | wc -l
Here, our program "doesn't care" where its output is going - to the default location (the screen), to a file, or through a pipe.
In our programs, we could choose to explicitly send to the stdout stream, instead of printf(), with:
fprintf(stdout, "a longhand mechanism, equivalent to calling printf\n");
stdout and stderr are two of the standard I/O streams that are created and initialized by the C runtime system when the execution of new C programs is commenced.
In general, we prefer to write "normal" output to stdout, and errors to stderr, and we'll adopt this practice in the rest of the unit.
FILE *fp = fopen("results.data", "w");
if(fp == NULL) {
fprintf(stderr, "Cannot open results.data\n");
}
else {
fprintf(fp, .....);
.....
}
The standard perror() function (from Lecture-11) also sends its output to stderr.
CITS2002 Systems Programming, Lecture 19, p1, 7th October 2019.
Reading input via stdin

We'll commence using an improved, more general, main() function, accepting zero or more command-line arguments representing filenames. If no file names are provided, our programs will process input from the stdin (pronounced standard-input) stream.
This is standard mechanism by which a single program receives its input via a named file, the contents of a redirected file, or through a pipe. We describe such a program as a filter [see The Art of Unix Programming, ch07]:
#include <stdio.h>
// process() READS FROM A FILE-POINTER, REGARDLESS OF ITS SOURCE

int process(FILE *infp)
{
int result = 0;
....
return result;
}

{
int result;
printf("program's name is %s\n", argv[0]);

// NO ARGUMENTS PROVIDED, ACCEPT DEFAULT (STANDARD) INPUT
if(argc == 1) {
result = process(stdin);
}
else {
// FILE NAME ARGUMENTS PROVIDED, READ FROM EACH OF THEM
for(int a=1 ; a<argc ; ++a) {
FILE *infp = fopen(argv[a], "r");
if(infp == NULL) {
fprintf(stderr, "%s: cannot open %s\n", argv[0], argv[a]);
}
result = process(infp);
fclose(infp); // WE OPENED, SO WE CLOSE IT
}
}
return result;
}
File streams and file descriptors

File streams such as stdin, stdout, and stderr are features of the C99 language. They hide from a (portable) C99 program the underlying operating system details.
Unix-based operating systems provide file descriptors, simple integer values, to identify 'communication channels' - such as files, interprocess-communication pipes, (some) devices, and network connections (sockets).
C99 defines the FILE * datatype (the file-pointer) as an abstraction over file descriptors to manage (simplify?) access to descriptors.
(As we know) when calling C99 standard functions we provide, or receive, a file-pointer. When calling OS system-calls we provide, or receive, a file-descriptor.
operation OS file-descriptors C99 file-pointers

opening and creating int open(char *pathname, int flags) FILE *fopen(char *pathname, char *mode)
closing int close(int fd) int fclose(FILE *fp)
size_t read(int fd, void *buffer, size_t nbytes) size_t fread(void *ptr, size_t size, size_t n, FILE *fp)
int fscanf(FILE *fp, char *format, ...)
reading
char *fgets(char *array, int size, FILE *fp)
int fgetc(FILE *fp)
size_t write(int fd, void *buffer, size_t nbytes) size_t fwrite(void *ptr, size_t size, size_t n, FILE *fp)
int fprintf(FILE *fp, char *format, ...)
writing
int fputs(char *string, FILE *fp)
int fputc(int ch, FILE *fp)
long lseek(int fd, long offset, int wrt); int fseek(FILE *fp, long offset, int wrt)
positioning long ftell(FILE *fp)
void rewind(FILE *fp)
Buffered Input
When reading from a file, the code for using file pointers and file descriptors can be very similar.
However, the standard I/O streams perform input buffering - the data read from a file is actually read into a large memory buffer (owned by the user's process and 'remembered' by the file-pointer) and then 'dished out' to the requesting application:
Remember that the read() function is a system-call, and that system-calls can be expensive because it permits the operating system to reschedule the requesting process from the RUNNING state to the BLOCKED state if the data is not ready.

#include <stdlib.h> #include <fcntl.h>
#include <unistd.h> #include <stdlib.h>
#include <unistd.h>
#define MYSIZE 10000 #define MYSIZE 10000
void file_pointer_read(char *filename) void file_descriptor_read(char *filename)

{ {
size_t got; size_t got;
char buffer[MYSIZE]; char buffer[MYSIZE];
FILE *fp = fopen(filename, "r"); int fd = open(filename, O_RDONLY);

if(fp == NULL) { if(fd == -1) {
} }
while((got = fread(buffer,1,sizeof buffer,fp)) > 0) { while((got = read(fd, buffer, sizeof buffer)) > 0) {
...... .....
} }
fclose(fp); close(fd);
} }
Consider what may happen in the above code if the value of MYSIZE is not always 10000, but is 1, 10, 1000, or 100000.
Buffered Output
Similarly, a big distinction between stdout and stderr is that the former is buffered (for efficiency), while the latter is unbuffered (to ensure output appears immediately).
A consequence of this is that, if all of your output is sent to stdout and your program crashes, you may not see all of your output.
The standard command-line arguments

In all of our C programs to date we've seen the use of, but not fully explained, command-line arguments.
We've noticed that the main() function receives command-line arguments from its calling environment (usually the operating system):
#include <stdio.h>

{
printf("program's name: %s\n", argv[0]);
for(int a=0 ; a < argc ; ++a) {

printf("%i: %s\n", a, argv[a] );
}
return 0;
}
We know:
that argc provides the count of the number of arguments, and

that all programs receive at least one argument (the program's name).
But what is argv?

If we read argv's definition, from right to left, it's
"an array of pointers to characters"
Or, try cdecl.org
While we typically associate argv with strings, we remember that C doesn't innately support strings. It's only by convention or assumption that we may assume that each value of argv[i] is a pointer to something that we'll treat as a string.
In the previous example, we print "from" the pointer. Alternatively, we can print every character in the arguments:
#include <stdio.h>

{
for(int a=0 ; a < argc ; ++a) {
printf("%i: ", a);
for(int c=0 ; argv[a][c] != '\0' ; ++c) {

printf("%c", argv[a][c] );
}
printf("\n");
}
return 0;
}
The operating system actually makes argv much more usable, too:
each argument is guaranteed to be terminated by a null-byte (because they are strings), and
the argv array is guaranteed to be terminated by a NULL pointer.
Parsing command-line arguments

By convention, most applications support command-line arguments (commencing with a significant character) that appear between a program's name and the "true" arguments.
For programs on Unix-derived systems (such as Apple's macOS and Linux), these are termed command switches, and their introductory character is a hyphen, or minus.
Keep in mind, too, that many utilities appear to accept their command switches in (almost) any order. For the common ls program to list files, each of these is equivalent:
ls -l -t -r files
ls -lt -r files
ls -ltr files
ls -rtl files
Of note, neither the operating system nor the shell know the switches of each program, so it's up to every program to detect them, and report any problems.
Parsing command-line arguments, continued

Consider the following program, that accepts an optional command switch, -d, and then zero or more filenames:
#include <stdio.h>
#include <stdlib.h>
char *progname;
bool dflag = false;

{
progname = argv[0];
--argc; ++argv;
while(argc > 0 && (*argv)[0] == '-') { // or argv[0][0]

if((*argv)[1] == 'd') // or argv[0][1]
dflag = !dflag;
else
argc = 0;
--argc; ++argv;
}
if(argc < 0) {
fprintf(stderr, "Usage : %s [-d] [filename]\n", progname);
}
if(argc > 0) {
while(argc > 0) {
process_file(*argv); // provide filename to function
--argc; ++argv;
}
}
else {
process_file(NULL); // no filename, use stdin or stdout?
}
return 0;
}
Parsing command-line arguments with getopt

As programs become more complicated, they often accept many command switches to define and constrain their execution (something casually termed creeping featurism [1]).
In addition, command switches do not just indicate, or toggle, a Boolean attribute to the program, but often provide additional string values and numbers to further control the program.
The correct parsing of command switches can quickly become complicated!
To simplify the task of processing command switches in our programs, we'll use the function getopt().
getopt() is not a function in the Standard C library but, like the function strdup(), it is widely available and used. In fact, getopt() conforms to a different standard - an POSIX standard [2], which provides functions enabling operating system portability.
[1] UNIX Style, or cat -v Considered Harmful

[2] The IEEE Std 1003.1 (POSIX.1-2008)
Parsing command-line arguments with getopt, continued

Let's repeat the previous example using getopt:
#include <stdio.h>
#include <unistd.h>
#include <getopt.h>
#define OPTLIST "d"

{
int o p t ;
.....
opterr = 0;
while((opt = getopt(argc, argv, OPTLIST)) != -1) {
if(opt == 'd')
dflag = !dflag;
else
argc = -1;
}
if(argc < 0) {
fprintf(stderr, "Usage: %s [-d] [filename]\n", progname);
}
while(optind < argc) {
process( argv[optind] );
++optind;
}
.....
return 0;
}
Parsing command-line arguments with getopt

Let's repeat the previous example, but now support an additional command switch that provides a number as well. The getopt function is informed, through the OPTLIST character string, that an argument is expected after the new -n switch.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <getopt.h>
#define OPTLIST "df:n:"

{
int opt;
bool dflag = false;
char *filenm = NULL;
int value = DEFAULT_VALUE;
opterr = 0;
while((opt = getopt(argc, argv, OPTLIST)) != -1) {
// ACCEPT A BOOLEAN ARGUMENT
if(opt == 'd') {
dflag = !dflag;
}
// ACCEPT A STRING ARGUMENT
else if(opt == 'f') {
filenm = strdup(optarg);
}
// ACCEPT A INTEGER ARGUMENT
else if(opt == 'n') {
value = atoi(optarg);
}
// OOPS - AN UNKNOWN ARGUMENT
else {
argc = -1;
}
}
if(argc <= 0) { // display program's usage/help

usage(1);
}
argc -= optind;
argv += optind;
.....
return 0;
}
getopt sets the global pointer variable optarg to point to the actual value provided after -n - regardless of whether any spaces appear between the switch and the value.
Dynamic data structures

Initially, we focused on scalar and array variables, whose size is known at compile-time.
More recently, we've focused on arrays of values, whose required size was only known at run-time.
In the case of dynamic arrays we've used C99 functions such as:
malloc(), calloc(), realloc(), and free()

to manage the required storage for us.
An extension to this idea is the use of dynamic data structures - collections of data whose required size is not known until run-time. Again, we'll use C99's standard memory allocation functions whenever we require more memory.
However, unlike our use of realloc() to grow (or shrink) a single data structure (there, an array), we'll see two significant differences:
we'll manage a complete data structure by allocating and deallocating its "pieces", and
we'll keep all of the "pieces" linked together by including, in each piece, a "link" to other pieces.
To implement these ideas in C99, we'll develop data structures that contain pointers to other data structures.
All code examples in this lecture are available from here: examples.zip
A simple dynamic data structure - a stack

We'll commence with a simple stack - a data structure that maintains a simple list of items by adding new items, and removing existing items, from the head of the list.
Such a data structure is also termed a first-in-last-out data structure, a FILO, because the first item added to the stack is the last item removed from it (not the sort of sequence you want while queueing for a bank's ATM!).
Let's consider the appropriate type definition in C99:
typedef struct _s {
int value;
struct _s *next;
} STACKITEM;
STACKITEM *stack = NULL;
Of note:
we haven't really defined a stack datatype, but a single item that will "go into" the stack.
the datatype STACKITEM contains a pointer field, named next, that will point to another item in the stack.
we've defined a new type, a structure named _s, so that the pointer field next can be of a type that already exists.
we've defined a single pointer variable, named stack, that will point to a stack of items.
Adding items to our stack data structure

As a program's execution progresses, we'll need to add and remove data items from the data structure.
The need to do this is not known until run-time, and data (perhaps read from file) will determine how large our stack eventually grows.
As its name suggests, when we add items to our stack, we'll speak of pushing new items on the stack, and popping existing items from the stack, when removing them.
typedef struct _s { // same definition as before

int value;
struct _s *next;
} STACKITEM;
STACKITEM *stack = NULL;
....
void push_item(int newvalue)

{
STACKITEM *new = malloc( sizeof(STACKITEM) );
if(new == NULL) { // check for insufficient memory

perror( __func__ );
}
new->value = newvalue;
new->next = stack;
stack = new;
}
The functions push_item and pop_item are quite simple, but in each case we must worry about the case when the stack is empty.
We use a NULL pointer to represent the condition of the stack being empty.
Removing items from our stack data structure

The function pop_item now removes an item from the stack, and returns the actual data's value.
In this example, the data held in each STACKITEM is just a single integer, but it could involve several fields of data. In that case, we may need more complex functions to return all of the data (perhaps using a structure or pass-by-reference parameters to the pop_item function).
Again, we must ensure that we don't attempt to remove (pop) an item from an empty stack:
int pop_item(void)
{
STACKITEM *old;
int oldvalue;
if(stack == NULL) {
fprintf(stderr, "attempt to pop from an empty stack\n");
}
oldvalue = stack->value;
old = stack;
stack = stack->next;
free(old);
return oldvalue;
}
Printing our stack data structure

To print out our whole data structure, we can't just use a standard C99 function as C99 doesn't know/understand our data structure.
Thus we'll write our own function, print_stack, to traverse the stack and successively print each item, using printf.
Again, we must check for the case of the empty stack:
void print_stack(void)
{
STACKITEM *thisitem = stack;
while(thisitem != NULL) {
printf("%i", thisitem->value);
thisitem = thisitem->next;
if(thisitem != NULL)
printf(" -> ");
}
if(stack != NULL)
printf("\n");
}
Again, our stack is simple because each node only contains a single integer. If more complex, we may call a different function from within print_stack to perform the actual printing:
....
print_stack_item( thisitem );
Using our stack in a Reverse Polish Calculator

Let's employ our stack data structure to evaluate basic integer arithmetic, as if using a Reverse Polish Calculator.
Each integer read from lines of a file is pushed onto the stack, arithmetic operators pop 2 integers from the stack, perform some arithmetic, and push the result back onto the stack.
int evaluate_RPN(FILE *fp) # Our input data:

{ 12
char line[BUFSIZ]; 3
int val1, val2; add
5
while( fgets(line, sizeof(line), fp) != NULL ) { div
if(line[0] == '#')
continue;
if(isdigit(line[0]) || line[0] == '-')
push_item( atoi(line) );
else if(line[0] == 'a') {

val1 = pop_item();
val2 = pop_item();
push_item( val1 + val2 );
}
....
else if(line[0] == 'd') {
val1 = pop_item();
val2 = pop_item();
push_item( val2 / val1 );
}
else
break;
}
return pop_item();
}
Using our stack in a Reverse Polish Calculator, continued

Careful readers may have noticed that in some cases we don't actually need the integer variables val1 and val2.
We can use the 2 results returned from pop_item as arguments to push_item:
int evaluate_RPN(FILE *fp)

{
char line[BUFSIZ];
while( fgets(line, sizeof line, fp) != NULL ) {

if(line[0] == '#')
continue;
if(isdigit(line[0]) || line[0] == '-')
push_item( atoi(line) );
else if(line[0] == 'a') {

push_item( pop_item() + pop_item() );
}
....
}
return pop_item();
}

{
printf("%i\n", evaluate_RPN( stdin ) );
return 0;
}
Problems with our stack data structure

As written, our stack data structure works, but may be difficult to deploy in a large program.
In particular, the whole stack was represented by a single global pointer variable, and all functions accessed or modified that global variable.
What if our program required 2, or more, stacks?

What if the required number of stacks was determined at run-time?
Could the stacks be manipulated by functions that didn't actually "understand" the data they were manipulating ?
Ideally we'd re-write all of our functions, push_item, push_item, and print_stack so that they received the required stack as a parameter, and used or manipulated that stack.
Techniques on how, and why, to design and implement robust data structures are a focus of the unit CITS2200 Data Structures & Algorithms.
Declaring a list of items

Let's develop a similar data structure that, unlike the first-in-last-out (FILO) approach of the stack, provides first-in-first-out (FIFO) storage - much fairer for queueing at the ATM!
We term such a data structure a list, and its datatype declaration is very similar to our stack:
typedef struct _l {
char *string;
struct _l *next;
} LISTITEM;
LISTITEM *list = NULL;
As with the stack, we'll need to support empty lists, and will again employ a NULL pointer to represent it.
This time, each data item to be stored in the list is string, and we'll often term such a structure as "a list of strings".
Adding (appending) a new item to our list

When adding (appending) new items to our list, we need to be careful about the special (edge) cases:
the empty list, and

when adding items to the end:
void append_item(char *newstring)

{
if(list == NULL) { // append to an empty list
list = malloc( sizeof(LISTITEM) );
if(list == NULL) {
}
list->string = strdup(newstring);
list->next = NULL;
}
else { // append to an existing list
LISTITEM *p = list;
while(p->next != NULL) { // walk to the end of the list

p = p->next;
}
p->next = malloc( sizeof(LISTITEM) );
if(p->next == NULL) {
}
p = p->next; // append after the last item
p->string = strdup(newstring);
p->next = NULL;
}
}
Notice how we needed to traverse the whole list to locate its end.
Such traversal can become expensive (in time) for very long lists.
Removing an item from the head our list

Removing items from the head of our list, is much easier.
Of course, we again need to be careful about the case of the empty list:
char *remove_item(void)
{
LISTITEM *old = list;
char *string;
if(old == NULL) {
fprintf(stderr, "cannot remove item from an empty list\n");
}
list = list->next;
string = old->string;
free(old);
return string;
}
Notice that we return the string (data value) to the caller, and deallocate the old node that was at the head of the list.
We say that the caller now owns the storage required to hold the string - even though the caller did not initially allocate that storage.
It will be up to the caller to deallocate that memory when no longer required.

Failure to deallocate such memory can lead to memory leaks, that may eventually crash long running programs.
Problems with our list data structure

As written, our list data structure works, but also has a few problems:
Again, our list accessing functions use a single global variable.

What if our program required 2, or more, lists?
Continually searching for the end-of-list can become expensive.
Could the lists be manipulated by functions that didn't actually "understand" the data they were manipulating?
We'll address all of these by developing a similar first-in-first-out (FIFO) data structure, which we'll name a queue.
A general-purpose queue data structure

Let's develop a first-in-first-out (FIFO) data structure that queues (almost) arbitrary data.
We're hoping to address the main problems that were exhibited by the stack and list data structures:
We should be able to manage the data without knowing what it is.

We'd like operations, such as appending, to be independent of the number of items already stored.
Such (highly desirable) operations are performed in a constant-time.
typedef struct _e {
void *data;
size_t datalen;
struct _e *next;
} ELEMENT;
typedef struct {
ELEMENT *head;
ELEMENT *tail;
} QUEUE;
Of note:
We've introduced a new datatype, ELEMENT, to hold each individual item of data.
Because we don't require our functions to "understand" the data they're queueing, each element will just hold a void pointer to the data it's holding, and remember its length.
Our "traditional" datatype QUEUE now holds 2 pointers - one to the head of the list of items, one to the tail.
Creating a new queue

We'd like our large programs to have more than a single queue - thus we don't want a single, global, variable, and we don't know until run-time how many queues we'll require.
We thus need a function to allocate space for, and to initialize, a new queue:
QUEUE *queue_new(void) QUEUE *queue_new(void) // same outcome, often seen

{ {
QUEUE *q = malloc( sizeof(QUEUE) ); QUEUE *q = calloc( 1, sizeof(QUEUE) );
if(q == NULL) { if(q == NULL) {

perror( __func__ ); perror( __func__ );
} }
q->head = NULL; return q;
q->tail = NULL; }
return q;
}
....
QUEUE *people_queue = queue_new();
QUEUE *truck_queue = queue_new();
If we remember that:
the calloc function both allocates memory and sets all of its bytes to the zero-bit-pattern, and
that (most) C99 implementations represent the NULL pointer as the zero-bit-pattern,
then we appreciate the simplicity of allocating new items with calloc.

Deallocating space used by our queue

It's considered a good practice to always write a function that deallocates all space used in our own user-defined dynamic data structures.
In the case of our queue, we need to deallocate 3 things:
1. the memory required for the data in every element,

2. the memory required for every element,
3. the queue itself.
void queue_free(QUEUE *q)

{
ELEMENT *this, *save;
this = q->head;
while( this != NULL ) {
save = this;
this = this->next;
free(save->data);
free(save);
}
free(q);
}
QUEUE *my_queue = queue_new();

....
// use my local queue
....
queue_free( my_queue );
Adding (appending) new items to our queue

Finally, we'll considered adding new items to our queue.
Remember two of our objectives:
To quickly add items - we don't wish appending to a very long queue to be slow.
We achieve this by remembering where the tail of the queue is, and quickly adding to it without searching.
To be able to queue data that we don't "understand".

We achieve this by treating all data as "a block of bytes", allocating memory for it, copying it (as we're told its length), all without ever interpreting its contents.
Adding (appending) new items to our queue, continued
void queue_add(QUEUE *Q, void *data, size_t datalen)

{
ELEMENT *newelement;
// ALLOCATE MEMORY FOR A NEW ELEMENT

newelement = calloc(1, sizeof(ELEMENT));
if(newelement == NULL) {
}
// ALLOCATE MEMORY FOR THE DATA IN THE NEW ELEMENT

newelement->data = malloc(datalen);
if(newelement->data == NULL) {
}
// SAVE (COPY) THE UNKNOWN DATA INTO OUR NEW MEMORY

memcpy(newelement->data, data, datalen);
newelement->datalen = datalen;
newelement->next = NULL;
// APPEND THE NEW ELEMENT TO AN EMPTY LIST

if(q->head == NULL) {
q->head = newelement;
q->tail = newelement;
}
// OR APPEND THE NEW ELEMENT TO THE TAIL OF THE LIST
else {
q->tail->next = newelement;
q->tail = newelement;
}
}
Writing a function to remove items from our queue, is left as a simple exercise.
Storing and searching ordered data - a binary tree

Each of the previous self-referential data-structures stored their values in their order of arrival, and accessed or removed them in the same order or the reverse. The actual time of insertion is immaterial, with the relative times 'embedded' in the order of the elements.
More common is to store data in a structure that embeds the relative magnitude or priority of the data. Doing so requires insertions to keep the data-structure ordered, but this makes searching much quicker as well.
Let's consider the type definition and insertion of data into a binary tree in C99:
typedef struct _bt { BINTREE *tree_insert(BINTREE *t, int value)

int value; {
struct _bt *left; if(t == NULL) {
struct _bt *right; BINTREE *new = calloc(1, sizeof(BINTREE));
} BINTREE;
if(new == NULL) {
BINTREE *tree_root = NULL; perror( __func__ );
}
new->value = value;
// the calloc() call has set both left and right to NULL
return new;
}
int order = (t->value - value);
if(order > 0) {
t->left = tree_insert(t->left, value);
}
else if(order < 0) {
t->right = tree_insert(t->right, value);
}
return t;
}
Of note:
we've defined a data-structure containing two pointers to other instances of the data-structure.
the use of the struct _bt data type is temporary, and never used again.
here, each element of the data-structure, each node of the tree, holds a unique instance of a data value - here, a single integer - though it's very common to hold multiple data values.
we insert into the tree with:
tree_root = tree_insert(tree_root, new_value);
the (magnitude of the) integer data value embeds the order of the structure - elements with lesser integer values are stored 'below' and to the left of the current node, higher values to the right.
unlike some (more complicated) variants of the binary-tree, we've made no effort to keep the tree balanced. If we insert already sorted elements into the tree, the tree will degenerate into a list, with every node having either a NULL left or a NULL right pointer.
Storing and searching ordered data - a binary tree, continued

Knowing that we've built the binary tree to maintain an order of its elements, we exploit this property to find elements:
bool find_recursively(BINTREE *t, int wanted) bool find_iteratively(BINTREE *t, int wanted)
{ {
if(t != NULL) { while(t != NULL) {
int order = (t->value - wanted); int order = (t->value - wanted);
if(order == 0) { if(order == 0) {
return true; return true;
} }
else if(order > 0) { else if(order > 0) {
return find_recursively(t->left, wanted); t = t->left;
} }
else { else {
return find_recursively(t->right, wanted); t = t->right;
} }
} }
return false; return false;
} }
Of note:
we search for a value in the tree with:

bool found = find_recursively(tree_root, wanted_value);
we do not modify the tree when searching, we simply 'walk' over its elements, determining whether to go-left or go-right depending on the relative value of each element's data to the wanted value.
some (more complicated) variants of the binary-tree re-balance the tree by moving recently found values (their nodes) closer to the root of the tree in the hope that they'll be required again, soon.
if the required value if found, the searching functions return true; otherwise we keep walking the tree until we find the value or until we can no longer walk in the required direction (because either the left or the right pointer is NULL).
Integers and their representation

All data in digital computers is stored as a sequence of bits, with (modern) computers aggregating 8 bits to a byte.
An integer on a Pentium, for example, requires 4 bytes, or 32 bits.
Consider how these bits form bytes, and how integers are represented:
Produces the output:

#include <stdio.h> 0: 00000000000000000000000000000000 00000000 0x0000
#include <sys/types.h> 1: 00000000000000000000000000000001 00000001 0x0001
#include <sys/param.h> 2: 00000000000000000000000000000010 00000002 0x0002
3: 00000000000000000000000000000011 00000003 0x0003
char *binary(unsigned int x) 4: 00000000000000000000000000000100 00000004 0x0004
{ 5: 00000000000000000000000000000101 00000005 0x0005
#define NBITS ((sizeof x) * NBBY) 6: 00000000000000000000000000000110 00000006 0x0006
7: 00000000000000000000000000000111 00000007 0x0007
static char bits[NBITS+1]; 8: 00000000000000000000000000001000 00000010 0x0008
char * b = b i t s ; 9: 00000000000000000000000000001001 00000011 0x0009
10: 00000000000000000000000000001010 00000012 0x000a
for(int n=NBITS-1 ; n>=0 ; --n) 11: 00000000000000000000000000001011 00000013 0x000b
{ 12: 00000000000000000000000000001100 00000014 0x000c
13: 00000000000000000000000000001101 00000015 0x000d
* b = ((x & (1<<n)) ? '1' : '0');
14: 00000000000000000000000000001110 00000016 0x000e
+ + b ;
15: 00000000000000000000000000001111 00000017 0x000f
} 16: 00000000000000000000000000010000 00000020 0x0010
*b = '\0';
return bits;
}
void display(unsigned int x)

{
printf("%2u:", x); // in decimal, base 10
printf(" %s", binary(x)); // in binary, base 2
printf(" %08o", x); // in octal, base 8
printf(" 0x%04x", x); // in hexadecimal, base 16
printf("\n");
}

{
for(unsigned int x=0 ; x<=16 ; ++x)
{
display(x);
}
return 0;
}
The constant NBBY is not a C99 symbol, but operating systems provide it to give the number of bits per byte.
Bitwise operators in C99

Knowing that data, such as integers, are sequences of bits, we may use the multiple bits of an integer to represent multiple pieces of data in a single integer.
Firstly, let's consider a few of C's operations on bits and bit patterns. When used in combination, these provide some powerful facilities not readily supported in some other programming languages.
The bitwise operators
Name Example Result Description

bitwise-and 3&5 1 1 if both bits are 1.
bitwise-or 3|5 7 1 if either bit is 1.
exclusive-or (xor) 3 ^ 5 6 1 if both bits are different.
not ~3 -4 Inverts the bits.
3 << 2
left shift 12 Shifts the bits of n left p positions. Zero bits are shifted into the low-order positions.
n << p
5 >> 2
right shift 1 Shifts the bits of n right p positions. If n is a 2's complement signed number, the sign bit is shifted into the high-order positions.
n >> p
Note: Don't confuse &&, the short-circuit logical-and, with &, which is the less common bitwise-and.
While the bitwise-and can also be used with Boolean operands, this is extremely rare and is almost always a programming error.
Storing multiple items in an integer

The most frequently seen example of C's bitwise operators, is the use of the left-shift operator to store multiple "items" in a single 32-bit integer variable.
Consider the following example, using an unsigned 32-bit integer (uint32_t) to represent colours in an RGB format of 24 bits-per-pixel (24bpp).
typedef unsigned uint32_t RGBCOLOUR;
RGBCOLOUR set_rgb(char red, char green, char blue)

{
return (red << 16) + (green << 8) + blue;
}
....
RGBCOLOUR white = set_rgb(255, 255, 255);
RGBCOLOUR black = set_rgb( 0, 0, 0);
RGBCOLOUR skyblue = set_rgb(135, 206, 235);
RGBCOLOUR yellow = set_rgb(255, 255, 0);
Here, the left-shift operator is used to quickly multiply a small value by a constant power of two.
On most modern architectures, bit-shifting operations are considerably faster than the equivalent multiplications (use left-shifting) and divisions (use right-shifting).
If unfamiliar with RGB colour representations, you may like to read:
www.rapidtables.com/web/color/RGB_Color.htm
www.w3schools.com/colors/colors_rgb.asp
Extracting multiple items from an integer

Storing multiple bit-wise items in an integer is simple.
Recovery of the individual components is more difficult (to understand).
There are a wide variety of ways to undertake this:
char get_blue(RGBCOLOUR rgb)

{
return (rgb & 0xff);
}
char get_green(RGBCOLOUR rgb)

{
return ((rgb >> 8) & 0xff);
}
char get_red(RGBCOLOUR rgb)

{
return ((rgb >> 16) & 0xff);
}
Such code often mixes octal and hexadecimal values, and left- and right-shift operations. Fortunately, modern C compilers are able to generate the optimal machine-level code for their targeted architecture.
Extracting multiple items from an integer, continued

In efforts to extract even more speed from programming involving bit-wise operations, advanced programmers will often use C preprocessor macros, providing inline functions, to reduce the (tiny) cost of the function call overhead:
#define set_rgb(r, g, b) (((b) << 16) + ((g) << 8) + (r)))
#define get_blue(rgb) ((rgb ) & 0xff)

#define get_green(rgb) (((rgb) >> 8) & 0xff)
#define get_red(rgb) (((rgb) >> 16) & 0xff)
void RGB_to_greyscale(RGBCOLOUR *greypixels[], RGBCOLOUR *colourpixels[], int width, int height)

{
for(int w=0 ; w<width ; ++w) {
for(int h=0 ; h<height ; ++h) {
RGBCOLOUR thispixel = colourpixels[w][h];
uint32_t grey = (0.3 * get_red (thispixel)) +

(0.6 * get_green(thispixel)) +
(0.1 * get_blue (thispixel)) ;
greypixels[w][h] = set_rgb(grey, grey, grey);

}
}
}
This simple program - greyscale.c - shows the idea (but doesn't use the RGBCOLOUR datatype, from above), converting a PPM-format colour image (as input) to a greyscale one (as output), with the command:
prompt> ./greyscale < rarebudgie.ppm > output.ppm
Unix File and Directory Permissions

Recall that:
Like most modern systems, Unix arranges its file system in a hierarchical structure of files and directories. Directories may contain entries for files and other directories.
File entries contain the file's name, together with a pointer to a structure termed the inode (the information node?), which represents the details of the file.
These are the familiar items shown by the standard ls -l command:
drwxr-xr-x 11 chris staff 1024 Jul 29 20:29 WWW

-rw------- 1 chris chris 53436 Aug 1 16:28 autonomous.pdf
-rw-r--r-- 1 chris chris 88 Dec 20 2016 scrolldown.gif
Multiple file entries, from possibly different directories, may point to the same inode. Thus, it is possible to have a single file with multiple names - we say that the names are links to the same file.
Of note, one 32-bit integer field in each inode contains the file's permission mode bits.
From history, the permission mode bits appear in the same integer defining the file's type (regular, directory, block device, socket, ...) -
See man 2 stat for details.
Octal Modes
You may also like to try the Unix Permissions Calculator.

Unix File and Directory Permissions, continued

When access requests are made by a process on behalf of a subject (a user) for an object (a file), the Unix kernel compares the effective user- and group-id attributes of the process against the permission mode bits of the file.
Of note, if the owner's permission bits of a file or directory are not set, then the owner cannot access the object by virtue of the 'group' or 'other' bits (can you think why?).
The inode structure also contains indication of the object's setuid and setgid status, together with a sticky bit having an overloaded meaning (historically, setting the sticky bit on an executable file requested that it not be swapped out of memory - requiring privilege to set the bit).
On different variants of Unix/Linux the permission mode bits, in combination, have some obscure meanings:
having execute access, but not read access, to a directory still permits an attacker to 'guess' filenames therein,
having the sticky bit set on a directory permits only the owner of a file, therein, to remove or modify the file,
having the setgid bit set on a directory means that files created in the directory receive the groupid of the directory, and not of their creator (owner).
A system administrator managing different operating systems (Unix/Linux, macOS, many flavours of Windows) needs be aware of these subtle differences.
Setting arbitrary bits in integers

The previous examples, using RGB colour values, only set individual bytes of the integers being considered.
Even more common is the setting and testing of individual bits in data, using each bit as if it were a complete Boolean value.
Consider the case of files in a file-system. We have already seen how we can determine if a directory entry is another directory or a file:
#include <sys/stat.h>
#include <unistd.h>
.....
struct stat stat_buffer;
if(stat(pathname, &stat_buffer) != 0)
{
perror( progname );
}
else if( S_ISDIR( stat_buffer.st_mode ))
{
printf( "%s is a directory\n", pathname );
}
else if( S_ISREG( stat_buffer.st_mode ))
{
printf( "%s is a file\n", pathname );
}
else
{
printf( "%s is unknown!\n", pathname );
}
How is this being achieved?

Setting arbitrary bits in integers, continued

As is typical of many operating system activities, information such as a file's filetype and access permissions are stored using multiple bits in an integer.
Individual bits in this integer are then used to represent a variety of possible combinations of permissible values.
These may be located by "wading" through a number of system-specific files in /usr/include :
// DEFINE A "NEW" TYPE TO REPRESENT A FILE'S MODE

typedef unsigned int mode_t;
struct stat {
....
mode_t st_mode;
....
}
#define S_IFMT 0170000 // THESE BITS DETERMINE FILE TYPE
// File types
#define S_IFDIR 0040000 // DIRECTORY
....
#define S_IFREG 0100000 // REGULAR FILE
// MACROS FOR TESTING FOR DIFFERENT FILE TYPES

#define S_ISDIR(mode) (((mode) & S_IFMT) == (S_IFDIR))
....
#define S_ISREG(mode) (((mode) & S_IFMT) == (S_IFREG))
This C program may better explain these concepts: showmode.c

Default Unix File Permissions and umask

Applications that create new directories or files can only do so by invoking the open() or creat() system calls, and will often use:
wide-open permissions, 0666 for files, 0777 for directories, or

closed permissions, 0600 for files, 0700 for directories
While the latter is preferable (because someone has to consciously issue chmod commands to make the files more accessible), most applications leave newly created files accessible by anyone.
Each Unix process also carries with it an integer attribute termed its umask, which is used by open() and creat() to set initial file permissions on newly-created files.
When new files or directories are created, their initial permissions are AND-ed with the Boolean-complement of the umask. Specifically, permissions set in the umask are turned off from the mode argument to open() or creat().
With an 'open' umask of 000, initial file and directory permissions are not modified on creation, and
With a umask of 022, write-permission is masked out for the 'group' and 'other' fields,
With a 'closed' umask of 077, all read, write, and execute permissions are disabled for the 'group' and 'other' fields.
It is important that a reasonable umask, ideally 077, is set upon login; the umask value is then inherited by all subprocesses spawned from the initial process (shell).
Additional file protections

Some Unix variants derived from BSD, and Linux ext2 (and later) file-systems, provide some additional file protection mechanisms that can enhance a system's robustness and security.
On macOS see: man chflags (in sections 1 and 2), and

on Linux see: man lsattr and man chattr.
As an example, on Linux the following 'attribute bits', when set, have the following meanings:
'A' - do not update the file's access-time when accessed,

'a' - the file may only be opened for appending (writing),
'c' - compress files before writing them to disk,
'i' - the file is immutable (may not be modified),
's' - when deleted, the file's data blocks are zeroed,
'S' - the file's data blocks are synchronously written to disk,
'u' - deleted files are actually saved elsewhere in the file-system.
Some of these attribute bits may only be set by the superuser, some may not be "compiled-in" to the kernel at all.
Unfortunately, their values (and meaning) are not honoured by most (standard) file-system utilities such as when being backed-up, are implemented differently across file-system versions, and care must be taken to restore their modes.
As these attributes are not as well known as the traditional protections (displayed using ls) they can provide additional protection against naive attackers - or greater challenges for naive administrators.
Users and their Operating System representation

Traditionally, an operating system positions itself directly between running processes and the hardware, and only permits information about the hardware, and software's queries and requests to pass through the kernel.
System calls are (supposed to be) the only mechanism by which processes may interact with the operating system and the resources it is protecting and managing (i.e. a process may not randomly read nor write the memory of the kernel, or of another process, itself).
Unix has adopted the approach that processes are the only active entities on a system (i.e. devices are passive). Processes act on the behalf of users, while accessing resources in a controlled fashion.
If these two approaches have been successful, we can study the basic security aspects of an operating system by examining:
the state of processes at runtime,

how a process's state may be changed,
how a process represents a user,
the mechanisms used to protect resources.
User IDs and system calls

Users are represented within the Unix kernel by 16-bit unsigned user identifiers, or userids (there are recent changes to 32-bits):
Operating systems support integral values for userids in preference to strings for their fixed length and their speed of comparison.
Although there may only be 216 possible values, even modern variants of Unix do not use the userid as an index into any kernel tables. Moreover, no userid is considered invalid.
The user-identifier is established when logging into a Unix system. A correct combination of user-name and password when logging in, or the validation of a network-based connection, set the user-identifier (uid) in the process control block of the user's login shell, or
command interpreter.
#include <stdio.h>
#include <unistd.h>
printf("user-id: %i\n", getuid() );
Unless modified, this user-identifier is inherited by all processes invoked from the initial login shell. Under certain conditions, the user-identifier may be changed and determined with the system calls setuid() and getuid().
Effective user IDs

A further property of each process is its effective user identifier:
The effective user-identifier is, by default, the same as the user-identifier, but may be temporarily changed to a different value to offer temporary privileges.
#include <stdio.h>
#include <unistd.h>
printf("real user-id: %i\n", getuid() );

printf("effective user-id: %i\n", geteuid() );
The successful invocation of set-user-id programs, such as passwd and login will, typically, set the effective user-identifier for the lifetime of that process.
Under certain conditions, the effective user-identifier may be changed and determined with the system calls seteuid() and geteuid().
The Unix kernel does not check userids

With the single exception of userid==0 (discussed later) the Unix kernel places no special significance of any userids.
In particular, because account- and user-management is not treated as an activity to the kernel, the kernel collapses users with the same userid to be a single user.
A frequently seen flaw is one where two (named) users have the same userid in the password file, or a user is incorrectly given a negative userid (such as a user named nobody with a userid of -2) on the assumption that all application software will consistently treat this as a
signed or unsigned integer.
User-level tools are required to check the consistency of password files.

An incorrect use of Unix userids?

Since its inception, Unix has also used distinct userids to represent activities or even programs, rather than 'real' users. Common examples include userids with names such as shutdown, mail, news, etc, typically with small userids.
Processes with these userids do not run with elevated privilege, but typically run in an environment where no human will notice them. These allocated userids do not invoke interactive sessions, (or are not supposed to!) and so no human operator is immediately notified if they do,
or if they try unsuccessfully several times.
Care must be taken to ensure that these 'users' do not have passwords, blank passwords, home directories, interactive login shells, etc. Typical approaches are to place an asterisk in their /etc/passwd password field, and set /sbin/nologin as their login shell.
Properties of the Unix Superuser

Unix uses the special userid value of 0 to represent its only special user, the superuser (or root) account.
Processes acting on behalf of the superuser can often access additional resources (often, incorrectly, stated as 'everything') because most system calls responsible for checking user permissions bypass their checks if being invoked with userid==0.
Unix is frequently criticized for both having a concept such as a

superuser, or for encouraging security practices which now rely on it.
It is thus the single greatest target of attack on a Unix system.

Properties of the Unix Superuser, continued

The result is that there appear to be no files, etc, that cannot be hidden from the superuser using the standard application programs which report and manipulate such resources.
For example, because the superuser can access all files and directories on a local system, it is simpler to use the superuser account/privilege to backup data volumes. It is cumbersome to direct the backup programs to not backup certain items.
Although the superuser has greater access to otherwise protected resources, the Unix kernel will not permit the superuser to undermine the integrity of the operating system itself. For example:
although the superuser can create a new file in any directory through a call to the open() or creat() system calls, and that details of this new file are written to the directory by the kernel itself, the superuser cannot open and explicitly write to the directory.
the superuser does not have the ability to write a file to a read-only file system, but can unmount the file system, re-mount it with an elevate read-write status, and then write the file.
Constraining a process's access with chroot

An additional approach to constraining how files and directories may be accessed, is to constrain how a process may access its resources.
Strictly speaking, this is not an alternative to setting file-access permissions, but may be used in combination with the previous techniques.
On behalf of each process, the kernel manages a current working directory, relative to which all file accesses are made, and a root directory 'above' which the process may not change directory, or open files.
The chroot system call changes the root directory of a calling process to be that specified by the function's argument.
This directory will be used for path names beginning with / and constrains traversal with ../../.. etc. The root directory attribute is inherited by all children of the current process.
Under normal execution, a process wishing to constrain the access of a subprocess would:
fork() a new (child) subprocess,

set the new root directory of the child, using chroot(),
execl() the required process.
Traditionally, only the super-user may change the root directory of the current process, and so the chroot application program exists with its setuid bit set to temporarily run as the superuser while making the system call.
Supporting Groups of Users

As well as the single userid of each of its process, each user is also represented by one of more group identifiers, or groupids.
groupids are similarly represented as 16-bit unsigned integral values. Each process maintains the primary groupid of its user (owner). The primary groupid for each user is initially located from the /etc/passwd file, typically when a user logs in.
chris:9HsKSemoJlxpA:333:110:Chris McDonald:/Users/chris:/bin/zsh
Under variants of Unix derived from the Berkeley (BSD) stream of Unix, each process also carries an array of 32 additional groupids, representing additional groups that the user is in. These are located from the /etc/group file, using the user-level library function
initgroups(), and set with the system call setgroups().
Variants of Unix derived from the AT&T System-V stream, support the newgrp command, which is similar to login in many respects but permits a process's primary group to be changed.
Additionally, the newgrp command permits an interactive process to change its primary group if it can provide the group's password (maintained in the /etc/group file).
Unix groups are used to constrain access to resources, such as files and some interprocess communication channels, to processes that must hold the required primary groupid, or (if BSD) hold it in its vector of groupids.
Supporting Groups of Users, continued

We recall that operating system kernels mostly deal with integer values, and not strings (file pathnames being an obvious exception).
Thus, while the operating system kernel will maintain group information (integers and arrays of integers) about users and their executing processes, additional information about these integers requires library functions (outside of the kernel).
For example, getgroups() is a system-call (with its documentation in section-2 of the online manual) to request all group-IDs of which the calling user-ID is a member.
We can then use the library function getgrgid() (with its documentation in section-3 of the online manual) to find information about each group-ID.
#include <stdio.h>
#include <unistd.h>
#include <grp.h>
#define MY_MAXGROUPS 100
void print_my_groups(void)
{
gid_t groups[MY_MAXGROUPS];
int ngroups = getgroups(MY_MAXGROUPS, groups);
for(int g=0 ; g<ngroups ; ++g) {

struct group *gp = getgrgid(groups[g]);
if(gp != NULL) {
printf("gid: %-8i name: %s\n", groups[g], gp->gr_name);
printf("\tmembers:");
for(int m=0 ; gp->gr_mem[m] != NULL ; ++m) {
printf(" %s", gp->gr_mem[m]);
}
printf("\n\n");
}
}
}
Changing and Setting User Information

As userids are used extensively to control and constrain a process's execution, and to provide possible auditing/logging of a user's activities, it is not possible for a non-superuser process to change its own userid.
If this were possible, an attacker could assume permissions of anyone!
However, there are occasions where a process must undertake an activity or temporarily have access to resources that it would not otherwise have. Examples include the passwd and newgrp programs already described. Each must have access to files containing passwords
(possibly shadowed versions of these), or require the ability to demonstrate that the activity is permitted.
In 1973, Dennis Ritchie, one of the original two designers of Unix filed a patent for the Unix set-uid mechanism:
US Patent 4,135,240: An improved arrangement for controlling access to data files by computer users. Access permission bits are used in the prior art to separately indicate permissions for the file owner and
nonowners to read, write and execute the file contents. An additional access control bit is added to each executable file. When this bit is set to one, the identification of the current user is changed to that of the owner
of the executable file. The program in the executable file then has access to all data files owned by the same owner. This change is temporary, the proper identification being restored when the program is
terminated.
In essence, program images are stored in files, and each file has an owner. Invoking a process, from a program file having its setuid bit set results in the (new) process executing with the privileges of the file's owner.
Changing and Setting User Information, continued

This gives us the ability for a process to run with another userid, of course provided that the 'owner' of that userid has established the appropriate permissions for the program's file.
chmod 4711 myprogram
or
chmod u+s,a+x myprogram
In effect, the owner of the (different) permissions is not explicitly changing the userid of a running process, but is permitting a (new) process to run with their permissions. Examples for its use include letting another user open a specific data file only by using your setuid program.
This facility takes on special significance when the owner of the program's file is the superuser -
any files then run as 'setuid root' run with the elevated permissions assigned to the superuser. (Hopefully) in all cases these are necessary:
-rws--x--x 1 root root 4764 Apr 2 07:26 /usr/bin/newgrp

-r-s--x--x 1 root root 15104 Mar 14 09:44 /usr/bin/passwd
setuid executable files, particularly if owned by root, are particular points of potential vulnerability on a Unix system, and regular checks with programs such as find should be made to locate (and justify) their existence.
A similarly extended execution state may be used with set-groupid permission mode bits on program files. Processes may execute as if the user running them were in the additional Unix group. This may provide access to resources that were otherwise constrained to the
particular group.
What have we learnt about C?

C is a good general purpose language (providing the necessary features to define and store data, perform calculations, and provide control-flow to sequence our programs), and an excellent systems programming language (providing well-defined and well-documented
interfaces to system-level APIs for system-calls and support functions).
C is a procedural programming language, not an object-oriented language like Java, Objective-C, or C#.
C programs can be described as "good" programs, if they are:
well designed,
clearly written and (hence) to read,
well documented,
use high level programming practices, and
well tested.
Of course, the above properties are independent of C, and are offered by many high level languages.
C has programming features provided by most procedural programming languages - strongly typed variables, constants, standard (or base) datatypes, enumerated types, user-defined types, aggregate structures, standard control flow, recursion, and program
modularization.
C does not offer tuples or sets, Java's concept of classes or objects, nested functions, subrange types, and has only recently added a Boolean datatype.
C does have, however, separate compilation, conditional compilation, bitwise operators, pointer arithmetic, and language independent input and output.
CITS2002 Systems Programming, Lecture 23, p1, 21st October 2019.
Why do we teach C?
small, imperative, procedural language
available on very wide-variety of hardware architectures
well defined by accessible standards
well supported by 3rd-party open-source and proprietary libraries
three contemporary compiler suites on three contemporary platforms
used to implement most other programming languages and libraries
software bridges to, and connecting, other programming languages

Features of C common to most programming languages

base data types - integers, characters, floating-point
conditions (evaluated as integers, 0 is false, anything other is true)
expressions with type promotion
control flow - if/else, switch; for, while, do...while loops
aggregate data-structures; one- and multi-dimensional arrays, structures
functions, accepting parameters, return value and pointer parameters

Less common C features supporting systems programming

systems programming demands speed and consistency
close mapping to underlying processor instruction set
representation/access to hardware characteristics, fixed-sized data objects
permits direct access to process's own memory through pointers

What's (still) missing from the C language?

During this unit, you will have naturally felt that the C language is missing some essential features for writing general programs, even general systems programs.
For example, we have seen that C does not provide features for graphics, networking, cryptography, or multimedia. Does this mean that C cannot be used to implement massive online multi-player games?
At one level, you probably appreciate that C doesn't have all these features (to learn). Instead, it is hoped that you have developed an appreciation that the "missing" features are provided by operating systems, other programs, 3rd-party libraries, and even bindings between C
and other programming languages.
C permits, enables, and encourages additional 3rd-party libraries (both open-source and commercial) to provide these facilities. The reason for these "omissions" is that C rigorously defines what it does provide, and rigorously defines how C must interact with external libraries.
We have previously listed some well-respected 3rd-party libraries, frequently employed in large C programs:

From this unit, it is hoped that you are now confident to investigate the use of these external libraries, be able to search for and read their documentation, and know how to incorporate their header files, functions, and libraries into your own projects.
The role and responsibilities of an OS

manage, arbitrate hardware resources - CPU, memory, devices
representation of processes, their resources, and state
efficient process switching and context switching
impose limits and accounting of resource usage
support exclusive access to resources
permit sharing of resources when requested
permit requests to obtain, increase, or relinquish resource ownership

The interface between an OS and programming languages

well-defined, orthogonal set of system-calls
system-calls receive parameters (typically integers and pointers)
system-calls return results through integer status and pointers
fixed-sized identifiers and parameters handled by the kernel
variable-sized objects handled by user-level libraries
collections - processes, memory, files, directories, users, networking
consistent use of file-descriptors for all 'types' of files and devices

C programs as good citizens

consistently indicate their success or failure with their exit status
consistenly accept command-line arguments (options, then data)
may receive more persistent configuration via environment variables
consistent representation of system-specific errors and reporting of error messages
consistent default data streams, stdin, stdout, stderr


Cits2002 Compressed PDF

Uploaded by

Copyright:

Available Formats

Cits2002 Compressed PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cits2002 Compressed PDF

Uploaded by

Copyright:

Available Formats

CITS2002 Systems Programming

CITS2002 CITS2002 schedule

Welcome to CITS2002 Systems Programming

Our UWA Handbook entry

CITS2002 Systems Programming, Lecture 1, p1, 29th July 2019.

Topics to be covered in CITS2002 Systems Programming

An introduction to the ISO-C99 programming language

An introduction to Operating Systems

An overview of computer hardware components

C programs in greater detail

Operating system services

Files and their use in programs

Here is our unit's schedule.

Is C still relevant? [ref: Tiobe survey]

Other interesting surveys:

Stackoverflow's Developer Survey Results 2019

C can be correctly described as a general purpose programming language -

C programs can be "good" programs, if they are:

A Systems Programming Language

Portability on different architectures

from 4-bit and 8-bit microcontrollers,

Compilers have been developed for:

All it requires is a ported C compiler

What about assembly languages?

The unreadability of C programs

Criticisms of C's execution model

C provides no runtime protection against arithmetic errors.

C does not hold the hand of lazy programmers.

What is the best programming language?

The important question is:

This unit will answer the questions:

"when is C the best language to use?" and

The Standardization of C - K&R C

CITS2002 Systems Programming, Lecture 1, p11, 29th July 2019.

The Standardization of C - ANSI-C (K&R-2)

CITS2002 Systems Programming, Lecture 1, p12, 29th July 2019.

The Standardization of C - ANSI/ISO-C99 and ISO/IEC 9899:2011 (C11)

In this unit we will focus exclusively on ANSI/ISO-C99,

What Standardization Provides

the representation of C programs;

They do not specify:

What's (deliberately) missing from the C language?

Function domain 3rd-party libraries

CITS2002 Systems Programming, Lecture 1, p15, 29th July 2019.

CITS2002 CITS2002 schedule

The structure of C programs

Of note in this example:

CITS2002 Systems Programming, Lecture 2, p1, 2nd August 2019.

The structure of C programs, continued

Same program, but more to note:

CITS2002 Systems Programming, Lecture 2, p2, 2nd August 2019.

Compiling and linking our C programs

Naming our variables

C variable names are case sensitive, thus:

MYLIMIT, mylimit, Mylimit and MyLimit

are four different variable names.

turn_nuclear_reactor_coolant_on and turn_nuclear_reactor_coolant_off

Let's look quickly at some of C's base datatypes:

typename description, and an example of variable initialization