Hidden Gems:

UNIX Shared Libraries

Way better than PROBUILD.

A Few Words About The Speaker

Tom Bascom, Roaming DBA & Progress User since 1987 President, DBAppraise, LLC
Remote Database Management Service. Simplifying the job of Managing and Monitoring The Worlds Best Business Applications. tom@dbappraise.com

VP, White Star Software, LLC

Expert Consulting Services related to all aspects of Progress and OpenEdge. tom@wss.com

What is a UNIX Shared Library?

A collection of executable code and data which is loaded into memory separately from the launch of the main program (_progres). Progress 4GL has supported Windows DLLs since (at least) version 8. UNIX DLL support arrived with Progress version 9. Ill be using Linux as an example but the principles apply to all UNIX platforms.
More Info: http://www.dwheeler.com/program-library/Program-Library-HOWTO/t1.html

Why Would You Use One?

To share code from 3rd party, non-Progress, sources. To leverage proprietary internal libraries. To gain performance improvements from routines written in faster languages. To access low-level OS features that are not otherwise available.

/lib (and /lib64)

/lib> ls *.so* ld-2.4.so ld-linux.so.2 ld-lsb.so.2 ld-lsb.so.3 libacl.so.1 libacl.so.1.1.0 libanl-2.4.so libanl.so.1 libattr.so.1 libattr.so.1.1.0 libaudit.so.0 libaudit.so.0.0.0 libblkid.so.1 libblkid.so.1.0 libBrokenLocale-2.4.so libBrokenLocale.so.1 libbz2.so.1 libbz2.so.1.0.0 libc-2.4.so libcap.so.1 libcap.so.1.92 libcidn-2.4.so libcidn.so.1 libcom_err.so.2 libcom_err.so.2.1 libcrypt-2.4.so libcrypt.so.1 libc.so.6 libdevmapper.so libdevmapper.so.1.02 libdl-2.4.so libdl.so.2 libe2p.so.2 libe2p.so.2.3 libext2fs.so.2 libext2fs.so.2.4 libgcc_s.so.1 libhistory.so.5 libhistory.so.5.1 libm-2.4.so libmemusage.so libm.so.6 libncurses.so.5 libncurses.so.5.5 libnscd.so.1 libnscd.so.1.0.0 libnsl-2.4.so libnsl.so.1 libnss_compat-2.4.so libnss_compat.so.2 libnss_dns-2.4.so libnss_dns.so.2 libnss_files-2.4.so libnss_files.so.2 libnss_hesiod-2.4.so libnss_hesiod.so.2 libnss_nis-2.4.so libnss_nisplus-2.4.so libnss_nisplus.so.2 libnss_nis.so.2 libnss_winbind.so.2 libnss_wins.so.2 libpamc.so.0 libpamc.so.0.81.0 libpam_misc.so.0 libpam_misc.so.0.81.2 libpam.so.0 libpam.so.0.81.5 libpcprofile.so libpthread-2.4.so libpthread.so.0 libreadline.so.5 libreadline.so.5.1 libresmgr.so libresmgr.so.0.9.8 libresmgr.so.1 libresmgr.so.1.0.0 libresolv-2.4.so libresolv.so.2 librt-2.4.so librt.so.1 libSegFault.so libss.so.2 libss.so.2.0 libsysfs.so.1 libsysfs.so.1.0.3 libthread_db-1.0.so libthread_db.so.1 libutil-2.4.so libutil.so.1 libuuid.so.1 libuuid.so.1.2 libwrap.so.0 libwrap.so.0.7.6 libxcrypt.so.1 libxcrypt.so.1.2.4 libz.so.1 libz.so.1.2.3

Can You Narrow That Down?

For backwards compatibility there are several versions of many libraries. Many libraries are niche libraries and not generally interesting. The most interesting (to me) are:
libc, the standard C library libm, the math library libz, the compression library libcrypt, the encryption library

What Functions are Inside?

$ nm -DC /lib/libz.so.1.2.3 | grep -v " _" | grep " T " 00000000000017b0 T adler32 0000000000001a70 T adler32_combine 0000000000001c10 T compress 0000000000001b40 T compress2 0000000000001b20 T compressBound 0000000000001c70 T crc32 0000000000002330 T crc32_combine 0000000000004220 T deflate 00000000000058b0 T deflateBound 0000000000005f70 T deflateCopy

A Few Words of Warning

The slides that you are about to see contain very little in the way of:
Error checking. Exception handling. Garbage collection. And other niceties.

While this may be customary for sample code it is very dangerous when working with real C code. You have been warned.

Calling Functions in libc

Problem: No OS-PUTENV()
NAME putenv - change or add an environment variable SYNOPSIS #include <stdlib.h> int putenv(char *string); DESCRIPTION The putenv() function adds or changes the value of environment variables. The argument string is of the form name=value. If name does not already exist in the environment, then string is added to the environment. If name does exist, then the value of name in the environment is changed to value. The string pointed to by string becomes part of the environment, so altering the string changes the environment. RETURN VALUE The putenv() function returns zero on success, or non-zero if an error occurs.

Calling putenv()
define variable x as integer no-undo. procedure putenv external "/lib64/libc.so.6": define input parameter env as character. define return parameter x as long. end. display os-getenv( "XYZZY" ). pause. run putenv( "XYZZY=pflugh", output x ). display os-getenv( "XYZZY" ). os-command value( 'echo "$XYZZY"' ). return.

Calling putenv()
? Press space bar to continue.

? pflugh
Press space bar to continue. pflugh

Press space bar to continue.

Problem: No Scientific Notation

NAME sprintf - formatted string output SYNOPSIS int sprintf( char *str, const char *format, ... ); THE CONVERSION SPECIFIER A character that specifies the type of conversion to be applied. The conversion specifiers and their meanings are: e,E The double argument is rounded and converted in the style [-]d.dddedd where there is one digit before the decimal-point character and the number of digits after it is equal to the precision; if the precision is missing, it is taken as 6; if the precision is zero, no decimal-point character appears. An E conversion uses the letter E (rather than e) to introduce the exponent. The exponent always contains at least two digits; if the value is zero, the exponent is 00.

Calling sprintf()
define variable x as integer no-undo. define variable c as memptr no-undo. define variable m as memptr no-undo. procedure sprintf external "/lib64/libc.so.6": define input-output parameter fStr as memptr. define input parameter mask as memptr. define input parameter arg as double. define return parameter x as long. end. set-size( c ) = 1024. set-size( m ) = 1024. put-string( m, 1 ) = "%1.4e". run sprintf( input-output c, m, 0.0123, output x ). display get-string( c, 1 ) format x(20). return.

Calling sprintf()
1.2300e-02

Procedure complete. Press space bar to continue.

Data Type Mapping

C Data Type char short unsigned short long int Procedure Parameter Definition BYTE SHORT UNSIGNED SHORT LONG LONG

float
double char* (read-only) char* (writeable) pointer to

FLOAT
DOUBLE CHARACTER or MEMPTR MEMPTR HANDLE TO

Creating Your Own Shared Library

Overview
Youll need some C source code. Compile to object code. Run the Linker and create a Shared Library. Test with Progress.

Problem: Sub-second PAUSE

define variable i as integer no-undo. define variable s as integer no-undo. s = time. do i = 1 to 1000: display i. etime( yes ). do while etime < 500: end. end.

display string( time - s, hh:mm:ss" ).

return.

Whats Wrong With That?

$ ps -ef | grep [_]progres tom 21243 27534 1 17:16 pts/0 00:02:30 /usr/pro102a/bin/_progres -1 $ pro p shortsleep.p i 1,000 08:20 $ ps -ef | grep [_]progres tom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1

8 minutes and 20 seconds of CPU time!

Source Code
#include <time.h>

long shortsleep( unsigned long milliseconds ) { struct timespec tmReq;

tmReq.tv_sec = (time_t)(milliseconds / 1000) ; tmReq.tv_nsec = (time_t)(milliseconds % 1000) * 1000 * 1000 ; nanosleep( &tmReq, NULL ) ; }

Compile to Object Code

$ cc c shortsleep.c -fPIC

$ ls shortsleep.c

shortleep.o

$ file shortleep.o ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped

Link to Create a Shared Lib

$ ld -shared shortsleep.o -o shortsleep.so

$ ls shortsleep* shortsleep.c shortsleep.o

shortsleep.so

$ file shortsleep.so shortsleep.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped $ nm -DC shortsleep.so shortsleep.so:0000000000100460 shortsleep.so:0000000000100460 shortsleep.so:0000000000100460 shortsleep.so: shortsleep.so:0000000000000280

A A A U T

__bss_start _edata _end nanosleep shortsleep

Progress 4GL Test Harness

define variable i as integer no-undo. define variable x as integer no-undo. define variable s as integer no-undo. procedure shortsleep external "./shortsleep.so": define input parameter ms as long. define return parameter x as long. end. s = time. do i = 1 to 1000: display i. run shortsleep( 500, output x ). end. display string( time - s, hh:mm:ss" ). return.

Test With Progress

$ pro p shortsleep.p Could not open Dynamic Library: ./shortsleep.so (8013) DLL Error : ./shortsleep.so: wrong ELF class: ELFCLASS32 (8014) Press space bar to continue. $ file $DLC/bin/_progres _progres: setuid ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), for GNU/Linux 2.4.0, not stripped

Test Again!
$ ps -ef | grep [_]progres tom 21243 27534 1 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1 $ pro p shortsleep.p i 1,000 08:20 $ ps -ef | grep [_]progres tom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1

No measurable CPU time!

Problem: Regular Expressions

NAME regcomp, regexec, regerror, regfree - POSIX regex functions SYNOPSIS #include <sys/types.h> #include <regex.h> int regcomp( regex_t *preg, const char *regex, int cflags ); int regexec( const regex_t *preg, const char *string, size_t nmatch, regmatch_t pmatch[], int eflags ); size_t regerror( int errcode, const regex_t *preg, char *errbuf, size_t errbuf_size ); void regfree( regex_t *preg );

Source Code
#include <regex.h>

long regex_match( char *pattern, char *string ) {

regex_t size_t regmatch_t int xpattern; matchDepth = 2; pmatch[2]; offset;

if ( regcomp(&xpattern, pattern, REG_EXTENDED|REG_ICASE ) != 0 ) { offset = -2; } else { if ( regexec(&xpattern, string, matchDepth, pmatch, 0 ) != 0 ) { offset = -1; } else { offset = pmatch[1].rm_so; } } regfree( &xpattern ); return offset; }

Compile and Link

$ cc c rx2.c -fPIC $ ld shared rx2.o o rx2.so $ file rx2.so rx2.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped $ nm -DC rx2.so 00000000001004e8 A __bss_start 00000000001004e8 A _edata 00000000001004e8 A _end U regcomp U regexec 0000000000000318 T regex_match U regfree

Progress 4GL Test Harness

define variable pattern as character no-undo format "x(60)". define variable string as character no-undo view-as editor inner-lines 10 inner-chars 60. define variable x as integer no-undo. procedure regex_match external "./rx2.so": define input parameter pattern as character. define input parameter string as character. define return parameter x as long. end. update pattern skip string skip with side-labels. run regex_match( pattern, string, output x ). display x. return.

Test!
$ pro p rx.p pattern: .*(Fox).* string: The quick brown fox jumped over the lazy dog! x: 16

Shared Library Challenges

Which Library to Use?

procedure putenv external "/lib64/libc.so.6": define input parameter env as character. define return parameter x as long. end.

Use a variable? NOT! Use a pre-processor. Use symbolic links. Wait for 10.2B.

Variable Argument Lists

int sprintf( char *str, const char *format, ... );

Isolate different cases in multiple .p files. Use wrapper libraries. Wait for 10.2B.

Call-Backs?
void qsort( void *base, size_t nmemb, size_t size, int(*compar)(const void *, const void *) );

Not supported .

Persistence
procedure putenv external "/lib64/libc.so.6" PERSISTENT: define input parameter env as character. define return parameter x as long. end.

About 25% faster in crude testing. RELEASE EXTERNAL /lib64/libc.so.6.

MEMPTRs
define variable x as integer no-undo. define variable c as memptr no-undo. define variable m as memptr no-undo. procedure sprintf external "/lib64/libc.so.6": define input-output parameter fStr as memptr. define input parameter mask as memptr. define input parameter arg as double. define return parameter x as long. end. set-size( c ) = 1024. set-size( m ) = 1024. put-string( m, 1 ) = "%1.4e". run sprintf( input-output c, m, 0.0123, output x ). display get-string( c, 1 ) format x(20). set-size( c ) = 0. set-size( m ) = 0. return.

Dynamic Shared Library Call

(Special Bonus Slide!)

Dynamic Shared Lib Call

define variable LIBC as character no-undo initial "/lib64/libc.so.6". function os-putenv returns integer ( input env as character ): define variable hCall as handle no-undo. define variable rVal as integer no-undo. create call hCall. assign hCall:call-name = "putenv" hCall:library = LIBC hCall:call-type = DLL-CALL-TYPE hCall:num-parameters = 1 hCall:return-value-dll-type = "LONG" . hCall:set-parameter( 1, "CHARACTER", "INPUT", env ). hCall:invoke(). rVal = hCall:return-value. delete object hCall. return rVal. end.

Conclusion
Share code from 3rd party, non-Progress, sources. Gain performance improvements from routines written in faster languages. Access low-level OS features that are not otherwise available. It beats the heck out of using PROBUILD.

Questions?

Thank-you!

dlopen() and Friends

NAME dladdr, dlclose, dlerror, dlopen, dlsym, dlvsym - programming interface to dynamic linking loader SYNOPSIS #include <dlfcn.h>

void *dlopen( const char *filename, int flag ); char *dlerror( void ); void *dlsym( void *handle, const char *symbol ); int dlclose( void *handle ); DESCRIPTION The four functions dlopen(), dlsym(), dlclose(), dlerror() implement the interface to the dynamic linking loader.