struct module in Python

The struct module in Python allows you to work with binary data by providing functionality to convert between Python values and C-style binary data. This is particularly useful when dealing with binary file formats or network protocols. It's key features include:

• Packing convert Python values into binary data (bytes).
• Unpacking convert binary data back into Python values.
• Format Strings define how data is packed/unpacked using format codes (e.g., i for integers, f for floats).

Methods in struct.pack()

1.Struct.pack(): It converts Python values into a packed binary format. The format string (fmt) specifies the layout of the packed data, and the subsequent values (v1, v2, ...) are packed according to this format. Syntax:

struct.pack(fmt, v1, v2, ...)

• fmt: A format string that specifies how the data will be packed.
• v1, v2, ...: The values that will be packed according to the specified format.

import struct

# pack values into binary
var = struct.pack('hhl', 1, 2, 3)
print(var)

var = struct.pack('iii', 1, 2, 3)
print(var)

b'\x01\x00\x02\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'
b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00'

Explanation: 'hhl' means two short integers (h, 2 bytes each) followed by a long (l, usually 4 or 8 bytes depending on platform). 'iii' packs three 4-byte integers. The output is in bytes (b''), representing the binary encoding of the values.

2.struct.unpack(): It convert packed binary data back into Python values. It takes a format string (fmt) and a packed binary string and returns a tuple of unpacked values. Syntax:

struct.unpack(fmt, string)

• fmt: A format string that specifies how the data should be unpacked.
• string: The packed binary data that needs to be unpacked.

import struct

var = struct.pack('?hil', True, 2, 5, 445)
print(var)

tup = struct.unpack('?hil', var)
print(tup)

var = struct.pack('qf', 5, 2.3)
print(var)

tup = struct.unpack('qf', var)
print(tup)

b'\x01\x00\x02\x00\x05\x00\x00\x00\xbd\x01\x00\x00\x00\x00\x00\x00'
(True, 2, 5, 445)
b'\x05\x00\x00\x00\x00\x00\x00\x0033\x13@'
(5, 2.299999952316284)

Explanation: This example first packs a boolean (?), a short (h), an integer (i), and a long (l) into bytes. Then, struct.unpack() is used to convert it back into Python values. The second part packs a long long integer (q) and a float (f), then unpacks them back. Note how 2.3 becomes 2.299999952... due to float precision.

3. struct.calcsize(): It returns the size (in bytes) of a struct corresponding to the format string. It is helpful for determining how much space is required to store packed data. Syntax:

struct.calcsize(fmt)

• fmt: A format string that specifies the data layout.

import struct

print(struct.calcsize('?hil'))
print(struct.calcsize('qf'))    

16
12

Explanation: '?hil' requires 16 bytes and 'qf' needs 12 bytes depending on alignment and platform.

4. struct.pack_into() and struct.unpack_from(): These methods allow you to directly pack and unpack data into/from a buffer starting at a given offset. These are particularly useful when dealing with pre-allocated memory buffers or when working with binary data stored in memory.

Syntax for struct.pack_into():

struct.pack_into(fmt, buffer, offset, v1, v2, ...)

• fmt: A format string specifying the data layout.
• buffer: A writable buffer (e.g., ctypes.create_string_buffer).
• offset: The starting position in the buffer where packing begins.
• v1, v2, ...: The values to be packed into the buffer.

Syntax for struct.unpack_from():

struct.unpack_from(fmt, buffer, offset=0)

• fmt: A format string specifying the data layout.
• buffer: The buffer containing the packed data.
• offset: The starting position from where unpacking begins (optional)

import struct
import ctypes

# Allocate buffer
size = struct.calcsize('hhl')
buff = ctypes.create_string_buffer(size)

# Pack into buffer
struct.pack_into('hhl', buff, 0, 2, 2, 3)

# Unpack from buffer
res = struct.unpack_from('hhl', buff, 0)
print(res)

(2, 2, 3)

Explanation: Here, a buffer is created using ctypes. struct.pack_into() inserts the values into this buffer at the specified offset (0 in this case). struct.unpack_from() then reads the data back from the buffer.

Effect of format order

The order of format characters can change the packed output due to padding and alignment. This affects both the byte content and size of the result.

import struct

var = struct.pack('bi', 56, 0x12131415)
print(var)
print(struct.calcsize('bi'))

var = struct.pack('ib', 0x12131415, 56)
print(var)
print(struct.calcsize('ib'))

b'8\x00\x00\x00\x15\x14\x13\x12'
8
b'\x15\x14\x13\x128'
5

Explanation: 'bi' (byte, int) might include padding after the byte, whereas 'ib' (int, byte) doesn’t need it. The size difference (8 vs 5) shows how alignment affects the memory layout.

Handling errors

If the wrong data type is used with struct.pack(), a struct.error occurs. Use try-except to handle such cases safely.

import struct

try:
    struct.pack('h', 'invalid')  # Wrong type, 'invalid' is a string, but 'h' expects an integer
except struct.error as e:
    print(f"Struct Error: {e}")

Output

Struct Error: required argument is not an integer

Explanation: This shows error handling when using struct. 'h' expects a short integer, but a string ('invalid') is given, causing a struct.error. The try-except block captures the error and prints a meaningful message.

Reference https://docs.python.org/2/library/struct.html