Select Numpy C API version

[mattiapenati]

This PR introduces the build configuration to select the target Numpy C API version and adjusts struct definitions to match the user selected version.

The default configuration has been chosen to match the setting in the first release of Numpy v2.

The only potentially breaking change is the new definition of PyUFuncObject. The struct PyUFuncObject has been updated to match the definition given in Numpy source. The previous definition matched the definition contained in documentation. I think the former is more accurate.

I understand that this PR is not small and that not all changes can be easily reviewed, it wasn't easy to put everything together either. If the PR is too large, I could split it into smaller changes. Anyway, I hope it will help improve the FFI.

[Icxolu]

Thanks for the PR and for looking into this. As I understand this tries to add version specific condition compilation similar to pyo3 itself, right? rust-numpy currently does this dynamically by reading the API_VERSION at runtime and dispatching based on that. For this reason the struct layouts have to be compatible with all supported versions at all times and we can't include fields introduced in newer versions based on compile time flags. I think the two approaches, at least as currently implemented, at not compatible to each other. I also think that the dynamic approach has the really nice property of leaving the version choice to the consumer. For example I could build an abi3 wheel across a large range of Python versions, some of which only have numpy v1 wheels and others only v2 wheel available. Currently this just works, I think this is a property worth keeping.

Additionally Python¹² (and numpy as well) is slowly moving towards a new stable abi build around opaque PyObjects, so I'm also not sure if it would be wise now to introduce a stronger dependency on PyObject layouts when in the future we probably want to move away from that.

So currently I'm inclined to say we should keep the current design.

cc @davidhewitt @ngoldbaum Maybe you have some opinions/comments for this one as well.

Footnotes

1. https://peps.python.org/pep-0803 ↩

2. https://peps.python.org/pep-0809/ ↩

[ngoldbaum]

I've personally been responsible for adding new structs and functions to the NumPy C API and it would be a shame if those functions can't be used by rust-numpy.

That said, this does introduce a significant amount of complexity. I also don't think it makes much sense to try to support very old versions of NumPy.

I would instead try to support only NumPy 1.25 and NumPy 2.1 and newer. I would also tend not to care much about supporting NumPy 1.x but I understand if that's important for people.

[ngoldbaum]

This was changed to dict only relatively recently in 2024, see numpy/numpy#27735. It was first available in NumPy 2.2.

[ngoldbaum]

This was changed to vectorcall in numpy/numpy#15271 which I believe was first included in NumPy 1.20.0.

[ngoldbaum]

Separately from the discussion in this PR, it makes sense to fix these incorrect bindings. Can you fix the issues I pointed out above and send in a PR that doesn't depend on details that changed in later NumPy versions? I think that means wrapping the fields that changed as *mut c_void but maybe @Icxolu has a way they'd prefer to see it spelled.

[mattiapenati]

Thank you for the feedback! I've updated this PR based on @ngoldbaum's comments. The definition of PyUFuncObject now conditionally matches NumPy's definitions from version 1.15 through version 2.4.

One thought: I believe supporting newer API versions would have a greater beneficial impact than having a support of wider set of versions. Additionally, the code could be significantly simplified if only newer versions of NumPy were supported, like >=1.25, but this choice is out of my responsibility.

Sorry @ngoldbaum, but I'm not sure I understand your last comment. Could you clarify what you mean by "details that changed in later NumPy versions"? The definition of PyUFuncObject on the main branch already matches the one in NumPy v1.15. Are you suggesting a separate PR that uses the same conditional compilation approach to implement only the changes of least recent versions of numpy?

[ngoldbaum]

Could you clarify what you mean by "details that changed in later NumPy versions"?

e.g. following what @Icxolu was saying and not having any conditional compilation or version-dependent bindings.

Another thing that I realized after writing my earlier comments is that NumPy itself, as of NumPy 2.0, supports building binaries that work with arbitrary older or newer numpy versions found at runtime. This is tricky and it requires a lot of conditional compilation in NumPy's headers. If it's possible to replicate that and allow setting something analogous to the NPY_FEATURE_VERSION and NPY_RUNTIME_VERSION macros. I suspect that might be tricky though.

[ngoldbaum]

@seberg knows more about how numpy handles forward and backward compatibility in its headers and might be interested in this thread too.

[mattiapenati]

Now I have a clearer understanding of what you meant. The mechanism you're referring to is the backward compatibility built into the NumPy API. Each version of NumPy exposes an API that remains compatible with older versions, and users can override the default by defining the NPY_TARGET_VERSION macro and updating pyproject.toml file.

This PR aims to replicate that mechanism. By default, the FFI exposed by rust-numpy targets NumPy v1.25, which is the same default used by NumPy v2.0. This may not be the ideal choice, and I'd welcome your input on what would work best. A set of features named target-npy### has been introduced to mirror the role of the NPY_TARGET_VERSION macro in NumPy. Users can opt into a newer API version by enabling one of these features and updating their pyproject.toml accordingly, or they can disable the default feature to target an older API version.

Consistency is ensured by the fact that all conditional compilation instructions have been written to mirror the conditionals involving the NPY_FEATURE_VERSION macro in the NumPy API.

[ngoldbaum]

I think there's no reason to try to support any numpy version in rust-numpy older than 1.25, so I'd suggest simplifying things here and removing the targets for older versions.

It's also not clear to me from the diff that you are handling forward and backward compatibility for different runtime NumPy versions like I think you just claimed - can you explain how you handle that? I'd think you'd need some functions to emulate the macros and static inline functions in the numpy headers. How are you handling the fact that the descriptor object, ufunc object and array object have different layouts in different numpy versions at runtime?

[ngoldbaum]

I would think you'd need some sort of automated testing where you build with one numpy version and run with another to validate all this.

[seberg]

@seberg knows more about how numpy handles forward and backward compatibility in its headers and might be interested in this thread too.

Some of these fields are a bit "why would you ever need them", and I guess those may be sloppy in NumPy (i.e. the dict is probably OK as it was NULL before, but just like vectorcall, you should use the Python APIs...).

The thing I think seems missing here is the runtime check (I am sure this already exists for the major C version, but that check needs to be expanded).
I.e. the vital thing with NPY_TARGET_VERSION (translated to NPY_FEATURE_VERSION in the headers) is that NumPy injects code to prevent running on older NumPy versions at runtime where the feature/field would not be valid.
(Yes, that can get in the way because it is tedious to conditionally use new features at runtime.)

Validating all fields seems rather tedious, but it seems important to try and validate the runtime version check. (Tedious because most of them just shouldn't be used and many that may be used are still niche to the point that my guess is there is maybe a single external user.)

(E.g. a good general candidate is the descr elsize Which I think here requires NumPy 2 or higher because to support both you would need to add runtime logic. NumPy does that with static inline functions in the npy2_compat.h.)

[Icxolu]

I think there's no reason to try to support any numpy version in rust-numpy older than 1.25, so I'd suggest simplifying things here and removing the targets for older versions.

We currently support the same Python versions as PyO3 itself. The latest numpy version for 3.7 is 1.21 and for 3.8 is is 1.24.
So if we increase the numpy target we effectively also drop support for 3.7 and 3.8 (which at some point we should, but so should pyo3 itself, see also PyO3/pyo3#4581).

Some of these fields are a bit "why would you ever need them", and I guess those may be sloppy in NumPy (i.e. the dict is probably OK as it was NULL before, but just like vectorcall, you should use the Python APIs...).

This is also something that I thought about. Some motivation for APIs that we want to build on top of these would be great. It would maybe that outline a more specific path to a design we could move towards.

It's also not clear to me from the diff that you are handling forward and backward compatibility for different runtime NumPy versions like I think you just claimed - can you explain how you handle that?

This is also unclear to me.

One way that I think could maybe work if we want too only add a few APIs that require only a small number of specific fields is to basically define both layouts like so:

#[repr(C)]
struct PyArray_Descr {
    ... // Like currently
}

#[repr(C)]
struct PyArray_DescrV2 {
    base: PyArray_Descr,
    ... // additional V2 fields
}

After a runtime version check we could then perform a pointer cast to access the fields or return an error for older numpy versions. But this blows up quite quickly if need a big range of versions and fields, so would be feasible for a small number of combinations.

[mattiapenati]

I spent some time studying the NumPy implementation more closely. This crate is implemented to match the NumPy ABI version 0x02000000, so looking the ABI, it is fully compatible with NumPy v1 and v2. A bit more attention is needed to verify the API compatibility. The chosen version range and the defaults are sufficient to keep the crate compatible with PyO3, as pointed out by @Icxolu.

In this PR, I updated the layout of the following three structures:

• PyArrayObject: All new fields are considered private, so I adjusted their visibility accordingly. I kept them in the definition because non-pointer instances of this type may be used in certain implementations, even if special care must be taken.
• PyArrayDescr: Non-pointer instances of this type should not be used, and new fields should be accessed with accessor functions, so I adjusted the visibility accordingly. Most accessor functions for these fields are already available in this crate, with the exception of PyDataType_GetArrFuncs, which I added in the latest changes.
• PyUFuncObject: This is the most complex of the three. Non-pointer instances should likewise not be used. The ufunc API is already available and it should be used to manipulate values of this type. I'm not sure about the change of visibility of fields.

I believe the changes to structures look good, since all accessor functions to ensure compatibility seem already available.

But I don't consider this PR ready to merge yet, since:

• As suggested by @seberg, the import mechanism of the API should be updated to support runtime check, like the one of NumPy. I plan to implement this change in this week.
• As suggested by @ngoldbaum, automatic tests should be added to check if everything works properly. This is harder to implement, but I will do my best.

I've found this discussion very valuable, thank you all!

[mattiapenati]

I think this PR should be closed without merging. Looking more closely at the code of this crate, I found that it is not ready to support multiple runtime versions in the way I implemented in this PR.

Currently, NumPy supports two different versions of the ABI (v1 and v2). Everything compiled targeting v1 cannot work on v2, a RuntimeError is raised on import. Conversely, if something is compiled targeting v2, it could work with v1, provided that the chosen runtime version matches the available NumPy package. These checks are not implemented in this crate in the same way. Instead, a lazy runtime check is implemented to ensure compatibility with both ABIs without compile-time checks.

As an extension creator this lack of clarity in the target ABI is a problem. I could call a function available only in NumPy v1 (for example, all the _ARRAY_API slots deprecated in the v2) without any warning, making the extension unusable with NumPy v2. So I would need to take extra care while writing an extension, checking the availability of each function or writing tests that cover all the NumPy API calls.

Some other issues related to the mechanism implemented in the crate are:

• Extension developers need to be aware of the definition of PyArray_DescrProto (not available in v1).
• In NumPy v2, the API call associated with slot 223 has been removed. From version v2.3, it has been reassigned to the NpyIter_GetTransferFlags function.

For me, the priority now is to implement a clear mechanism to target a specific ABI version. By default, the crate should target the latest version of the ABI, but it should be possible to switch to the older one if needed by enabling a feature flag. This would avoid the possibility of calling unavailable functions and accidentally making the extension unusable.

I can work on this feature in the coming weeks. But before starting this work, I want to be sure that this is the right approach, so I would like to ask what do you think about this?

It will make sense to talk about the choice of runtime only once the target ABI has been clarified.

[ngoldbaum]

I can work on this feature in the coming weeks. But before starting this work, I want to be sure that this is the right approach, so I would like to ask what do you think about this?

Personally, I wouldn't spend much effort to specifically target the NumPy 1.x ABI. I don't think we'd serving anyone by making it easier to use old versions of NumPy. But that's also just my opinion.

[Icxolu]

First of all, thanks for continuing to looking into this ❤️

I found that it is not ready to support multiple runtime versions in the way I implemented in this PR.

I agree, this was my initial feeling as well.

As an extension creator this lack of clarity in the target ABI is a problem. I could call a function available only in NumPy v1 (for example, all the _ARRAY_API slots deprecated in the v2) without any warning, making the extension unusable with NumPy v2. So I would need to take extra care while writing an extension, checking the availability of each function or writing tests that cover all the NumPy API calls.

Do we talk about the raw ffi bindings or rust-numpys high level API here. The high level APIs should be implemented in a way that is version agnostic, and even the raw ffi layer has version checks built in where necessary. Calling a v1 only slot on v2 will cause a panic at runtime. Definitely not perfect, but also no UB and no tapping in the dark why it does not work. I guess from my mindset numpy is a runtime dependency that is chosen by the end user of my extension and I have only limited control over that, instead of a compile time depedency like the interpreter that I choose as the extension auther.

Some other issues related to the mechanism implemented in the crate are:

• Extension developers need to be aware of the definition of PyArray_DescrProto (not available in v1).
• In NumPy v2, the API call associated with slot 223 has been removed. From version v2.3, it has been reassigned to the NpyIter_GetTransferFlags function.

This raises the question for me again, why do you need to interact with them, at all. When using the high level API these are not needed and using the raw bindings my question would be why? Could we in this case add to the high level API to make this easier? (223 for example is marked as v1 only and will panic on v2)

For me, the priority now is to implement a clear mechanism to target a specific ABI version. By default, the crate should target the latest version of the ABI, but it should be possible to switch to the older one if needed by enabling a feature flag. This would avoid the possibility of calling unavailable functions and accidentally making the extension unusable.

I'm asking my self here what is the benefit I get from targeting a specific abi over the current approach (from the perspective of both developing (maintainance) and using the high level api (downstream user)). Maybe you can paint a picture for me here, as I'm still struggling a bit to see what problem we are trying to solve, what the solution should look like and how that solution affect the rest of the api and user experience. I'm open to changes but I would like to understand what we are doing and why we are doing it (mostly this one is missing for my I think).

The other thing is that I don't think we can or should use feature flags for this. Cargo features are supposed to be additive and much like the different Python versions for PyO3 itself, this likely requires us using raw cfg flags instead, so that we can model functionality getting changed or removed in later versions. I also don't think we should use a moving target as a default, picking a specific one and only bumping it occasionally when needed seems more desirable to me. Ideally I think this should be the one that provides the most compatibility (so likely the fewest features) and options to in enable newer features by reducing compatibility.

Personally, I wouldn't spend much effort to specifically target the NumPy 1.x ABI. I don't think we'd serving anyone by making it easier to use old versions of NumPy. But that's also just my opinion.

I agree, but I would also make the argument that we should drop v1 support together with PyO3 dropping support for Python 3.7 and 3.8. We could do it sooner I guess, if there is a clear advantage we get that we could not get otherwise.