Add support for python 3.11 and 3.12; switch default to 3.9#1126
Merged
Add support for python 3.11 and 3.12; switch default to 3.9#1126
Conversation
added 2 commits
August 21, 2024 18:58
github-merge-queue bot
pushed a commit
that referenced
this pull request
Aug 26, 2024
SQLAlchemy 2.0 came out in January of 2023 (>1.5 years ago). It introduced a variety of breaking changes. Sematic uses SQLAlchemy heavily, and in complex ways (ORM type introspection for dynamic serialization/deserialization, ORM type mixins, complex and dynamic query builds). This meant we had to wait to upgrade until sufficient time could be dedicated to migrating and testing it. We don't want to wait too much longer, it's only a matter of time before 1.x releases stop getting published and SQLAlchemy's development has largely shifted to >=2.0. So this PR bites the bullet and does the upgrade. This PR also upgrades the server image python version to 3.12. While not technically related to the SQLAlchemy changes, it follows closely on the heels of #1126 , and since that did not change anything running on the server (but this does) it seemed like an OK place to add so the 3.12 server and SQLAlchemy 2.0 changes could be tested together. Finally, there was actually a bug for custom metrics usage that was not caught in unit tests. This PR fixes that bug, confirmed in remote testing. Testing -------- In addition to all unit tests passing, I also tested running a local server with bazel and running the MNIST example in it. I confirmed built-in metrics worked and the pipeline passed with the expected resulting UI. I then built a wheel with these changes and published a release candidate Docker image for the standard and EE images. In a remote environment, I: - Confirmed the built-in system pipeline metrics (ex: success rate and timings) worked - ran a pipeline end-to-end - ran a pipeline that used a Ray cluster - ran a pipeline that used custom metrics and confirmed the UI works - cofirmed that the cleaner is working - tested run search extensively --------- Co-authored-by: Josh Bauer <josh@sematic.dev>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python 3.8 reaches EOL in October. Meanwhile, python 3.11 and 3.12 have been released since we last updated our supported python versions. This PR adds support for 3.11 and 3.12, and changes the default bazel targets to use 3.9 instead of 3.8. Some things of note:
The bazel dependency for torchvision in python3.10 is no longer resolving (and doesn't resolve for 3.11 or 3.12 either). For now, this PR handles that by making all bazel targets depending on torch vision only get built/tested for 3.8 and 3.9 . Note that this does NOT mean the examples using torch vision will not be runnable in higher python versions outside the context of bazel.
recent
rules_pythonintroduce a complex bazel workspace dependency issue so we can only have bazel targets for building and testing with interpreters up to python3.11. Note again that this is a limitation of what we can do in bazel and not a limitation of what interpreters the resulting wheel can be run with.Note that this does not change what version of python the server or default worker images run with (python3.9 for both). So this is essentially an sdk and local-server only change.
Testing