We are going to be updating our production spark distribution to version 3.5.3.
As such, we need to update the spark shuffler version that is deployed to match it.
We already have a mechanism to deploy multiple spark shuffler versions concurrently.
The decision has been made to update the default scheduler to be 3.5.3 and to run this on the default port of 3773.
We will run the version 3.1.2 shuffler on a custom port and provide an option for users to update their DAGs to use this port, if necessary.
This may be necessary for anyone who has a custom conda environment that is cloned from the old base conda-analytics environment.
Once all of these custom environments have been upgraded, we will be able to remove the shufflers prior to version 3.5.3.