Use color to highlight error locations

[pablogsal]

This has several advantages:

• Will help a lot with readability as parsing the error lines is easier if color highlights the error ranges.
• In the future we can optionally (via config) drop the ranges and only use color, recovering back the extra lines that the carets are taking.
• All the cool kids are doing it: This feature has already been successfully implemented in various tools. It has proven to be an effective aid for developers in quickly identifying the source and location of errors.

Control features:

• The feature will use 16 ANSI color scapes so the actual color is configured via the terminal emulator.
• Users can set the following env vars to control it (in the following order):
  • PY_COLORS=1 activates the feature (used by pytest: https://github.com/pytest-dev/pytest/blob/022f1b4de546c8b3529e071965555888ecf01cb4/src/_pytest/_io/terminalwriter.py#L28)
  • PY_COLORS=0 deactivates the feature
  • NO_COLOR=1 deactivates the feature
  • FORCE_COLOR=1 activates the feature
  • Feature is deactivated if terminal is not a tty or if TERM is set to dumb.

Linked PRs

• gh-112730: Use color to highlight error locations #112732
• gh-112730: Update docs for colour env vars #112837
• gh-112730: Make the test suite resilient to color-activation environment variables #117672
• gh-112730: use keyword only argument for colorize #118086
• gh-112730: Respect tests that require environment variables with no-colorize fixes #118288

[hugovk]

This looks really good, thanks!

Perhaps something for a followup, can we customise the colouring a bit?

PR #112732 right now:

GitHub / MagicPython

GitHub markdown (with pytb after the triple backticks):

Traceback (most recent call last):
  File "/Users/hugo/github/python/cpython/main/1.py", line 1, in <module>
    1/0
    ~^~
ZeroDivisionError: division by zero

(GitHub screenshot)

(Uses https://github.com/MagicStack/MagicPython for Python tracebacks: https://github.com/github-linguist/linguist/blob/master/vendor/README.md)

Sphinx / pygments

Sphinx with .. code-block:: pytb, for example on the PEPs repo:

(Uses Pygments' Native style)

[pablogsal]

Perhaps something for a followup, can we customise the colouring a bit?

Yup, see #112732 (comment).

I want to explore customization options AFTER we land the basic version first. EDIT Or are you referring to customizing the coloring as "adding more colors"?

On the other hand, are you suggesting to also colorize the name of the exception, the file, the number and the Traceback text?

I have been told that yellow and cian are really bad for light mode, so we should do something else there.

[pablogsal]

@hugovk What about like this?

[hugovk]

I wasn't thinking about allowing the user to customise them (although that's something to consider) but about having different colours for the different elements (exception name, file, etc).

Yes, we should check the contrast is good for both light and dark modes.

The GitHub colours (ignoring the plain black/white text) look the same for each mode and display well for both:

edit: and the blues are different, but you get the idea :)

[pablogsal]

I am quite colorblind myself 😅 Do you mind telling me what colors are supposed to be every piece? As in "filename is ..., line number is ...., function name is ..., source code is ..., error location is ...)

[ethanfurman]

Where are we on allowing user-specified colors? On my system the REPL background is black, and the exception text is a dark purple which is extremely hard to read.

[pablogsal]

Where are we on allowing user-specified colors? On my system the REPL background is black, and the exception text is a dark purple which is extremely hard to read.

Currently the only option is to use PYTHON_COLORS=0 (or any other of the environment variables here: https://docs.python.org/3.13/using/cmdline.html#using-on-controlling-color) to deactivate the feature. We have not developed or discussed any API to configure the colours yet.

[hugovk]

@ethanfurman Please could you post a screenshot? What OS/terminal/etc do you use?

[ambv]

Note that this still needs support for -E.

[gvanrossum]

Also I don't see NO_COLORS and FORCE_COLORS listed in python --help-env.

[AlexWaygood]

Note that this still needs support for -E.

@gpshead wrote in #117605:

python -E is defined as only ignoring PYTHON* environment variables. This is a good thing. https://docs.python.org/3/using/cmdline.html#cmdoption-E

I believe tests either need to be happy in the presence of arbitrary color control codes (barfing rainbows...), or our make test aka python -m test aka regrtest runner should explicitly un-configure the community standard not language specific COLOR variables.

[gvanrossum]

But then shouldn’t the argument be that Python shouldn’t be sensitive to env vars that don’t start with PYTHON?

[AlexWaygood]

I personally think it's very useful for Python to use the same env vars that are being increasingly widely adopted elsewhere. So I suppose I lean towards changing -E so that it also ignores the FORCE_COLOR variable. But anyway, I was mainly flagging that Greg had voiced a different opinion on another issue; having people comment on two different issues at the same time can cause confusion :-)

[hugovk]

The "community standards" like https://no-color.org and https://force-color.org are trying to solve the problem that every tool has their own env vars, which makes it hard to configure them all individually.

And having our own PYTHON_* env vars that override the global standard allows for finer tuning of what can have colour.

[gpshead]

@hugovk mentioned "GitHub Actions is another good place to use FORCE_COLOR. It's not a tty so autodetect doesn't work, but in most cases colour really helps you find errors in long logs." in chat...

If so, resolving the test failure issue by disabling them during a regrtest run, while technically correct, misses out on a potentially nice new feature (granted we haven't had color there before...).

[pablogsal]

Yeah I think we can just skip these particular tests if the environment variables are set (or check if we can make them work regardless of what is set) while keeping the whole test suite working with whatever env is set to benefit from the colours in the logs.

[pablogsal]

Maybe we could just add a support decorator that just does the appropriate monkey patching in the colorise function to avoid being affected by the force colours variable. To avoid regressions, setting FORCE_COLORS in GH actions sounds like a good idea.

[pablogsal]

I personally think it's very useful for Python to use the same env vars that are being increasingly widely adopted elsewhere. So I suppose I lean towards changing -E so that it also ignores the FORCE_COLOR variable.

I think if we don't do this it's going to be a full pain for us and a lot of users that have the variable set in CI and are running subprocesses in tests

[encukou]

#117672 broke several stable buildbots, see e.g. here. As Jakub noted they seem to be --enable-shared builds that rely on LD_LIBRARY_PATH to find libpython.

Do you have time to look into this?

[pablogsal]

I will look into this today. We can revert in the meanwhile if you wish

[pablogsal]

I think this will be fixed by #118288

[hugovk]

#118288 has been merged, closing.