Commits

History for

Commits on Dec 8, 2025

Fix pyflakes warnings: variable is assigned to but never used (#142294)
vstinner
authored

Commits on Oct 20, 2025

gh-63161: Fix tokenize.detect_encoding() (GH-139446)
serhiy-storchaka
authored

Commits on May 26, 2025

gh-134675: Add t-string prefixes to tokenizer module, lexical analysis doc, and add a test to make sure we catch this error in the future. (#134734)
ericvsmith
authored

Commits on May 25, 2025

gh-134582: Fix t-strings untokenize() roundtrip removing space between braces (#134603)
loic-simon
authored

Commits on May 5, 2025

gh-130645: Add color to stdlib argparse CLIs (gh-133380)
hugovk
authored

Commits on May 3, 2025

gh-133306: Use \z instead of \Z in regular expressions in the stdlib (GH-133337)
serhiy-storchaka
authored

Commits on Apr 30, 2025

gh-132661: Implement PEP 750 (#132662)

authored

Commits on Mar 27, 2025

gh-131178: Add tests for `tokenize` command-line interface (#131274)
donbarbos
authored

Commits on Jan 21, 2025

gh-125553: Fix backslash continuation in `untokenize` (#126010)
tomasr8
authored

Commits on Jan 6, 2025

gh-128519: Align the docstring of untokenize() to match the docs (#128521)
tomasr8
authored

Commits on Oct 9, 2024

gh-91818: Use default program name in the CLI of many modules (GH-124867)
serhiy-storchaka
authored

Commits on Oct 6, 2024

gh-125008: Fix `tokenize.untokenize` roundtrip for `\n{{` (#125013)
tomasr8
authored

Commits on Feb 19, 2024

gh-115154: Fix untokenize handling of unicode named literals (#115171)
pablogsal
authored

Commits on Oct 12, 2023

gh-104169: Fix test_peg_generator after tokenizer refactoring (#110727)
lysnikolaou
authored

Commits on Jun 19, 2023

gh-71299: Fix __all__ in tokenize (#105907)
lysnikolaou
authored

Commits on Jun 7, 2023

gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors (#105399)
pablogsal
authored

Commits on Jun 5, 2023

gh-105324: Fix tokenize module main function for stdin (#105325)
pablogsal
authored

Commits on May 30, 2023

gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070)
pablogsal
authored

Commits on May 26, 2023

gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer (#104980)
pablogsal
authored

Commits on May 22, 2023

gh-102856: Tokenize performance improvement (#104731)
mgmacias95
authored

Commits on May 21, 2023

Commits on Feb 3, 2022

bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993)
sobolevn
authored

Commits on Aug 24, 2021

Add tests for the C tokenizer and expose it as a private module (GH-27924)
pablogsal
authored

Commits on Jul 31, 2021

bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499)
pablogsal
authored

Commits on Jan 24, 2021

bpo-43014: Improve performance of tokenize.tokenize by 20-30%
asottile
authored

Commits on May 30, 2019

bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)

asottile
authored and
miss-islington
committed
bpo-5028: fix doc bug for tokenize (GH-11683)

andrewnc
authored and
miss-islington
committed

Commits on May 6, 2019

bpo-36766: Typos in docs and code comments (GH-13116)

GiantRobato
authored and
matrixise
committed

Commits on Dec 22, 2018

bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
serhiy-storchaka
authored

Commits on Jul 6, 2018

bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)

ammaraskar
authored and
taleinat
committed

Commits on Jun 5, 2018

bpo-12486: Document tokenize.generate_tokens() as public API (#6957)

takluyver
authored and
willingc
committed

Commits on Apr 23, 2018

bpo-33338: [tokenize] Minor code cleanup (#6573)
ambv
authored

Commits on Apr 11, 2018

bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447)
serhiy-storchaka
authored

Commits on Oct 6, 2017

bpo-30406: Make async and await proper keywords (#1669)

JelleZijlstra
authored and
1st1
committed