lezer/python
1
0
Fork 0
Code Issues 1 Pull requests Activity

Python string escape tokens #22

Closed
blast-hardcheese wants to merge 5 commits from python-string-escape-tokens into main
pull from: python-string-escape-tokens
merge into: lezer:main
Conversation 4 Commits 5 Files changed 3 +66 -24
blast-hardcheese commented 2024-03-12 01:42:03 +01:00 (Migrated from github.com)
Copy link

My attempt at resolving #20. This should be ready for review.

Thank you!

My attempt at resolving #20. This should be ready for review. Thank you!
blast-hardcheese commented 2024-03-12 01:44:26 +01:00 (Migrated from github.com)
Copy link

This is to support Replit efforts, fwiw. Thank you again!

This is to support Replit efforts, fwiw. Thank you again!
blast-hardcheese (Migrated from github.com) reviewed 2024-03-12 03:47:20 +01:00
src/tokens.js
@ -110,6 +114,25 @@ export const legacyPrint = new ExternalTokenizer(input => {
}
blast-hardcheese (Migrated from github.com) commented 2024-03-12 03:47:15 +01:00
Copy link

this should probably not be a regex, as I'm reviewing it. I'll change it to char === quote || char === '\\' as well

this should probably not be a regex, as I'm reviewing it. I'll change it to `char === quote || char === '\\'` as well
blast-hardcheese (Migrated from github.com) reviewed 2024-03-12 05:34:18 +01:00
src/tokens.js
@ -110,6 +114,25 @@ export const legacyPrint = new ExternalTokenizer(input => {
}
blast-hardcheese (Migrated from github.com) commented 2024-03-12 05:34:18 +01:00
Copy link

Resolved

Resolved
marijnh commented 2024-03-12 19:42:38 +01:00 (Migrated from github.com)
Copy link

I've pushed an alternative patch 798669d6ab that properly handles multi-character escapes (\u, \N, etc) and raw strings. Had to use a kludge with a context to keep the amount of different string tokenizers under control (raw/non-raw, single/double quoted, format/non-format, long/short = 16 different types of string content).

I've pushed an alternative patch 798669d6ab3ccdba21a8ad1876c04da0fd0764a9 that properly handles multi-character escapes (`\u`, `\N`, etc) and raw strings. Had to use a kludge with a context to keep the amount of different string tokenizers under control (raw/non-raw, single/double quoted, format/non-format, long/short = 16 different types of string content).
👍 1 🚀 1

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lezer/python!22
No description provided.