Skip to content

Improve path conversions and add new string utilities#913

Merged
narknon merged 1 commit intomainfrom
utf8-path-utils
Jun 9, 2025
Merged

Improve path conversions and add new string utilities#913
narknon merged 1 commit intomainfrom
utf8-path-utils

Conversation

@narknon
Copy link
Copy Markdown
Collaborator

@narknon narknon commented Jun 9, 2025

Description
This PR enhances the string and path conversion utilities in the Helpers library with proper UTF-8 and UTF-16 support. The main improvements include rewriting the path conversion logic to handle Unicode correctly and adding several new utility functions for common string/path operations.

The key motivation is to ensure consistent and correct handling of Unicode paths across different platforms.

Necessary to be merged prior to #884

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Is/requires documentation update

How has this been tested?
Conversion tests/tested with updated lua system.

Checklist

  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • I have added the necessary description of this PR to the changelog, and I have followed the same format as other entries.

Additional context
Breaking Change: The to_charT_string_path() function now returns UTF-8 encoded strings when the target type is char, instead of using locale-dependent encoding. This ensures consistent behavior across platforms but may affect code that relied on the previous locale-specific behavior, but I doubt there is any code that did.

New Functions Added:
ensure_str_as() - Allows explicit target character type conversion
to_utf8_string() - Convenient UTF-8 string conversion
normalize_path_for_lua() - Converts paths to UTF-8 with forward slashes for Lua compatibility
utf8_to_wpath() - Converts UTF-8 paths to Windows wide strings for API compatibility

- Rewrite to_charT_string_path() to properly handle UTF-8 and UTF-16 encodings
- Add ensure_str_as<CharT>() for explicit target char type conversion
- Add to_utf8_string() for UTF-8 string conversion
- Add normalize_path_for_lua() to convert paths to UTF-8 with forward slashes
- Add utf8_to_wpath() to convert UTF-8 paths to Windows wide strings

BREAKING CHANGE: to_charT_string_path() now returns UTF-8 for char type instead of locale-dependent encoding
@narknon narknon force-pushed the utf8-path-utils branch from ab808e1 to 61ee57e Compare June 9, 2025 19:45
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 9, 2025

MSVC-Game__Debug__Win64 Download Logs
Build Details
Name Information
PR Commit 61ee57e
Merge Commit 40e2bd5
Size 47.81 MB
Last Updated Jun 9, 25, 8:06:28 PM UTC
Expires At Jun 23, 25, 8:06:23 PM UTC

MSVC-Game__Shipping__Win64 Download Logs
Build Details
Name Information
PR Commit 61ee57e
Merge Commit 40e2bd5
Size 29.41 MB
Last Updated Jun 9, 25, 8:09:35 PM UTC
Expires At Jun 23, 25, 8:09:32 PM UTC

@narknon narknon merged commit 2b73f36 into main Jun 9, 2025
14 checks passed
@narknon narknon deleted the utf8-path-utils branch June 9, 2025 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant