Skip to content

DEL-2261: Set LargeUtf8 in ArrowDataType - #3

Merged
eshaanmangal-tf merged 3 commits into
tf-0.22.2from
em/DEL-2261
Sep 10, 2026
Merged

eshaanmangal-tf merged 3 commits into
tf-0.22.2from
em/DEL-2261

Conversation

@eshaanmangal-tf

@eshaanmangal-tf eshaanmangal-tf commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

What changes are proposed in this pull request?

How was this change tested?


Note

Medium Risk
Runtime behavior of kernel→Arrow typing and expression evaluation changes when the env var is set; default behavior is unchanged but the new code paths affect string/binary handling across compaction-related flows.

Overview
Adds an opt-in path to emit LargeUtf8 / LargeBinary when converting kernel String/Binary to Arrow types, controlled by the environment variable DELTA_KERNEL_PREFER_LARGE_VARLEN=true (default remains Utf8/Binary).

Downstream Arrow expression code is updated so this choice does not break evaluation: map-to-struct reads map keys/values via as_string_accessor instead of assuming StringArray, scalar append_to / append_null accept both standard and large string/binary builders, and tests cover LargeUtf8 maps and large builder appends.

Reviewed by Cursor Bugbot for commit 9655a75. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

Copy link
Copy Markdown

PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: DEL-2261: Set LargeUtf8 in ArrowDataType

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d50b59c. Configure here.

@eshaanmangal-tf
eshaanmangal-tf merged commit 60593e0 into tf-0.22.2 Sep 10, 2026
8 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants