[pcbeingused333]

Fixes #7658

Cause

When a filter is pushed down to the parquet reader, substitute_missing_cols replaces columns absent from the file schema with null literals. It only handled top-level columns. A predicate such as s.y.is_null() where struct s exists in the file but lacks field y (schema evolution) was left as-is and failed when bound/evaluated against the file schema.

Fix

substitute_missing_cols now also rewrites struct.get(field) to a null literal when the struct exists but has no such field, or when the parent was already substituted with null (nested access). Fields that are present are untouched.

Tests

Rust unit tests in daft-parquet helpers::tests: missing field, nested access under a missing struct, and a present field left alone. The first two fail without the change and pass with it.

Disclosure

This change was prepared with the help of an AI coding assistant (Claude Code).

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The change affects predicate handling in Parquet reads.

Fix constant mask lengths before merging so missing-field filters work for multi-row reads.

Findings

1.  Missing-field filters lose rows ▶

Summary

Extends substitute_missing_cols to replace missing struct fields with null, including nested accesses.

• Parquet filters replace missing struct fields with null.

_{Reviews (1) · Last reviewed commit: "fix(parquet): treat a struct field missi..." · Reviewed by Greptile}

[greptile-apps]

Missing-field filters lose rows

For a multi-row file, rewriting s.y.is_null() to null_lit().is_null() produces a one-element boolean array. eval_predicate_mask passes it unchanged to array filtering and bool_array_to_row_selection, so the selection covers only the first row instead of keeping every row.

Expand constant masks to the batch length before using them, and add a multi-row read test. The new tests only check rewriting and binding.

[codspeed]

Merging this PR will improve performance by 10.08%

2 warnings

⚠️ 40 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran in a different software environment

Its base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the change might come from this difference rather than from the code. Open the benchmark to see what changed.

See affected benchmarks

⚡ 1 improved benchmark
✅ 39 untouched benchmarks
⏩ 10 skipped benchmarks¹

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	test_clickbench_sql[0]	11.6 ms	10.5 ms	+10.08%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing pcbeingused333:fix/parquet-missing-struct-field-pushdown (07c802c) with main (bffd834)}

Footnotes

1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩