Repository navigation
feat(cache): Add CacheEntrySizePolicy to reuse shorter cache entries as prefixes - #19126
Open
kingcrimsontianyu wants to merge 8 commits into
Open
kingcrimsontianyu wants to merge 8 commits into
kingcrimsontianyu wants to merge 8 commits into
Conversation
✅ Deploy Preview for meta-velox canceled.
|
meta-cla
Bot
added
the
CLA Signed
Co-authored-by: Kyle Hubert <kjmph@users.noreply.github.com>
kingcrimsontianyu
force-pushed
the
cache-size-policy
branch
from
September 22, 2026 02:04
6299296 to
6d6ed96
Compare
Collaborator
|
Is there a reason to have a policy? Why not always reuse? |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
|
@majetideepak Sorry I forgot to reply. So my understanding is that a cache hit under the default cc @kjmph |
Selective Build Plan
Affected targets (549)Directly changed (336)
Transitively affected (213)
Fast path • Graph from main@485e24f05450 |
Collaborator
|
@kingcrimsontianyu Is it accurate that |
Author
|
@majetideepak Right, the new |
majetideepak
left a comment
majetideepak
left a comment
Collaborator
There was a problem hiding this comment.
@kingcrimsontianyu this makes sense. Just one more comment.
| ASSERT_EQ(stats.numStales, 1); | ||
| ASSERT_EQ(stats.numEntries, 1); | ||
| ASSERT_EQ(stats.numHit, 1); | ||
| ASSERT_EQ(stats.numHit, 2); |
Collaborator
There was a problem hiding this comment.
Why would this change to 2?
Author
There was a problem hiding this comment.
So cache_->refreshStats(); accumulates the statistics. We inserted a small chunk of test above, where under kAllowSmaller a hit happened, making the counter 2. Here the counter stays 2.
Collaborator
There was a problem hiding this comment.
@kingcrimsontianyu let's reset the stats before each chunk so we can test independently. Thanks!
Author
There was a problem hiding this comment.
@majetideepak Thanks. I feel that the original two (now three) chunks of tests are placed one after another intentionally, so instead of placing them in independent tests, I asserted on the stats delta (using CacheStats::operator-) for each chunk to have the same effect of "stats resetting". Let me know if the test now is what you have in mind.
Collaborator
There was a problem hiding this comment.
@kingcrimsontianyu Your current test change is under TEST_P(AsyncDataCacheTest, staleEntry). This PR should get its own test TEST_P(AsyncDataCacheTest, cacheEntrySizePolicy)?
Author
There was a problem hiding this comment.
Done. Used a separate test cacheEntrySizePolicy.
Collaborator
|
@xiaoxmeng, @oerling, @Yuhta any comments from you? |
majetideepak
approved these changes
Oct 19, 2026
majetideepak
left a comment
majetideepak
left a comment
Collaborator
There was a problem hiding this comment.
This API extension makes sense. It trades a new read vs more cache entries.
I will wait for a day or two for @xiaoxmeng to respond as well.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Today
findOrCreatetreats an existing entry at the requested key as stale when the entry is shorter than the requested size. The entry is evicted and the whole range is loaded again, even though the shorter entry already holds a valid prefix of the requested bytes.This PR adds a
CacheEntrySizePolicy { kRequireAtLeast, kAllowSmaller }parameter tofindOrCreate, defaulting tokRequireAtLeastso that current callers behave exactly as before. UnderkAllowSmaller, the shorter entry is returned as a shared pin and counted as a hit. The caller reads the entry's actual size and requests the remaining bytes under a new key starting where the entry ends.This PR is extracted from the WXD-launch integration branch (https://github.com/rapidsai/velox/tree/WXD-launch), where the cache size policy feature is originally made by @kjmph.