Repository navigation
streamly-core 0.3.2: backport bug fixes from master - #3262
Merged
Merged
Conversation
cabal-3.18 on the GitHub runner rejects cabal.project.markdown-doctest because it has no packages field. Also use the same ghcup version as the packdiff CI.
The base branch was hardcoded to master, whose benchmark suites differ from this branch: the benchmark targets of this branch, e.g. Data.Array.Stream, do not exist on master and the run failed.
(cherry picked from commit 133059d)
Port the fixes in master commits e5c153d and fa98ea8. Git for Windows 2.56 moved /mingw64 to /ucrt64, so curl and other tools were not found in the PATH passed to packcheck. The macOS image ships Command Line Tools newer than the default Xcode; linking against the CLT SDK fails with a tapi "unknown architecture arm64e.x1-macos" error. Select the default Xcode, and point the CLT MacOSX.sdk symlink at the Xcode SDK because the GHC bindists hardcode the CLT SDK path. Add the Xcode version and build to the macOS GHC and dependency cache keys.
cabal-3.10.1.0 failed to update the hackage index: "root.json does not have enough signatures signed with the appropriate keys". Use a newer ghcup that can install cabal-3.16.1.0.
harendra-kumar
force-pushed
the
core-0.3.2-backports
branch
from
October 19, 2026 13:18
cdd2aa8 to
9271594
Compare
The network package fails to build on the old Windows images that AppVeyor uses, the AppVeyor webhook is deactivated. The GitHub Actions Windows build covers Windows. (cherry picked from commit 433bc28)
(cherry picked from commit 9641fed)
(cherry picked from commit 6d2f228)
unsafeByteCmp returned the result of compareByteArrays# as a thunk, the bytes were compared when the result was evaluated. If either array was modified in between, the result was that of the modified bytes. (cherry picked from commit 2865acf)
Deserializing a list with a negative length did not stop at zero elements, it read elements until it ran past the end of the buffer, or never terminated if the elements took no space to deserialize. (cherry picked from commit a2573cf)
With encodeConstrNames, the derived deserializeAt compared the bytes of the constructor names with the buffer without checking that the name length read from the buffer fits in it. With encodeRecordFields, it compared the record header with the buffer without checking that the header fits in the buffer. A truncated or corrupt buffer made it read beyond the end of the buffer. When the constructor name does not fit, raise an error. When the record header does not fit, take the slow path that reads the header field by field with bounds checks. (cherry picked from commit bc75701)
(cherry picked from commit 94d3f07, core/src changes only)
(cherry picked from commit 980fc3e, core/src changes only)
(cherry picked from commit 044486a, reimplemented as a direct Unfold because 0.3.1 has no Unfold takeEndBy or carryInput)
(cherry picked from commit 8f4987c, core/src changes only)
asyncClock and timer used forkIOManaged, which kills the thread when the ThreadId box returned by fork becomes unreachable. The consumers do not keep the ThreadId, so the thread was killed at the next major GC: the clock stopped advancing and waitTimer failed with "thread blocked indefinitely in an MVar operation". The GHC JavaScript RTS runs a GC every second, which makes the clock stop after about a second. The threads now hold a weak reference to the IORef that the consumers use, and exit when it is no longer reachable. The timer thread holds its MVar strongly, so that a thread blocked in waitTimer stays reachable, and waitTimer keeps the IORef alive. (cherry picked from commit 34f852f, core/src changes only)
chunksOf emitted an empty array when the stream ended on a chunk boundary, e.g. chunksOf 3 on [1..9] gave [[1,2,3],[4,5,6],[7,8,9],[]], and [[]] for an empty stream. Fixed in both the unboxed and generic MutArray versions, the Array versions are defined using them. (cherry picked from commit 5b32080, core/src changes only)
unsafeAppendMax computed the bound of the result from the end of the input array. When the input array was reallocated its start became 0, so for a slice with a non-zero start the bound pointed beyond the new allocation, e.g. appending 3 Ints to a 2 Int slice at offset 5 gave a bound of 80 bytes on a 40 byte allocation. A later snoc on the result would write out of bounds. When the requested count needed less space than the array already had free, the bound was computed from the requested size and the returned array lost the remaining reserved capacity. Use at least the free space when computing the new bound. (cherry picked from commit f868ae0, core/src changes only)
When the array was full writeWithAs grew it to max (oldSize * 2) 1 bytes. For an empty array, e.g. createMinOf 0, that is 1 byte, which realloc rounds down to 0 bytes for any element larger than a byte, and the first element was written past the end of the allocation. Grow to at least one element instead. (cherry picked from commit fd76a80, core/src changes only)
unsnoc [1,2,3] returned (fromList [2,3], 3) instead of (fromList [1,2], 3). (cherry picked from commit 38cc25b, core/src changes only)
getIndex compared the element index with the ring size in bytes, so an index from the length up to the byte size wrapped around instead of returning Nothing. getIndex 5 on a 5-element Int ring returned the head element. (cherry picked from commit 9bf4966, core/src changes only)
cast checked only that the ring byte length is a multiple of the target element size. A ring head at a byte offset that is not a multiple of the target element size makes elements straddle the end of the ring, so reads go past the end of the memory. cast now returns Nothing in that case. (cherry picked from commit ea8dc84, core/src changes only)
fold read the element at the head before checking the ring size, so a zero-sized ring folded one garbage element. createOfLast returns a zero-sized ring for an empty stream, so RingArray.fold Fold.toList on it returned [0] instead of []. foldlM' uses fold and had the same problem. (cherry picked from commit e59724d, core/src changes only)
eqArray and eqArrayN compared the ring with the bytes from offset 0 of the array's memory instead of from the array's start offset, so they gave wrong results when the array is a slice. (cherry picked from commit fc37e26, core/src changes only)
When the requested range did not wrap around the end of the ring, copyToMutArray still copied from the start index to the end of the ring into an array of the requested length, writing past its end, and then copied a negative number of elements. copyToMutArray 0 2 on a full 5-element ring failed an assertion in debug builds and corrupted memory otherwise. (cherry picked from commit 59af8ca, core/src changes only)
refold used the deprecated extractM to get the initial value of the Refold, calling the source fold's extract instead of its final. The source fold was never finalized, and refold failed for folds whose extract is an error, e.g. splitWith, split_, concatMap, longest and duplicate. For example, refold Refold.sconcat over splitWith (++) (fromPure [1]) (take 1 (foldl' (++) [])) failed with "splitWith: cannot be used for scanning" instead of folding [[2]] to [1,2]. (cherry picked from commit d8f8315, core/src changes only)
bracketIO stored the result of the inner fold's step, Partial or Done, and always returned Partial. Done was returned only on the next input, which was dropped: Stream.foldBreak with bracketIO over Fold.take 2 on [1..5] left [4,5] instead of [3,4,5]. If the stream ended right after the inner fold terminated, final returned the result without running the release action, which then ran only on GC. (cherry picked from commit 9f025ef, core/src changes only)
finallyIO ran the action only in final. When the inner fold returned Done from initial or step, e.g. Fold.take 1 on [1,2,3], the action ran only when the IORef holding it was garbage collected. (cherry picked from commit 2dc4882, core/src changes only)
sampleFromthen offset stride kept the elements at the indices i with (i + offset) mod stride == 0, so sampleFromthen 1 3 on [0..9] returned [2,5,8] instead of [1,4,7]. Keep the indices i >= offset with (i - offset) mod stride == 0. The existing test used offset 0, where the two agree. (cherry picked from commit 86758bd, core/src changes only)
distributeScan kept a fold whose step returned Done in the list of running folds, with Done as its initial action. Its result was added to the output only on the next input, and was lost if the stream ended: with the folds [Fold.take 2 Fold.sum, Fold.take 2 Fold.length] on [1,2], Fold.fromScanl (distributeScan gen) returned [] instead of [3,2]. (cherry picked from commit 73688f4, core/src changes only)
When the fold for a key terminated on the first input of that key, the
key was neither kept in the map of running folds nor added to the set of
finished keys. The next input with the same key started the fold again
and produced another result. For example, classifyScan fst with the
inner fold "take 1 sum" on [("A",1),("A",2)] emitted ("A",1) and
("A",2) instead of only ("A",1).
Fix classifyGeneric, classifyGenericIO, classifyScanGeneric and
classifyScanGenericIO, and with them classify, classifyIO, classifyScan
and classifyScanIO.
(cherry picked from commit bd5c7ba, core/src changes only)
sampleFromthen offset stride kept the elements at the indices i with (i + offset) mod stride == 0, so sampleFromthen 1 3 on [0..9] sampled [2,5,8] instead of [1,4,7]. Keep the indices i >= offset with (i - offset) mod stride == 0. The existing test used offset 0, where the two agree. (cherry picked from commit 7d18bef, core/src changes only)
The instances defined only show, so the default showsPrec ignored the precedence: show (Just n) produced "Just fromList [1,2]", which the Read instance cannot parse. Delegate showsPrec to the Stream and StreamK instances, which use showParen, the same as the Array fix. Keep show, delegating to an explicit show in the Stream and StreamK instances, the same as the Array instance. (cherry picked from commit bbfe688, core/src changes only)
foldr1 used the head of the stream as the seed of a right fold over the tail, so the head was combined last: foldr1 (-) on [1,2,3] computed 2 - (3 - 1) = 0 instead of 1 - (2 - 3) = 2, and foldr1 const returned the second element. The existing tests used only (+). Fold the whole stream with a Maybe accumulator instead. (cherry picked from commit 97edd2e, core/src changes only)
isInfixOf splits the stream on the infix and checks for a second segment. An empty pattern splits the stream at every element, so a stream of fewer than two elements has fewer than two segments: isInfixOf nil nil and isInfixOf nil (fromList [1]) returned False. An empty infix occurs in every stream; return True for it directly. (cherry picked from commit 1729a70, core/src changes only)
* A file opened in WriteMode was not truncated, the old contents beyond the newly written data remained. Truncate a regular file after it is locked, like base does. * When making the Handle failed, e.g. for a directory or a locked file, the file descriptor leaked. Close it. * Use an interruptible open like base, open can be slow, e.g. on NFS or FUSE file systems, or block. The fd based openAt is unchanged. * ReadWriteMode did not set O_NOCTTY and O_NONBLOCK like the other modes. Add tests for the truncation and the file descriptor leak. (cherry picked from commit 382f4d8, core/src changes only, adapted to the 0.3.1 module paths)
When the stream ended before the Karp-Rabin buffer was full, the last segment was dropped if the buffer matched the pattern. The last slot of the buffer then still holds data from the previous segment, so after a match, separator "abc" and input "abcab" gave ["abc"] instead of ["abc","ab"]. A pattern at the end of the stream never reaches this branch, a complete match goes through SplitOnSuffixSeqKRCheck, so remove the check. (cherry picked from commit 66686f5, core/src changes only, applied to Stream/Nesting.hs)
(cherry picked from commit fed8ae8, core/src changes only, uses new instead of emptyOf as in 0.3.1)
When the scan for a key terminated on the first input of that key, the
key was neither kept in the map of running scans nor added to the set of
finished keys. The next input with the same key started the scan again
and produced another result. For example, classify fst with the inner
scan "take 1 sum" on [("A",1),("A",2)] emitted Just ("A",1) and
Just ("A",2) instead of Just ("A",1) and Nothing.
Fix classifyGeneric and classifyGenericIO, and with them classify and
classifyIO.
(cherry picked from commit 213df86, core/src changes only, applied by hand to the 0.3.1 code)
uniqBy was rollingMap, whose initial result is error "Empty stream". Using the result before any input failed: Stream.scanl Scanl.uniq on "aab" had the error as its first output, and Scanl.compose Scanl.uniq Scanl.toList failed when it was initialized, compose forces the initial result of the first scan. Implement uniqBy with scanl', the initial result is Nothing. (cherry picked from commit da60994, uniqBy change only, uses mkScanl as 0.3.1 has no scanl')
Port the tests of the backported master commits to the 0.3.1 test layout. Tests whose master module does not exist on this branch go to the suite of the tested module: Stream and StreamK tests to Data.Stream, Fold and Scanl tests to Data.Fold, RingArray.Generic tests to Data.RingArray and Generic MutArray tests to Data.MutArray. The Scanl sampleFromthen test checks the final result instead of the postscan output, a Scanl filter on this branch emits the previous output for filtered-out inputs. Also add tests for Unfold.before, refoldIterateM and the enumeration overflow fixes.
Bump version in streamly-core/configure.ac
harendra-kumar
force-pushed
the
core-0.3.2-backports
branch
from
October 19, 2026 15:49
9271594 to
869a742
Compare
The docspec job fails to link network's hsc2hs helper: ghc passes -fuse-ld=gold and gcc cannot find the linker. The ubuntu:latest image has binutils 2.46, which no longer includes gold.
The 2021 cabal-docspec looks for packages only in ~/.cabal/store, cabal 3.16 installs them in the XDG store, ~/.local/state/cabal/store.
packcheck writes the cabal config to ~/.config/cabal and links it from ~/.cabal/config. With both present cabal 3.16 uses the XDG store in ~/.local/state/cabal/store, cabal-docspec looks in ~/.cabal/store.
cabal-docspec 0.0.0.20250606 starts GHCi with the package's Haskell2010 language, the doctests in MkType, StreamK, WindowsPath and Unbox need FlexibleContexts, ScopedTypeVariables and DeriveGeneric. Master enables them with :set in the sources, done here in the CI to keep the streamly-core sources unchanged.
The mkZipType and mkCrossType examples need FlexibleInstances, GeneralizedNewtypeDeriving and TypeOperators, as in master's doctest setup for Streamly.Data.Stream.MkType.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports bug fixes from master for streamly-core 0.3.2 release. See the changelog for the list.
Unfold.before (pre-release) added a Monad m constraint, since the older version was buggy, this will let users know if there is buggy code.
streamly 0.11.0 and 0.11.1 are API compatible with 0.3.2, but will need Hackage revisions to allow it.