Write-heavy setup

[GrzegorzGawlikH]

I would like to ask about good practice for write-heavy setup.
I've deployed in my k8s cluster and using it for shared volume where people can upload big files (these files are stored, read and then deleted). When there's a big load of files - rate of zerofs_write_operations_total metrics keep going up (rate - not total number) and after 2-3 hours of big load - there start big delays on write (read works fine) - up to minutes for single small file.
As this is some sort of temporary data - I can remove them and restart app - and it works fine again for 2-3 hours and problem occurs again.
I've got 2 instances of gateway - working in active-standby with replication.
My settings are:
[cache]
dir = "/data/cache"
disk_size_gb = 300
memory_size_gb = 4.0

[filesystem]
compression = "lz4"

[lsm]
flush_interval_secs = 15
max_concurrent_compactions = 8

Is there something that I can change to make it work better? Add separate compactor instance(-s)? Or change my config to reduce or increase cache?

[Barre]

What version are you running?

[GrzegorzGawlikH]

2.1.1

[GrzegorzGawlikH]

I've noticed also that metric zerofs_dedup_retained_results grows in the same way as rate of zerofs_write_operations_total

[Barre]

zerofs_dedup_retained_results is just your write rate: every mutation with a client operation id is kept for 150s so a retry can be answered with its original result, so the gauge sits at roughly (mutations per second * 150).

On the write path, data is staged in memory and uploaded to the object store when a segment fills (256 MiB) or on fsync, and an fsync waits for the uploads already in flight, and for a lsm l0 flush. Reads are served from your local cache, so they stay fast independently of that. Slow writes with fast reads usually means the uploads aren't keeping up.

Worth watching on the node during a slow period: network throughput to the object store, CPU, and the disk behind the disk cache.

[GrzegorzGawlikH]

If I would like to buffer this heavy upload - should I increase disk cache only or l0_max_ssts as well? Or something else?

[Barre]

This is not currently configurable and is already buffered 256MB*8 so it's unlikely buffering further would provide any benefit.