I don't know if Haskell etc affords you such control

You can have immutable arrary with vectors, but to mutate them you will need to wrap your action in a Monad. It even supports unboxed values.

https://hackage.haskell.org/package/vector

But I agree boxed default actually causes a lot of performance overhead in many high-level languages.