Reading a file in chunks? Swallow it whole!

Recently, I had to render multiple high-resolution images on a Rust + Egui stack for Hestia. The bottleneck was obviously the process being choked by decoding and rendering, but let's shift our focus elsewhere: reading the image files.

For nearly 2 decades, I have been reading files by chunking them into smaller buffers. Nothing is wrong with that, really...

use std::fs::File;
use std::io::{self, Read};

fn main() {
    // Open the file
    let mut file = File::open("image.png").unwrap();

    // Collect all bytes into 'image'
    let mut image = Vec::new();

    // Prepare chunk size of 4kb
    let mut buffer = [0; 4096];

    // Loop-read operation
    loop {
        let b = file.read(&mut buffer).unwrap();
        if b == 0 {
          break; 
        }
        image.extend_from_slice(&buffer[..b]);
    }
}

In this scenario, I needed to squeeze performance wherever possible. This led me to memory mapping, which is also available in Rust under the memmap2 crate.

use std::fs::File;
use memmap2::MmapOptions;

fn main() {
    // Open the file
    let file = File::open("image.png").unwrap();

    // Map the file into memory & assign to 'image'
    let image = unsafe{ 
        MmapOptions::new().map(&file).unwrap()
    };
}

The block is wrapped in unsafe because it contradicts Rust's memory-safe philosophy, and it won't compile unless we explicitly mark it as unsafe.

Adopting this technique has significantly improved speed. Under the hood, it doesn't actually read the whole file yet, it only registers the file's location in memory and loads chunks of the file when needed, and the OS can efficiently unload it if memory is tight. And the whole thing is done with zero-copy, avoiding wasted operations or memory usage. Beautiful.