How can I decrease memory usage?

[s3rius]

Hi to everyone. I have a problem with memory usage. It doesn't look like a leak, but more like internal caches or something.

I'm developing TUS protocol implementation and I've chosen actix-web as a framework. It's really nice but memory usage is high and I can't get over this issue.

Here are some charts of load testing:

As you can see, when the number of active connections (opened file descriptors) grows, memory usage goes up. When number of active connections decreases, memory usage is the same. And only when all connections are closed memory usage goes down, but it doesn't return to the same value at the start of load testing.

Also, I'm interested why after some warmup connections the minimal amount of memory used by my application is about 200MB. Isn't it high?

I have some ideas but I can't really find a answer.

Boxes in state

I use Boxes with dynamic traits as a state. My state is defined like this:

pub struct State {
    pub config: RustusConf,
    pub data_storage: Box<dyn Storage + Send + Sync>,
    pub info_storage: Box<dyn InfoStorage + Send + Sync>,
    pub notification_manager: NotificationManager,
}

Also Storage and InfoStorage traits are using the #[async_trait] annotation.

Too many blocking tasks

I use spawn_blocking for writing and reading files to make the app faster since tokio::fs is not so fast.
So basically at almost all requests, I call something like this:

        if file_info.path.is_none() {
            return Err(RustusError::FileNotFound);
        }
        let path = String::from(file_info.path.as_ref().unwrap());
        let force_sync = self.force_fsync;
        tokio::task::spawn_blocking(move || {
           // Open file and write bytes to it.
            Ok(())
        })
        .await?

I write lots of bytes

TUS is a protocol for file uploads. Maybe memory consumption is huge because I use a bytes extractor and move bytes to sync closure with spawn_blocking. But, as the documentation says, bytes cloning must be cheap.

Source code of a project: https://github.com/s3rius/rustus

P.S. Of course, I understand that I'm doing something wrong and this behavior is the result of my code. But I would appreciate some help in finding the root cause.

Script for recording process info:

from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter, Namespace
import math

import psutil
import time

import matplotlib.pyplot as plt
from datetime import datetime


def parse_args() -> Namespace:
    parser = ArgumentParser()

    parser.add_argument("--pid", "-p", dest="pid", type=int, required=True)
    parser.add_argument("--save", "-s", dest="save", action="store_true", default=False)
    parser.add_argument(
        "--no-show", "-ns", dest="no_show", action="store_true", default=False
    )

    return parser.parse_args()


def main():
    args = parse_args()
    process = psutil.Process(pid=args.pid)

    process_info = {
        "meaurements": [],
        "rss": [],
        "data": [],
        "shared": [],
        "num_fds": [],
        "num_threads": [],
    }

    try:
        while process.is_running():
            mem = process.memory_info()
            process_info["meaurements"].append(datetime.now())
            process_info["rss"].append(mem.rss / 1024**2)
            process_info["shared"].append(mem.shared / 1024**2)
            process_info["data"].append(mem.data / 1024**2)
            process_info["num_fds"].append(process.num_fds())
            process_info["num_threads"].append(process.num_threads())
            time.sleep(1)
    except KeyboardInterrupt:
        print("Stopping watcher")

    fig, (memory, files, threads) = plt.subplots(3, 1, sharex=True)

    memory.plot(
        process_info["meaurements"],
        process_info["rss"],
        color="green",
        label="rss",
    )
    memory.plot(
        process_info["meaurements"],
        process_info["shared"],
        color="purple",
        label="shared",
    )
    memory.plot(
        process_info["meaurements"],
        process_info["data"],
        color="blue",
        label="data",
    )
    files.plot(
        process_info["meaurements"],
        process_info["num_fds"],
        color="black",
        label="Opened file descriptors",
    )
    threads.plot(
        process_info["meaurements"],
        process_info["num_threads"],
        color="pink",
        label="Number of threads",
    )

    memory.legend()
    files.legend()
    threads.legend()

    if args.save:
        plt.savefig(f"{args.pid}.png", format="png")

    if not args.no_show:
        plt.show()


if __name__ == "__main__":
    main()

To run it simply install dependencies and run like this:

pip install psutil matplotlib
python ./script.py --pid "<pid>"

[robjtede]

There is no easy answer to "why is usage high". The framework indeed caches objects for re-use, but it does not account for the whole picture. Your program/system allocator is likely holding on to blocks of memory for re-use, too.

[fggarcia]

Do you try to use jemalloc as allocator? I have the same issue about memory consume (always increase and never reclaim this). After I add jemalloc as allocator, app always consume the same memory size.

[s3rius]

I ran heaptrack to find leaks and I guess I found it.

After killing the program with kill -USR1 <pid>, I found this in leakage report:

Full stack trace gave me more information. I guess it's an issue of a bytes' crate or misuse of the reserve which can cause allocations.

Full heaptrack report:
leak.zip