LoRA Adapters from PVC and S3 Compatible Storage Server

[tuckkin]

Hi.. may I know if there is any timeline for LoRA adapters to be loaded from PVC (PV with NFS) and also the S3 Compatible Storage Server such as MinIO? Thank you.
https://www.kubeai.org/how-to/serve-lora-adapters/

[nstogner]

Hey, we should support s3:// urls today (screenshot from that link):

We don't currently have PV/PVC (NFS) based adapters on the roadmap at the moment. We have mainly seen adapters stored in buckets. Is NFS a hard requirement for you or will S3 do?

Note: We have an open bug in adapter support today, linking in case you run into it. We plan to tackle this bug soon: #347

Reach out here or on the discord channel if you encounter any issues with adapters. We need to put some more focus on this part of the project.

[tuckkin]

Thanks Nick for your response. Yeah I know s3:// is supported today but only applicable to the s3 bucket from AWS. However, if the s3 bucket is from other s3 compatible storage server, such as MinIO, then I will have the following error due to the hostname in the s3 URI. So is there a way for me to specify the hostname of the s3 storage server other than AWS? Currently the MLOps solution that I am using support the upload of the lora adapters to the local s3 server with MinIO, and we would expect the KubeAI operator running on-premise, sitting close to the local s3 server as well.

fatal error: Parameter validation failed:
Invalid bucket name "minio.cnasg.local:443": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
2025-03-07T05:33:04Z    ERROR   Reconciler error        {"controller": "model", "controllerGroup": "kubeai.org", "controllerKind": "Model", "Model": {"name":"llama-3.2-1b-instruct","namespace":"kubeai"}, "namespace": "kubeai", "name": "llama-3.2-1b-instruct", "reconcileID": "c8e48d2f-b107-4bd4-b1bb-c676754a1aff", "error": "reconciling adapters: exec adapter load for pod \"kubeai/model-llama-3.2-1b-instruct-79698bd769-t88fg\": exec adapter load: streaming: command terminated with exit code 1"}

Thank you.

[samos123]

Based on the logic in the code, I think @tuckkin is right that there is no way to use s3 compatible storage (non AWS) today because you can't override the S3 endpoint url in the loader container.

For loading from S3 we use aws cli: https://github.com/substratusai/kubeai/blob/7505cce83a19758ae66022a4aed6d77501fe777f/components/model-loader/load.sh#L51

Currently the loader container won't take any env variables defined in the model: https://github.com/substratusai/kubeai/blob/7505cce83a19758ae66022a4aed6d77501fe777f/internal/modelcontroller/adapters.go#L191

The aws cli can support minio by setting the AWS_ENDPOINT_URL env variable.

Would this be a workable solution for you?

apiVersion: kubeai.org/v1
kind: Model
metadata:
  name: tinyllama-chat
spec:
  features: [TextGeneration]
  owner: meta-llama
  url: hf://TinyLlama/TinyLlama-1.1B-Chat-v0.3
  # Note this doesn't work today because the loader port doesn't use the env variables
  env:
    AWS_ENDPOINT_URL: minio.cnasg.local:443
  adapters: # <--
  - name: colorist
    url: s3://jashing/tinyllama-colorist-lora
  engine: VLLM
  resourceProfile: nvidia-gpu-l4:1
  minReplicas: 1

[tuckkin]

Thx @samos123, yup sounds good to me. :)

[samos123]

I created this issue to track this work: #451

This should be a fairly easy fix. Thanks for raising this.

[tuckkin]

Cool, thank you very much @samos123! :)