How To: Identify and Fix a Bug, Build Your Own Cosmo

[BradHutchings]

While the README walkthrough was helpful getting started with Cosmo, it took a lot of digging around and playing around to figure out the path from "I think there's a problem" to fixing it. So here is my journey.

Inspired by llamafile, I forked llama.cpp to give it the Cosmo treatment — one binary cross architecture and platform + package some supporting files.

My project is Mmojo-Server. It's open source + I periodically publish builds on HuggingFace. I also package Mmojo Server in an appliance based currently on Raspberry Pi. I hope to offer Intel NUC, AMD Ryzen AI, and Mac Mini appliances soon.

The problem was that when run on x86, it was crashing after loading some specific models: IBM Granite v3.3 and Meta Llama v3.2.

---

Run in gdb to catch the crash:

gdb --args .\mmojo-server.exe -m IBM-Granite-2B-Instruct-v3.3-q8_0.gguf

It shows me:

common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
[New LWP 1407]
[New LWP 1408]

Thread 1 "mmojo-server" received signal SIGSEGV, Segmentation fault.
0x00000000013a8c38 in _mm_load_si128 (__P=0x0) at ./third_party/intel/emmintrin.internal.h:536
warning: 536    ./third_party/intel/emmintrin.internal.h: No such file or directory
(gdb) bt
#0  0x00000000013a8c38 in _mm_load_si128 (__P=0x0) at ./third_party/intel/emmintrin.internal.h:536
#1  memchr_sse (s=0x0, c=0 '\000', n=0) at libc/intrin/memchr.c:42
#2  0x0000004100000041 in ?? ()
#3  0x0000000000000000 in ?? ()

Alternatively, I can run with the --strace flag, redirect to a file:

.\mmojo-server.exe -m IBM-Granite-2B-Instruct-v3.3-q8_0.gguf --ftrace > server.ftrace 2>&1

I see this at the end:

FUN �[1;30m 27832�[0m �[1;35m 14544�[0m    136'447'679'490 20'320                   std::__1::__bracket_expression<char, std::__1::regex_traits<char>>::__exec(std::__1::__state<char>&) const
FUN �[1;30m 27832�[0m �[1;35m 14544�[0m    136'447'689'138 20'560                     isascii
FUN �[1;30m 27832�[0m �[1;35m 14544�[0m    136'447'694'506 20'560                     memchr
FUN �[1;30m 27832�[0m �[1;35m 14544�[0m    136'447'708'646 24'352                     __sig_unmaskable
FUN �[1;30m 27832�[0m �[1;35m 14544�[0m    136'447'714'142 24'416                       __sig_death

---

The first thing to figure out is why it's crashing. gdb tells us that memchr_sse is being called with NULL, 0, 0 params. Looking at the implementation of memchr_sse on libc/intrin/memchr.c, you can see there is no checking for NULL before derefencing s in a call to _mm_cmpeq_epi8 (inlined?):

#if defined(__x86_64__) && !defined(__chibicc__)
static const char *memchr_sse(const char *s, char c, size_t n) {
  const char *e = s + n;
  __m128i t = _mm_set1_epi8(c);
  unsigned m, k = (uintptr_t)s & 15;
  m = _mm_movemask_epi8(
      _mm_cmpeq_epi8(_mm_load_si128((const __m128i *)((uintptr_t)s & -16)), t));

Meanwhile, the non-SSE version effectively checks for n == 0 in the for statement:

static inline const unsigned char *memchr_pure(const unsigned char *s,
                                               unsigned char c, size_t n) {
  size_t i;
  for (i = 0; i < n; ++i) {
    if (s[i] == c) {
      return s + i;
    }
  }
  return 0;
}

That explains why there's a crash here on x86 but not ARM.

So, do I fix it here or find something upstream? The function trace log, when I scroll up, tells me this is happening in processing a regular expression as part of Google Minja. Due to the size of that function trace log, it proved too diffcult to figure out the culprit calling that in the llama.cpp code.

I decided to put a parameter check in memchr_sse():

static const char *memchr_sse(const char *s, char c, size_t n) {
  if ((s == NULL) || (n == 0)) return 0;
  const char *e = s + n;
  __m128i t = _mm_set1_epi8(c);
  unsigned m, k = (uintptr_t)s & 15;

---

Now I have to clone the Cosmopolitan repo, make a change, and build the repo. I hadn't had to do that before, and this took me a little more than a day to figure out. Specifically, I stumbled on the tool/cosmocc/package.sh script only by digging through the repo file by file looking for some clue. LOL.

BUILD_COSMOPOLITAN_DIR="1-BUILD-cosmopolitan"
git clone https://github.com/jart/cosmopolitan.git ~/$BUILD_COSMOPOLITAN_DIR
cd ~/$BUILD_COSMOPOLITAN_DIR
# Edit the memchr_sse() function to check params.
sed -i '39i \  if ((s == NULL) || (n == 0)) return 0;' libc/intrin/memchr.c
# Build  cosmo -- This takes 20ish minutes.
tool/cosmocc/package.sh

At this point, there is a directory with everything built: ~/$BUILD_COSMOPOLITAN_DIR/cosmocc.

Instead of downloading the latest from cosmo.zip, copy that directory to where you need it locally. I like to make a copy so there's no chance of wrecking what I just built.

---

I build Mmojo-Server as detailed here. After that, I configure the Cosmo build as detailed here.

Voila, no more crash when loading those particular models.

---

Hopefully this will save someone some time. Or could be integrated clearly into the repo README.

Cosmopolitan is an amazing little project! If I had $1M to throw at it, I would find a business model. The key point is that making all platform decisions at compile time (like llama.cpp does) limits the user base to people who know how to build. My user base isn't even great with running stuff from a command line. And why I'm selling an appliance. Also, CPU isn't dead, especially with small LLM inference.

-Brad

[moonbeam5115]

You're my hero, Brad. Good to see the debugging process and how to push through initially unknown issues