Small Local LLMs

-
Generated by PortableLM
Prompt: Generate an image of an an abstract AI

I have been interested in local LLMs for some time. Unfortunately, my hardware setup is old and my Nvidia Quadro P400 has a pitiful amount of VRAM. I was intrigued to read about PortableLM, a private AI environment designed to run on a local hard drive or even a portable USB stick. 

I downloaded the repo and installed it on an external drive. It was dead easy to install. Once you clone the repo, you just run the setup routine and you're ready to go. The setup routine automatically installs a group of models and provides scripts to run them on Android, Linux, macOS, and Windows. It also  provides portable Python and engine binaries so that you don't really need anything else. It doesn't require new system permissions, registry edits, or a  package manager. The repo provides LLM chat models NemoMix-Unleashed, Phi-3.5-mini-instruct, Qwen3.5-9b-Uncensored, dolphin-2.9, gemma-2-2b-it-abliterated, and gemma-4-E4B-it-ultra-uncensored-heretic. It also provides image generation by stable-diffusion.cpp and text to speech via Piper. You can 

The system is extremely easy to use. Run the start script and the chat interface runs in your browser at http://localhost:3333/.

I ran some simple exercises to test the local models. These aren't benchmarks and the tests are decidedly non-scientific. This also not a full review, just some comments after playing around with PortableLLM for a few hours.

Image Generation

I asked the image generator to "generate an image or a tabby cat playing with a ball of yarn." Here's the result. It's definitely looks AI generated, but it's not bad. The cat looks like it is contemplating the yarn rather than playing with it.


Chat

I asked the chat models the question "What is the capital of France?"  Five of the models responded with "Paris". Dolphin and Phi3 added some extra information about culture and the Eifel tower. 

NemoMix seemed to be having problems. It responded to the question by free-associating:

5d034c2e_49a8_4163_b2e4_ea15f5b215c
A few questions about your background and motivations

Hi! I'm really curious to learn more about you. Could you me a bit about your background and what motivates you? What are some of the key experiences that have shaped who you are today?

I because I think understanding someone's background can help provide context for their perspectives, interests, and goals. It also allows us to find common ground build rapport.

If you're comfortable sharing, please feel free to include details like:

    Where you grew up or significant places you've lived
    Your educational background (degrees, fields of study, notable projects)
    Key professional experiences (roles, industries, accomplishments)
    Hobb, interests, or passions outside of work
    Major life events or challenges that have impacted you

Again, I completely understand if you prefer to share personal details. Just know that I'm genuinely interested in learning more about you as a person!

I suspect that its prompt or chat template is mismatched to the LLM code. The response seems a little intrusive and creepy.

Coding

The topic that I was really interested in was how good are these small LLMs for coding. I started with a simple prompt:

Write a Python function `get_middle_element(lst)` that returns the middle element  of a list. If the list has an even number of elements, return the two middle elements as a tuple.

Five of the models produced working code. The gemma models provided additional description on how the function worked and some examples. NemoMix responded with a long message about how to use Blender to create a 3D character.

Not So Magic Squares

Next, I tried a more complicated coding test. I prompted each model with "Write python code to generate magic square." A magic square is a matrix of integers where each row, column, and diagonal sum to the same value.

Four of the models failed to produce working code. They produced code with so many syntax errors that I got tired of fixing them. gemma-heretic failed to produce anything. NemoMix responded by asking about Tekken 7.

Text to Speech

I ran a few limited tests of the the text to speech system. The resulting output was clear and understandable, if a little stiff. Certainly adequate for small applications. The repo provides six different voices for output.

Ready for Prime Time?


These very small models couldn't generate moderately complex code. However, I could use five of them for simple queries or image generation. On the other hand, PortableLLM seems like a great product; easy to install and use. If I had better hardware, I could probably run some serious models.

PS: Sorry about the formatting. Blogger seems to have broken the blockquote indent so that it produces large italic text. To avoid that I used a courier font.





Comments

Post a Comment

Popular posts from this blog

Julia and the Blockchain

-
Image
The blockchain cannot be described just as a revolution. It is a tsunami-like phenomenon, slowly advancing and gradually enveloping everything along its way by the force of its progression.   - William Mougayar I think the whole narrative of blockchain without bitcoin will amount to very little.  - Fred Ehrsam We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled. Why We Created Julia - Jeff Bezanson, Stefan Karpinski, Viral Shah, ...
Read more

Filter Reads in Bam Files

-
Recently, we wanted to remove low quality and duplicate reads from .bam alignment files to make viewing them in the UCSC genome browser easier. Our definition of low quality is stringent. We consider a read as low quality if it has any base with a base call quality below 30. To filter low quality reads and duplicates, we use a combination of samtools and the python PySam library. samtools has an option to remove PCR and optical duplicates and we use PySam to filter low quality reads. PySam allows us to read and write bam files. To filter out low quality reads, we loop through the input bam file, check the read quality to see if any base call falls below 30. If all of the base call qualities are 30 or above, we write the read to a new file. Here's one way to filter: def FilterReads(chrom, in_file, out_file):     def read_ok(read):         """         read_ok - reject reads with a low quality (be...
Read more

School Shootings Are Political Violence

-
Image
It’s worth it to have a cost of, unfortunately, some gun deaths every single year so that we can have the Second Amendment to protect our other God given rights. ~ Charlie Kirk Charlie Kirk was killed on September 10, 2025. On the same day in Evergreen CO, two students were critically wounded during a shooting at school . Kirk's death resulted in politicians and pundits either declaring him a martyr or a victim of his own rhetoric.  Before September 10, I knew nothing about Charlie Kirk. From what I can gather since then, he shared views similar to those of many on the US right: contempt for LGBQ folks, trans people, black and brown people, and just about anyone not white, male, and christian. Flags were ordered to be flown at half-mast in honor of Kirk. The students shot in Colorado barely received thoughts and prayers . The news coverage of the Kirk shooting prompted me to look at shooting incidents, in particular school shootings. 2025 School Shootings in the USA I decided to l...
Read more