A follow-up to Local LLM coding agent weekend. Last time the model picked one config and we moved on. This time I actually measured the tuning parameters, and then capped the GPUs to save power.

In short, raising max-num-seqs from 2 to 8 was the big win: four-request throughput jumped 55%. Speculative decoding (MTP) made short responses feel quicker but did not move throughput, and it shrank the KV cache. Doubling context to 256k fit, though it only allows just over one full-context request at a time. Capping the GPUs to 250 W cost a single user nothing.

Setup

• 2x RTX 3090 (24 GB) in one machine, tensor-parallel 2, no NVLink.
• vLLM 0.28.0, running as a vllm systemd service.
• Model: Avuja/Qwen3.8-27B-int4-AutoRound (INT4, Marlin kernels, BF16 activations).
• The model reports a 262144-token context and ships MTP tensors (explained below).

I benchmarked with a small Python harness that streams /v1/completions at temperature 0, always requesting exactly 256 output tokens with ignore_eos, plus a separate context probe that feeds a repeated token-id sequence up to the context limit. It's deliberately a synthetic comparison, not a broad workload study: enough to tell me which setting matters, not enough to publish a leaderboard.

The lever that mattered: max-num-seqs

My production config had been --max-num-seqs 2 since last weekend (conservative, "one agent, queue up the rest"). The obvious thing to try was letting more requests run concurrently:

--max-num-seqs 8 --max-num-batched-tokens 8192

max-num-seqs is the active-sequence cap; max-num-batched-tokens is how much prefill+decode work one iteration can do. The result:

Three things stand out. First, single-request decode speed is unchanged at about 58 tok/s; concurrency costs a single client nothing, which makes this a pure win. Second, aggregate throughput scales roughly in line with concurrency up to the cap, from 77 to 120 to 174 tok/s. Third, first-token actually improves as concurrency rises (3.90 s to 2.29 s at four requests), because chunked prefill spreads the long-prompt work across the extra sequences. That last one is counterintuitive; I'd expected the opposite.

So I kept it: the service now runs --max-num-seqs 8 --max-num-batched-tokens 8192.

Speculative decoding (MTP): nice latency, weaker throughput

MTP stands for Multi-Token Prediction. Rather than emitting one token per forward pass, the transformer gets a small draft head on top that predicts the next few tokens in advance. Those drafts are checked against the main model, and when they match they are accepted at no extra cost: the same forward pass that would have produced one token now turns out several. It is speculative decoding baked into the checkpoint itself, not a separate draft model that vLLM has to manage. Qwen3.8 ships the MTP tensors, and vLLM exposes it as speculative_config; the relevant field is num_speculative_tokens, how many draft tokens to attempt per step. I started with one:

--speculative-config '{"method":"mtp","num_speculative_tokens":1}'

At one speculative token the draft acceptance rate was around 86 to 92 percent, with a mean acceptance length of about 1.9. On its face that is a good number. Two things hold it back in practice. The first is that the speculative state eats into the KV cache: the pool shrank from about 280k to 242k tokens. The second is that it never beat the plain 8-seq configuration on throughput.

Going higher with num_speculative_tokens could in principle accept more tokens for free, but every extra draft token also carries its own chance of a rejection that wastes the step, and it costs more state. It is not automatically better, so I would benchmark it per model rather than assume.

In the end it does make short responses feel a touch snappier for a single user, but it steals cache from the thing I care about more, which is long agent context. I left it off by default and left it available as an opt-in profile.

Single GPU: it loads, don't rely on it

To free up a GPU for other work I tried running TP=1 on just GPU1 with a 32k context and a 0.95 memory budget. It loads and passes a 32k probe, and at 95% budget it fits ~31.7k tokens (a 90% budget only gets ~16.5k).

But two things keep it off the critical path: single-GPU decode is slower than TP2 (47 vs 58 tok/s), and the run threw an asynchronous engine-dead error under a heavier workload even though the benchmark files completed. Workable as a fallback to park GPU0, not something I would serve the main agent from.

256k context: it fits, but barely, for full-context requests

The model advertises 262144 tokens, so I raised it:

--max-model-len 262144

vLLM fit a KV pool of about 288k tokens, which is only 1.1x a full 256k request. So I can do one full-context request at a time, which is fine for my workload (a single big agent conversation), but not several. The original 128k setting gave me roughly 2.1x. The trade worked in my favor: I kept 8 active sequences for short turns while still holding a full 256k ceiling for the long ones.

The launcher (vllm-start) selects a profile through an environment variable, so switching is a one-liner, not a recompile:

Capping the power: 250 W

Once the throughput was sorted, the next lever was energy and thermals. The 3090s were sitting well under their default ceiling, and a benchmark with the limit set to 275 W came back clean: no HTTP 500 responses, no OOM, no engine faults, and decode at or above the earlier run. So I went further, to 250 W:

/usr/bin/nvidia-smi -i 0,1 -pl 250

It applied cleanly and the server stayed healthy. Single-request decode peaked at about 61.5 tok/s at the cap, no measurable loss for a single user.

The real question was whether the cap would bite under concurrent load, where a fixed power budget matters more than it does for a single stream. I reran the concurrency benchmark against the capped server over the network. Four concurrent requests came in at roughly 146 tok/s, statistically the same as the about 141 tok/s the 275 W run produced at that point. Pushing to eight concurrent requests gave about 214 tok/s, with no sign of throttling. In fact the capped numbers ran slightly higher than the earlier ones, which is run-to-run variance, not a real gain.

This is where a caution I had logged in an earlier draft paid off. I first tried putting that nvidia-smi line at the top of the service's startup script. It died on Insufficient Permissions, because it is a privileged operation; the set -e at the top then killed the whole startup. Wrapping it in sudo did not help either: a unit's main process has no TTY, so there is nothing to read a password from. The fix was to move the privilege into its own root unit, ordered to run before the model:

# gpu-power-limit.service
[Unit]
Description=Limit GPU power
Before=vllm-qwen.service
[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -i 0,1 -pl 250
[Install]
WantedBy=multi-user.target

with Wants=gpu-power-limit.service and After=gpu-power-limit.service in the vLLM unit. Wants (rather than Requires) is deliberate: if the cap ever fails to apply, the model still starts. Power is a nicety; inference is not. The vLLM script itself does no privileged work, so nothing in it can die on a permission error.

Wrapping up

A few things I would carry into the next model run. Start with the concurrency settings, the max-num-seqs and max-num-batched-tokens pair: that was the biggest single win and it cost a lone user nothing, so there is no reason not to raise it before touching anything else. Be more careful with speculative decoding. If a model ships MTP or EAGLE, an acceptance-rate number will always look impressive; the real cost is in the KV cache it steals, so profile it against your actual context budget and judge it on end-to-end throughput, not on how often the drafts hit. Watch the KV pool size rather than the max-model-len flag: a 256k context can fit in memory and still leave you just over a single full request from capacity, a detail nvidia-smi will not volunteer and only the startup report will show. And keep the two problems separate: a config meant to free a GPU and one meant to serve the agent are not the same job, and a profile that merely loads does not deserve to front a daemon until it has survived a sustained, heavier workload.

One last, small thing. While all of this was going on I also came across nvtop, a top-style live monitor for GPUs. It refreshes constantly and shows per-GPU utilization, power draw, and memory, with a process view underneath so you can see what is actually occupying the cards. It is not a substitute for nvidia-smi, which is still what I reach for when I need an exact field or am scripting something, but for the constant "what is the GPU doing right now" questions during a tuning session it is much easier on the eyes.

Goal

Run coding agents from VS Code while doing all LLM inference on a Linux workstation:

• Threadripper Pro workstation
• 128 GB RAM
• 2× RTX 3090 24 GB
• GPUs connected over PCIe (PHB), no NVLink
• Ubuntu 24.04 HWE
• vLLM providing an OpenAI-compatible API over the LAN
• VS Code Copilot using a custom model endpoint

Final daily-driver setup

The best-performing model tested was:

ulkaa/Qwen3.8-27B-AWQ-INT4

Working vLLM configuration:

CUDA_VISIBLE_DEVICES=0,1 \
vllm serve ulkaa/Qwen3.8-27B-AWQ-INT4 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 131072 \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Important: do not use --enforce-eager for this model.

With eager mode enabled, simple generation was around 16 tokens/s.

With CUDA graphs enabled by removing --enforce-eager, it jumped to roughly 66 tokens/s.

CUDA graph memory itself was tiny: about 0.07 GiB/GPU.

Memory at 128k context

At startup vLLM reported approximately:

• model/runtime: ~9.2 GiB/GPU
• peak activation: ~1.6 GiB/GPU
• KV cache: ~9.25 GiB/GPU
• CUDA graphs: ~0.07 GiB/GPU
• total visible usage: ~19–20 GiB/GPU

KV capacity:

GPU KV cache size: 290,768 tokens
Maximum concurrency at 131,072 tokens: 2.22x

So 128k context with max-num-seqs=2 fits surprisingly comfortably.

A key lesson: increasing --max-model-len does not necessarily increase idle nvidia-smi memory usage. vLLM allocates a KV-cache pool according to its GPU memory budget; max-model-len mostly determines how much of that pool one request is allowed to consume.

VS Code configuration

[
  {
    "name": "Workstation vLLM",
    "vendor": "customendpoint",
    "apiKey": "dummy",
    "apiType": "chat-completions",
    "models": [
      {
        "id": "ulkaa/Qwen3.8-27B-AWQ-INT4",
        "name": "Qwen3.8 27B INT4",
        "url": "http://192.168.0.167:8000/v1/chat/completions",
        "toolCalling": true,
        "vision": true,
        "maxInputTokens": 120000,
        "maxOutputTokens": 8000
      }
    ]
  }
]

Advertising slightly less than the server's hard 131 072-token limit gives Copilot room for output and avoids requests landing exactly on the context ceiling.

Model lessons

Parameter count is a terrible proxy for deployment size

We managed to run GPT-OSS-120B, yet initially struggled with a dense 27B Qwen model.

Why?

GPT-OSS-120B is an MoE model shipped heavily quantized. Although it has ~120B total parameters, only a small fraction are active per token, and the distributed checkpoint is around 60 GB.

The stock Qwen 27B model was dense BF16:

27B × ~2 bytes ≈ 54 GB

before runtime overhead, KV cache, activations, vision components, etc.

48 GB of physical VRAM simply isn't enough.

The useful questions are therefore:

• Dense or MoE?
• Bits per weight?
• Checkpoint size?
• Active parameters per token?
• KV/state requirements?
• CPU offload?
• Architecture/kernel support?

Not merely: "How many billion parameters?"

Quantization was the right answer for Qwen

The AWQ INT4 version is dramatically more practical on 2×3090:

• fits entirely on GPUs
• large KV-cache budget remains
• 128k context works
• no CPU weight streaming
• ~66 tok/s
• vision
• structured tool calling

This is much better hardware utilization than forcing a high-precision checkpoint onto insufficient VRAM.

CPU offload works, but PCIe becomes the bottleneck

GPT-OSS-120B ran successfully with roughly 14 GB of CPU offload.

It achieved around 11 tok/s after tuning.

nvidia-smi dmon showed huge sustained PCIe reads — around 8–12 GB/s — while the GPUs were at 100% SM utilization.

The limiting resource wasn't GPU compute; it was moving offloaded weights from RAM to VRAM.

Reducing offload from 20 GB to 14 GB improved generation from roughly 7 → 11 tok/s.

12 GB offload didn't boot.

Conclusion: CPU offload is useful for making otherwise impossible models run, but it is not equivalent to having enough VRAM.

A third 3090 would potentially be a much more meaningful upgrade for those huge models than small software tweaks.

Tool calling matters more than expected

A model merely producing plausible JSON/XML is not sufficient.

Direct curl tests against /v1/chat/completions were invaluable because they separated:

model/vLLM problem

from:

VS Code / Continue problem

Before connecting a new model to an agent, test:

1. plain completion
2. reasoning output
3. OpenAI-style tool_calls
4. only then the agent/client

For Qwen3.8:

--tool-call-parser qwen3_coder
--reasoning-parser qwen3

produced proper OpenAI tool-call structures.

Copilot has enormous context overhead

VS Code Copilot Agent mode can consume a large fraction of the context window before actual project content appears.

Observed contributors included:

• system instructions
• tool definitions
• conversation
• workspace/browser context

At smaller context sizes this caused hard vLLM context errors and frequent compaction.

32k was technically usable but restrictive.

64k worked with no noticeable performance or memory penalty.

128k also works and is a much more sensible target for real agent work.

max-num-seqs

--max-num-seqs controls how many sequences vLLM can actively schedule concurrently.

It does not reserve a complete max-length context for each request ahead of time.

For a personal agent server:

1 = extremely conservative
2 = good default
4+ = useful for real parallel subagents/multiple clients

Extra requests normally queue rather than immediately fail when all sequence slots are occupied.

2 is a good fit because Copilot occasionally performs concurrent/auxiliary calls even without intentionally launching subagents.

Vision

The earlier models were text-only:

• Qwen2.5-Coder-7B
• Qwen3-Coder-Next
• GPT-OSS-20B
• GPT-OSS-120B

Qwen3.8-27B is multimodal.

Its vision capability means:

image/screenshot → model → text/tool calls

It does not generate images.

For Copilot this allows screenshots/images to be included in prompts, hence:

"vision": true

vLLM vs Ollama

Ollama's major operational advantage is not necessarily inference speed. It is model lifecycle management.

One Ollama API can expose many installed models and load/swap them automatically.

A normal vLLM process serves one loaded model.

That matters when accessing the workstation remotely: manually SSHing in to stop/start model servers is undesirable.

Long-term architecture:

VS Code
      |
      v
stable OpenAI-compatible gateway
      |
      +---- Qwen fast/vision
      |
      +---- GPT-OSS smart/slow
      |
      +---- future models

The gateway/model manager can start the requested vLLM backend and stop or sleep whichever model currently owns the GPUs.

This gives the Ollama-style UX while retaining vLLM's performance and tuning control.

systemd is the right way to run the main server

The Qwen server now runs as a system service and starts at boot, not user login.

Important service environment:

User=user
WorkingDirectory=/home/user

Environment="HOME=/home/user"
Environment="PATH=/home/user/miniforge3/envs/vllm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="CUDA_HOME=/home/user/miniforge3/envs/vllm"
Environment="CUDA_VISIBLE_DEVICES=0,1"

Then call vLLM directly from the Conda environment:

ExecStart=/home/user/miniforge3/envs/vllm/bin/vllm ...

No conda activate is required.

Conda activation is mostly environment-variable manipulation; systemd can call the environment's executable directly.

Conda was preferable for CUDA management

Using a Conda/Miniforge environment made the CUDA toolkit much less intrusive than installing another system-wide CUDA toolchain.

Current environment ended up with:

• CUDA toolkit 13.2
• PyTorch built for CUDA 13.2
• vLLM 0.28
• CUDA_HOME=$CONDA_PREFIX

vLLM/PyTorch pip packages still install many NVIDIA CUDA runtime wheels. This is intentional: Python packages often depend on specific CUDA user-space libraries even when a toolkit is already available.

Hugging Face downloads

hf download reuses completed cached blobs, but interrupted large/Xet-backed files may appear to restart from the beginning rather than resume byte-for-byte.

Don't assume a progress bar restarting at zero means the complete model is being downloaded again.

Useful cache inspection:

du -sh ~/.cache/huggingface/hub/models--OWNER--MODEL
find ~/.cache/huggingface/hub/models--OWNER--MODEL \
  -name '*.incomplete' -ls

Network/access

For LAN testing:

http://192.168.0.167:8000

works.

For access away from home, don't expose vLLM directly through router port forwarding.

The intended setup is Tailscale on the laptop and workstation, then use the workstation's Tailscale 100.x.x.x address as the permanent model endpoint.

This gives an encrypted WireGuard-based path and lets the same coding setup work away from home.

Biggest lessons

1. Quantization and architecture matter more than parameter count.
2. Enough VRAM beats CPU offload.
3. CUDA graphs can transform performance.
4. Large context is essential for modern coding-agent harnesses.
5. Direct API tests should precede debugging the client.
6. Tool-call parser compatibility is a hard requirement for agents.
7. Ollama solves lifecycle management; vLLM solves high-performance serving.
8. A gateway/model manager is the natural next piece of infrastructure.
9. Treat NVIDIA/kernel upgrades conservatively and preserve a known-good kernel.
10. 2× used RTX 3090 remains an absurdly capable local inference setup when the model is chosen appropriately.

Prerequisites

Activate developer mode and enable USB debugging. Connect the phone via USB to a computer that has adb installed.

Create unencrypted backup

We temporarily downgrade to a version of the WhatsApp app that still supports adb backup. This way, we do not need to root our phone.

adb devices
adb shell getprop ro.build.version.sdk
adb shell am force-stop com.whatsapp
adb shell pm path com.whatsapp
adb shell cp {whatsapp_apk_path} /data/local/tmp/WhatsAppbackup.apk  # TODO also cp the two other files?
adb ls /data/local/tmp
adb shell pm uninstall -k com.whatsapp
adb reboot
adb devices
adb install -r -g --bypass-low-target-sdk-block LegacyWhatsApp.apk
adb shell am start -n com.whatsapp/.Main  # ignore any warnings you get
adb backup -f whatsapp.ab com.whatsapp
adb shell pm uninstall -k com.whatsapp
adb shell pm install /data/local/tmp/WhatsAppbackup.apk  # or just reinstall from Play Store

Extract backup

Using Android Backup Extractor (ABE)

curl https://github.com/YuvrajRaghuvanshiS/WhatsApp-Key-Database-Extractor/blob/master/bin/abe.jar -o abe.jar
# or via <https://github.com/nelenkov/android-backup-extractor/releases/tag/latest>
java -jar abe.jar unpack whatsapp.ab whatsapp.tar {backup_password}
tar -xf whatsapp.tar
mkdir extract
cp apps/com.whatsapp/f/key extract/key  # encryption key
cp apps/com.whatsapp/db/msgstore.db extract/msgstore.db  # decrypted messages
cp apps/com.whatsapp/db/wa.db extract/wa.db  # decrypted contact info

View backup

git clone https://github.com/absadiki/whatsapp-msgstore-viewer
cd whatsapp-msgstore-viewer
conda create -n venv-whatsapp python=3.9
conda activate venv-whatsapp
pip install -r requirements.txt
python main.py

Portal, a puzzle-platform game developed by Valve, was first released in 2007 as part of The Orange Box. It quickly became a cult classic, known for its unique mechanics and dark humor. Set in a mysterious research facility, the game revolves around the use of a "portal gun" to create linked portals that allow the player to navigate through the environment and solve puzzles. It’s widely regarded as one of the best games of the 2000s and remains a staple of gaming culture.

In recent years, Portal became available on multiple platforms, including macOS. However, with the increasing focus on 64-bit computing, Valve's macOS version of Portal remained stuck in the 32-bit era, making it incompatible with the latest macOS updates. In 2020, Apple announced that macOS Catalina (10.15) and beyond would no longer support 32-bit applications, forcing many older games and software to be abandoned or require updates to work on newer systems.

For those looking to play Portal on macOS today, there is a workaround: using the leaked Source Engine code to build the game from scratch. The source code was made public years ago, and while it has some issues out of the box, with a bit of tinkering, it can be compiled to work on modern macOS versions.

In this post, I'll walk you through the steps to build Portal on macOS using the leaked Source Engine, specifically for users who are dealing with Apple’s 64-bit-only policy. The process involves downloading the leaked source code, building the engine, downloading the necessary game assets, and combining everything to make Portal run on your system.

Step 1: Download the Leaked Source Code

The leaked source code for the Source Engine is available on GitHub. However, you'll want to use a specific fork for it to build successfully on macOS.

• Main repository: nillerusr/source-engine
  • Unfortunately, this repository doesn’t build successfully on macOS by default
• Working fork: er2off/source-engine
  • This commit fixes the build issues

Step 2: Build the Source Code

Once you've downloaded the source, follow these instructions to build it.

Prerequisites

Install the required dependencies:

xcode-select --install
brew install sdl2 freetype2 fontconfig pkg-config opus libpng libedit jpeg jpeg-turbo python3

Next, set up your workspace:

cd ~/workspace
git clone --recursive https://github.com/er2off/source-engine.git
cd source-engine
git checkout clang19

Build the Engine

Now you can configure and build the source:

python3 waf configure -T release --prefix='' --build-games=portal
python3 waf build
python3 waf install --destdir='~/Documents/Gaming/Portal'

Step 3: Download Game Assets from Steam

Portal for macOS is still available on Steam today, but only as 32-bit version. This means you can download it, but you cannot run in. Our goal is to combine the assets from this download with our own 64-bit game engine build. Unfortunately, recent updates to the game have made it incompatible with the leaked source engine. The last version of Portal from 2024 that works with the leaked engine can be found on SteamDB:

• Patch Notes
• Depot Info

Luckily, the current beta branch "SteamPipe Beta" points to this older version, so it is very easy to download from Steam.

Step 4: Combine the Engine and Assets

Now that you’ve built the engine and downloaded the necessary game files, it’s time to combine them. First we back up Steam's Portal folder, then delete the 32-bit binaries, and finally replace them with our own 64-bit versions.

cd ~/Library/Application\ Support/Steam/steamapps/common/Portal
cp -r . ~/Portal_backup
rm -rf ./bin ./portal/bin ./hl2_osx
cp -r ~/Documents/Gaming/Portal/bin ./bin
cp -r ~/Documents/Gaming/Portal/portal/bin ./portal/bin
cp ~/Documents/Gaming/Portal/hl2_launcher ./hl2_osx

Note how hl2_launcher gets renamed to hl2_osx.

Step 5: Run the Game

Finally, you're ready to run the game!

./hl2_osx -game Portal

This should launch the game using the custom-built engine. In my experiments it runs flawlessly.

The launch button in Steam should now work as well.

What about Portal 2?

While this approach works for the original Portal (and Half Life 2), it does not work for Portal 2. Portal 2 requires a more recent version of the Source Engine, so it is not compatible with the leaked code we used. If you're looking to play Portal 2 or need a more straightforward way to play Portal on newer versions of macOS, there are several alternatives you can explore using emulation. I have not tried these myself, but googling for terms such as Wine, Whisky, Crossover and Proton should get you started.

Happy gaming!

PS: in retrospect it might be safer to combine the engine and the assets in a folder outside of Steam, so that it will not accidentally get overwritten by any incoming updates.

Introduction

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform that allows developers to harness the power of NVIDIA GPUs for general-purpose computing (GPGPU). CUDA provides a suite of tools and libraries that enable high-performance computing on GPUs, making it a go-to solution for a wide range of computational tasks, including deep learning.

CUDA competes with other GPU computing platforms, such as AMD's ROCm and Intel's OneAPI. Both ROCm and OneAPI are open-source platforms that offer similar capabilities to CUDA. However, CUDA remains dominant, especially in the AI and deep learning space, due to its mature ecosystem and widespread support.

CUDA can be deployed on various operating systems, including Linux, Windows, and macOS. However, it is worth noting that CUDA support for macOS was discontinued after version 12.5 due to Apple's transition to the new ARM-based architecture (Apple Silicon).

In this blog post, we will dive into CUDA by exploring it across three layers:

1. System-wide setup: We will cover the installation and configuration of the graphics driver.
2. CUDA and cuDNN setup: We will discuss two different approaches: system-wide or isolated.
3. Using CUDA: How to leverage CUDA in your projects, including the installation of GPU-accelerated libraries and frameworks.

System overview

For this guide, we will walk through setting up CUDA on a Linux system. Specifically, we will be using Ubuntu 23.10. While it would have been ideal to use a long-term support (LTS) version like 24.04, the differences in setup will be minimal.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 23.10
Release:        23.10
Codename:       mantic

$ uname -m
x86_64

$ uname -r
6.5.0-44-generic

$ ldd --version
ldd (Ubuntu GLIBC 2.38-1ubuntu6.3) 2.38
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Linux display driver Setup

When setting up CUDA on Linux, one of the first steps is ensuring that your display driver is correctly installed. On Linux, this is always a system-wide process, and you have two main options: the proprietary NVIDIA driver or the open-source Nouveau driver.

• Proprietary drivers: The official drivers provided by NVIDIA, offering the best performance and full support for CUDA. Common versions include 470, 525, 535, 545 and 550.

• Nouveau drivers: The Nouveau driver is an open-source alternative to NVIDIA's proprietary driver. While it provides basic functionality and is a good choice for general use, it does not support CUDA.

In this guide, we will be using the proprietary NVIDIA drivers. These drivers also include the nvidia-smi tool, which is vital for managing and monitoring your GPU.

There are two main procedures for installing the drivers: automatic or manual.

Automatic Installation

The easiest way to install the appropriate NVIDIA driver is through the automatic installation process, which detects your GPU and recommends the best driver.

$ ubuntu-drivers devices
== /sys/devices/pci0000:40/0000:40:01.1/0000:41:00.0 ==
modalias : pci:v000010DEd00002204sv000010DEsd0000147Dbc03sc00i00
vendor   : NVIDIA Corporation
model    : GA102 [GeForce RTX 3090]
driver   : nvidia-driver-470 - distro non-free
driver   : nvidia-driver-470-server - distro non-free
driver   : nvidia-driver-545 - distro non-free
driver   : nvidia-driver-545-open - distro non-free
driver   : nvidia-driver-535-server - distro non-free
driver   : nvidia-driver-535 - distro non-free recommended
driver   : nvidia-driver-535-open - distro non-free
driver   : nvidia-driver-535-server-open - distro non-free
driver   : xserver-xorg-video-nouveau - distro free builtin

$ ubuntu-drivers list
$ ubuntu-drivers install

Manual Installation

For those who prefer more control over the installation process, or if you want the latest drivers not available in the default Ubuntu repositories, you can manually install the driver.

You can install the display drivers either from the default Ubuntu repositories or from the additional PPA (Personal Package Archive) provided by Ubuntu's graphics drivers team if you want the latest and greatest.

$ sudo add-apt-repository ppa:graphics-drivers/ppa && sudo apt update

$ sudo apt install nvidia-driver-535

This command installs the NVIDIA driver version 535, but you can replace "535" with your desired version number.

Once installed, you can verify that the driver is correctly set up using the nvidia-smi tool:

$ nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        Off | 00000000:41:00.0  On |                  N/A |
|  0%   45C    P8              32W / 350W |    541MiB / 24576MiB |      7%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off | 00000000:42:00.0 Off |                  N/A |
|  0%   40C    P8              19W / 350W |     10MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2635      G   /usr/lib/xorg/Xorg                          192MiB |
|    0   N/A  N/A      2944      G   /usr/bin/gnome-shell                         82MiB |
|    0   N/A  N/A      3607      G   ...irefox/3626/usr/lib/firefox/firefox      134MiB |
|    0   N/A  N/A      4701      G   ...erProcess --variations-seed-version      116MiB |
|    1   N/A  N/A      2635      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+

Important Notes

• Stability vs. latest features: When choosing your driver version, consider the trade-off between stability and access to the latest features. Older drivers like version 470 are more stable and widely tested, while newer versions like 550 offer the latest updates and support for newer hardware.
• GPU compatibility: Ensure that the driver you select is compatible with your GPU model. The automatic detection method mentioned above typically handles this well.
• Bundled CUDA runtime: The NVIDIA driver comes with a minimal CUDA runtime (i.e., version 12.2) necessary for running basic CUDA applications. However, it does not include the full CUDA toolkit required for development purposes. The runtime version bundled with the driver will not change even if you separately install a proper CUDA runtime or toolkit.

To see where the CUDA runtime is located on your system, you can run:

$ find /usr -name libcuda.so*
/usr/lib/x86_64-linux-gnu/libcuda.so.1
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so.535.171.04
/usr/lib/i386-linux-gnu/libcuda.so.1
/usr/lib/i386-linux-gnu/libcuda.so
/usr/lib/i386-linux-gnu/libcuda.so.535.171.04

This command will display the paths to the installed CUDA runtime libraries, which are essential for running CUDA-enabled applications.

Compute capability

Compute capability determines the hardware capabilities of your GPU and cannot be upgraded through software. Here's a table summarizing the compute capabilities of various NVIDIA GPU architectures:

To check the compute capability of your GPU, you can use the following command:

$ nvidia-smi --query-gpu=compute_cap --format=csv
compute_cap
8.6
8.6

For more detailed information on compute capabilities and their implications, refer to the NVIDIA CUDA C Programming Guide.

CUDA and cuDNN

When working with CUDA, it is important to distinguish between the CUDA runtime and the CUDA toolkit, similar to the difference between the Java Runtime Environment (JRE) and the Java Development Kit (JDK). The NVIDIA driver includes a minimal CUDA runtime that enables you to run basic CUDA-enabled applications. However, this runtime is limited and does not include all the components needed for more advanced CUDA tasks.

The CUDA toolkit, on the other hand, is a comprehensive package that provides all the development tools necessary for creating, compiling, and running CUDA applications. It also includes a more complete runtime, which provides additional libraries and features needed for more complex applications. Most deep learning tasks will require this toolkit to function correctly.

When installing the CUDA toolkit, ensure that you only install versions that are less than or equal to the runtime version bundled with your display driver. Installing a higher version without updating the driver first can lead to instability.

In addition to the CUDA toolkit, some deep learning frameworks also require the CUDA Deep Neural Network (cuDNN) package.

System-wide installation

To set up CUDA and cuDNN system-wide, start by ensuring that the NVIDIA proprietary drivers are installed, as discussed earlier. Next, you will need to install a C++ compiler, which is required for compiling CUDA code. This can be done with sudo apt install gcc g++.

Once GCC is installed, you can proceed to install CUDA. You have two main options for this:

• The easiest approach is to use the official Ubuntu repository by running sudo apt install nvidia-cuda-toolkit
• Alternatively, for more control or to get the latest version, you can install CUDA directly from NVIDIA’s official source. This involves following the detailed instructions provided in the NVIDIA CUDA Installation Guide for Linux.

After installing CUDA, the next step is to set up cuDNN, which is essential for deep learning applications. You can do this by following the instructions in the NVIDIA cuDNN Installation Guide.

Isolated installation (recommended)

For situations where you need to manage multiple projects with different CUDA or cuDNN versions, setting up an isolated environment using Conda or Mamba is the recommended option. This approach keeps the system-wide components minimal and allows each environment to have its own specific setup.

Start by ensuring that the NVIDIA proprietary drivers are installed as discussed earlier, since they are the only system-wide component required. Next, install Mamba, which is a faster alternative to Conda, via Miniforge. You can verify that the installation was successful by running mamba info:

$ mamba info

          mamba version : 1.5.5
     active environment : None
            shell level : 0
       user config file : /home/user/.condarc
 populated config files : /home/user/miniforge3/.condarc
          conda version : 23.11.0
    conda-build version : not installed
         python version : 3.10.13.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen3
                          __conda=23.11.0=0
                          __cuda=12.2=0
                          __glibc=2.38=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /home/user/miniforge3  (writable)
      conda av data dir : /home/user/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/user/miniforge3/pkgs
                          /home/user/.conda/pkgs
       envs directories : /home/user/miniforge3/envs
                          /home/user/.conda/envs
               platform : linux-64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.10.13 Linux/6.5.0-44-generic ubuntu/23.10 glibc/2.38 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.5
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Relevance of virtual packages in mamba environments

Mamba environments utilize virtual packages to dynamically detect and represent certain system features that are critical for package resolution and compatibility. These virtual packages include system-specific details such as architecture, the operating system, the version of the GNU C Library (glibc), and, importantly, the version of CUDA supported by your installed NVIDIA drivers.

Virtual packages are not installed in the traditional sense but are automatically detected by Mamba. They help the package manager resolve dependencies by ensuring that the packages you install are compatible with your system's underlying hardware and software.

Among these virtual packages, __cuda is particularly important when working with CUDA. It represents the maximum version of CUDA that your NVIDIA driver officially supports. This information is automatically detected and provided by Mamba, assisting in the selection of the appropriate CUDA toolkit and related packages for your environment.

The output above indicates that our system's NVIDIA drivers support CUDA up to version 12.2. While Mamba does not strictly enforce this version when installing the CUDA toolkit, it serves as a guideline. You can technically install lower or higher versions of the toolkit, but installing a version higher than what __cuda indicates is generally not recommended, as it could lead to instability or compatibility issues. If your project requires a higher than supported version of the toolkit, consider upgrading your graphics driver first.

Creating a new mamba environment

Once Mamba is set up, we can proceed to install the CUDA toolkit within an isolated environment. We can do this by creating a new environment and specifying the CUDA version we need, along with any other packages such as Python or cuDNN. For example:

$ mamba create -n my-environment python=3.12 cuda-toolkit=12.2 cudnn
$ mamba activate my-environment

This command sets up a new environment named my-environment with Python 3.12, CUDA toolkit 12.2, and a recent, compatible version of cuDNN.

After activating your environment, you can verify that CUDA is correctly installed by checking the version of the NVIDIA CUDA compiler:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

Package overview

When working with CUDA 12, it is important to be aware of the significant changes in package names and structures compared to earlier versions. In CUDA 11 and earlier versions, the installation was typically done using the cudatoolkit or cudatoolkit-dev package. However, starting with CUDA 12, there has been a restructuring of package names. This shift makes installation trickier, especially as much of the older documentation still references these outdated package names.

The following meta-packages can be used to setup a CUDA 12 environment:

• cuda-toolkit: This package (notice the hyphen) is now the primary way of installing CUDA development tools. Running mamba install cuda-toolkit=12.2, for instance, will typically provide all the necessary components for CUDA 12.2 development, including the compiler, libraries, and headers. The downside is that it is fairly big, and it will likely include much more than what is strictly necessary for your use case.
• cuda-runtime: If you only need the runtime components, install this package. Note that this refers to the complete runtime, not the minimal one bundled with the driver.
• cuda: This is a meta-package that pulls in both the toolkit and runtime. Seeing as the runtime contains a subset of the packages in the toolkit, this is in effect functionally equivalent to the cuda-toolkit package.
• Other meta packages such as cuda-libraries, cuda-libraries-dev, cuda-compiler and cuda-tools can be used to specify more precisely what your project needs. Below is a simplified hierarchy of the most relevant packages. Check the appendix for a full list.

cuda
├── cuda-runtime
│   └── cuda-libraries
└── cuda-toolkit
    ├── cuda-compiler
    ├── cuda-libraries
    ├── cuda-libraries-dev
    └── cuda-tools
        └── cuda-command-line-tools

Finally, for the fullest possible control, refer to the actual CUDA packages instead of these meta-packages (which are simply groups of packages).

Note that neither cuda, cuda-runtime, nor cuda-toolkit include cuDNN. It is a separate package specifically tailored for deep learning applications, which needs to be installed independently as shown earlier.

Channel selection

When setting up CUDA and related packages in a Mamba environment, the various channels that offer similar packages might cause confusion. The two primary channels to consider are conda-forge (default in Miniforge) and nvidia, both of which offer nearly identical CUDA-related packages, including the CUDA toolkit, cuDNN, and other NVIDIA libraries. We will ignore the anaconda channel (default in Anaconda) because it typically hosts somewhat outdated package versions. Many mamba commands accept a -c <channel> option to include an extra channel on top of the default channels configured in the .condarc file.

The conda-forge channel is a widely used, community-driven repository known for its extensive package coverage beyond CUDA. This makes conda-forge particularly suitable for projects that require a mix of CUDA and other libraries. Additionally, conda-forge is continuously updated and maintained, ensuring that you have access to recent versions of packages.

In contrast, the official nvidia channel, maintained directly by NVIDIA, is dedicated specifically to CUDA and other NVIDIA tools. To install the complete CUDA suite from this channel, you can use mamba install -c nvidia cuda. Although the CUDA-related (meta-)packages in the nvidia channel are almost identical to those found in conda-forge, the nvidia channel provides slightly earlier access to the latest versions and includes some less common versions that may not be available on conda-forge.

In most cases, if your environment requires a wide range of software, conda-forge is likely the better option due to its extensive package offerings. It also helps to avoid some minor hiccups that can result from multi-channel package resolution.

Note that certain packages, such as pytorch also provide their own dedicated channel to install packages from.

Setting environment variables

For certain applications, you might need to manually set additional environment variables:

• CUDA_HOME and CUDA_PATH
  • These are interchangeable and typically point to the root of the CUDA toolkit folder, which contains the lib and bin directories.
  • In the case of a conda environment, they should point to the environment's root.
• LD_LIBRARY_PATH
  • Add $CUDA_HOME/lib to this path to ensure your system can locate the necessary libraries.

$ echo $CONDA_PREFIX
/home/user/miniforge3/envs/my-environment

$ which nvcc
/home/user/miniforge3/envs/my-environment/bin/nvcc

$ export CUDA_HOME=$CONDA_PREFIX
$ export CUDA_PATH=$CUDA_HOME

Install frameworks and libraries

Once CUDA and cuDNN are set up, the next step is installing the python packages that leverage GPU acceleration. Depending on your development environment, you can install these using either pip in a virtual environment or mamba.

Here are some popular libraries and frameworks:

• PyCUDA: Python wrapper for CUDA.
• CuPy: NumPy-compatible library that runs on CUDA.
• cuNumeric: A drop-in replacement for NumPy, optimized for CUDA.
• RAPIDS: A suite of libraries for data science and analytics on GPUs, including cuDF (a faster pandas) and cuML (a faster scikit-learn).
• Deep learning frameworks: TensorFlow, PyTorch, ONNX

For example, to install PyTorch with CUDA support using mamba:

$ mamba create -n torch-env -c pytorch -c nvidia python=3.12 pytorch-cuda=12.1 torchvision torchaudio


Looking for: ['python=3', 'pytorch-cuda=12.1', 'torchvision', 'torchaudio']

...

  Package                          Version  Build                         Channel           Size
──────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────────────────────

  + libcublas                    12.1.0.26  0                             nvidia           345MB
  + libcufft                      11.0.2.4  0                             nvidia           108MB
  + libcusolver                  11.4.4.55  0                             nvidia           103MB
  + libcusparse                  12.0.2.55  0                             nvidia           171MB
  + libnpp                       12.0.2.50  0                             nvidia           147MB
  + cuda-cudart                   12.1.105  0                             nvidia           193kB
  + cuda-nvrtc                    12.1.105  0                             nvidia            21MB
  + libnvjitlink                  12.1.105  0                             nvidia            18MB
  + libnvjpeg                    12.1.1.14  0                             nvidia             3MB
  + cuda-cupti                    12.1.105  0                             nvidia            16MB
  + cuda-nvtx                     12.1.105  0                             nvidia            58kB
  ...
  + libcurand                    10.3.7.37  0                             nvidia            54MB
  + libcufile                    1.11.0.15  0                             nvidia             1MB
  + cuda-opencl                    12.6.37  0                             nvidia            27kB
  + cuda-libraries                  12.1.0  0                             nvidia             2kB
  + cuda-runtime                    12.1.0  0                             nvidia             1kB
  ...
  + pytorch                          2.4.0  py3.12_cuda12.1_cudnn9.1.0_0  pytorch            1GB
  + torchtriton                      3.0.0  py312                         pytorch          245MB
  + torchaudio                       2.4.0  py312_cu121                   pytorch            7MB
  + torchvision                     0.19.0  py312_cu121                   pytorch            9MB

  Summary:

  Install: 180 packages

  Total download: 3GB

───────────────────────────────────────────────────────────────────────────────────────────────────


Confirm changes: [Y/n]

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To activate this environment, use

     $ mamba activate torch-env

To deactivate an active environment, use

     $ mamba deactivate

Here, the meta-package pytorch-cuda allows us to specify the required CUDA version. Since version 12.2 is not available, we settle for version 12.1. If we had installed the regular pytorch package, we would have downloaded the CPU version without CUDA acceleration. We can double check the build string of the pytorch package in the command output: py3.12_cuda12.1_cudnn9.1.0_0.

Notice how we need two extra channels: pytorch and nvidia. The first attempt without the nvidia channel failed because pytorch-cuda=12.1 has a dependency on a very specific version of cuBLAS that is unavailable in conda-forge.

Furthermore, notice how we did not specify cudnn this time. PyTorch with CUDA support includes a statically linked version of this library, so we don't need to include it separately.

When the environment is created and activated, we can test whether CUDA support is enabled.

>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.device_count()
2
>>> torch.cuda.current_device()
0
>>> torch.backends.cudnn.version()
90100

Topics for another time

• more elaborate pytorch example
  • or tensorflow vs tensorflow-gpu
• NVIDIA TensorRT

Appendix

Glossary

• cudaRT - CUDA runtime
• cuBLAS - CUDA BLAS
• cuFFT - CUDA Fast Fourier Transform
• cuDPP - CUDA Data Parallel Primitives
• cuDNN - CUDA Deep Neural Network
• cuRAND - CUDA Random Number Generation library
• cuSOLVER - CUDA based collection of dense and sparse direct solvers
• cuSPARSE - CUDA Sparse Matrix library
• NPP - NVIDIA Performance Primitives library
• nvGRAPH - NVIDIA Graph Analytics library
• NVML - NVIDIA Management Library
• NVRTC - NVIDIA Runtime Compilation library for CUDA C++
• NVCC - Nvidia CUDA Compiler
  • based on LLVM
  • source file extension: *.cu
• NCCL - NVIDIA Collective Communications Library
  • multi-GPU setup
• Thrust: open source C++ library of parallel algorithms and data structures

Mamba: CUDA 12 package overview

• useful commands
  • mamba search -c <channel> --override-channels [--info] <package-spec>
  • mamba repoquery whoneeds --tree --recursive -c <channel> <package>
  • mamba repoquery depends --tree --recursive -c <channel> <package>

Hierarchical meta package list

• cuda
  • cuda-runtime
    • cuda-libraries
      • (see below)
  • cuda-toolkit
    • cuda-compiler
      • c-compiler
      • cuda-cuobjdump
      • cuda-cuxxfilt
      • cuda-nvcc
      • cuda-nvprune
      • cxx-compiler
    • cuda-libraries
      • cuda-cudart
      • cuda-nvrtc
      • cuda-opencl
      • libcublas
      • libcufft
      • libcufile
      • libcurand
      • libcusolver
      • libcusparse
      • libnpp
      • libnvfatbin
      • libnvjitlink
      • libnvjpeg
    • cuda-libraries-dev
      • cuda-cccl
      • cuda-cudart-dev
      • cuda-driver-dev
      • cuda-nvrtc-dev
      • cuda-opencl-dev
      • cuda-profiler-api
      • libcublas-dev
      • libcufft-dev
      • libcufile-dev
      • libcurand-dev
      • libcusolver-dev
      • libcusparse-dev
      • libnpp-dev
      • libnvfatbin-dev
      • libnvjitlink-dev
      • libnvjpeg-dev
    • cuda-nvml-dev
    • cuda-tools
      • cuda-command-line-tools
        • cuda-cupti-dev
        • cuda-gdb
        • cuda-nvdisasm
        • cuda-nvprof
        • cuda-nvtx
        • cuda-sanitizer-api
      • cuda-visual-tools
      • gds-tools
• cuda-minimal-build
  • cuda-cccl
  • cuda-compiler
    • ...
  • cuda-cudart-dev
  • cuda-profiler-api
• not part of any meta package
  • cuda-compat
  • cuda-crt
  • cuda-nsight
  • cuda-nvvm
  • cuda-nvvp
  • cuda-python
  • cudnn
  • cuquantum
  • cutensor
  • nccl

Flat package list

• cuda-cccl
• cuda-compat
• cuda-crt
• cuda-crt-dev_linux-64
• cuda-crt-tools
• cuda-cudart
• cuda-cudart-dev
• cuda-cuobjdump
• cuda-cupti
• cuda-cupti-dev
• cuda-cupti-doc
• cuda-cuxxfilt
• cuda-driver-dev
• cuda-gdb
• cuda-gdb-src
• cuda-nsight
• cuda-nvcc
• cuda-nvcc-dev_linux-64
• cuda-nvcc-impl
• cuda-nvcc-tools
• cuda-nvdisasm
• cuda-nvml-dev
• cuda-nvprof
• cuda-nvprune
• cuda-nvrtc
• cuda-nvrtc-dev
• cuda-nvtx
• cuda-nvtx-dev
• cuda-nvvm
• cuda-nvvm-dev_linux-64
• cuda-nvvm-impl
• cuda-nvvm-tools
• cuda-nvvp
• cuda-opencl
• cuda-opencl-dev
• cuda-profiler-api
• cuda-python
• cuda-sanitizer-api
• cuda-visual-tools
• cudnn
• cupti
• cuquantum
• cutensor
• libcublas
• libcublas-dev
• libcufft
• libcufft-dev
• libcuquantum
• libcurand
• libcurand-dev
• libcusolver
• libcusolver-dev
• libcusparse
• libcusparse-dev
• libcutensor
• nccl

Repoqueries

• output has been slightly edited for clarity

$ mamba repoquery depends cuda=12.6 -c conda-forge --tree --recursive

cuda[12.6.0]
  ├─ cuda-runtime[12.6.0]
  │  └─ cuda-libraries[12.6.0]
  │     ├─ cuda-cudart[12.6.37]
  │     │  ├─ cuda-cudart_linux-64[12.6.37]
  │     │  │  └─ cuda-version[12.6]
  │     │  ├─ libgcc-ng[14.1.0]
  │     │  │  ├─ _libgcc_mutex[0.1]
  │     │  │  └─ _openmp_mutex[4.5]
  │     │  │     ├─ _libgcc_mutex already visited
  │     │  │     └─ llvm-openmp[18.1.8]
  │     │  │        ├─ libzlib[1.3.1]
  │     │  │        └─ zstd[1.5.6]
  │     │  │           ├─ libzlib already visited
  │     │  │           └─ libstdcxx-ng[14.1.0]
  │     │  └─ libstdcxx-ng already visited
  │     ├─ cuda-nvrtc[12.6.20]
  │     │  ├─ libgcc-ng already visited
  │     │  └─ libstdcxx-ng already visited
  │     ├─ cuda-opencl[12.6.37]
  │     │  ├─ libgcc-ng already visited
  │     │  ├─ libstdcxx-ng already visited
  │     │  └─ ocl-icd[2.3.2]
  │     │     └─ libgcc-ng already visited
  │     ├─ libcublas[12.6.0.22]
  │     │  ├─ libgcc-ng already visited
  │     │  ├─ libstdcxx-ng already visited
  │     │  └─ cuda-nvrtc[12.0.76]
  │     │     ├─ libgcc-ng already visited
  │     │     ├─ libstdcxx-ng already visited
  │     │     └─ cuda-version[12.0.0]
  │     ├─ libcufft[11.2.6.28]
  │     │  ├─ libgcc-ng already visited
  │     │  └─ libstdcxx-ng already visited
  │     ├─ libcufile[1.11.0.15]
  │     │  ├─ libgcc-ng already visited
  │     │  └─ libstdcxx-ng already visited
  │     ├─ libcurand[10.3.7.37]
  │     │  ├─ libgcc-ng already visited
  │     │  └─ libstdcxx-ng already visited
  │     ├─ libcusolver[11.6.4.38]
  │     │  ├─ libgcc-ng already visited
  │     │  ├─ libstdcxx-ng already visited
  │     │  ├─ libcublas already visited
  │     │  ├─ libcusparse[12.5.2.23]
  │     │  │  ├─ libgcc-ng already visited
  │     │  │  ├─ libstdcxx-ng already visited
  │     │  │  └─ libnvjitlink[12.6.20]
  │     │  │     ├─ libgcc-ng already visited
  │     │  │     └─ libstdcxx-ng already visited
  │     │  └─ libnvjitlink already visited
  │     ├─ libcusparse already visited
  │     ├─ libnvjitlink already visited
  │     ├─ libnpp[12.3.1.23]
  │     │  ├─ libgcc-ng already visited
  │     │  └─ libstdcxx-ng already visited
  │     ├─ libnvfatbin[12.6.20]
  │     │  ├─ libgcc-ng already visited
  │     │  └─ libstdcxx-ng already visited
  │     └─ libnvjpeg[12.3.3.23]
  │        ├─ libgcc-ng already visited
  │        └─ libstdcxx-ng already visited
  └─ cuda-toolkit[12.6.0]
     ├─ cuda-libraries already visited
     ├─ cuda-compiler[12.6.0]
     │  ├─ c-compiler[1.0.0]
     │  │  ├─ libgcc-ng already visited
     │  │  └─ gcc_linux-64[10.3.0]
     │  │     ├─ binutils_linux-64[2.36]
     │  │     │  ├─ binutils_impl_linux-64[2.36.1]
     │  │     │  │  ├─ ld_impl_linux-64[2.36.1]
     │  │     │  │  └─ sysroot_linux-64[2.12]
     │  │     │  │     └─ kernel-headers_linux-64[2.6.32]
     │  │     │  └─ sysroot_linux-64 already visited
     │  │     ├─ sysroot_linux-64 already visited
     │  │     └─ gcc_impl_linux-64[10.3.0]
     │  │        ├─ libgcc-ng already visited
     │  │        ├─ libstdcxx-ng already visited
     │  │        ├─ binutils_impl_linux-64 already visited
     │  │        ├─ sysroot_linux-64 already visited
     │  │        ├─ libgcc-devel_linux-64[10.3.0]
     │  │        ├─ libgomp[14.1.0]
     │  │        │  └─ _libgcc_mutex already visited
     │  │        └─ libsanitizer[10.3.0]
     │  │           └─ libgcc-ng already visited
     │  ├─ cuda-cuobjdump[12.6.20]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ cuda-nvdisasm[12.0.76]
     │  │     ├─ libgcc-ng already visited
     │  │     └─ libstdcxx-ng already visited
     │  ├─ cuda-cuxxfilt[12.6.20]
     │  │  ├─ libgcc-ng already visited
     │  │  └─ libstdcxx-ng already visited
     │  ├─ cuda-nvcc[12.6.20]
     │  │  ├─ gcc_linux-64 already visited
     │  │  ├─ cuda-nvcc_linux-64[12.6.20]
     │  │  │  ├─ cuda-cudart-dev_linux-64[12.6.37]
     │  │  │  │  ├─ cuda-cccl_linux-64[12.0.90]
     │  │  │  │  ├─ cuda-cudart-static_linux-64[12.0.107]
     │  │  │  │  └─ cuda-cudart_linux-64[12.0.107]
     │  │  │  │     ├─ libgcc-ng already visited
     │  │  │  │     └─ libstdcxx-ng already visited
     │  │  │  ├─ cuda-driver-dev_linux-64[12.6.37]
     │  │  │  ├─ cuda-nvcc-dev_linux-64[12.6.20]
     │  │  │  │  ├─ libgcc-ng already visited
     │  │  │  │  ├─ cuda-crt-dev_linux-64[12.6.20]
     │  │  │  │  └─ cuda-nvvm-dev_linux-64[12.6.20]
     │  │  │  ├─ cuda-nvcc-impl[12.6.20]
     │  │  │  │  ├─ cuda-cudart already visited
     │  │  │  │  ├─ cuda-nvcc-dev_linux-64 already visited
     │  │  │  │  ├─ cuda-cudart-dev[12.0.107]
     │  │  │  │  │  ├─ libgcc-ng already visited
     │  │  │  │  │  ├─ libstdcxx-ng already visited
     │  │  │  │  │  ├─ cuda-cudart[12.0.107]
     │  │  │  │  │  │  ├─ libgcc-ng already visited
     │  │  │  │  │  │  └─ libstdcxx-ng already visited
     │  │  │  │  │  ├─ cuda-cudart-dev_linux-64[12.0.107]
     │  │  │  │  │  │  ├─ cuda-cccl_linux-64 already visited
     │  │  │  │  │  │  └─ cuda-cudart-static_linux-64 already visited
     │  │  │  │  │  └─ cuda-cudart-static[12.0.107]
     │  │  │  │  │     ├─ libgcc-ng already visited
     │  │  │  │  │     ├─ libstdcxx-ng already visited
     │  │  │  │  │     └─ cuda-cudart-static_linux-64 already visited
     │  │  │  │  ├─ cuda-nvcc-tools[12.6.20]
     │  │  │  │  │  ├─ libgcc-ng already visited
     │  │  │  │  │  ├─ libstdcxx-ng already visited
     │  │  │  │  │  ├─ cuda-crt-tools[12.6.20]
     │  │  │  │  │  └─ cuda-nvvm-tools[12.6.20]
     │  │  │  │  │     ├─ libgcc-ng already visited
     │  │  │  │  │     └─ libstdcxx-ng already visited
     │  │  │  │  └─ cuda-nvvm-impl[12.6.20]
     │  │  │  │     ├─ libgcc-ng already visited
     │  │  │  │     └─ libstdcxx-ng already visited
     │  │  │  ├─ cuda-nvcc-tools already visited
     │  │  │  └─ sysroot_linux-64[2.28]
     │  │  │     ├─ _sysroot_linux-64_curr_repodata_hack[3]
     │  │  │     └─ kernel-headers_linux-64[4.18.0]
     │  │  │        └─ _sysroot_linux-64_curr_repodata_hack already visited
     │  │  └─ gxx_linux-64[10.3.0]
     │  │     ├─ gcc_linux-64 already visited
     │  │     ├─ binutils_linux-64 already visited
     │  │     ├─ sysroot_linux-64 already visited
     │  │     └─ gxx_impl_linux-64[10.3.0]
     │  │        ├─ sysroot_linux-64 already visited
     │  │        ├─ gcc_impl_linux-64 already visited
     │  │        └─ libstdcxx-devel_linux-64[10.3.0]
     │  ├─ cuda-nvprune[12.6.20]
     │  │  ├─ libgcc-ng already visited
     │  │  └─ libstdcxx-ng already visited
     │  └─ cxx-compiler[1.0.0]
     │     ├─ libgcc-ng already visited
     │     ├─ libstdcxx-ng already visited
     │     └─ gxx_linux-64 already visited
     ├─ cuda-libraries-dev[12.6.0]
     │  ├─ cuda-cccl[12.6.37]
     │  │  ├─ cccl[2.5.0]
     │  │  └─ cuda-cccl_linux-64[12.6.37]
     │  ├─ cuda-cudart-dev[12.6.37]
     │  │  ├─ cuda-cudart already visited
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  ├─ cuda-cudart-dev_linux-64 already visited
     │  │  └─ cuda-cudart-static[12.6.37]
     │  │     ├─ libgcc-ng already visited
     │  │     ├─ libstdcxx-ng already visited
     │  │     └─ cuda-cudart-static_linux-64[12.6.37]
     │  ├─ cuda-driver-dev[12.6.37]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ cuda-driver-dev_linux-64[12.0.107]
     │  ├─ cuda-nvrtc-dev[12.6.20]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ cuda-nvrtc already visited
     │  ├─ cuda-opencl-dev[12.6.37]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ cuda-opencl already visited
     │  ├─ cuda-profiler-api[12.6.37]
     │  │  └─ cuda-cudart-dev already visited
     │  ├─ libcublas-dev[12.6.0.22]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libcublas already visited
     │  ├─ libcufft-dev[11.2.6.28]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libcufft already visited
     │  ├─ libcufile-dev[1.11.0.15]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libcufile already visited
     │  ├─ libcurand-dev[10.3.7.37]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libcurand already visited
     │  ├─ libcusolver-dev[11.6.4.38]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libcusolver already visited
     │  ├─ libcusparse-dev[12.5.2.23]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  ├─ libcusparse already visited
     │  │  └─ libnvjitlink already visited
     │  ├─ libnpp-dev[12.3.1.23]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libnpp already visited
     │  ├─ libnvfatbin-dev[12.6.20]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libnvfatbin already visited
     │  ├─ libnvjitlink-dev[12.6.20]
     │  │  ├─ libgcc-ng already visited
     │  │  ├─ libstdcxx-ng already visited
     │  │  └─ libnvjitlink already visited
     │  └─ libnvjpeg-dev[12.3.3.23]
     │     ├─ libnvjpeg already visited
     │     └─ cuda-cudart-dev already visited
     ├─ cuda-nvml-dev[12.6.37]
     │  ├─ libgcc-ng already visited
     │  └─ libstdcxx-ng already visited
     └─ cuda-tools[12.6.0]
        ├─ cuda-command-line-tools[12.6.0]
        │  ├─ cuda-cupti-dev[12.6.37]
        │  │  ├─ libgcc-ng already visited
        │  │  ├─ libstdcxx-ng already visited
        │  │  └─ cuda-cupti[12.6.37]
        │  │     ├─ libgcc-ng already visited
        │  │     └─ libstdcxx-ng already visited
        │  ├─ cuda-gdb[12.6.37]
        │  │  ├─ libgcc-ng already visited
        │  │  ├─ libstdcxx-ng already visited
        │  │  └─ gmp[6.3.0]
        │  │     ├─ libgcc-ng already visited
        │  │     └─ libstdcxx-ng already visited
        │  ├─ cuda-nvdisasm[12.6.20]
        │  │  ├─ libgcc-ng already visited
        │  │  └─ libstdcxx-ng already visited
        │  ├─ cuda-nvprof[12.6.37]
        │  │  ├─ libgcc-ng already visited
        │  │  ├─ libstdcxx-ng already visited
        │  │  └─ cuda-cupti[12.0.90]
        │  │     ├─ libgcc-ng already visited
        │  │     └─ libstdcxx-ng already visited
        │  ├─ cuda-nvtx[12.6.37]
        │  │  ├─ libgcc-ng already visited
        │  │  └─ libstdcxx-ng already visited
        │  └─ cuda-sanitizer-api[12.6.34]
        │     ├─ libgcc-ng already visited
        │     └─ libstdcxx-ng already visited
        ├─ cuda-visual-tools[12.6.0]
        │  ├─ cuda-libraries-dev already visited
        │  ├─ cuda-nvml-dev already visited
        │  ├─ cuda-nsight[12.6.20]
        │  ├─ cuda-nvvp[12.6.37]
        │  │  ├─ libgcc-ng already visited
        │  │  ├─ libstdcxx-ng already visited
        │  │  ├─ cuda-nvdisasm already visited
        │  │  └─ cuda-nvprof[12.0.90]
        │  │     ├─ libgcc-ng already visited
        │  │     ├─ libstdcxx-ng already visited
        │  │     └─ cuda-cupti already visited
        │  └─ nsight-compute[2024.3.0.15]
        │     └─ ...
        └─ gds-tools[1.11.0.15]
           ├─ libgcc-ng already visited
           ├─ libstdcxx-ng already visited
           └─ libcufile already visited

$ mamba repoquery depends cuda-minimal-build=12.6 -c conda-forge --tree --recursive

cuda-minimal-build[12.6.0]
  ├─ cuda-cccl[12.6.37]
  │  ├─ cccl[2.5.0]
  │  └─ cuda-cccl_linux-64[12.6.37]
  ├─ cuda-compiler[12.6.0]
  │  ├─ c-compiler[1.0.0]
  │  │  ├─ gcc_linux-64[10.3.0]
  │  │  │  ├─ binutils_linux-64[2.36]
  │  │  │  │  ├─ binutils_impl_linux-64[2.36.1]
  │  │  │  │  │  ├─ ld_impl_linux-64[2.36.1]
  │  │  │  │  │  └─ sysroot_linux-64[2.12]
  │  │  │  │  │     └─ kernel-headers_linux-64[2.6.32]
  │  │  │  │  └─ sysroot_linux-64 already visited
  │  │  │  ├─ sysroot_linux-64 already visited
  │  │  │  └─ gcc_impl_linux-64[10.3.0]
  │  │  │     ├─ binutils_impl_linux-64 already visited
  │  │  │     ├─ sysroot_linux-64 already visited
  │  │  │     ├─ libgcc-devel_linux-64[10.3.0]
  │  │  │     ├─ libgcc-ng[14.1.0]
  │  │  │     │  ├─ _libgcc_mutex[0.1]
  │  │  │     │  └─ _openmp_mutex[4.5]
  │  │  │     │     ├─ _libgcc_mutex already visited
  │  │  │     │     └─ llvm-openmp[18.1.8]
  │  │  │     │        ├─ libzlib[1.3.1]
  │  │  │     │        └─ zstd[1.5.6]
  │  │  │     │           ├─ libzlib already visited
  │  │  │     │           └─ libstdcxx-ng[14.1.0]
  │  │  │     ├─ libstdcxx-ng already visited
  │  │  │     ├─ libgomp[14.1.0]
  │  │  │     │  └─ _libgcc_mutex already visited
  │  │  │     └─ libsanitizer[10.3.0]
  │  │  │        └─ libgcc-ng already visited
  │  │  └─ libgcc-ng already visited
  │  ├─ cuda-cuobjdump[12.6.20]
  │  │  ├─ libgcc-ng already visited
  │  │  ├─ libstdcxx-ng already visited
  │  │  └─ cuda-nvdisasm[12.0.76]
  │  │     ├─ libgcc-ng already visited
  │  │     ├─ libstdcxx-ng already visited
  │  │     └─ cuda-version[12.0.0]
  │  ├─ cuda-cuxxfilt[12.6.20]
  │  │  ├─ libgcc-ng already visited
  │  │  └─ libstdcxx-ng already visited
  │  ├─ cuda-nvcc[12.6.20]
  │  │  ├─ gcc_linux-64 already visited
  │  │  ├─ cuda-nvcc_linux-64[12.6.20]
  │  │  │  ├─ cuda-cudart-dev_linux-64[12.6.37]
  │  │  │  │  ├─ cuda-cccl_linux-64[12.0.90]
  │  │  │  │  ├─ cuda-cudart-static_linux-64[12.0.107]
  │  │  │  │  └─ cuda-cudart_linux-64[12.0.107]
  │  │  │  │     ├─ libgcc-ng already visited
  │  │  │  │     └─ libstdcxx-ng already visited
  │  │  │  ├─ cuda-driver-dev_linux-64[12.6.37]
  │  │  │  ├─ cuda-nvcc-dev_linux-64[12.6.20]
  │  │  │  │  ├─ libgcc-ng already visited
  │  │  │  │  ├─ cuda-crt-dev_linux-64[12.6.20]
  │  │  │  │  └─ cuda-nvvm-dev_linux-64[12.6.20]
  │  │  │  ├─ cuda-nvcc-impl[12.6.20]
  │  │  │  │  ├─ cuda-nvcc-dev_linux-64 already visited
  │  │  │  │  ├─ cuda-cudart[12.6.37]
  │  │  │  │  │  ├─ libgcc-ng already visited
  │  │  │  │  │  ├─ libstdcxx-ng already visited
  │  │  │  │  │  └─ cuda-cudart_linux-64[12.6.37]
  │  │  │  │  ├─ cuda-cudart-dev[12.0.107]
  │  │  │  │  │  ├─ libgcc-ng already visited
  │  │  │  │  │  ├─ libstdcxx-ng already visited
  │  │  │  │  │  ├─ cuda-cudart[12.0.107]
  │  │  │  │  │  │  ├─ libgcc-ng already visited
  │  │  │  │  │  │  └─ libstdcxx-ng already visited
  │  │  │  │  │  ├─ cuda-cudart-dev_linux-64[12.0.107]
  │  │  │  │  │  │  ├─ cuda-cccl_linux-64 already visited
  │  │  │  │  │  │  └─ cuda-cudart-static_linux-64 already visited
  │  │  │  │  │  └─ cuda-cudart-static[12.0.107]
  │  │  │  │  │     ├─ libgcc-ng already visited
  │  │  │  │  │     ├─ libstdcxx-ng already visited
  │  │  │  │  │     └─ cuda-cudart-static_linux-64 already visited
  │  │  │  │  ├─ cuda-nvcc-tools[12.6.20]
  │  │  │  │  │  ├─ libgcc-ng already visited
  │  │  │  │  │  ├─ libstdcxx-ng already visited
  │  │  │  │  │  ├─ cuda-crt-tools[12.6.20]
  │  │  │  │  │  └─ cuda-nvvm-tools[12.6.20]
  │  │  │  │  │     ├─ libgcc-ng already visited
  │  │  │  │  │     └─ libstdcxx-ng already visited
  │  │  │  │  └─ cuda-nvvm-impl[12.6.20]
  │  │  │  │     ├─ libgcc-ng already visited
  │  │  │  │     └─ libstdcxx-ng already visited
  │  │  │  ├─ cuda-nvcc-tools already visited
  │  │  │  └─ sysroot_linux-64[2.28]
  │  │  │     ├─ _sysroot_linux-64_curr_repodata_hack[3]
  │  │  │     └─ kernel-headers_linux-64[4.18.0]
  │  │  │        └─ _sysroot_linux-64_curr_repodata_hack already visited
  │  │  └─ gxx_linux-64[10.3.0]
  │  │     ├─ gcc_linux-64 already visited
  │  │     ├─ binutils_linux-64 already visited
  │  │     ├─ sysroot_linux-64 already visited
  │  │     └─ gxx_impl_linux-64[10.3.0]
  │  │        ├─ sysroot_linux-64 already visited
  │  │        ├─ gcc_impl_linux-64 already visited
  │  │        └─ libstdcxx-devel_linux-64[10.3.0]
  │  ├─ cuda-nvprune[12.6.20]
  │  │  ├─ libgcc-ng already visited
  │  │  └─ libstdcxx-ng already visited
  │  └─ cxx-compiler[1.0.0]
  │     ├─ libgcc-ng already visited
  │     ├─ libstdcxx-ng already visited
  │     └─ gxx_linux-64 already visited
  ├─ cuda-cudart-dev[12.6.37]
  │  ├─ libgcc-ng already visited
  │  ├─ libstdcxx-ng already visited
  │  ├─ cuda-cudart-dev_linux-64 already visited
  │  ├─ cuda-cudart already visited
  │  └─ cuda-cudart-static[12.6.37]
  │     ├─ libgcc-ng already visited
  │     ├─ libstdcxx-ng already visited
  │     └─ cuda-cudart-static_linux-64[12.6.37]
  └─ cuda-profiler-api[12.6.37]
     └─ cuda-cudart-dev already visited

Intro

I have been keeping a close eye on the evolutions in AI/ML for a while now. Whenever I come across an interesting demo, I of course like to try it out. Because my main computer at home only has a weak iGPU, I often resort to running workloads in the cloud (mostly Google Colab or AWS). While that works reasonably well, there are some downsides:

• risk of going over budget when an instance accidentally keep running after use
• general mild inconvenience of working with remote systems
• cloud defeats the purpose of running a private/local LLM
• more expensive in the long run

That is why I decided to build my own ML system last month. I am not sure if I will actually end up saving money this way, but it is going to be an educational experience regardless. It is still early days, but this post contains my lessons learned so far.

Component selection

GPU

The core of any ML workstation is the GPU. Due to the ubiquity of CUDA requirements in deep learning, there is only a single viable brand: Nvidia. Their offerings can be categorized in three categories:

If you have tens of thousands of dollars to burn, you will want to look at Nvidia's enterprise offerings and more specifically at the A100 or newer H100 GPUs. These options come with abundant VRAM (40 to 80GB), which we can put to good use in a deep learning context. Additionally, they are very power efficient with a lower TDP compared to consumer-grade GeForce cards. This translates to a smaller physical footprint, so that multiple cards can fit in a single server case. Be careful when installing these GPUs in a regular desktop case though: they only have passive (i.e., fanless) cooling so they require very intensive external ventilation as is standard in a typical, temperature-controlled data center.

Notice how I focus on VRAM memory above all else. The reasoning behind this is simple: if you do not have enough memory, your model simply will not run. The other specs will only determine how patient you will have to be to see the result.

One step down in the price range (5 000EUR - 10 000EUR) we find their workstation offerings. Here, the RTX A6000 and RTX 6000 Ada with 48GB VRAM both look appealing. These come in a "blower style" form factor instead of the more traditional "open air" form factor, meaning they exhaust hot air via the back instead of spreading it back into the case. This again makes it possible to install many cards in a limited physical space without having to worry too much about heat dissipation. Unfortunately, these cards make a lot of noise, and this type of cooling is not suitable for more power-hungry consumer-grade cards (250W+).

The price range of up to 2000EUR makes those consumer-grade cards a lot more viable for most people. Conveniently, the current and last generation flagships - RTX 4090 and RTX 3090 (Ti) respectively - both have 24GB VRAM. In conclusion, buying a (lightly) used RTX 3090 (700EUR - 800EUR) might be the most budget-friendly option out there. Downsides of these consumer grade cards are their high power usage and their unwieldy form factor. (You will have a hard time fitting an RTX 4090 in a 4U server case.)

While Nvidia produces all of its own cards in the datacenter and workstation segments, there is a lot more competition in the consumer space. While Nvidia releases Founders Edition (FE) cards for soms of its own GPU chips, many other companies (Asus, MSI, Gigabyte, ...) build their own alternative cards around those same chips. They all have their own peculiarities:

• factory overclocking
  • not at all relevant for us, if anything we might end up underclocking our card to keep power usage and temperature over long time spans under control
• cooling method (1-3 fans, optional watercooling)
• physical size
  • typically 3+ expansion slots
  • watercooled cards can be slimmed down to 1 slot height
• power usage
• power connectors
  • most cards need 2 to 3 PCIe 6+2pin connectors
• looks
  • especially RGB lights, if you are into that
• ...

For my build, I am going to start out with a single RTX 3090 FE, but I want to select the other components carefully so that I can expand to 2x3090 or even 3x3090 in the future. Specifications:

• Ampere architecture
• 24GB GDDR6X VRAM (= 12 chips x 2GB/chip?)
  • have a tendency to run hot (>100 degrees Celcius)
  • might need to replace the thermal pads
  • using a GPU brace is also reported to fix some heat issues
  • or consider underclocking with sudo nvidia-smi -i <GPU_index> -pl <power_limit>
  • or look into custom water cooling blocks, if that is your thing
• memory bus: 384bit (= 12 chips x 32bits/chip)
• dimensions: 313 mm x 138 mm x 3 expansion slots
• PCIe connector: PCIe Gen 4 x16
  • note: PCIe Gen 5 is the latest standard, but there are not Gen 5 GPUs yet
• power
  • 350W
  • connector
    • placed on long edge of card (instead of short edge in higher segments)
    • (variant of) new 12Vhpwr connector found on new ATX 3.0 PSUs
    • including conversion cable to 2xPCIe 6+2pin connectors
• last consumer card to support NVLink

For a much more in-depth analysis of GPUs for deep learning, check out https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/.

Multi-GPU considerations

We need a motherboard and CPU combination that has enough PCIe Gen 4 x16 slots with enough PCIe lanes and enough physical spacing in between to make this possible.

• space
• heat
• power
• PCIe lanes
• SLI / NVLink
  • bridge sold separately
  • requires fixed amount of space between cards
    • not compatible with "creative" (i.e., vertical) GPU placement options
• use same make and model for all cards
  • else computation will often wait for the slowest card to finish
• note: this kind of setup only makes sense for deep learning, not for gaming

CPU

For a CPU we have to choose between Intel and AMD. While Intel used to be a no-brainer in the not-too-distant past, the tables have turned in recent years. I knew this was true in the consumer space, but as it turns out it is also valid in high-end segments such as HEDT, workstation and server CPUs.

For a ML workstation, the CPU is not nearly as important as the GPU. When possible, extra budget should go to the GPU instead. However, the CPU has an important role in making sure the GPUs can reach their full potential. To do this, it has to be able to supply them with enough data so that they are not sitting idle. This memory bandwidth will play a crucial role in our choice of CPU segment.

Some background on PCI slots: each has a physical size (i.e., width) and a number of communication lanes that are both expressed with indicators such as "x1", "x2", "x4", "x8" or "x16". A GPU typically occupies a physical x16 slot because a lot of data has to be transferred back and forth. Note that smaller expansion cards can fit in larger slots but not the other way around. For example, a single x4 card can plug in a x16 PCIe slot, thereby forfeiting the other x12 lanes. Typically, a slot that is x4 wide and contains an x4 expansion card will use all x4 lanes and a slot that is x16 wide will use all x16 lanes. However, the two factors can in practice diverge. When either the CPU or motherboard are not able to handle the combined amount of lanes over all slots, they can decide to run one or more slot at half the number of lanes. So an x16 GPU slot can run with x8 lanes.

Furthermore, each PCIe generation roughly works double as fast as the previous one. So PCIe Gen 5x8 can work as fast as PCIe Gen 4x16. You might now be thinking: it evens out if we put a Gen 4x16 GPU in a Gen 5x8 slot. Unfortunately, that is not the case. The Gen 5 slot has full backwards compatibility with Gen 4 expansion cards, but it will also be limited to Gen 4 speeds in that case. Effectively, it will still be running at Gen4x8 if only 8 lanes are available. To be clear: running at half the amount of lanes does not halve the effective speed of the GPU. The number of lanes is not the main bottleneck in most systems, so the performance penalty will be much lower.

Consumer-grade CPUs such as Intel Core i3-i9 and AMD Ryzen 3-9 have a very limited number of available PCIe lanes. For example, the top end AMD Ryzen 9 7900X can only manage 28 PCIe lanes (of which 4 are reserved to communicate with the motherboard chipset). That leaves 24 lanes (e.g., x16 + x8) for our GPUs. For almost all consumers - who are only ever interested in having a single GPU - this is plenty. However, we have to ask ourselves if running our second GPU with only x8 instead of x16 lanes is worth it. For many people the answer will be "yes" and they should stick to this segment. The alternative is looking at HEDT, workstation or server segment CPUs, as we will do below.

HEDT or high-end desktop started with Intel Extreme Edition CPUs, and later Intel Core X CPUs. These days the HEDT segment of Intel has been integrated in their Xeon lineup of workstation and server processors. Specifically, the Xeon W9 2400 and 3400 series. The category sits somewhere between consumer-grade hardware and workstation hardware, offering more multithreading performance and more PCIe lanes. AMD is going back and forth with regard to their HEDT support. Threadripper CPUs are in the HEDT segment, while Threadripper PRO CPUs are in the workstation segment. AMD had not released a non-PRO Threadripper in a while, but at CES 2024 they announced a new lineup (e.g., AMD Ryzen Threadripper 7970X with 32 cores and 92 PCIe 5.0 lanes). In summary, HEDT is a good match for our build but the segment is being squeezed by high-end consumer hardware and lower-end workstation hardware.

Specifically, the AMD Ryzen Threadripper PRO 5000WX series (based on the older Zen 3 architecture) is very competitively priced these days. It offers workstation CPUs with up to 64 cores, 2TB of DDR4 RAM and 128 PCIe lanes. As we will see in the motherboard section, these builds come with typical enterprise features that are redundant for our target audience, to the point where a HEDT build would be a better match if properly priced. An additional benefit of using somewhat older (i.e., 2022) hardware is that the DDR4 memory and PCIe 4.0 SSDs that come with it are cheaper than the recent DDR5/PCIe 5.0 counterparts.

I also briefly looked at the server segment (Intel Xeon and AMD EPYC) but found no better offerings there. In the end I settled for a Threadripper PRO 5955WX 16 core CPU with a TDP of 280W and 128 PCIe lanes that I could get a decent deal on. The Intel counterparts have fewer cores, lower clock rates, fewer PCIe lanes for the same or more money.

Another fun fact about CPUs: you can buy them boxed (default) or as "tray". Tray refers to the tray with multiple CPUs that are typically bought by OEMs for use in prebuilt their systems. As such, these don't come with any extras (no box, no manual, no stock cooler, ...). OEMs are not supposed to resell these to consumers, but it sometimes happens regardless. Manufacturers like Intel and AMD will typically not provide factory warranty for such products, so you will have to talk to the intermediary (i.e., the OEM) instead in case of problems. The main advantage of tray CPUs is their lower price. If the discount is significant enough, it is worth considering. However, I learned the hard way that Threadripper CPUs are supposed to come with a torque wrench to fix the CPU mount precisely as tight as prescribed. The tray version of these CPUs obviously also do not include this tool.

CPU cooler

For workstation-grade builds, I prefer aircooling over watercooling. It requires barely any maintenance and it can run without failure for years on end. We do however need a cooler with a sizable heatsink to be able to dissipate the TDP of our CPU. When choosing one, make sure it does not occlude any RAM of PCIe slots that you intend to use.

Specifically for Threadripper PRO builds, pay attention to the orientation of the cooler. In desktops and workstations, coolers are supposed to blow air from front to back in the case. However, our CPU is from the server segment - contrary to the non-PRO Threadrippers in the HEDT segment - where a horizontal socket orientation is more common. In that case a regular cooler will blow air from bottom to top. The Noctua NH-U14S and NH-U12S both suffer from this. Eventually, I discovered the Arctic Freezer 4U-M, which has the correct orientation and also matches all other requirements (i.e., socket and TDP). The "4U" terminology refers to server height in a rack.

Motherboard

Our choice of CPU (or more precisely its sWRX80 socket) limits our choice of motherboards quite significantly. Again, we select in function of our multi-GPU setup. Specifically, we are looking for plenty of PCI Gen 4x16 slots that can run at full speed and are also spaced far enough apart. Additionally, we would like a few M.2 slots with heatsink that have a direct connection to the CPU and that are placed far enough away from hot GPUs. Finally, make sure to check the connectivity options (USB, USB-C, WiFi, Bluetooth). Fortunately, most motherboards in the short list fit the bill. I eventually went with the ASUS Pro WS WRX80E-SAGE SE WIFI because I saw it being used in Lambda Labs builds and I could get a good deal on one. It was only afterwards that I realized the awkward dimensions of this board (see Case section). It is worth looking for smaller alternatives, but make sure to look at the block diagram showing all interconnections before making a decision. The ASRock WRX80 Creator comes to mind, although it seems hard to come by and does not support x16 lanes in all PCIe slots.

Some random notes on the ASUS Pro WS WRX80E-SAGE SE WIFI board:

• requires lots of power cables
  • 1 x 24-pin ATX connector
  • 2 x 8-pin CPU/EPS connector
  • 2 x 6-pin PCIe connector
  • 1 x 6+2-pin PCIe connector
• main feature: 7 PCIe 4.0 x16 slots
• built-in power and reset button
  • works without connecting front panel headers of case
• built-in VGA output
  • useful when you don't have discrete GPU yet
    • note: AMD Threadripper PRO does not have iGPU
  • makes linux crash on boot
    • first attempt: add acpi=off to grub bootloader options list
      • makes it possible to boot into live environment
      • also disables all but two USB ports
      • also crashes KVM via BMC
      • also disables all NVMe SSD drives
    • proper fix 1: add pci=nommconf to grub bootloader options list
      • in bootloader: E, add option, F10
        • linux /boot/vmlinuz-x.y.z-... ro quiet splash pci=nommconf
      • make permanent when booted
        • vi /etc/default/grub
          • add pci=nommconf to GRUB_CMDLINE_LINUX variable
        • sudo update-grub
        • reboot
    • proper fix 2: disable VGA header via physical switch
  • still works with BMC disabled
• Q-code display output does not seem to match with table in manual
• BMC / IPMI
  • typical server-level feature
  • access
    • can only be accessed over ethernet (not WiFi) via one of the two ports
    • make sure to use HTTPS
    • check IP address in BIOS
      • typically https://192.168.0.x
    • user: admin
    • password: admin
  • makes system take minutes to boot after complete power down (e.g., after unplugging)
    • much faster after regular shutdown and start
      • but still slow compared to regular desktop
    • fixed when BMC is disabled
  • LEDs
    • stay on when system is off
    • green (blinking): BMC is up and running
    • orange: on iff new warning in system event log
      • possibly about fans with RPM below threshold
  • control fan curves via web portal
    • or via BIOS (after firmware update)
    • (non-PWM?) fans will run at max speed when BMC is disabled
  • built-in KVM
• contains two small fans
• bottom pins are oriented south instead of up
  • pro: allows large GPU to hang off bottom edge of motherboard
  • con: many cases have limited space near bottom to connect everything
• WiFi
  • 6, not 6E
  • shark-shaped WiFi antenna is very unpractical
    • alternative: aftermarket antennas that attach directly to connectors
• no thunderbolt header (as is typical in AMD builds)
• sound when running Ubuntu
  • chip: Realtek ALC4080
  • back panel 3.5mm AUX jack works fine
  • front audio requires recent driver
    • requires alsa-ucm-conf>=1.2.9
      • commit: https://github.com/alsa-project/alsa-ucm-conf/commit/c79e8c18c6d08e3298bb4073a5429c17c1ff2b7b
      • not in Ubuntu 22.04 LTS
      • OK in Ubuntu 23.10

Other

RAM

• check QVL list of motherboard for compatibility
  • mine insisted specifically on DDR4-3200 RAM
• amount
  • more is better (up to a point)
  • at least 20% more than total VRAM
• type: DDR4 (cheaper) or DDR5
• form factor
  • DIMM (desktop)
  • make sure they fit under the CPU cooler
• speed, timings, latency: not important
• overclocking profiles (Intel XMP/AMD EXPO): not important
• heatsink: not important
• mostly works best with two modules in dual channel mode
• ECC
  • nice to have
  • more expensive
  • more difficult to find
• warranty: lifetime

Storage

• main SSD
  • type: NVMe M.2 SSD
  • size: 1TB+
    • models take up a lot of room
  • PCIe
    • typically uses x4 lanes
    • ideally directly connected to CPU instead of via motherboard chipset
    • both Gen 4 and Gen 5 options are available
  • no need for heatsink if you motherboard already has one
  • warranty (5+ years)
• optional 5400RPM HDD(s) for cheap extra storage

PSU

• must haves
  • power rating
    • rule of thumb: $1.2 \times (TDP_{CPU} + \sum_i TDP_{GPU_i})$
  • right type and amount of connectors
    • ATX24 for motherboard
    • EPS for CPU
    • PCIe depending on motherboard, GPU and other components
    • warning: never daisy chain GPUs
• nice to haves
  • 80+ efficiency rating (gold < platinum < titanium)
    • note: small percentual differences become relevant when using lots of power
  • 12Vhpwr connector
    • supports up to 600W
    • plug these in properly, or you risk melting the plug
  • modular design
  • silent
  • warranty (10+ years)

Case

• volume
  • for aircooled multi-GPU setup, disregard any case with a volume below 60 liters
• constraints
  • supports motherboard form factor
  • CPU cooler height
  • GPU length
  • PSU length
• nice to haves
  • dust filters
  • cable management options
  • easy to open
  • built-in GPU brace(s)

I realized fairly late in the process that my motherboard has an unusual form factor: EEB (12.2" x 13") instead of the far more common ATX (12" x 9.6"). This severely limited the number of compatible cases I could choose from. Even cases that officially claimed to support EEB form factors had some caveats. For example, because of downward-facing connectors on the bottom edge of the motherboard, I had to make sure I had spare room in that area to be able to connect all cables. Note that this extra space is useful regardless if you plan on installing a large GPU in the bottom PCIe slot. Furthermore, the standard cable management holes in the backplate of many cases get covered by the much wider motherboard. This results in some unconventional cable management practices. If I would do this build over again, I would put a much stronger emphasis on selecting a standard ATX motherboard.

Some feasible options:

• Corsair 7000D Airflow (very tight, not recommended)
• Fractal Design Define 7 XL
• Fractal Design Meshify 2 XL
• Lian Li O11 Dynamic XL
• Phanteks Enthoo Pro 2

Cooling fans

Ventilation is very important is a high-powered system, especially if the goal is to sustain long duration workloads. Do not cheap out on fans after building a $2000+ computer.

• case should have positive pressure ($Q_{in} \geq Q_{out}$)
• size
  • use 120mm or 140mm fans
  • not 200mm, they fail first due to high torque
• RPM trade-off
  • high RPM = more airflow
  • low RPM = less noise
• purpose
  • static pressure: for radiators, meshes, filters
  • airflow: elsewhere
  • hybrid
• connector
  • 3 pin (voltage regulated)
  • 4 pin (PWM regulated, better)
• bearings
  • fluid
    • cheap
    • mineral oil for lubrication
    • dust sensitive
    • go bad after a while
    • sensitive to orientation (avoid horizontal)
  • ball
    • expensive
    • long lasting
    • more noisy
    • ideal for servers
    • any orientation
  • sleeve
    • hybrid between ball and fluid bearing
    • closed system
    • prefers horizontal orientation
  • rifle
    • like sleeve
    • with Archimedes screw
      • prefers horizontal orientation
    • used in be quiet! fans
  • magnetic / maglev
    • lowest noise
    • expensive
    • any orientation
• fan orientation
  • vertical: fails first
  • horizontal
• recommendations
  • Noctua
    • favorite among enthusiasts
      • good airflow
      • low noise
      • long lifespan
    • very expensive compared to competition
    • 120mm, all round
      • https://noctua.at/en/nf-a12x25-pwm
      • https://noctua.at/en/products/fan/nf-a12x25-pwm-chromax-black-swap
    • 140mm, all round
      • https://noctua.at/en/nf-a14-pwm
      • https://noctua.at/en/nf-a14-pwm-chromax-black-swap
  • Corsair ML140 (maglev)

Further reading

• https://lambdalabs.com/blog/deep-learning-hardware-deep-dive-rtx-30xx
• https://forums.fast.ai/t/recommendations-on-new-2-x-rtx-3090-setup/78202/1
• https://www.reddit.com/r/LocalLLaMA/comments/15zx322/ideal_setup_for_dual_4090/
• https://medium.com/@chankhavu/meet-iva-my-mini-deep-learning-rig-f5588588ca8a
• https://nonint.com/2022/05/30/my-deep-learning-rig/
• https://www.youtube.com/playlist?list=PLBoQnSflObcnZktTLziVYvXg-8imDiLhy

Intro

• open source code autocomplete model
  • cf. OpenAI Codex, GitHub Copilot, DeepMind AlphaCode
• https://github.com/bigcode-project/starcoder
• https://huggingface.co/spaces/HuggingFaceH4/starchat-playground
• integrates with VSCode
  • https://marketplace.visualstudio.com/items?itemName=HuggingFace.huggingface-vscode
• goals
  • run model in own AWS account instead of on HuggingFace
  • let VSCode extension use own custom endpoint
  • figure out finetuning

How to set up starcoder in AWS

• create S3 bucket
• create policy that allows read/write to that bucket
• create EC2 role containing that policy
• start a new EC2 instance
  • TODO select right instance type
  • t2.micro for now to set up S3 properly
  • use newly created IAM role
• sudo yum install git
• Amazon Linux 2023 does not support git-lfs out of the box, workaround:
  • curl -LO https://github.com/git-lfs/git-lfs/releases/download/v3.3.0/git-lfs-linux-amd64-v3.3.0.tar.gz
  • tar xvfz git-lfs-linux-amd64-v3.3.0.tar.gz
  • sudo ./install.sh instead of git lfs install
  • git lfs version
• git clone https://huggingface.co/bigcode/starcoder
  • takes a while, needs to download 65GB
• cd starcoder
• TODO save to S3
• don't forget to stop the instance when you're done

Local, out of the box usage

• conda create -n starcoder python=3.11
• conda activate starcoder
• git clone https://github.com/bigcode-project/starcoder.git
• cd starcoder
• pip install -r requirements.txt
• set ENV HUGGING_FACE_HUB_TOKEN

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
checkpoint = "bigcode/starcoder"

model = AutoModelForCausalLM.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, device=0)
print( pipe("def hello():") )

Regularly updated sources

• https://www.reddit.com/r/LocalLLaMA/
  • https://old.reddit.com/r/LocalLLaMA/wiki/models
• https://github.com/nichtdax/awesome-totally-open-chatgpt
• https://github.com/zhengzangw/awesome-huge-models
• https://huggingface.co/blog
• https://eng.lyft.com

Big tech news

Adobe

• 2023-03-21 🧨 https://firefly.adobe.com/
• 2023-05-23 🧨 https://www.adobe.com/products/photoshop/generative-fill.html
• 2023-10-10 🧨 Firefly 2

AI21 labs

• Jurassic models
• 2024-03-28 https://www.ai21.com/jamba

Amazon

• 2012-12-03 https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-amazon-sagemaker-autopilot/
• 2021-11-30 https://aws.amazon.com/blogs/aws/announcing-amazon-sagemaker-canvas-a-visual-no-code-machine-learning-capability-for-business-analysts/
• 2023-04-13 https://aws.amazon.com/blogs/machine-learning/announcing-new-tools-for-building-with-generative-ai-on-aws
  • Amazon Bedrock
  • 🖥️ Amazon CodeWhisperer
• 2023-04-17 https://aws.amazon.com/blogs/machine-learning/deploy-large-models-at-high-performance-using-fastertransformer-on-amazon-sagemaker/
• 2023-05-03 https://aws.amazon.com/blogs/machine-learning/quickly-build-high-accuracy-generative-ai-applications-on-enterprise-data-using-amazon-kendra-langchain-and-large-language-models/
• 2023-05-10 https://aws.amazon.com/blogs/machine-learning/announcing-new-jupyter-contributions-by-aws-to-democratize-generative-ai-and-scale-ml-workloads/
• 2023-06-29 https://aws.amazon.com/blogs/machine-learning/interactively-fine-tune-falcon-40b-and-other-llms-on-amazon-sagemaker-studio-notebooks-using-qlora/
• 2023-07-05 https://github.com/aws-samples/amazon-bedrock-samples
• 2023-07-18 https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/
• 2023-08-16 https://aws.amazon.com/blogs/machine-learning/how-thomson-reuters-developed-open-arena-an-enterprise-grade-large-language-model-playground-in-under-6-weeks
• 2023-09-28 https://www.aboutamazon.com/news/aws/aws-amazon-bedrock-general-availability-generative-ai-innovations
• 2023-12-01 AWS re:Invent
  • https://aws.amazon.com/q
  • https://press.aboutamazon.com/2023/11/aws-unveils-next-generation-aws-designed-chips
  • https://aws.amazon.com/sagemaker/clarify/
  • https://aws.amazon.com/sagemaker/canvas/
  • https://aws.amazon.com/blogs/aws/introducing-amazon-sagemaker-hyperpod-a-purpose-built-infrastructure-for-distributed-training-at-scale/
• 2024-07-10 https://aws.amazon.com/appstudio/

Andrej Karpathy / Eureka Labs

• 2015-05-21 https://karpathy.github.io/2015/05/21/rnn-effectiveness/
  • https://github.com/karpathy/char-rnn
• 2022-08-16 https://karpathy.ai/zero-to-hero.html
  • https://github.com/karpathy/micrograd
  • https://github.com/karpathy/nanoGPT
• 2023-04-09 BabyGPT
• 2023-05-23 State of GPT
• 2023-07-23 https://github.com/karpathy/llama2.c
• 2023-11-23 1h intro to LLMs
• 2024-02-20 Let's build the GPT Tokenizer
• 2024-02-20 https://github.com/karpathy/minbpe
  • https://www.youtube.com/watch?v=zduSFxRajkE
• https://github.com/karpathy/llm.c
• 2024-07-11 train GPT-2 for $672
• 2024-07-16 Announcing Eureka Labs
  • https://github.com/karpathy/LLM101n

Anthropic

• 2023-03-14 https://www.anthropic.com/index/introducing-claude
• 2023-05-11 https://www.anthropic.com/index/100k-context-windows
• 2023-07-11 https://www.anthropic.com/index/claude-2
• 2023-09-25 https://www.anthropic.com/index/anthropic-amazon
• 2023-10-27 https://www.wsj.com/tech/ai/google-commits-2-billion-in-funding-to-ai-startup-anthropic-db4d4c50
• 2023-11-21 https://www.anthropic.com/index/claude-2-1
• 2024-03-04 https://www.anthropic.com/news/claude-3-family
  • Haiku
  • Sonnet
  • Opus
• 2024-03-13 https://www.anthropic.com/news/claude-3-haiku
• 2024-03-27 https://www.aboutamazon.com/news/company-news/amazon-anthropic-ai-investment
• https://docs.anthropic.com/en/prompt-library/library
• https://github.com/anthropics/anthropic-cookbook
• 2024-06-21 https://www.anthropic.com/news/claude-3-5-sonnet
• 2024-07-16 https://www.anthropic.com/news/android-app

Apple

• https://developer.apple.com/metal
• https://github.com/ml-explore/mlx
• https://github.com/ml-explore/mlx-examples
• https://github.com/apple/ml-ferret
• 2024-03-14 https://machinelearning.apple.com/research/mm1-methods-analysis-insights
• 2024-04-22 https://machinelearning.apple.com/research/openelm
  • https://huggingface.co/apple/OpenELM-3B

Black Forest Labs

• 2024-08--1 https://blackforestlabs.ai/announcing-black-forest-labs/
  • Stability.ai spin-off

Chip Huyen

• https://huyenchip.com/ml-interviews-book/
• 2022-01-03 https://stanford-cs329s.github.io
• 2022-05-01 https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/
• 2023-04-11 https://huyenchip.com/2023/04/11/llm-engineering.html
• 2023-05-02 https://huyenchip.com/2023/05/02/rlhf.html
• 2023-06-07 https://huyenchip.com/2023/06/07/generative-ai-strategy.html
• 2023-08-16 https://huyenchip.com/2023/08/16/llm-research-open-challenges.html
• 2024-01-16 https://huyenchip.com/2024/01/16/sampling.html
• 2024-07-25 https://huyenchip.com/2024/07/25/genai-platform.html
• https://www.oreilly.com/library/view/ai-engineering/9781098166298/

Cohere

• 2024-02-13 https://txt.cohere.com/aya/
• 2024-03-11 https://cohere.com/blog/command-r
• 2024-04-04 https://cohere.com/blog/command-r-plus-microsoft-azure

Databricks

• 2023-06-09 https://github.com/databrickslabs/pyspark-ai
• 2023-06-26 https://www.wsj.com/articles/databricks-strikes-1-3-billion-deal-for-generative-ai-startup-mosaicml-fdcefc06
• 2024-03-27 https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

Google

• https://www.tensorflow.org
• 2017-06-12 Attention is all you need
  • https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/
• 2022-12-13 🧨 https://imagen.research.google/editor/
  • https://arxiv.org/abs/2212.06909
• 2023-05-01 https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html
• 2023-05-04 ✨ https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
• 2023-05-10 https://blog.google/technology/ai/google-io-2023-keynote-sundar-pichai/#palm-2-gemini
• 2023-05-10 https://cloud.google.com/ai/generative-ai
• 2023-05-11 https://workspace.google.com/blog/product-announcements/duet-ai
  • 2023-08-29 https://workspace.google.com/blog/product-announcements/duet-ai-in-workspace-now-available
• https://developers.google.com/mediapipe/solutions
• 2023-06-29 🧨 https://ai.googleblog.com/2023/06/on-device-diffusion-plugins-for.html
• 2023-07-12 https://blog.google/technology/ai/notebooklm-google-ai/
  • https://notebooklm.google.com/signup
• 2023-08-08 https://idx.dev
• 2023-08-29 🧨 https://www.deepmind.com/blog/identifying-ai-generated-images-with-synthid
• 2023-08-29 https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-colab-enterprise-and-mlops
• 2023-10-12 https://blog.google/products/search/google-search-generative-ai-october-update/
• 2023-11-06 https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf
• 2023-11-16 https://deepmind.google/discover/blog/transforming-the-future-of-music-creation/
  • https://blog.youtube/inside-youtube/ai-and-music-experiment/
  • DeepMind Lyria
  • no paper or code yet
• 2023-12-06 https://blog.google/technology/ai/google-gemini-ai/
• 2023-12-06 https://blog.google/products/bard/google-bard-try-gemini-ai/
• 2023-12-08 https://blog.google/technology/ai/notebooklm-new-features-availability/
• 2023-12-13 https://blog.google/technology/ai/google-gemini-pro-imagen-duet-ai-update/
• 2023-12-13 https://blog.google/technology/ai/gemini-api-developers-cloud/
• 2023-12-13 https://cloud.google.com/blog/products/ai-machine-learning/gemini-support-on-vertex-ai
• 2023-12-13 https://cloud.google.com/blog/topics/healthcare-life-sciences/introducing-medlm-for-the-healthcare-industry
• 2023-12-13 https://cloud.google.com/blog/products/ai-machine-learning/duet-ai-for-developers-and-in-security-operations-now-ga
• 2023-12-14 🧨 https://cloud.google.com/blog/products/ai-machine-learning/imagen-2-on-vertex-ai-is-now-generally-available
• 2024-01-03 🧨 https://instruct-imagen.github.io/
• 2024-01-17 https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/
• 2024-01-23 📽️ https://lumiere-video.github.io/
• 2024-02-08 https://blog.google/products/gemini/bard-gemini-advanced-app/
  • Bard renamed to Gemini
  • launch of Gemini mobile apps
  • launch of Gemini Advanced (cf. ChatGPT Plus?) with Ultra 1.0 model
• 2024-02-15 https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/
• 2024-02-21 https://blog.google/technology/developers/gemma-open-models/
• https://aitestkitchen.withgoogle.com/tools/image-fx
• https://deepmind.google/discover/blog/sima-generalist-ai-agent-for-3d-virtual-environments/
• 2024-02-23 https://sites.google.com/view/genie-2024
• 2024-05-08 https://blog.google/technology/ai/google-deepmind-isomorphic-alphafold-3-ai-model/
• 2024-05-14 https://blog.google/technology/ai/google-gemini-update-flash-ai-assistant-io-2024/
• 2024-05-14 https://developers.googleblog.com/en/start-building-with-project-idx-today/
  • https://deepmind.google/technologies/gemini/flash/
• 2024-06-17 🎧 https://deepmind.google/discover/blog/generating-audio-for-video/
• 2024-07-25 https://blog.google/products/gemini/google-gemini-new-features-july-2024/
• 2024-07-25 https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/

Huggingface

• 2023-04-25 https://huggingface.co/chat
  • https://github.com/huggingface/chat-ui
• 2023-05-09 https://huggingface.co/blog/starchat-alpha
• 2023-05-10 https://huggingface.co/docs/transformers/transformers_agents
• 2023-05-31 https://huggingface.co/blog/sagemaker-huggingface-llm
• 2024-02-02 https://huggingface.co/chat/assistants
  • https://twitter.com/_philschmid/status/1753429249363452274

Inflection

• https://inflection.ai
• https://pi.ai
• 2023-06-22 https://inflection.ai/inflection-1
• 2023-11-22 https://inflection.ai/inflection-2
• 2024-03-07 https://inflection.ai/inflection-2-5

Lightning AI

• 2023-04-26 https://lightning.ai/pages/community/tutorial/lora-llm/
• 2023-04-28 https://lightning.ai/pages/community/community-discussions/the-ultimate-battle-of-language-models-lit-llama-vs-gpt3.5-vs-bloom-vs/
  • 
• 2023-05-05 https://github.com/Lightning-AI/lit-gpt
• 2024-02-18 https://lightning.ai/lightning-ai/studios/code-lora-from-scratch?view=public&section=all
• https://github.com/Lightning-AI/lightning-thunder

Meta

• 2016-07-15 https://github.com/facebookresearch/fastText/
• 2016-09-xx ✨ https://pytorch.org
• 2017-02-28 https://github.com/facebookresearch/faiss
• 2017-04-18 https://caffe2.ai/blog/2017/04/18/caffe2-open-source-announcement.html
• 2017-09-07 https://research.facebook.com/blog/2017/9/facebook-and-microsoft-introduce-new-open-ecosystem-for-interchangeable-ai-frameworks/
  • ONNX
• 2018-05-02 https://caffe2.ai/blog/2018/05/02/Caffe2_PyTorch_1_0.html
• 2021-07-15 https://engineering.fb.com/2021/07/15/open-source/fsdp/
• 2021-xx-xx https://github.com/facebookresearch/fairscale
• 2022-xx-xx https://github.com/facebookresearch/xformers
• 2022-09-12 https://pytorch.org/blog/PyTorchfoundation/
• 2022-05-03 https://huggingface.co/facebook/opt-30b
• 2022-05-03 https://ai.meta.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/
• 2022-11-16 https://huggingface.co/facebook/galactica-120b
• 2023-02-24 ✨ https://ai.meta.com/blog/large-language-model-llama-meta-ai/
• 2023-03-15 https://pytorch.org/blog/pytorch-2.0-release/
• 2023-04-05 https://ai.meta.com/blog/segment-anything-foundation-model-image-segmentation/
• 2023-04-25 https://ai.facebook.com/blog/self-supervised-learning-practical-guide
• 2023-04-17 https://ai.meta.com/blog/dino-v2-computer-vision-self-supervised-learning/
• 2023-05-09 https://ai.meta.com/blog/imagebind-six-modalities-binding-ai/
• 2023-07-18 https://ai.meta.com/blog/llama-2/
• 2023-08-22 https://ai.meta.com/blog/seamless-m4t/
• 2023-08-24 https://ai.meta.com/blog/code-llama-large-language-model-coding/
• 2023-08-25 https://facebookresearch.github.io/nougat/
• 2023-09-27 https://about.fb.com/news/2023/09/introducing-ai-powered-assistants-characters-and-creative-tools/
• 2023-11-16 🧨 https://emu-video.metademolab.com
• 🧨 https://imagine.meta.com
• 2023-11-30 https://ai.meta.com/blog/seamless-communication/
• 2024-01-29 CodeLlama-70B
• 2024-03-12 https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/
• 2024-03-19 https://www.bloomberg.com/news/articles/2024-03-19/microsoft-hires-deepmind-co-founder-suleyman-to-run-consumer-ai
• 2024-03-21 https://www.bloomberg.com/news/articles/2024-03-21/microsoft-to-pay-inflection-ai-650-million-after-scooping-up-most-of-staff
• 2024-04-18 https://ai.meta.com/blog/meta-llama-3/
• 2024-06-12 https://engineering.fb.com/2024/06/12/data-infrastructure/training-large-language-models-at-scale-meta/
• 2024-07-02 https://ai.meta.com/research/publications/meta-3d-gen/
• 2024-07-23 https://ai.meta.com/blog/meta-llama-3-1/
• 2024-07-29 https://ai.meta.com/research/publications/sam-2-segment-anything-in-images-and-videos/

Microsoft

• 2021-06-17 https://github.com/microsoft/LoRA
• 2022-02-23 🖥️ https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/
• 2023-02-07 New Bing
• 2023-03-16 https://www.microsoft.com/en-us/microsoft-365/blog/2023/03/16/introducing-microsoft-365-copilot-a-whole-new-way-to-work/
• 2023-03-22 🖥️ https://github.blog/2023-03-22-github-copilot-x-the-ai-powered-developer-experience/
• 2023-03-28 https://blogs.microsoft.com/blog/2023/03/28/introducing-microsoft-security-copilot-empowering-defenders-at-the-speed-of-ai/
• 2023-03-21 Bing Image Creator
• 2023-07-18 https://www.microsoft.com/en-us/microsoft-365/blog/2023/07/18/introducing-bing-chat-enterprise-microsoft-365-copilot-pricing-and-microsoft-sales-copilot/
• 2023-07-20 https://github.com/microsoft/promptflow
• 2023-09-07 https://blogs.microsoft.com/on-the-issues/2023/09/07/copilot-copyright-commitment-ai-legal-concerns/
• 2023-09-20 🖥️ https://github.blog/2023-09-20-github-copilot-chat-beta-now-available-for-all-individuals/
• 2023-12-12 https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
• 2023-12-29 🖥️ https://github.blog/2023-12-29-github-copilot-chat-now-generally-available-for-organizations-and-individuals/
• 2024-01-15 https://blogs.microsoft.com/blog/2024/01/15/bringing-the-full-power-of-copilot-to-more-people-and-businesses/

Midjourney

• 2023-12-21 🧨 https://mid-journey.ai/midjourney-v6-release/

Mistral

• 2023-09-27 https://mistral.ai/news/announcing-mistral-7b/
• 2023-12-11 https://mistral.ai/news/mixtral-of-experts/
• 2023-12-11 https://mistral.ai/news/la-plateforme/
• 2024-02-26 https://mistral.ai/news/mistral-large/
• 2024-04-17 https://mistral.ai/news/mixtral-8x22b/
• 2024-05-29 https://mistral.ai/news/codestral/
• 2024-07-16 https://mistral.ai/news/codestral-mamba/
• 2024-07-16 https://mistral.ai/news/mathstral/
• 2024-07-18 https://mistral.ai/news/mistral-nemo/
  • https://colab.research.google.com/drive/17d3U-CAIwzmbDRqbZ9NnpHxCkmXB6LZ0
• 2024-07-24 https://mistral.ai/news/mistral-large-2407/

NVIDIA

• https://github.com/NVIDIA/TensorRT-LLM
• 2024-02-13 https://blogs.nvidia.com/blog/chat-with-rtx-available-now/
• NVIDIA NIM

Ollama

• 2023-07-08 https://github.com/jmorganca/ollama
  • 2024-01-23 https://ollama.ai/blog/python-javascript-libraries
  • https://github.com/ollama/ollama
• 2024-02-08 https://ollama.com/blog/openai-compatibility
• 2024-02-15 https://ollama.com/blog/windows-preview
• 2024-03-14 https://ollama.com/blog/amd-preview

OpenAI

• models
  • GPT
  • GPT 2
  • GPT 3
  • GPT 3.5
  • GPT 3.5 Turbo
  • GPT 4
  • GPT 4 Turbo
  • GPT 4o
  • GPT 4o mini
  • o1
  • o1 mini
  • GPT 4.5
  • o3 mini
  • GPT 4.1
  • GPT 4.1 mini
  • GPT 4.1 nano
  • o3
  • o4 mini
• 2021-07-27 https://openai.com/research/triton
• 2023-01-31 https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text
  • 2023-07-20 offline due to low accuracy
• 2023-03-02 ChatGPT political bias
• 2023-03-14 https://openai.com/index/gpt-4-research/
• 2023-04-17 https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/
• 2023-04-25 https://openai.com/blog/new-ways-to-manage-your-data-in-chatgpt
• 2023-05-09 https://openai.com/research/language-models-can-explain-neurons-in-language-models
• 2023-05-18 https://apps.apple.com/app/openai-chatgpt/id6448311069
• 2023-05-31 https://openai.com/research/improving-mathematical-reasoning-with-process-supervision
• 2023-06-11 https://openai.com/blog/function-calling-and-other-api-updates
• 2023-07-06 https://openai.com/blog/gpt-4-api-general-availability
• 2023-07-06 ChatGPT Code interpreter
• 2023-07-11 https://investor.shutterstock.com/news-releases/news-release-details/shutterstock-expands-partnership-openai-signs-new-six-year
• 2023-07-18 How is ChatGPT's behavior changing over time?
• 2023-07-20 https://openai.com/blog/custom-instructions-for-chatgpt
• 2023-07-25 https://play.google.com/store/apps/details?id=com.openai.chatgpt
• 2023-07-26 https://openai.com/blog/frontier-model-forum
  • Anthropic, Google, Microsoft, and OpenAI
• 2023-08-28 https://openai.com/blog/introducing-chatgpt-enterprise
• 2023-08-31 https://openai.com/blog/teaching-with-ai
• 2023-09-25 https://openai.com/blog/chatgpt-can-now-see-hear-and-speak
  • https://cdn.openai.com/papers/GPTV_System_Card.pdf
    • TODO add to models
• 2023-09-27 ChatGPT can now browse the internet
• 2023-10-03 https://cdn.openai.com/papers/DALL_E_3_System_Card.pdf
  • https://cdn.openai.com/papers/dall-e-3.pdf
  • TODO add to models
• 2023-10-11 ChatGPT system messages
• 2023-10-19 https://openai.com/blog/dall-e-3-is-now-available-in-chatgpt-plus-and-enterprise
• 2023-10-29 ChatGPT PDF support preview
• 2023-11-06 https://openai.com/blog/new-models-and-developer-products-announced-at-devday
  • GPT-4 Turbo
  • GPT-4 Turbo with Vision
  • GPT-4 fine tuning
  • Assistants API
  • DALL-E 3 API
  • Whisper v3
  • Copyright Shield
• 2023-11-06 https://openai.com/blog/introducing-gpts
• 2023-11-17 https://openai.com/blog/openai-announces-leadership-transition
• 2023-11-21 ChatGPT with voice public availability
• 2023-11-29 https://openai.com/blog/sam-altman-returns-as-ceo-openai-has-a-new-initial-board
• https://platform.openai.com/docs/guides/prompt-engineering/strategy-use-external-tools
• 2024-01-10 https://openai.com/blog/introducing-chatgpt-team
• 2024-01-25 https://openai.com/blog/new-embedding-models-and-api-updates
• 2024-02-13 https://openai.com/blog/memory-and-new-controls-for-chatgpt
• 2024-02-15 📽️ https://openai.com/research/video-generation-models-as-world-simulators
  • https://openai.com/sora
• 2024-03-12 https://github.com/openai/transformer-debugger
• 2024-03-29 https://openai.com/index/navigating-the-challenges-and-opportunities-of-synthetic-voices/
• 2024-04-01 https://openai.com/index/start-using-chatgpt-instantly/
• 2024-04-03 🧨 edit DALL·E images in ChatGPT
• 2024-04-04 https://openai.com/index/introducing-improvements-to-the-fine-tuning-api-and-expanding-our-custom-models-program/
• 2024-05-08 https://openai.com/index/introducing-the-model-spec/
• 2024-05-13 https://openai.com/index/hello-gpt-4o/
• 2024-07-18 https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/
• 2024-07-25 https://openai.com/index/searchgpt-prototype/
• 2024-09-12 https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/
• 2024-09-12 https://openai.com/index/learning-to-reason-with-llms/
• 2024-12-05 https://openai.com/index/introducing-chatgpt-pro/
• 2024-12-05 https://openai.com/index/openai-o1-system-card/
• 2024-12-18 https://help.openai.com/en/articles/10193193-1-800-chatgpt-calling-and-messaging-chatgpt-with-your-phone
• 2025-01-31 https://openai.com/index/openai-o3-mini/
• 2025-02-02 https://openai.com/index/introducing-deep-research/
• 2025-02-23 https://openai.com/index/introducing-operator/
• 2025-02-27 https://openai.com/index/introducing-gpt-4-5/
• 2025-02-27 https://openai.com/index/gpt-4-5-system-card/
• 2025-03-25 https://openai.com/index/introducing-4o-image-generation/
• 2025-04-14 https://openai.com/index/gpt-4-1/
• 2025-04-16 https://openai.com/index/introducing-o3-and-o4-mini/

Runway ML

• 2023-02-06 https://research.runwayml.com/gen1
  • https://arxiv.org/abs/2302.03011
• https://runwayml.com/ai-magic-tools/gen-2/

Stability AI

• 🧨 https://clipdrop.co
  • 2023-03-17 https://stability.ai/blog/stable-diffusion-reimagine
  • 2023-05-25 https://stability.ai/blog/stability-ai-clipdrop-launches-reimagine-xl
  • 2023-06-08 https://stability.ai/blog/clipdrop-launches-uncrop-the-ultimate-aspect-ratio-editor
  • 2023-07-13 https://stability.ai/blog/clipdrop-launches-stable-doodle
• 2023-05-11 🧨 https://stability.ai/blog/stable-animation-sdk
• 🧨 https://dreamstudio.ai
  • 2023-05-17 https://github.com/Stability-AI/StableStudio
• 2023-08-11 https://research.stability.ai/chat
• 2023-11-01 https://stability.ai/news/stability-ai-enhanced-image-apis-for-business-features
• 2023-11-21 https://stability.ai/news/stable-video-diffusion-open-ai-video-model
• 2023-11-28 https://stability.ai/news/stability-ai-sdxl-turbo
• 2023-12-07 https://stability.ai/news/stablelm-zephyr-3b-stability-llm
• 2024-01-16 https://stability.ai/news/stable-code-2024-llm-code-completion-release
• 2024-02-12 https://stability.ai/news/introducing-stable-cascade
• 2024-02-22 🧨 https://stability.ai/news/stable-diffusion-3
• 2024-03-13 https://stability.ai/news/celebrating-one-year-of-medarc
• 2024-03-18 https://stability.ai/news/introducing-stable-video-3d
• 2024-03-23 https://stability.ai/news/stabilityai-announcement
• 2024-03-25 https://stability.ai/news/introducing-stable-code-instruct-3b
• 2024-04-03 🎧 https://stability.ai/news/stable-audio-2-0
  • https://stability-ai.github.io/stable-audio-demo/?
• 2024-04-08 https://stability.ai/news/introducing-stable-lm-2-12b
• 2024-04-17 🧨 https://stability.ai/news/stable-diffusion-3-api
• 2024-06-05 🎧 https://stability.ai/news/introducing-stable-audio-open
• 2024-06-12 🧨 https://stability.ai/news/stable-diffusion-3-medium
• 2024-08-01 https://stability.ai/news/introducing-stable-fast-3d

xAI

• 2023-07-12 xAI Grok
  • https://github.com/xai-org/grok-1
  • weights
• 2023-11-06 https://x.ai/prompt-ide/

Other news

• 2019-10-18 https://thegradient.pub/understanding-evaluation-metrics-for-language-models/
• 2020-07-28 https://dugas.ch/artificial_curiosity/GPT_architecture.html
• 2022-03-30 https://kipp.ly/transformer-param-count
• 2023-02-14 https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
• 2023-02-21 https://www.pinecone.io/learn/langchain/
• 2023-03-02 https://www.activeloop.ai/resources/ultimate-guide-to-lang-chain-deep-lake-build-chat-gpt-to-answer-questions-on-your-financial-data/
• 2023-03-04 https://confusedbit.dev/posts/how_does_gpt_work/
• 2023-03-29 https://github.com/xtekky/gpt4free
• 2023-03-29 https://ai.v-gar.de/ml/transformer/timeline/
  • TODO update csv
• 2023-03-30 https://kipp.ly/transformer-taxonomy
• 2023-04-05 ChaosGPT
• 2023-04-10 https://www.izzy.co/blogs/robo-boys.html
• 2023-04-15 https://www.activeloop.ai/resources/lang-chain-gpt-4-for-code-understanding-twitter-algorithm/
• 2023-04-16 🧨 https://www.shruggingface.com/blog/how-i-used-stable-diffusion-and-dreambooth-to-create-a-painted-portrait-of-my-dog
• 2023-04-19 https://www.similarweb.com/blog/insights/ai-news/stack-overflow-chatgpt/
• 2023-04-20 https://github.com/Vision-CAIR/MiniGPT-4
• 2023-04-21 https://github.com/brexhq/prompt-engineering
• 2023-04-21 https://gist.github.com/timesler/4b244a6b73d6e02d17fd220fd92dfaec
• 2023-04-22 https://magazine.sebastianraschka.com/p/finetuning-large-language-models
• 2023-04-28 https://github.com/jostmey/NakedAttention
• 2023-04-29 https://thegradient.pub/in-context-learning-in-context/
• 2023-04-30 https://github.com/Mooler0410/LLMsPracticalGuide
• 2023-04-30 https://agi-sphere.com/llama-models/
• 2023-04-30 https://www.wired.com/story/how-chatgpt-works-large-language-model/
• 2023-05-01 https://www.kdnuggets.com/2023/05/machine-learning-chatgpt-cheat-sheet.html
• 2023-05-02 https://www.philschmid.de/sagemaker-fsdp-gpt
• 2023-05-03 https://www.assemblyai.com/blog/the-full-story-of-large-language-models-and-rlhf/
• 2023-05-05 https://www.pinecone.io/learn/vector-database
• 2023-05-09 https://newsroom.ibm.com/2023-05-09-IBM-Unveils-the-Watsonx-Platform-to-Power-Next-Generation-Foundation-Models-for-Business
• 2023-05-12 https://www.kdnuggets.com/2023/05/8-free-ai-llms-playgrounds.html
• 2023-05-15 https://erichartford.com/uncensored-models
• 2023-05-15 https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-context-window-all-tricks-in-one-place-ffd40577b4c
• 2023-05-17 https://github.com/ray-project/llm-numbers
• 2023-05-25 ✨ https://a16z.com/2023/05/25/ai-canon
• 2023-06-05 🧨 https://www.reddit.com/r/StableDiffusion/comments/141hg9x/controlnet_for_qr_code/
• 2023-06-12 https://www.evidentlyai.com/ml-system-design
• 2023-06-14 https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier
• 2023-06-15 https://crfm.stanford.edu/2023/06/15/eu-ai-act.html
• 2023-06-20 https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/
• 2023-06-27 https://mlops.community/unraveling-gpu-inference-costs-for-fine-tuned-open-source-models-v-s-closed-platforms/
• 2023-06-30 ✨ https://neptune.ai/blog/mlops-tools-platforms-landscape
• 2023-07-03 https://www.similarweb.com/blog/insights/ai-news/chatgpt-traffic-drops/
• 2023-07-09 https://blog.mithrilsecurity.io/poisongpt-how-we-hid-a-lobotomized-llm-on-hugging-face-to-spread-fake-news/
• 2023-07-10 https://www.semianalysis.com/p/gpt-4-architecture-infrastructure
• 2023-07-20 https://www.cursor.so/blog/llama-inference
• 2023-07-20 ✨ https://github.com/tikkuncreation/llama-2-resources
• 2023-07-22 https://replicate.com/blog/run-llama-locally
• 2023-07-23 https://willthompson.name/what-we-know-about-llms-primer
• 2023-07-27 https://stackoverflow.blog/2023/07/27/announcing-overflowai/
  • TODO
• 2023-07-27 https://hacks.mozilla.org/2023/07/so-you-want-to-build-your-own-open-source-chatbot/
• 2023-07-27 https://llm-attacks.org
• 2023-07-30 https://betterprogramming.pub/frameworks-for-serving-llms-60b7f7b23407
• 2023-07-30 ✨ https://eugeneyan.com/writing/llm-patterns/
• 2023-07-31 https://arstechnica.com/science/2023/07/a-jargon-free-explanation-of-how-ai-large-language-models-work/
• 2023-08-08 https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/wormgpt-and-fraudgpt-the-rise-of-malicious-llms/
• 2023-08-09 ✨ https://blog.briankitano.com/llama-from-scratch/
• 2023-08-11 https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications
• 2023-08-24 https://scale.com/blog/open-ai-scale-partnership-gpt-3-5-fine-tuning
• 2023-09-06 raccoons.be - gen AI playbook
• 2023-09-12 Fine-tune your own Llama 2 to replace GPT-3.5/4
• 2023-09-13 https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1
• 2023-09-14 LLM agent survey
  • https://github.com/WooooDyy/LLM-Agent-Paper-List
• 2023-09-18 https://www.anyscale.com/anyscale-launches-new-service-anyscale-endpoints-10x-more-cost-effective-for-most-popular-open-source-llms
• 2023-10-09 https://blog.replit.com/ai4all
• 2023-10-18 How ChatGPT Vision Works - YouTube
• 2023-11-08 Samsung Gauss
• 2023-11-16 https://pytorch.org/blog/accelerating-generative-ai
• 2023-11-29 https://blog.perplexity.ai/blog/introducing-pplx-online-llms
• 2023-11-30 https://pytorch.org/blog/accelerating-generative-ai-2
• 2023-12-09 https://www.europarl.europa.eu/news/en/press-room/20231206IPR15699/artificial-intelligence-act-deal-on-comprehensive-rules-for-trustworthy-ai
• 2024-01-03 https://pytorch.org/blog/accelerating-generative-ai-3
• 2024-02-01 https://shyam.blog/posts/beyond-self-attention/
• 2024-02-01 https://allenai.org/olmo
• 2024-02-19 Transformers demystified
• https://www.chenyang.co/diffusion.html
• 2024-05-29 https://foojay.io/today/indexing-all-of-wikipedia-on-a-laptop/
• 2024-06-05 https://opening-up-chatgpt.github.io/
• 2024-06-08 https://applied-llms.org/
• 2024-06-13 https://huggingface.co/blog/mlabonne/abliteration
• 2024-06-19 https://epochai.org/data/notable-ai-models
• https://situational-awareness.ai/
• 2024-06-25 https://imbue.com/research/70b-infrastructure/
• https://fastvoiceagent.cerebrium.ai/
• 2024-08-09 LLM price-performance graph

Tools

• https://onnx.ai/
• 🖥️ https://www.tabnine.com
• https://www.baseten.co
• http://vectors.nlpl.eu/explore/embeddings/en/
• https://github.com/togethercomputer/OpenChatKit
• https://github.com/streamlit/streamlit
• 2020-07-09 https://github.com/gradio-app/gradio
• 2022-05-27 https://github.com/Dao-AILab/flash-attention
  • https://arxiv.org/abs/2205.14135
  • https://arxiv.org/abs/2307.08691
• 2022-09-02 🧨 https://promptbase.com
• 2021-12-21 🧨 https://github.com/invoke-ai/InvokeAI
• 2022-09-12 🧨 https://github.com/brycedrennan/imaginAIry
• 2022-10-24 https://github.com/hwchase17/langchain
  • https://www.langchain.com/langsmith
• 2022-12-21 🧨 https://github.com/oobabooga/text-generation-webui
• 2023-01-17 🧨 https://github.com/comfyanonymous/ComfyUI
• 2023-01-24 🧨 https://github.com/AUTOMATIC1111/stable-diffusion-webui
• 2023-01-25 🧨 https://www.shutterstock.com/ai-image-generator
  • uses DALL-E
• 2023-01-29 https://github.com/arc53/DocsGPT
• 2023-03-13 https://github.com/ShreyaR/guardrails
• 2023-03-16 🖥️ https://github.com/TabbyML/tabby
• 2023-03-19 https://chat.lmsys.org
• 2023-03-30 https://github.com/Significant-Gravitas/Auto-GPT
• 🧨 https://github.com/apple/ml-stable-diffusion
• 2023-03-10 https://github.com/ggerganov/llama.cpp
  • 2023-05-14 now supports partial GPU usage
  • 2023-06-12 CUDA support
• 2023-03-14 https://github.com/getumbrel/llama-gpt
• https://github.com/bigscience-workshop/petals
  • distributed LLM finetuning and inference
• https://github.com/jerryjliu/llama_index
• https://github.com/jupyterlab/jupyter-ai
  • Codex clone using https://huggingface.co/Salesforce/codegen-350M-multi
• https://kubiya.ai
• https://www.chatpdf.com
• https://pdf.ai
• 2023-03-30 https://www.cursor.so
• 2023-04-18 https://github.com/h2oai/h2o-llmstudio
• 2023-04-19 ✨ https://github.com/smallcloudai/refact
  • https://refact.ai
  • https://marketplace.visualstudio.com/items?itemName=smallcloud.codify
• https://github.com/paulpierre/RasaGPT
• 2023-04-29 https://github.com/mlc-ai/mlc-llm
• 2023-05-02 https://www.modular.com/mojo
• 2023-05-02 https://heypi.com
• 2023-05-02 https://github.com/imartinez/privateGPT
• 2023-05-06 https://github.com/nadermx/backgroundremover
• 2023-05-12 https://github.com/assafelovic/gpt-researcher
  • https://app.tavily.com
• https://github.com/LucienShui/huggingface-vscode-endpoint-server
• https://gandalf.lakera.ai
• https://faraday.dev
• https://sourcegraph.com
• 2023-05-13 🧨 https://github.com/varunshenoy/opendream
• 2023-05-13 https://github.com/StanGirard/quivr
• 2023-05-24 https://github.com/artidoro/qlora
• 🖥️ https://github.com/continuedev/continue
• 2023-06-08 https://simonwillison.net/2023/Jun/8/gpt-tokenizers/
• 2023-06-11 https://github.com/AntonOsika/gpt-engineer
• 2023-06-20 https://github.com/embedchain/embedchain
• 2023-06-29 https://github.com/ShishirPatil/gorilla
• 2023-07-14 https://github.com/KillianLucas/open-interpreter
• 2023-07-18 https://github.com/Pythagora-io/gpt-pilot
• 2023-07-24 https://github.com/kuafuai/DevOpsGPT
• 2023-07-26 https://github.com/Alpha-VLLM/LLaMA2-Accessory
• 2023-08-10 https://github.com/modelscope/facechain
• 2023-08-27 https://github.com/turboderp/exllamav2
• 2023-09-07 https://labs.heygen.com/video-translate
• https://www.trulens.org
• https://www.phind.com/search?home=true
  • https://www.phind.com/blog/introducing-phind-70b
• 2023-10-05 GenAI Stack - Docker
• 2023-10-20 https://neuralmagic.com/blog/building-sparse-llm-applications-on-cpus-with-langchain-and-deepsparse/
  • https://github.com/neuralmagic/deepsparse
  • people behind DeepSparse were also behind GPTQ
• 2023-11-01 🧨 https://lumalabs.ai/genie
• 2023-11-18 https://github.com/tldraw/make-real
• https://www.perplexity.ai
• 2023-11-28 📽️ https://pika.art/launch
• 2023-11-29 https://hacks.mozilla.org/2023/11/introducing-llamafile/
• https://github.com/ishan0102/vimGPT
• https://github.com/BuilderIO/gpt-crawler
• https://github.com/Vaibhavs10/insanely-fast-whisper
• https://github.com/intel/intel-extension-for-transformers
• https://github.com/coqui-ai/TTS
• https://github.com/Niek/chatgpt-web
• https://lmstudio.ai
• https://www.suno.ai
• https://github.com/imoneoi/openchat
• 2023-12-03 https://github.com/myshell-ai/OpenVoice
• 2023-12-16 https://github.com/SJTU-IPADS/PowerInfer
• 2023-12-19 🧨 https://github.com/cumulo-autumn/StreamDiffusion
• https://github.com/dvmazur/mixtral-offloading
• 2024-01-15 🧨 https://github.com/InstantID/InstantID
• https://jan.ai/
• https://github.com/danswer-ai/danswer
• https://github.com/KillianLucas/open-interpreter
• 2024-02-19 https://groq.com
• 2024-03-12 https://www.cognition-labs.com/introducing-devin
• 2024-03-28 https://github.com/jasonppy/VoiceCraft
• 🧨 https://github.com/lllyasviel/Fooocus
• 📽️ https://blog.lumalabs.ai/p/dream-machine
• 2024-07-11 https://tridao.me/blog/2024/flash3/
  • https://pytorch.org/blog/flashattention-3/
• https://github.com/saoudrizwan/claude-dev
• https://github.com/OpenInterpreter/open-interpreter