llvm 19 support #227

brandonros · 2025-06-08T22:27:27Z

potentially addresses all of:

Update rustc_llvm_wrapper to optionally support LLVM v19 #226 (Updates rustc_llvm_wrapper)
GitHub Codespaces/VSCode Devcontainer support #224 (Adds Codespaces, optional, we could remove)
sha2 crate = runtime error #207 (if LLVM v19 will compile and not have same issues as LLVM v7)
CUDA 12.8.1 and LLVM 18.1.8 #197 (we can put this behind a feature flag to optionally support Blackwell+)

brandonros · 2025-06-08T22:30:49Z

i'm on the fence about renaming llvm to llvm7 and llvm19

i think it might not actually be needed and i might put it back

brandonros · 2025-06-08T22:33:38Z

@LegNeato my measuring stick here is does vecadd example build with LLVM v19

could you tell me if I'm close or if I'm actually missing something huge like a mountain of work I'm not seeing?

brandonros · 2025-06-10T04:30:35Z

 DEBUG: About to call LLVMRunPasses - THIS IS THE CRITICAL POINT
  DEBUG: Parameters:
  DEBUG:   llmod: 0x770f3d33a580
  DEBUG:   pipeline: "default<O0>"
  DEBUG:   tm: 0x770f27dafb00
  DEBUG:   pass_options: 0x770f27e00060
  DEBUG: LLVMRunPasses returned: 0
  DEBUG: LLVMRunPasses completed successfully
  DEBUG: Cleaning up pass builder options
  DEBUG: Pass builder options disposed
  DEBUG: optimize function completed successfully
  DEBUG: About to verify module before prepare_thin
  DEBUG: Module verification result: 0
  DEBUG: LLVMRustThinLTOBufferCreate called with is_thin=1, emit_summary=1
  DEBUG: Taking ThinLTO path
  DEBUG: About to run ThinLTO pass
  error: rustc interrupted by SIGSEGV, printing backtrace

(gdb) bt
#0  0x0000772679b89e10 in llvm::ValueEnumerator::EnumerateType(llvm::Type*) ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#1  0x0000772679b8d240 in llvm::ValueEnumerator::incorporateFunction(llvm::Function const&) ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#2  0x0000772679b615b2 in (anonymous namespace)::ModuleBitcodeWriter::write() ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#3  0x0000772679b5b341 in llvm::BitcodeWriter::writeModule(llvm::Module const&, bool, llvm::ModuleSummaryIndex const*, bool, std::array<unsigned int, 5ul>*) () from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#4  0x0000772679b66802 in llvm::WriteBitcodeToFile(llvm::Module const&, llvm::raw_ostream&, bool, llvm::ModuleSummaryIndex const*, bool, std::array<unsigned int, 5ul>*) () from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#5  0x000077267944549c in llvm::ThinLTOBitcodeWriterPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#6  0x00007726786b34a7 in llvm::detail::PassModel<llvm::Module, llvm::ThinLTOBitcodeWriterPass, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#7  0x000077267a2e0dc8 in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
--Type <RET> for more, q to quit, c to continue without paging--
#8  0x000077267867e178 in LLVMRustThinLTOBufferCreate ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#9  0x000077267859d7be in rustc_codegen_nvvm::lto::ThinBuffer::new ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#10 0x000077267858512a in <rustc_codegen_nvvm::NvvmCodegenBackend as rustc_codegen_ssa::traits::write::WriteBackendMethods>::prepare_thin () from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#11 0x00007726785ec605 in rustc_codegen_ssa::back::write::execute_optimize_work_item ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#12 0x00007726785e3cd5 in rustc_codegen_ssa::back::write::spawn_work::{{closure}} ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#13 0x0000772678404d97 in std::sys::backtrace::__rust_begin_short_backtrace ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#14 0x000077267847357f in std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#15 0x0000772678520144 in <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
    () from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#16 0x0000772678466e24 in std::panicking::try::do_call ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#17 0x000077267847381b in __rust_try () from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#18 0x00007726784730f1 in std::thread::Builder::spawn_unchecked_::{{closure}} ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#19 0x000077267848f987 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
   from /workspaces/Rust-CUDA/target/debug/deps/librustc_codegen_nvvm.so
#20 0x000077268c55b8ab in std::sys::pal::unix::thread::Thread::new::thread_start ()
   from /root/.rustup/toolchains/nightly-2025-03-02-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-e3b06f91230294e6.so
#21 0x000077268668aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#22 0x0000772686717a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

pain

LegNeato · 2025-06-10T14:53:08Z

I don't know, llvm is an area of the project I have not touched.

LegNeato

This should stay the same, right? The newer support should be optional / only enabled when

LegNeato · 2025-06-10T14:57:35Z

crates/rustc_codegen_nvvm/build.rs

 static PREBUILT_LLVM_URL: &str =
    "https://github.com/rust-gpu/rustc_codegen_nvvm-llvm/releases/download/LLVM-7.1.0/";

-static REQUIRED_MAJOR_LLVM_VERSION: u8 = 7;
+static REQUIRED_MAJOR_LLVM_VERSION: u8 = 19;


The requirement doesn't bump unless targeting a higher arch? So the logic is:

7 if targeting arch supported by 7 and 19

19 if targeting arch not supported by 7

LegNeato · 2025-06-10T15:02:28Z

We should probably break this down, seeing as neither of us understand this space.

First, we should probably add various values for arch and stuff to enums on the rust side. Some of these might need to be gated if on 7 vs 19.

Then, we should get switching between 7 and a stubbed out / non working 19 via target arch.

Then, we should systematically fix each issue and refactor common code on the way.

brandonros · 2025-06-10T15:49:12Z

seeing as neither of us understand this space.

ok... kind of a strange remark...

i'm just going to work on my local branch and make kernels compile with 19.1

then i was going to work backwards and "make it upstream worthy"...

brandonros · 2025-06-14T17:33:02Z

@LegNeato

i rented some powerful big aws spot VM in the cloud and built llvm-19 debug with assertions enabled and am using it in devcontainer (it's a 15gb .tar.xz and 67gb uncompressed)

it's helping find issues more than a typical llvm release build fyi, just a little tip i'd share

i have the "shaved yaks" opinionated way to get that instance if you want, cost me about $3 total

https://github.com/brandonros/cloud-llvm-build

brandonros · 2025-06-15T01:41:31Z

@LegNeato i'm on the fence about having two rustc_codegen_nvvm_v{{version}} crates 85% copy and pasted but.... i got this to work. vecadd is working at least, going to see if ed25519_vanity_rs compiles later

proof:
vecadd_kernel.ptx.txt

…lidate

brandonros · 2025-06-20T14:41:21Z

@LegNeato i've read multiple conflicting things from multiple "official nvidia sources/documentation" that the new cuda 12.9 toolkit is based on/adds support for either llvm 18, llvm 19, or llvm 20.

i can see in their cicc binary references of llvm 20, so....

i also question if we need this. this might sound dumb but... what if we used a simple official rust target like riscv64gc-unknown-none-elf, use official rust compiler (no custom nightly, no custom codegen llvm integration) to spit out llvm ir, and then patch it to work with cuda...

i totally agree with you/the project's view on "use nvvm to compile llvm ir to ptx"

https://github.com/brandonros/vanity-miner-rs/pull/8/files i haven't had a chance to test it yet (working on it) but in terms of thinking outside the box, i was for sure able to get nvvm to accept llvm ir and make what seems to be a "valid ptx"

brandonros · 2025-06-22T20:30:41Z

i got this working for blackwell a different way. this branch/pr/LLVM v19 integration might work just fine but it's kind of a lot to maintain if there's an "easier" (albeit hackier) way to solve this

https://github.com/brandonros/vanity-miner-rs/actions/runs/15809309968/job/44558442212

Build Pipeline

compile no_std Rust logic + kernels libraries (specifically 1.86.0 because it was built against LLVM 19) targeting riscv64gc-unknown-none-elf due to its simplicity in instruction set
make it emit LLVM IR instead of an actual binary
Adapt the RISC-V LLVM IR to NVPTX64 LLVM IR
assemble the NVPTX64 LLVM IR to NVPTX64 LLVM bitcode
feed the NVPTX64 LLVM bitcode to new CUDA toolkit 12.9 libNVVM which adds support for LLVM19 for Blackwell (previous architectures only support LLVM v7 which is very old) to get Nvidia's PTX (Parallel Thread Execution)
feed the PTX to ptxas to get CUBIN SaSS (Streaming ASSembler)
run the CUBIN on device with gpu_runner

let me know if you actually want this/to put time into it, otherwise blackwell+ might be able to avoid cuda_builder or i'd need to make a cuda_builder that makes this super opinionated rust -> cubin pipeline i made

Fix read_volatile intrinsic

caaef11

LegNeato reviewed Jun 10, 2025

View reviewed changes

brandonros closed this Jun 10, 2025

brandonros reopened this Jun 15, 2025

brandonros force-pushed the llvm-19 branch 3 times, most recently from 6b5fd6b to 3cac1eb Compare June 15, 2025 01:08

support sm_100 and llvm v19

f50708b

brandonros force-pushed the llvm-19 branch from 3cac1eb to f50708b Compare June 15, 2025 01:14

brandonros added 2 commits June 15, 2025 01:32

feature flags

025af1f

less codegen logs

1f79d0c

brandonros added 11 commits June 15, 2025 16:55

compute_120?

ad4181e

no default cpu

f38390b

make bitcast work for new "opaque pointers"?

d7ad3fe

temporarily switch fatal to bug to try to track something down

32d42e3

add logging to cuda_builder as a test

6bfe464

trying to get logs

6cdb9e5

missing debug

47ca0a2

special build.rs loggic

1712528

LLVMRustInlineAsm CanThrow

a87cb5f

no other AsmDialect

f21da08

try to fix an atomic intrinsic problem and add ptxas in builder to va…

d1d2e39

…lidate

brandonros added 11 commits June 16, 2025 01:47

v7 needs new verify

8d9ff2a

try v7 as a default

dc018fb

make llvm-v7 run llvm-as as well

33c2c6d

bump libintrinsics as a test

f014de4

put libinstrincs back

5e2462b

panic on ptxas failure

180e447

default to 61

a787d2c

put dockerfile back

c2e374d

undo scope_asm change

97ff3f8

panic on signal

7440c1b

no ptxas

52791ef

brandonros closed this Jun 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

llvm 19 support #227

llvm 19 support #227

Uh oh!

brandonros commented Jun 8, 2025 •

edited

Loading

Uh oh!

brandonros commented Jun 8, 2025

Uh oh!

brandonros commented Jun 8, 2025

Uh oh!

brandonros commented Jun 10, 2025

Uh oh!

LegNeato commented Jun 10, 2025

Uh oh!

LegNeato left a comment

Uh oh!

LegNeato Jun 10, 2025

Uh oh!

LegNeato commented Jun 10, 2025

Uh oh!

brandonros commented Jun 10, 2025

Uh oh!

brandonros commented Jun 14, 2025 •

edited

Loading

Uh oh!

brandonros commented Jun 15, 2025 •

edited

Loading

Uh oh!

brandonros commented Jun 20, 2025 •

edited

Loading

Uh oh!

brandonros commented Jun 22, 2025

Uh oh!

Uh oh!

llvm 19 support #227

llvm 19 support #227

Uh oh!

Conversation

brandonros commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brandonros commented Jun 8, 2025

Uh oh!

brandonros commented Jun 8, 2025

Uh oh!

brandonros commented Jun 10, 2025

Uh oh!

LegNeato commented Jun 10, 2025

Uh oh!

LegNeato left a comment

Choose a reason for hiding this comment

Uh oh!

LegNeato Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

LegNeato commented Jun 10, 2025

Uh oh!

brandonros commented Jun 10, 2025

Uh oh!

brandonros commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brandonros commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brandonros commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brandonros commented Jun 22, 2025

Build Pipeline

Uh oh!

Uh oh!

brandonros commented Jun 8, 2025 •

edited

Loading

brandonros commented Jun 14, 2025 •

edited

Loading

brandonros commented Jun 15, 2025 •

edited

Loading

brandonros commented Jun 20, 2025 •

edited

Loading