CSL: Potential crt version mismatch

[inkydragon]

Background

Trying to reproduce #55900.
I'm using gcc 14 in MSYS2 (mingw32), and build from source with master fe5ed17.

Then I got a undef error from ld: "undefined reference to `_initterm_e'"

Details

$ make -j16
echo '@"%~dp0/'"$(echo 'usr/bin/julia.exe')"'" %*' | tr / '\\' > /d/jl/julia/julia.bat
chmod a+x /d/jl/julia/julia.bat
    LINK usr/lib/julia/sys.dll
D:/env/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/14.2.0/../../../../i686-w64-mingw32/bin/ld.exe: D:/env/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/14.2.0/../../../../lib/dllcrt2.o: in function `CRT_INIT@12':
C:/M/B/src/mingw-w64/mingw-w64-crt/crt/crtdll.c:100:(.text+0x17a): undefined reference to `_initterm_e'
collect2.exe: error: ld returned 1 exit status
make[1]: *** [sysimage.mk:18: /d/jl/julia/usr/lib/julia/sys.dll] Error 1
make: *** [Makefile:120: julia-sysimg-release] Error 2

I know julia carries a specific version of crt.
Check the exported symbols:

$ nm -g /d/jl/julia/usr/lib/gcc/i686-w64-mingw32/13/libmsvcrt.a | grep initterm
00000000 I __imp___initterm
00000000 T __initterm

$ nm -g /mingw32/lib/libmsvcrt.a  | grep initterm
lib32_libmsvcrt_extra_a-_initterm_e.o:
00000000 D __imp___initterm_e
00000000 T __initterm_e
00000000 I __imp___initterm
00000000 T __initterm

$ gcc --version
gcc.exe (Rev2, Built by MSYS2 project) 14.2.0

$ ld --version
GNU ld (GNU Binutils) 2.43.1

This ld error is easily fixed by not using the libraries carried by CSL.
But it also brought a problem to my attention.

Problem

On Windows, we ship crt from GCC 13.

julia/deps/csl.mk

Lines 107 to 115 in 006f19c

	ifeq ($(OS),WINNT)
	install-csl:
	mkdir -p $(build_private_libdir)/
	cp -a $(build_libdir)/gcc/$(BB_TRIPLET)/13/libgcc_s.a $(build_private_libdir)/
	cp -a $(build_libdir)/gcc/$(BB_TRIPLET)/13/libgcc.a $(build_private_libdir)/
	cp -a $(build_libdir)/gcc/$(BB_TRIPLET)/13/libmsvcrt.a $(build_private_libdir)/
	cp -a $(build_libdir)/gcc/$(BB_TRIPLET)/13/libssp.dll.a $(build_private_libdir)/
	cp -a $(build_libdir)/gcc/$(BB_TRIPLET)/13/libssp.dll.a $(build_libdir)/
	endif

And we are using GCC 14 to build julia x64 version:

• Docker: juliapackaging/package-windows-x86_64:v7.10
• gcc version 14.2.0 (Rev1, Built by MSYS2 project)
• GNU ld (GNU Binutils) 2.43
• Buildkite Job: build x86_64-w64-mingw32
  https://buildkite.com/julialang/julia-master/builds/42913#0193b141-b435-4f5c-901f-b08890a33337/700-710

Using GCC 12 to build julia x86 version:

• Docker: juliapackaging/package-windows-i686:v7.10
• gcc version 12.2.0 (i686-posix-sjlj-rev0, Built by MinGW-W64 project)
• GNU ld (GNU Binutils) 2.38
• Buildkite Job: build i686-w64-mingw32
  https://buildkite.com/julialang/julia-master/builds/42913#0193b141-b8f7-42d8-ba40-4485e8bb901e/701-711

I'm wondering if the old gcc + new crt libraries are causing some problems.
Or if this is related to the stack trace error in #55900.

[Zentrik]

Just hit the background issue I'm guessing this broke in between mingw-w64-x86_64-crt-git-12.0.0.r423.g8bcd5fc1a-1 and mingw-w64-x86_64-crt-git-12.0.0.r446.ga2d19218d-1

EDIT: Yep clearly some suspicious commits in mingw-w64/mingw-w64@8bcd5fc...a2d1921.
Seems unfortunate USE_BINARYBUILDER_CSL=0 didn't get automatically set here.

[Zentrik]

Wait actually how did you get your build working USE_BINARYBUILDER_CSL=0 doesn't work with my 64 bit build which is now failing with an undefined reference to __imp___stack_chk_guard and it broke my 32 bit one where it's not able to load codegen.dll somehow.
Logs at https://github.com/Zentrik/llvm_julia_tester/actions/runs/12458779495.

[inkydragon]

1. Build with USE_BINARYBUILDER_CSL=1,
   we want to link to libssp (Windows build fails on MinGW GCC 13 due to libssp changes #51740).
2. use the new version of libmsvcrt.a (and other libs if you want) on your system instead of the CSL one.
   See https://github.com/JuliaLang/julia/pull/56875/files#diff-f2cc386b53273a3d09652ab8c81e0f0bd3ab5d97d1b4c3a947ac6628b4f0ef4e

OR: Build with USE_BINARYBUILDER_CSL=0, use libssp from CSL

I'm not sure to use an unsupported lib if it's a good idea, but it just works.