Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebase to v3.5.5 #249

Merged
merged 49 commits into from
Dec 24, 2024
Merged

Rebase to v3.5.5 #249

merged 49 commits into from
Dec 24, 2024

Conversation

dscho
Copy link
Collaborator

@dscho dscho commented Dec 21, 2024

range-diff relative to v3.5.4
  • 1: 51380ec = 1: bdad035 Add MSYS2 triplet

  • 2: a116f41 = 2: 900232b Fix msys library name in import libraries

  • 3: f858c02 ! 3: 585fadf Rename dll from cygwin to msys

    @@ winsup/cygwin/syscalls.cc: try_to_bin (path_conv &pc, HANDLE &fh, ACCESS_MASK ac
          {
     -      /* Create unique filename.  Start with a dot, followed by "cyg"
     +      /* Create unique filename.  Start with a dot, followed by "msys"
    - 	 transposed into the Unicode low surrogate area (U+dc00) on file
    - 	 systems supporting Unicode (except Samba), followed by the inode
    - 	 number in hex, followed by a path hash in hex.  The combination
    + 	 transposed to the Unicode private use area in the U+f700 area
    + 	 on file systems supporting Unicode (except Samba), followed by
    + 	 the inode number in hex, followed by a path hash in hex.  The
     @@ winsup/cygwin/syscalls.cc: try_to_bin (path_conv &pc, HANDLE &fh, ACCESS_MASK access, ULONG flags)
            RtlAppendUnicodeToString (&recycler,
      				(pc.fs_flags () & FILE_UNICODE_ON_DISK
      				 && !pc.fs_is_samba ())
    --				? L".\xdc63\xdc79\xdc67" : L".cyg");
    -+				? L".\xdc6d\xdc73\xdc79\xdc73" : L".msys");
    +-				? L".\xf763\xf779\xf767" : L".cyg");
    ++				? L".\xf76d\xf773\xf779\xf773" : L".msys");
            pfii = (PFILE_INTERNAL_INFORMATION) infobuf;
            status = NtQueryInformationFile (fh, &io, pfii, sizeof *pfii,
      				       FileInternalInformation);
  • 4: b084abe = 4: d076660 Add functionality for converting UNIX paths in arguments and environment variables to Windows form for native Win32 applications.

  • 5: f160385 = 5: 35a477b Add functionality for changing OS name via MSYSTEM environment variables.

  • 6: 6393488 = 6: 41ed62e - Move root to /usr. - Change sorting mount points. - By default mount without ACLs. - Can read /etc/fstab with short mount point format.

  • 7: 06ea20e = 7: 8096583 Instead of creating Cygwin symlinks, use deep copy by default

  • 8: 222fd37 = 8: 0ba54e8 Automatically rewrite TERM=msys to TERM=cygwin

  • 9: 41e8ada = 9: e28b742 Do not convert environment for strace

  • 10: 766e55c = 10: 4a24304 strace.cc: Don't set MSYS=noglob

  • 11: 4f01c3f = 11: b3a4a5b Add debugging for strace make_command_line

  • 12: d43f1ee = 12: f8c0980 strace --quiet: be really quiet

  • 13: 0053f7b = 13: c4d9d83 path_conv: special-case root directory to have trailing slash

  • 14: 1e8a891 = 14: 1c5a6e8 When converting to a Unix path, avoid double trailing slashes

  • 15: 673065d = 15: 8b5e9b5 msys2_path_conv: pass PC_NOFULL to path_conv

  • 16: e1db148 = 16: b09fb7f path-conversion: Introduce ability to switch off conversion.

  • 17: f6e516d = 17: b239180 dcrt0.cc: Untangle allow_glob from winshell

  • 18: 1059fbb = 18: 1e94f3e dcrt0.cc (globify): Don't quote literal strings differently when dos_spec

  • 19: f8da08f = 19: e6c5e3e Add debugging for build_argv

  • 20: 03d6fe1 = 20: d44fc10 environ.cc: New facility/environment variable MSYS2_ENV_CONV_EXCL

  • 21: 90d66ee = 21: 98c1641 Fix native symbolic link spawn passing wrong arg0

  • 22: 7b84fed = 22: d12ab6d Introduce the enable_pcon value for MSYS

  • 23: 6f87e23 = 23: 865ad5d popen: call /usr/bin/sh instead of /bin/sh

  • 24: df7a56e = 24: 5f36fdb Disable the 'cygwin' GitHub workflow

  • 25: cf56ccd ! 25: 862887f CI: add a GHA for doing a basic build test

    @@ .github/workflows/build.yaml (new)
     +
     +    steps:
     +      - name: Checkout code
    -+        uses: actions/checkout@v3
    ++        uses: actions/checkout@v4
     +
     +      - name: setup-msys2
     +        uses: msys2/setup-msys2@v2
    @@ .github/workflows/build.yaml (new)
     +          make DESTDIR="$(pwd)"/_dest install
     +
     +      - name: Upload
    -+        uses: actions/upload-artifact@v3
    ++        uses: actions/upload-artifact@v4
     +        with:
     +          name: install
     +          path: _dest/
  • 26: 083c245 = 26: 764bd7f CI: fix the build with gcc 13

  • 27: 91f46df = 27: bea6a07 Set up a GitHub Action to keep in sync with Cygwin

  • 28: 4dd6aad = 28: 080e2b0 Expose full command-lines to other Win32 processes by default

  • 29: 6ef58ed = 29: 9f45f58 Add a helper to obtain a function's address in kernel32.dll

  • 30: 38a8880 = 30: f6f7e2c Emulate GenerateConsoleCtrlEvent() upon Ctrl+C

  • 31: b9d1fad = 31: 86cd50d kill: kill Win32 processes more gently

  • 32: 715f6ce = 32: 4b228ef Cygwin: make option for native inner link handling.

  • 33: abeeb31 = 33: 6e0c3b5 docs: skip building texinfo and PDF files

  • 34: fbe8ae3 = 34: ecf50e6 install-libs: depend on the "toollibs"

  • 35: 38abc7c = 35: 8cb4ff2 POSIX-ify the SHELL variable

  • 36: 9c758e8 = 36: 1f6020c Handle ORIGINAL_PATH just like PATH

  • 37: 312f766 = 37: 79e3880 uname: allow setting the system name to CYGWIN

  • 38: e19af20 = 38: ef59c45 Pass environment variables with empty values

  • 39: 4deb751 = 39: c347d6c Optionally disallow empty environment values again

  • 40: a1f282d = 40: 210e5f3 build_env(): respect the MSYS environment variable

  • 41: d1b382b = 41: bef1e6b Revert "Cygwin: Enable dynamicbase on the Cygwin DLL by default"

  • 42: 997f6f3 = 42: 8103a09 CI: set -Wno-error=maybe-uninitialized

  • 43: e347515 = 43: 6e10b1b Avoid sharing cygheaps across Cygwin versions

  • 44: dd9d970 = 44: f61f7fb uname: report msys2-runtime commit hash, too

  • 45: 2bfb773 < -: ---------- Cygwin: pipe: Fix a regression that raw_write() slows down

  • 46: 16dfe94 = 45: debafbf Cygwin: find_fast_cwd: don't run assembler checking code on ARM64

  • 47: e09c64e ! 46: 2c55ca5 cygthread: suspend thread before terminating.

    @@ winsup/cygwin/cygthread.cc: cygthread::terminate_thread ()
        WaitForSingleObject (h, INFINITE);
        CloseHandle (h);
     
    + ## winsup/cygwin/pinfo.cc ##
    +@@ winsup/cygwin/pinfo.cc: proc_waiter (void *arg)
    + 
    +   for (;;)
    +     {
    +-      DWORD nb;
    ++      DWORD nb, err;
    +       char buf = '\0';
    + 
    +       if (!ReadFile (vchild.rd_proc_pipe, &buf, 1, &nb, NULL)
    +-	  && GetLastError () != ERROR_BROKEN_PIPE)
    ++	  && (err = GetLastError ()) != ERROR_BROKEN_PIPE)
    + 	{
    +-	  system_printf ("error on read of child wait pipe %p, %E", vchild.rd_proc_pipe);
    ++	  /* ERROR_OPERATION_ABORTED is expected due to the possibility that
    ++	     CancelSynchronousIo interruped the ReadFile call, so don't output
    ++	     that error */
    ++	  if (err != ERROR_OPERATION_ABORTED)
    ++	    system_printf ("error on read of child wait pipe %p, %E", vchild.rd_proc_pipe);
    + 	  break;
    + 	}
    + 
    +
      ## winsup/cygwin/sigproc.cc ##
     @@ winsup/cygwin/sigproc.cc: proc_terminate ()
    + 	     to 1 iff it is a Cygwin process.  */
      	  if (!have_execed || !have_execed_cygwin)
      	    chld_procs[i]->ppid = 1;
    - 	  if (chld_procs[i].wait_thread)
    --	    chld_procs[i].wait_thread->terminate_thread ();
    -+	    if (!CancelSynchronousIo (chld_procs[i].wait_thread->thread_handle ()))
    -+	      chld_procs[i].wait_thread->terminate_thread ();
    +-	  if (chld_procs[i].wait_thread)
    ++	  /* Attempt to exit the wait_thread cleanly via CancelSynchronousIo
    ++	     before falling back to the (explicitly dangerous) cross-thread
    ++	     termination */
    ++	  if (chld_procs[i].wait_thread
    ++	      && !CancelSynchronousIo (chld_procs[i].wait_thread->thread_handle ()))
    + 	    chld_procs[i].wait_thread->terminate_thread ();
      	  /* Release memory associated with this process unless it is 'myself'.
      	     'myself' is only in the chld_procs table when we've execed.  We
    - 	     reach here when the next process has finished initializing but we
    +@@ winsup/cygwin/sigproc.cc: remove_proc (int ci)
    + {
    +   if (have_execed)
    +     {
    +-      if (_my_tls._ctinfo != chld_procs[ci].wait_thread)
    ++      /* Attempt to exit the wait_thread cleanly via CancelSynchronousIo
    ++	 before falling back to the (explicitly dangerous) cross-thread
    ++	 termination */
    ++      if (_my_tls._ctinfo != chld_procs[ci].wait_thread
    ++	  && !CancelSynchronousIo (chld_procs[ci].wait_thread->thread_handle ()))
    + 	chld_procs[ci].wait_thread->terminate_thread ();
    +     }
    +   else if (chld_procs[ci] && chld_procs[ci]->exists ())
  • 48: eafd9a2 < -: ---------- fixup! cygthread: suspend thread before terminating.

  • 49: 7829673 < -: ---------- fixup! cygthread: suspend thread before terminating.

  • 50: 3cd8609 < -: ---------- fixup! CI: add a GHA for doing a basic build test

  • 51: 2eb6be1 = 47: e5dc132 Cygwin: revert use of CancelSyncronousIo on wait_thread.

  • 52: fe383b0 = 48: 5453f9f Cygwin: cache IsWow64Process2 host arch in wincap.

  • 53: 8d847f4 = 49: 985e265 Cygwin: uname: add host machine tag to sysname.

Alexpux and others added 30 commits December 21, 2024 17:48
Cygwin's speclib doesn't handle dashes or dots. However, we are about to
rename the output file name from `cygwin1.dll` to `msys-2.0.dll`.

Let's preemptively fix up all the import libraries that would link
against `msys_2_0.dll` to correctly link against `msys-2.0.dll` instead.
…ent variables to Windows form for native Win32 applications.
…t without ACLs. - Can read /etc/fstab with short mount point format.
The new `winsymlinks` mode `deepcopy` (which is made the default) lets
calls to `symlink()` create (deep) copies of the source file/directory.

This is necessary because unlike Cygwin, MSYS2 does not try to be its
own little ecosystem that lives its life separate from regular Win32
programs: the latter have _no idea_ about Cygwin-emulated symbolic links
(i.e. system files whose contents start with `!<symlink>\xff\xfe` and
the remainder consists of the NUL-terminated, UTF-16LE-encoded symlink
target).

To support Cygwin-style symlinks, the new mode `sysfile` is introduced.

Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With MSys1, it was necessary to set the TERM variable to "msys". To
allow for a smooth transition from MSys1 to MSys2, let's simply handle
TERM=msys as if the user had not specified TERM at all and wanted us to
use our preferred TERM value.
Strace is a Windows program so MSYS2 will convert all arguments and environment vars and that makes debugging msys2 software with strace very tricky.
Commit message for this code was:

* strace.cc (create_child): Set CYGWIN=noglob when starting new process so that

  Cygwin will leave already-parsed the command line alonw."

I can see no reason for it and it badly breaks the ability to use
strace.exe to investigate calling a Cygwin program from a Windows
program, for example:
strace mingw32-make.exe
.. where mingw32-make.exe finds sh.exe and uses it as the shell.
The reason it badly breaks this use-case is because dcrt0.cc depends
on globbing to happen to parse commandlines from Windows programs;
irrespective of whether they contain any glob patterns or not.

See quoted () comment:
"This must have been run from a Windows shell, so preserve
 quotes for globify to play with later."
The biggest problem with strace spitting out `create_child: ...` despite
being asked to be real quiet is that its output can very well interfere
with scripts' operations.

For example, when running any of Git for Windows' shell scripts with
`GIT_STRACE_COMMANDS=/path/to/logfile` (which is sadly an often needed
debugging technique while trying to address the many MSYS2 issues Git for
Windows faces), any time the output of any command is redirected into a
variable, it will include that `create_child: ...` line, wreaking havoc
with Git's expectations.

So let's just really be quiet when we're asked to be quiet.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When converting `/c/` to `C:\`, the trailing slash is actually really
necessary, as `C:` is not an absolute path.

We must be very careful to do this only for root directories, though. If
we kept the trailing slash also for, say, `/y/directory/`, we would run
into the following issue: On FAT file systems, the normalized path is
used to fake inode numbers. As a result, `Y:\directory\` and
`Y:\directory` have different inode numbers!!!

This would result in very non-obvious symptoms. Back when we were too
careless about keeping the trailing slash, it was reported to the Git
for Windows project that the `find` and `rm` commands can error out on
FAT file systems with very confusing "No such file or directory" errors,
for no good reason.

During the original investigation, Vasil Minkov pointed out in
git-for-windows/git#1497 (comment),
that this bug had been fixed in Cygwin as early as 1997... and the bug
was unfortunately reintroduced into early MSYS2 versions.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When calling `cygpath -u C:/msys64/` in an MSYS2 setup that was
installed into `C:/msys64/`, the result should be `/`, not `//`.

Let's ensure that we do not append another trailing slash if the
converted path already ends in a slash.

This fixes #112

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In theory this doesn't make a difference because posix_to_win32_path()
is only called with rooted/absolute paths, but as pointed out in
#103 PC_NOFULL will preserve
the trailing slash of unix paths (for some reason).

See "cygpath -m /bin/" (preserved) vs "cygpath -am /bin/" (dropped)

One use case where we need to trailing slashes to be preserved is the GCC build
system:
https://github.com/gcc-mirror/gcc/blob/6d82e0fea5f988e829912a/gcc/Makefile.in#L2314

The Makefile appends a slash to the prefixes and the C code doing relocation will
treat the path as a directory if there is a trailing slash. See
msys2/MINGW-packages#14173 for details.

With this change all our MSYS2 path_conv tests pass again.
When calling windows native apps from MSYS2, the runtime tries to
convert commandline arguments by a specific set of rules. This idea was
inherited from the MSys/MinGW project (which is now seemingly stale, yet
must be credited with championing this useful feature, see MinGW wiki
https://web.archive.org/web/20201112005258/http://www.mingw.org/wiki/Posix_path_conversion).

If the user does not want that behavior on a big scale, e.g. inside a
Bash script, with the changes introduced in this commit, the user can
now set the the environment variable `MSYS_NO_PATHCONV` when calling
native windows commands.

This is a feature that has been introduced in Git for Windows via
git-for-windows/msys2-runtime#11 and it predates
support for the `MSYS2_ENV_CONV_EXCL` and `MSYS2_ARG_CONV_EXCL`
environment variables in the MSYS2 runtime; Many users find the
simplicity of `MSYS_NO_PATHCONV` appealing.

So let's teach MSYS2 proper this simple trick that still allows using
the sophisticated `MSYS2_*_CONV_EXCL` facilities but also offers a
convenient catch-all "just don't convert anything" knob.

Signed-off-by: 마누엘 <nalla@hamal.uberspace.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Otherwise if globbing is allowed and we get called from a
Windows program, build_argv thinks we've been called from
a Cygwin program.
…spec

Reverts 25ba8f3. I can't figure out what
the intention was. I'm sure I'll find out soon enough when everything breaks.

This change means that input of:
  '"C:/test.exe SOME_VAR=\"literal quotes\""'

becomes:
  'C:/test.exe SOME_VAR="literal quotes"'

instead of:
  'C:/test.exe SOME_VAR=\literal quotes\'

.. which is at least consistent with the result for:
  '"no_drive_or_colon SOME_VAR=\"literal quotes\""'

The old result of course resulted in the quoted string being split into
two arguments at the space which is clearly not intended.

I *guess* backslashes in dos paths may have been the issue here?
If so I don't care since we should not use them, ever, esp. not at
the expense of sensible forward-slash-containing input.
Works very much like MSYS2_ARG_CONV_EXCL. In fact it uses the same
function, arg_heuristic_with_exclusions (). Also refactors parsing
the env. variables to use new function, string_split_delimited ().

The env. that is searched through is the merged (POSIX + Windows)
one. It remains to be seen if this should be made an option or not.

This feature was prompted because the R language (Windows exe) calls
bash to run configure.win, which then calls back into R to read its
config variables (LOCAL_SOFT) and when this happens, msys2-runtime
converts R_ARCH from "/x64" to an absolute Windows path and appends
it to another absolute path, R_HOME, forming an invalid path.
It is simply the negation of `disable_pcon`, i.e. `MSYS=enable_pcon` is
equivalent to `MSYS=nodisable_pcon` (the former is slightly more
intuitive than the latter) and likewise `MSYS=noenable_pcon` is
equivalent to `MSYS=disable_pcon` (here, the latter is definitely more
intuitive than the former).

This is needed because we just demoted the pseudo console feature to be
opt-in instead of opt-out, and it would be awkward to recommend to users
to use "nodisable_pcon"... "nodisable" is not even a verb.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We mount /usr/bin to /bin, but in a chroot this is broken and we
have no /bin, so try to use the real path.

chroot is used by pacman to run install scripts when called with --root
and this broke programs in install scripts calling popen()
(install-info from texinfo for example)

There are more paths hardcoded to /bin in cygwin which might also be broken
in this scenario, so this maybe should be extended to all of them.
It does not work at all. For example, `rpm -E %fedora` says that there
should be version 33 of rpmsphere at
https://github.com/rpmsphere/noarch/tree/master/r, but there is only
version 32.

Another thing that is broken: Cygwin now assumes that a recent
mingw-w64-headers version is available, but Fedora apparently only
offers v7.0.0, which is definitely too old to accommodate for the
expectation of cygwin/cygwin@c1f7c4d1b6d7.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Build with --disable-dependency-tracking because we only build once
and this saves 3-4 minutes in CI.
This will help us by automating an otherwise tedious task.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In the Cygwin project, it was decided that the command-line of Cygwin
processes, as shown in the output of `wmic process list`, would suffer
from being truncated to 32k (and is transmitted to the child process via
a different mechanism, anyway), and therefore only the absolute path of
the executable is shown by default.

Users who would like to see the full command-line (even if it is
truncated) are expected to set `CYGWIN=wincmdln` (or, in MSYS2's case,
`MSYS=wincmdln`).

Seeing as MSYS2 tries to integrate much better with the surrounding
Win32 ecosystem than Cygwin, it makes sense to turn this on by default.

Users who wish to suppress it can still set `MSYS=nowincmdln`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In particular, we are interested in the address of the CtrlRoutine
and the ExitProcess functions. Since kernel32.dll is loaded first thing,
the addresses will be the same for all processes (matching the
CPU architecture, of course).

This will help us with emulating SIGINT properly (by not sending signals
to *all* processes attached to the same Console, as
GenerateConsoleCtrlEvent() would do).

Co-authored-by: Naveen M K <naveen@syrusdark.website>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This patch is heavily inspired by the Git for Windows' strategy in
handling Ctrl+C.

When a process is terminated via TerminateProcess(), it has no chance to
do anything in the way of cleaning up. This is particularly noticeable
when a lengthy Git for Windows process tries to update Git's index file
and leaves behind an index.lock file. Git's idea is to remove the stale
index.lock file in that case, using the signal and atexit handlers
available in Linux. But those signal handlers never run.

Note: this is not an issue for MSYS2 processes because MSYS2 emulates
Unix' signal system accurately, both for the process sending the kill
signal and the process receiving it. Win32 processes do not have such a
signal handler, though, instead MSYS2 shuts them down via
`TerminateProcess()`.

For a while, Git for Windows tried to use a gentler method, described in
the Dr Dobb's article "A Safer Alternative to TerminateProcess()" by
Andrew Tucker (July 1, 1999),
http://www.drdobbs.com/a-safer-alternative-to-terminateprocess/184416547

Essentially, we injected a new thread into the running process that does
nothing else than running the ExitProcess() function.

However, this was still not in line with the way CMD handles Ctrl+C: it
gives processes a chance to do something upon Ctrl+C by calling
SetConsoleCtrlHandler(), and ExitProcess() simply never calls that
handler.

So for a while we tried to handle SIGINT/SIGTERM by attaching to the
console of the command to interrupt, and generating the very same event
as CMD does via GenerateConsoleCtrlEvent().

This method *still* was not correct, though, as it would interrupt
*every* process attached to that Console, not just the process (and its
children) that we wanted to signal. A symptom was that hitting Ctrl+C
while `git log` was shown in the pager would interrupt *the pager*.

The method we settled on is to emulate what GenerateConsoleCtrlEvent()
does, but on a process by process basis: inject a remote thread and call
the (private) function kernel32!CtrlRoutine.

To obtain said function's address, we use the dbghelp API to generate a
stack trace from a handler configured via SetConsoleCtrlHandler() and
triggered via GenerateConsoleCtrlEvent(). To avoid killing each and all
processes attached to the same Console as the MSYS2 runtime, we modify
the cygwin-console-helper to optionally print the address of
kernel32!CtrlRoutine to stdout, and then spawn it with a new Console.

Note that this also opens the door to handling 32-bit process from a
64-bit MSYS2 runtime and vice versa, by letting the MSYS2 runtime look
for the cygwin-console-helper.exe of the "other architecture" in a
specific place (we choose /usr/libexec/, as it seems to be the
convention for helper .exe files that are not intended for public
consumption).

The 32-bit helper implicitly links to libgcc_s_dw2.dll and
libwinpthread-1.dll, so to avoid cluttering /usr/libexec/, we look for
the helped of the "other" architecture in the corresponding mingw32/ or
mingw64/ subdirectory.

Among other bugs, this strategy to handle Ctrl+C fixes the MSYS2 side of
the bug where interrupting `git clone https://...` would send the
spawned-off `git remote-https` process into the background instead of
interrupting it, i.e. the clone would continue and its progress would be
reported mercilessly to the console window without the user being able
to do anything about it (short of firing up the task manager and killing
the appropriate task manually).

Note that this special-handling is only necessary when *MSYS2* handles
the Ctrl+C event, e.g. when interrupting a process started from within
MinTTY or any other non-cmd-based terminal emulator. If the process was
started from within `cmd.exe`'s terminal window, child processes are
already killed appropriately upon Ctrl+C, by `cmd.exe` itself.

Also, we can't trust the processes to end it's subprocesses upon receiving
Ctrl+C. For example, `pip.exe` from `python-pip` doesn't kill the python
it lauches (it tries to but fails), and I noticed that in cmd it kills python
also correctly, which mean we should kill all the process using
`exit_process_tree`.

Co-authored-by: Naveen M K <naveen@syrusdark.website>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho and others added 12 commits December 21, 2024 17:48
There is a difference between an empty value and an unset environment
variable. We should not confuse both; If the user wants to unset an
environment variable, they can certainly do so (unsetenv(3), or in the
shell: 'unset ABC').

This fixes Git's t3301-notes.sh, which overrides environment variables
with empty values.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We just disabled the code that skips environment variables whose values
are empty.

However, this code was introduced a long time ago into Cygwin in
d6b1ac7 (* environ.cc (build_env): Don't put an empty environment
variable into the environment.  Optimize use of "len". * errno.cc
(ERROR_MORE_DATA): Translate to EMSGSIZE rather than EAGAIN.,
2006-09-07), seemingly without any complaints.

Meaning: There might very well be use cases out there where it makes
sense to skip empty-valued environment variables.

Therefore, it seems like a good idea to have a "knob" to turn it back
on. With this commit, we introduce such a knob: by setting
`noemptyenvvalues` the `MSYS` variable (or appending it if that variable
is already set), users can tell the MSYS2 runtime to behave just like in
the olden times.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With this commit, you can call

	MSYS=noemptyenvvalues my-command

and it does what is expected: to pass no empty-valued environment
variables to `my-command`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
After the update of msys2-w32api from v11.0.1 to current master (and soon to be v12)
we get: winsup/cygwin/exceptions.cc:1736:33: error: '<anonymous>' may be used uninitialized [-Werror=maybe-uninitialized]

Ignore it like the rest.

Fixes #214
It frequently leads to problems when trying, say, to call from MSYS2's
Bash into Cygwin's or Git for Windows', merely because sharing that data
is pretty finicky.

For example, using the MSYS2' Bash using the MSYS2 runtime version that
is current at time of writing, trying to call Cygwin's programs fails
in manners like this:

    $ /c/cygwin64/bin/uname -r
      0 [main] uname (9540) child_copy: cygheap read copy failed, 0x800000000..0x800010BE0, done 0, windows pid 9540, Win32 error 6
    680 [main] uname 880 C:\cygwin64\bin\uname.exe: *** fatal error - couldn't create signal pipe, Win32 error 5

with the rather misleading exit code 127 (a code which is reserved to
indicate that a command was not found).

Let's just treat the MSYS2 runtime and the Cygwin runtime as completely
incompatible with one another, by virtue of using a different
magic constant than merely `CHILD_INFO_MAGIC`.

By using the msys2-runtime commit to modify that magic constant, we can
even spawn programs using a different MSYS2 runtime (such as Git for
Windows') because the commit serves as the tell-tale whether two MSYS2
runtime versions are compatible with each other. To support building in
the MSYS2-packages repository (where we do not check out the
`msys2-runtime` but instead check out Cygwin and apply patches on top),
let's accept a hard-coded commit hash as `./configure` option.

One consequence is that spawned MSYS processes using a different MSYS2
runtime will not be visible as such to the parent process, i.e. they
cannot share any resources such as pseudo terminals. But that's okay,
they are simply treated as if they were regular Win32 programs.

Note: We have to use a very rare form of encoding the brackets in the
`expr` calls: quadrigraphs (for a thorough explanation, see
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs).
This is necessary because it is apparently impossible to encode brackets
in `configure.ac` files otherwise.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Having just Cygwin's version in the output of `uname` is not helpful, as
both MSYS2 as well as Git for Windows release intermediate versions of
the MSYS2 runtime much more often than Cygwin runtime versions are
released.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
https://cygwin.com/pipermail/cygwin/2024-February/255397.html
reports a crash on ARM64 probably related to checking x86_64
code on the x86_64 emulator on AArch64.

At least for testing, pull the code checking the host HW
up to be called before trying to evaluate assembler code.

This fixes git-for-windows/git#4808

Backported from 4e77fa9 (Cygwin: find_fast_cwd: don't run assembler
checking code on ARM64, 2024-02-13).

Signed-off-by: Corinna Vinschen <corinna@vinschen.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This addresses an extremely difficult to debug deadlock when running
under emulation on ARM64.

A relatively easy way to trigger this bug is to call `fork()`, then within the
child process immediately call another `fork()` and then `exit()` the
intermediate process.

It would seem that there is a "code emulation" lock on the wait thread at
this stage, and if the thread is terminated too early, that lock still exists
albeit without a thread, and nothing moves forward.

It seems that a `SuspendThread()` combined with a `GetThreadContext()`
(to force the thread to _actually_ be suspended, for more details see
https://devblogs.microsoft.com/oldnewthing/20150205-00/?p=44743)
makes sure the thread is "booted" from emulation before it is suspended.

Hopefully this means it won't be holding any locks or otherwise leave
emulation in a bad state when the thread is terminated.

Also, attempt to use `CancelSynchonousIo()` (as seen in `flock.cc`) to avoid
the need for `TerminateThread()` altogether.  This doesn't always work,
however, so was not a complete fix for the deadlock issue.

Addresses: https://cygwin.com/pipermail/cygwin-developers/2024-May/012694.html
Signed-off-by: Jeremy Drake <cygwin@jdrake.com>
It appears this is causing hangs on native x86_64 in similar scenarios
as the hangs on ARM64, because `CancelSynchronousIo` is returning `TRUE`
but not canceling the `ReadFile` call as expected.

Addresses: msys2/MSYS2-packages#4340 (comment)
Fixes: b091b47 ("cygthread: suspend thread before terminating.")
Signed-off-by: Jeremy Drake <cygwin@jdrake.com>
This was already used in the FAST_CWD check, and could be used in a
couple other places.

I found the "emulated"/process value returned from the function largely
useless, so I did not cache it.  It is useless because, as the docs say,
it is set to IMAGE_FILE_MACHINE_UNKNOWN (0) if the process is not
running under WOW64, but Microsoft also doesn't consider x64-on-ARM64 to
be WOW64, so it is set to 0 regardless if the process is ARM64 or x64.
You can tell the difference via
GetProcessInformation(ProcessMachineTypeInfo), but for the current
process even that's overkill: what we really want to know is the
IMAGE_FILE_MACHINE_* constant for the Cygwin dll itself, which is
conveniently located in memory already, so cache that in wincap also for
easy comparisons.

Signed-off-by: Jeremy Drake <cygwin@jdrake.com>
(cherry picked from commit 46f7bcc)
If the Cygwin dll's architecture is different from the host system's
architecture, append an additional tag that indicates the host system
architecture (the Cygwin dll's architecture is already indicated in
machine).

Signed-off-by: Jeremy Drake <cygwin@jdrake.com>
(cherry picked from commit 7923059)
@jeremyd2019
Copy link
Member

  • 3: f858c02 ! 3: 585fadf Rename dll from cygwin to msys
    @@ winsup/cygwin/syscalls.cc: try_to_bin (path_conv &pc, HANDLE &fh, ACCESS_MASK ac
          {
     -      /* Create unique filename.  Start with a dot, followed by "cyg"
     +      /* Create unique filename.  Start with a dot, followed by "msys"
    - 	 transposed into the Unicode low surrogate area (U+dc00) on file
    - 	 systems supporting Unicode (except Samba), followed by the inode
    - 	 number in hex, followed by a path hash in hex.  The combination
    + 	 transposed to the Unicode private use area in the U+f700 area
    + 	 on file systems supporting Unicode (except Samba), followed by
    + 	 the inode number in hex, followed by a path hash in hex.  The
     @@ winsup/cygwin/syscalls.cc: try_to_bin (path_conv &pc, HANDLE &fh, ACCESS_MASK access, ULONG flags)
            RtlAppendUnicodeToString (&recycler,
      				(pc.fs_flags () & FILE_UNICODE_ON_DISK
      				 && !pc.fs_is_samba ())
    --				? L".\xdc63\xdc79\xdc67" : L".cyg");
    -+				? L".\xdc6d\xdc73\xdc79\xdc73" : L".msys");
    +-				? L".\xf763\xf779\xf767" : L".cyg");
    ++				? L".\xf76d\xf773\xf779\xf773" : L".msys");
            pfii = (PFILE_INTERNAL_INFORMATION) infobuf;
            status = NtQueryInformationFile (fh, &io, pfii, sizeof *pfii,
      				       FileInternalInformation);

Huh, there was more discussion about the unpaired surrogates causing issues, but I didn't notice a cygwin-patches patch changing them, let alone backporting the change to 3.5.

@lazka
Copy link
Member

lazka commented Dec 22, 2024

Huh, there was more discussion about the unpaired surrogates causing issues, but I didn't notice a cygwin-patches patch changing them, let alone backporting the change to 3.5.

cygwin/cygwin@bb854e0 for reference

@lazka
Copy link
Member

lazka commented Dec 22, 2024

lgtm

@dscho dscho merged commit 985e265 into msys2-3.5.5 Dec 24, 2024
3 checks passed
@dscho dscho deleted the tentative/msys2-3.5.5 branch December 24, 2024 21:05
dscho added a commit to dscho/MSYS2-packages that referenced this pull request Dec 24, 2024
This corresponds to msys2/msys2-runtime#249.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho
Copy link
Collaborator Author

dscho commented Dec 24, 2024

➡️msys2/MSYS2-packages#5095

lazka pushed a commit to msys2/MSYS2-packages that referenced this pull request Dec 24, 2024
This corresponds to msys2/msys2-runtime#249.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@jeremyd2019
Copy link
Member

There are some reports of hangs on the Cygwin mailing list, possibly due to changes in signal handling that are messing up SIGCHLD somehow (it looked like Takashi had a patch in https://cygwin.com/pipermail/cygwin-patches/2024q4/013145.html)

@lazka
Copy link
Member

lazka commented Dec 25, 2024

I'll keep it in staging for now.

@jeremyd2019
Copy link
Member

A build was hung on arm64, I had to kill a sh.exe process (I tried kill -CHLD on the sh and parent make process, but neither worked)

@dscho
Copy link
Collaborator Author

dscho commented Dec 25, 2024

A build was hung on arm64, I had to kill a sh.exe process (I tried kill -CHLD on the sh and parent make process, but neither worked)

@jeremyd2019 should we fast-track #251 and see whether it addresses the hangs? In case Cygwin decides to go with a different version of the patch, we can always revert and apply the new one...

@jeremyd2019
Copy link
Member

I don't know - it seems like reports are still coming in of issues, but most of them Takashi suggests will be fixed by that patch. Currently outstanding is https://inbox.sourceware.org/cygwin/20241225230834.b21c465455c4542854464d07@nifty.ne.jp/T/#t

@jeremyd2019
Copy link
Member

it seems to have hung again building imagemagick on arm64, so might be worth a try (think I'll try to hold back msys2-runtime and make sure it doesn't hang with 3.5.4)

@jeremyd2019
Copy link
Member

Hung 3 times with 3.5.5, worked first try with 3.5.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants