http.request crash nodejs 32 bit

[mohammadmoth]

Version

14.17.0 32 bit in this debug , But it happens with

versions crashed
node 14.x.x 32 bit
node 16.x.x 32 bit
node 17.x.x 32 bit

It was working well on
node 10.x.x 32 bit 
node 12.x.x 32 bit

It works perfectly with all node 64 bit

Platform

Microsoft Windows NT 10.0.19042.0 x64

Subsystem

No response

What steps will reproduce the bug?

We have servers forward request to the back-end side that worked through http .
Some requests are repeated every 10 to 20 seconds and All requests are localhost .
What this means is that the node works as a proxy at a high level(Application layer).

What happens in particular is that the nodejs CRASH without any message error, even though we followed the rules for the wrong documentation.

After following we put nodejs through WinDbg or qt creator.
This error has been thrown.

1  __report_gsfailure                                              gs_report.c           220   0x198bdd3  
2  uv_tcp_try_connect                                              tcp.c                 879   0xe4c84b   
3  TpSetPoolWorkerThreadIdleTimeout                                ntdll                       0x77858a7b 
4  RtlSwitchedVVI                                                  ntdll                       0x7785718c 
5  v8::base::Thread::GetThreadLocal                                platform-win32.cc     1387  0x17bc69a  
6  v8::HandleScope::Initialize                                     api.cc                1075  0x14ef384  
7  node::tracing::TraceEventHelper::GetCategoryGroupEnabled        trace_event.h         326   0xdae35a   
8  node::AsyncWrap::AsyncReset                                     async_wrap.cc         636   0xe26faf   
9  node::AsyncWrap::AsyncWrap                                      async_wrap.cc         506   0xe24f01   
10 uv_tcp_connect                                                  uv-common.c           361   0xe61127   
11 node::TCPWrap::Connect<sockaddr_in>                             tcp_wrap.cc           314   0xd0c733   
12 node::TCPWrap::Connect                                          tcp_wrap.cc           273   0xd0cfd6   
13 v8::internal::FunctionCallbackArguments::Call                   api-arguments-inl.h   159   0x14b8871  
14 v8::internal::`anonymous namespace'::HandleApiCallHelper<0>     builtins-api.cc       113   0x14b7fdf  
15 v8::internal::Builtin_Impl_HandleApiCall                        builtins-api.cc       141   0x14b8263  
16 v8::internal::Builtin_HandleApiCall                             builtins-api.cc       129   0x14b8108  
17 _Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit embedded.S            13174 0x1571f67  
18 _Builtins_InterpreterEntryTrampoline                            embedded.S            1081  0x1515509  
19 _Builtins_ArgumentsAdaptorTrampoline                            embedded.S            317   0x150ffe2  
20 _Builtins_InterpreterEntryTrampoline                            embedded.S            1081  0x1515509  
21 _Builtins_ArgumentsAdaptorTrampoline                            embedded.S            317   0x150ffe2  
22 _Builtins_InterpreterEntryTrampoline                            embedded.S            1081  0x1515509  
23 _Builtins_InterpreterEntryTrampoline                            embedded.S            1081  0x1515509  
24 _Builtins_JSEntryTrampoline                                     embedded.S            827   0x151368a  
25 _Builtins_JSEntry                                               embedded.S            803   0x15134ab  
26 v8::internal::`anonymous namespace'::Invoke                     execution.cc          367   0x141f9f0  
27 v8::internal::Execution::Call                                   execution.cc          461   0x141f3c9  
28 v8::Function::Call                                              api.cc                4875  0x14de344  
29 node::InternalMakeCallback                                      callback.cc           200   0xe33790   
30 node::AsyncWrap::MakeCallback                                   async_wrap.cc         691   0xe2ced6   
31 node::AsyncWrap::MakeCallback                                   async_wrap-inl.h      80    0xcbda75   
32 node::cares_wrap::`anonymous namespace'::AfterGetAddrInfo       cares_wrap.cc         1498  0xe18728   
33 uv__getaddrinfo_done                                            getaddrinfo.c         232   0xe552c6   
34 uv__work_done                                                   threadpool.c          311   0xe620f4   
35 uv_process_async_wakeup_req                                     async.c               96    0xe5e49b   
36 uv_process_reqs                                                 req-inl.h             195   0xe5dcf7   
37 uv_run                                                          core.c                610   0xe5de71   
38 node::NodeMainInstance::Run                                     node_main_instance.cc 134   0xd84cf3   
39 node::Start                                                     node.cc               1107  0xde4b52   
40 wmain                                                           node_main.cc          88    0xc9936d   


--------------------------------------------
	Local Variables:	
		addr	@0x93edd8	sockaddr
		addrlen	9694756	unsigned int
		bind_addr	<unzugänglich>	sockaddr*
		bytes	0	unsigned long
		cb	2005234107	<function>
		converted	@0x93ec1c	sockaddr_storage
		err	1	int
		handle	@0x77805750	uv_tcp_s
			activecnt	1197426003	int
			alloc_cb	1801676136	<function>
			close_cb	1683164813	<function>
			data	0x6425	void*
			delayed_error	25701	int
			endgame_next	0x6c	uv_handle_s*
			flags	1752199233	unsigned int
			handle_queue	@0x77805760	void*[2]
			loop	0x901d2afa	uv_loop_s*
			read_cb	1969516365	<function>
			read_req	@0x77805794	uv_read_s
			reqs_pending	25964	unsigned int
			socket	1987013989	unsigned int
			stream	@0x778057d8	<unnamed-tag>
			tcp	@0x778057e8	<unnamed-tag>
			type	0n1188515830 (No matching enumerant)	uv_handle_type
			u	@0x77805768	<unnamed-tag>
			write_queue_size	1131441253	unsigned int
		loop	<unzugänglich>	uv_loop_s*
		req	@0x93edd8	uv_connect_s
			cb	10	<function>
			data	0x0	void*
			handle	0x0	uv_stream_s*
			next_req	0x11c	uv_req_s*
			reserved	@0x93ede0	void*[6]
			type	0n9694732 (No matching enumerant)	uv_req_type
			u	@0x93edf8	<unnamed-tag>
		retransmit_ioctl	@0x93ec14	_TCP_INITIAL_RTO_PARAMETERS
			MaxSynRetransmissions	'È' 	200	0xc8	unsigned char
			Rtt	27064	unsigned short
		success	<unzugänglich>	int
	Inspector		
	Ausdrücke		
	Rückgabewert		
	Tooltip		

handle->delayed_error = 25701
err = 1

Note that it does not happen continuously and we do not know when it will happen.
What i find here is that it is an error in winsock and we were unable to remedy the error, it may be useful to redirect the same error.

How often does it reproduce? Is there a required condition?

It happens randomly, the more servers you get, the more, i.e. 10 to 20%
no required condition ....

What is the expected behavior?

No response

What do you see instead?

....

Additional information

[bnoordhuis]

From your description of the problem, it seems you've deduced it's a runtime error (and I concur, the stack trace makes it clear creating the connection fails.)

I don't see anything that's actionable as a bug report for us. Do you want me to convert it to a discussion?

[mohammadmoth]

From your description of the problem, it seems you've deduced it's a runtime error (and I concur, the stack trace makes it clear creating the connection fails.)

I don't see anything that's actionable as a bug report for us. Do you want me to convert it to a discussion?

• I think so .

Yes please

[mohammadmoth]

Does anyone have an idea of how to reproduce the same error?

[starwed]

I can't really help you reproduce it, but I did observe what looks like the same thing.

Running a node application on a windows Azure ASP, I'd see relatively frequent crashes in node 16.13.0. If I attached remotely to the process, the exit code was 0xc0000409 ( STATUS_STACK_BUFFER_OVERRUN, although that can apparently mean many things.) I didn't get to the point of pulling out more information about the crashes; running on Azure made this tricky.

I could reproduce in node 14, but not with node 12. I was never able to reproduce it on a locally running node instance with the same traffic. On Azure it crashed perhaps 1 out of a few hundred requests when under load; it seemed to matter how many outbound HTTP requests the app was making per inbound request, which kind of matches the cause indicated here. (That is, I'd disable the application from making certain http calls, and the crash rate seemed to decrease.)

We switched to the 64 bit version of node and things seem to have dramatically improved -- we have no particular need to run on the 32 bit version. Once I realized that, I was able to locate this issue/discussion.

Sorry I can't add much more detail, but hopefully this at least confirms others see the crash.

[LWERGIN]

@mohammadmoth and @starwed ,
We just experienced what appears to be the corrupted stack crash again but this time using 32-bit node v12.22.12 (final build of nodejs v12)!
We had not experienced the corrupted stack problem with 32-bit node v12.18.3.
So, in trying to deduce where the corrupted stack behavior is introduced, I'm left with 2 thoughts as to where the root cause was introduced:

1. The problem is in NodeJS v14 and later and perhaps was backported into the last build of v12.
2. The problem may arise from the use of the pkg NodeJS application packager we use. Both of our previous attempts to upgrade NodeJS to v14 and v16--and now this latest attempt to upgrade to v12.22.12--all required using a later version of pkg (5.6.0). Our 32-bit NodeJS v12.18.3 solution--which did not exeprience this problem--used pkg@^4.4.0.)

Question: Might either of your projects use the pkg packager for your NodeJS applications?

[mohammadmoth]

@LWERGIN @starwed
I think the first step to solving this problem is to write a code that enables us to reproduce the problem within minutes at most.

[mohammadmoth]

@LWERGIN
What do you mean by "crash while load testing"
Do you mean it crashes the server that responds,
Or Nodejs that simulates a high load?

[LWERGIN]

@LWERGIN What do you mean by "crash while load testing" Do you mean it crashes the server that responds, Or Nodejs that simulates a high load?

@mohammadmoth I believe you meant to address this question to @starwed.

[mohammadmoth]

@LWERGIN Yes . Thank you for attention . I meant @starwed

[spinard65]

Reviving this discussion, as we have seen this same crash in our application intermittently several times, using 32-bit Node v18.16.1 on Windows. I have been able to reproduce it using just a simple Javascript program that loops on net.connect. It is not easy to reproduce, as it often takes outrageous numbers of iterations before it will happen.
Test.zip
(Replace the IP addresses and port number in parameters.json with something in your environment that is listening)

I do have multiple crash dump files, only one of which is from just the simple JS program. Crash dumps are too big to upload, but I can find a way to share if someone wants it.

Here is what I have learned so far:

1. It always happens after a call to uv__tcp_try_connect. That is, the stack corruption (/GS) is detected at the return from uv__tcp_try_connect.
2. I have tried, but have been unsuccessful so far, to reproduce it with a local build of node, with libuv and node compiled with /Od and /RTCs. Hoping to "catch it in the act" with /RTCs.
3. Quite often, the (corrupted) return function is node::tracing::TraceEventHelper::GetCategoryGroupEnabled. The (corrupted) stack variables (local variables and function arguments) when viewed in the Visual Studio debugger, often contain string data like "node,node.async_hooks", "c:\\ws\\src\\env.cc:587", and "(task_queues_async_refs_) >= (0)".
4. The above clues seem to suggest that the corruption is somehow related to tracing, but could be a red herring.

I am willing to keep trying to discover more, but I would really like suggestions about how to make it more reproducible based on what I have so far.

Thanks.

[spinard65]

FYI -- Like everyone else who has encountered this problem, we have worked around it. We moved the functionality we were using net.connect for to a C++ addon and implemented it using Winsock. This avoids the node.js / libuv crash.

[LWERGIN]

Update: We've moved to 64-bit node and that appeared to address the problem.