Strings in luajit are infinitely bloated, even if they are not rereferenced in the lua vm

[fairyqb]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

kong all

Current Behavior

Minimal example code

resty example.lua

	local function get_memory_usage()  
	    local b = collectgarbage("count")  * 1024
	    return b  
	end  

	local my_table = {}  
	for i = 1, 10000000 do  
	    my_table[i] =  string.rep("A" , 256) .. i  

	end

	my_table = nil
	print("start gc:", "mem:",get_memory_usage())
	collectgarbage("collect") 
	print("end gc:", "mem:",get_memory_usage())
	print("sleep 10m")

	print("done")
	ngx.sleep(60*10*6)

Strings in luajit are infinitely bloated, even if they are not rereferenced in the lua vm. As a result, the memory resources of the nginx process RES are increasing to become larger and larger.

Steps To Reproduce

No response

Anything else?

No response

[chobits]

What i got is as following:

(kong-dev) xc kong $ resty example.lua
start gc:mem:3148645258
end gc:mem:67322434
sleep 10m
done

Are you concerned about the remaining 64MB of memory usage after end gc:mem:67322434?"

I suspect this is caused by two factors:

1. a form of memory fragmentation after LuaJIT's garbage collection.
2. cached string based on Luajit's string interning mechanism

So I think we might not have a good to optimize for your dedicated designed case, maybe you need to seek help from Luajit and openresty upstream. But I think it is still hard for them to optimize this special case.

---

BTW, if anyone has a good idea of this problem, feel free to left a comment

[fairyqb]

I am concerned about the memory RES (resident memory usage) value after running.

[bungle]

@fairyqb it is known that gc runs in steps. E.g. here is your original example slightly modified:

local function get_memory_usage()
  local b = collectgarbage("count")  * 1024
  return b
end


print("before gc:", "mem:",get_memory_usage())

local my_table = {}
for i = 1, 10000000 do
  my_table[i] =  string.rep("A" , 256) .. i

end

my_table = nil
print("start gc:", "mem:",get_memory_usage())
collectgarbage("collect")
collectgarbage("collect")
collectgarbage("collect")
collectgarbage("collect")
collectgarbage("collect")
collectgarbage("collect")
collectgarbage("collect")
print("end gc:", "mem:",get_memory_usage())
print("sleep 10m")

print("done")
ngx.sleep(60*10*6)

And here is its output:

before gc:mem:221877
start gc:mem:3148639829
end gc:mem:224333
sleep 10m
done

In general if you were to run such code, I would highly consider using something else than Lua strings in it. The RES is another complex topic (don't blindly look the number, it might be that OS doesn't bother on it).