Public chat and news/updates

[hamadmarri]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[hamadmarri]

Please test the recent changes.
I have updated the linux tree and cacule patches.

hamadmarri/linux@add54f0

https://github.com/hamadmarri/cacule-cpu-scheduler

I did an experiment where I run many tasks and count how many iterations when enqueue starting from head vs start from tail. Starting from head is ~0.77 where from tail is ~0.69 which is less iterations. So I made enqueue starts to find position from tail.
enqueue_counter.txt

Thank you.

[hamadmarri]

Please test the recent changes on RDB.
I have updated the linux tree and cacule patches.

hamadmarri/linux@9cd8a98

85d2950

Thank you.

[hamadmarri]

https://github.com/hamadmarri/cacule-cpu-scheduler/blob/master/cachy%20debug%20helper%20files/apply_suggested_configs.sh

Helper bash script to auto set suggested configs

[ptr1337]

Thanks for the config. Worked like a charm.
CONFIG_RCU_CPU_STALL_TIMEOUT 4 is doubled in the config.

Because the Number of CPU's, is there mean the CPU Cores or also Threads?
For example ive got 6 Core and with Hyperthread (SMT) 12 Threads.
Should i write in 12 or 6 ?

Im just getting slowly in since some days to work with building own kernels and so on. Sorry for the questions and Thanks for your great work!

[hamadmarri]

No worries my friend we are all on the same ship.

CONFIG_RCU_CPU_STALL_TIMEOUT this is just a timeout in seconds I don't think it is related to the number of cpus

if you mean CONFIG_NR_CPUS which is the number of cpus, set the total number of threads (in your case it's 12). Example, I have 2 cores and each core has two cpus/threads. I set 4.

[ptr1337]

We got now ARM Support to work - so far only for the raspberry pi4 aarch64 and armv7

For manjaro rpi4 you will find the aur/PKGBUILDS here:

• https://aur.archlinux.org/packages/linux-manjaro-rpi4-cacule-mainline/
• https://aur.archlinux.org/packages/linux-manjaro-rpi4-cacule-rdb-mainline/ (RDB)

For a archlinuxarm they are available for armv7 and aarch64. Read the instructions please!

• https://aur.archlinux.org/packages/linux-raspberrypi4-cacule-stable/
• https://aur.archlinux.org/packages/linux-raspberrypi4-cacule-lts/

Debian/Bullseye and Ubuntu Images are comming these days for the pi4 devices.

We need some tester who are using the desktop frequently if they notice any improvements and so on.
Would be nice for feedback.

[hamadmarri]

Could you please try these changes on CacULE (without RDB):

In this patch I am trying to enhance the interactivity so when the task is interactive based on the threshold ("sched_interactivity_threshold" <- you can adjust via sysctl) the cpu is selected such that the highest Interactivity Score IS is the max (max IS value means least interactive)

commit 87ef6d833095525154242f34d5e32a1999e61aac
Author: Hamad Marri <hamad.s.almarri@gmail.com>
Date:   Tue Apr 27 12:23:40 2021 +0300

    select_task_rq interactive aware

diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h
index 774de59e8111..f8db1089180b 100644
--- a/include/linux/sched/sysctl.h
+++ b/include/linux/sched/sysctl.h
@@ -33,6 +33,7 @@ extern unsigned int sysctl_sched_child_runs_first;
 
 #ifdef CONFIG_CACULE_SCHED
 extern int interactivity_factor;
+extern unsigned int interactivity_threshold;
 extern int cacule_max_lifetime;
 extern int cacule_harsh_mode;
 #endif
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ba07c1571b82..3282441b6732 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -130,6 +130,7 @@ int __weak arch_asym_cpu_priority(int cpu)
 int cacule_max_lifetime					= 30000; // in ms
 int cacule_harsh_mode					= 0;
 int interactivity_factor				= 32768;
+unsigned int interactivity_threshold			= 20480;
 #endif
 
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -630,6 +631,14 @@ calc_interactivity(u64 now, struct cacule_node *se)
 	return score_se;
 }
 
+static inline int is_interactive(struct cacule_node *cn)
+{
+	if (cn->vruntime == 0)
+		return 0;
+
+	return calc_interactivity(sched_clock(), cn) < interactivity_threshold;
+}
+
 static inline int
 entity_before_cached(u64 now, unsigned int score_curr, struct cacule_node *se)
 {
@@ -7085,7 +7094,55 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	/* SD_flags and WF_flags share the first nibble */
 	int sd_flag = wake_flags & 0xF;
 
-#if !defined(CONFIG_CACULE_SCHED)
+#ifdef CONFIG_CACULE_SCHED
+	struct cfs_rq *cfs_rq;
+	struct sched_entity *se = &p->se;
+	unsigned int max_IS = 0;
+	unsigned int IS, IS_c, IS_h;
+	struct sched_entity *curr_se;
+	struct cacule_node *cn, *head;
+	int cpu_i;
+
+	if (!is_interactive(&se->cacule_node))
+		goto cfs_way;
+
+	for_each_online_cpu(cpu_i) {
+		if (!cpumask_test_cpu(cpu_i, p->cpus_ptr))
+			continue;
+
+		cn = NULL;
+		cfs_rq = &cpu_rq(cpu_i)->cfs;
+
+		curr_se = cfs_rq->curr;
+		head = cfs_rq->head;
+
+		if (!curr_se && head)
+			cn = head;
+		else if (curr_se && !head)
+			cn = &curr_se->cacule_node;
+		else if (curr_se && head) {
+			IS_c = calc_interactivity(sched_clock(), &curr_se->cacule_node);
+			IS_h = calc_interactivity(sched_clock(), head);
+
+			IS = IS_c > IS_h? IS_c : IS_h;
+			goto compare;
+		}
+
+		if (!cn)
+			return cpu_i;
+
+		IS = calc_interactivity(sched_clock(), cn);
+
+compare:
+		if (IS > max_IS) {
+			max_IS = IS;
+			new_cpu = cpu_i;
+		}
+	}
+
+	return new_cpu;
+cfs_way:
+#else
 	if (wake_flags & WF_TTWU) {
 		record_wakee(p);
 
@@ -7098,7 +7155,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 
 		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
 	}
-#endif
+#endif /* CONFIG_CACULE_SCHED */
 
 	rcu_read_lock();
 	for_each_domain(cpu, tmp) {
@@ -7134,7 +7191,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	rcu_read_unlock();
 
 	return new_cpu;
-#endif
+#endif /* CONFIG_CACULE_RDB */
 }
 
 #if !defined(CONFIG_CACULE_RDB)
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index c6b24b552656..e70d4946080d 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1667,6 +1667,13 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec,
 	},
+	{
+		.procname	= "sched_interactivity_threshold",
+		.data		= &interactivity_threshold,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
 	{
 		.procname	= "sched_max_lifetime_ms",
 		.data		= &cacule_max_lifetime,

Please let me know if there is any differences in performance/responsiveness

Thank you
select_task_interactive_aware.patch.zip

[hamadmarri]

Please try twice on autogroup+fairgroup enabled and disabled. I noticed when I disabled autogroup and fair_group this patch shows high response. About 13ms compared with 16ms before

[hamadmarri]

https://forum.xanmod.org/thread-2-page-24.html

Some discussions on CacULE

[B3HOID]

Ok, now with 5.12 version of CacULE and these following sysctl settings
kernel.sched_interactivity_factor=7
kernel.sched_interactivity_threshold=10240
kernel.sched_autogroup=0
kernel.sched_rr_timeslice_ms=4
I am having the highest frames in glxgears test, bypassing 7500 which I did not expect.
Haven't done tests with actual games or compile speeds but it seems like now CacULE is working very well with my specific workload compared to the previous 5.10 and 5.11 versions that I did not have good luck with.
Thanks for the continued development on CacULE.
رمضان كريم btw

[hamadmarri]

Could you please test this patch on mainline kerne (not on CacULE). Please let me know if this patch makes cfs as responsive as cacule.

cfs_no_balance_wake.txt

[hamadmarri]

Added interactivity levels which are used to avoid fake interactive tasks. Fake interactive task are detected by the number of forks of their parent in a period of time. Introduced new sysctls: sched_nr_fork_threshold which is the max number of forks in period of time to consider all children fake interactive. This happnes for example when run make -j4 where make creates new threads for each job. All make children/threads are fake interactive, therefore, they are (any fake interactive tasks) put in lower interactivity regions based on how often this task is becoming fake interactive in a period of time. When a child exits or sleep, then the number of forks and the fake interactive level are reduced by 1. Also, when a maximum slid window is reached without any new fork, the fake interactive level is reduced by 1. The second sysctl is sched_fake_interactive_win_time_ms which is 1s by default. Each 1s (if a task is in runqueue, not sleeping) if the task is fake interactive already and the time slice ended without any new forks, then interactivty level is reduced by 1. For each fork, interactivity level is increased by 1. Higher levels here means lower interactive scores they got.

Please test
interactivity_levels.txt

[hamadmarri]

Sure @raykzhao I will do

Thank you

[raykzhao]

Hi @hamadmarri,

I got kernel warning in sched_tick_remote() for the check of delta when using NO_HZ_FULL. Based on how the delta is computed in fair.c, I tried the following patch and it fixed the problem on my computer:

diff -Naur a/kernel/sched/core.c b/kernel/sched/core.c
--- a/kernel/sched/core.c	2021-05-07 20:53:26.000000000 +1000
+++ b/kernel/sched/core.c	2021-05-09 05:56:57.237314488 +1000
@@ -4624,7 +4624,7 @@
 		 * Make sure the next tick runs within a reasonable
 		 * amount of time.
 		 */
-		delta = rq_clock_task(rq) - curr->se.exec_start;
+		delta = sched_clock() - curr->se.exec_start;
 		WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
 	}
 	curr->sched_class->task_tick(rq, curr, 0);

EDIT: For now I guess I will comment this check out, since it seems that this delta is not used anywhere else and solely for checking purpose. After changing to sched_clock() I still occasionally got the same kernel warning. The kernel warning I got is similar to the comment at torvalds/linux@d9c0ffc and I guess it might be false positive.

[raykzhao]

Hi @hamadmarri,

I enabled the full dynticks on all cores of a machine with hyperthreading except CPU0, and I have noticed that the count of Local timer interrupts (LOC) at CPU0 in /proc/interrupts quickly reaches some very large number (tens or hundreds of million) after booting the system for a while. I don't have this issue on my laptop without hyperthreading.

I suspect for some reason the full dynticks may not work well with SMT. Not sure if the same issue exists on CFS though.

EDIT: It seems that mixing ticking methods on a CPU core and its sibling might cause performance regression. Some discussions on LKML suggested that full dynticks configuration should be consistent between a CPU core and its sibling i.e. full dynticks must be enabled/disabled on a CPU and its sibling on the same time. In that case, I think we should disable full dynticks on the hyperthread from CPU0.

[hamadmarri]

Hi @hamadmarri,

I got kernel warning in sched_tick_remote() for the check of delta when using NO_HZ_FULL. Based on how the delta is computed in fair.c, I tried the following patch and it fixed the problem on my computer:

diff -Naur a/kernel/sched/core.c b/kernel/sched/core.c
--- a/kernel/sched/core.c	2021-05-07 20:53:26.000000000 +1000
+++ b/kernel/sched/core.c	2021-05-09 05:56:57.237314488 +1000
@@ -4624,7 +4624,7 @@
 		 * Make sure the next tick runs within a reasonable
 		 * amount of time.
 		 */
-		delta = rq_clock_task(rq) - curr->se.exec_start;
+		delta = sched_clock() - curr->se.exec_start;
 		WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
 	}
 	curr->sched_class->task_tick(rq, curr, 0);

EDIT: For now I guess I will comment this check out, since it seems that this delta is not used anywhere else and solely for checking purpose. After changing to sched_clock() I still occasionally got the same kernel warning. The kernel warning I got is similar to the comment at torvalds/linux@d9c0ffc and I guess it might be false positive.

Hi @raykzhao

Does this warning happen with CFS or with using rq_clock instead of sched_clock?
I see that it seems something related to waker/wakee CPUS relationship.

Thank you

[hamadmarri]

Hi @hamadmarri,

I enabled the full dynticks on all cores of a machine with hyperthreading except CPU0, and I have noticed that the count of Local timer interrupts (LOC) at CPU0 in /proc/interrupts quickly reaches some very large number (tens or hundreds of million) after booting the system for a while. I don't have this issue on my laptop without hyperthreading.

I suspect for some reason the full dynticks may not work well with SMT. Not sure if the same issue exists on CFS though.

EDIT: It seems that mixing ticking methods on a CPU core and its sibling might cause performance regression. Some discussions on LKML suggested that full dynticks configuration should be consistent between a CPU core and its sibling i.e. full dynticks must be enabled/disabled on a CPU and its sibling on the same time. In that case, I think we should disable full dynticks on the hyperthread from CPU0.

Hi @raykzhao

full dynticks must be enabled/disabled on a CPU and its sibling on the same time.

Could you please explain more about this point. It is not clear to me what are the sibling of a CPU? threads on the same core?

Thank you

[hamadmarri]

CacULE revision 2

1- Added 2000HZ as an option
2- Added Responsive wakup/new task to be placed in a CPU with least Interactive score (to balance out interactive tasks among cpus)
3- Added Interactivity levels mechanism
4- Increased the precision of lifetime normalize from x8 to x1024
5- Decoupled se.vruntime from cn.vruntime

[raykzhao]

https://jeremyeder.com/2013/11/15/nohz_fullgodmode/

I think we need to read more about isolcpus= too

Here is the description of isolcpus=:

Remove the specified CPUs, as defined by the cpu_number values, from the general kernel SMP balancing and scheduler algroithms. The only way to move a process onto or off an "isolated" CPU is via the CPU affinity syscalls. cpu_number begins at 0, so the maximum value is 1 less than the number of CPUs on the system.

This option is the preferred way to isolate CPUs. The alternative, manually setting the CPU mask of all tasks in the system, can cause problems and suboptimal load balancer performance.

From my understanding, isolcpus= will isolate a CPU core from userspace tasks and maybe even some IRQs or kthreads. I usually use isolcpus= for microbenchmarks of my program in order to avoid interference from other userspace tasks. What I do is put one CPU core after isolcpus= and pin my program to that core.

[terencode]

Hi, I was wondering what would motivate the use of a 2000Hz tickrate.

[hamadmarri]

Hi, I was wondering what would motivate the use of a 2000Hz tickrate.

Hi @terencode

High frequencies have pros and cons. Many believe that 1000HZ is quite enough. One important advantage of high Hz is faster interrupt action. Interrupts could be from user actions, network signals, or other things. Faster process of the interrupt could increase throughput. However, for tasks with longer bursts, high HZ will reduce the performance since CPU got interrupt a lot within a slice of time where as in low HZ such as 100HZ the CPU got 10ms without any interrupts which could save more cache and increase the throughput.

Having said that, I suggest using NO_HZ_FULL feature with proper kernel parameters settings for nohz_full=... with high HZ values >=500HZ. This will make 1 or 2 CPUs handle all the interrupts on behalf other adaptive ticks cpus. However, this feature works only when a CPU has 1 or 0 running tasks, which is normally happens a lot in desktop normal usages. So probably you would have a kind of balance with high HZ+NO_HZ_FULL.

Why 2000HZ instead of 1000HZ? I am able to see more smooth window animation on older tech such as Unity7 DE on Ubunut. But currently on KDE with Opensuse Tumbleweed, no obvious differences. I believe using 2000HZ with nohz_full can have advantages with some work loads, that's why I added it as an optional HZ.

Thank you

[terencode]

Thank you very much for your detailed answer!
I am using the nohz like so: nohz_full=1-5,7-11 (I have a ryzen 3600).
I wonder if 2000Hz makes an improvement for low latency audio (using jack2). Are there specific settings you recommend for this kind of workload?

[hamadmarri]

Hi @terencode

I recommend using RT kernel. You may need to try xanmod-rt:
https://github.com/xanmod/linux/releases/tag/5.10.35-rt39-xanmod1

with cacule for xanmod-rt:
https://github.com/hamadmarri/cacule-cpu-scheduler/blob/master/patches/CacULE/v5.10/xanmod-rt/cacule-5.10-rt.patch

Thank you

[hamadmarri]

https://github.com/hamadmarri/linux/tree/cacule-5.12-rdb

https://github.com/hamadmarri/cacule-cpu-scheduler/tree/master/patches/CacULE/RDB

RDB patch update:
Please disable
CONFIG_RCU_BOOST

You can use the patch with your previous config for comparing purposes.
Please compare with previous RDB, cacule-12 without RDB, CFS.

rdb-changes.txt
The changes in recent update

Thank you

[hamadmarri]

RDB update:
hamadmarri/linux@3eefd15

https://github.com/hamadmarri/cacule-cpu-scheduler/tree/master/patches/CacULE/RDB

RDB: rdb disables autogroup, which can be a problem because sometimes autogroup
provides better interactivity since it gathers tasks in groups and count their
vruntime as one entity. I have made something similar to autogroups for rdb where
update_curr updates all parents with delta_fair (func. update_parents()). Also
normalize_lifetime updates the diff of vruntime to all parents. The use is in
calc_interactivity where instead of taking cn.vruntime we take the average vruntime
from the task to its highest root parent (func. average_vruntime). Since every
parent/task has the total of vruntime of its children (due to update_curr updating parents)
the average is taking of the task would take on the account that if this task belongs
to a very busy group (based on fork hierarchy) or not. The task will have higher vruntime
average if it is a child of busy group (going thru the path to root parent).

[raykzhao]

Hi @hamadmarri,

Unfortunately this patch is very laggy on my laptop when compiling at background (both w/ and w/o IL). I will try to recompile the kernel with CONFIG_NO_HZ_FULL and/or CONFIG_PREEMPT disabled and see if there is any difference.

[hamadmarri]

Hi @raykzhao

I have updated RDB:
fix no need for trigger_nohz since it calls loadbalancer

hamadmarri/linux@e906f0f

62dede3

--
I believe you are using 100HZ
You may try change scale_down_hz = 10 to 100
or comment out the RDB stuff here in core.c

#if defined(CONFIG_CACULE_RDB) && !defined(CONFIG_NO_HZ_FULL)
	int scale_down_hz = 10;
	/* skip HZ/scale_down_hz ticks */
	int skip = HZ / scale_down_hz;
#endif

	arch_scale_freq_tick();
	sched_clock_tick();

#if defined(CONFIG_CACULE_RDB) && !defined(CONFIG_NO_HZ_FULL)
	rq->ticks++;
	if (rq->nr_running == 1 && rq->ticks < skip)
		/* skip */
		return;

	rq->ticks = 0;
#endif

[hamadmarri]

Hi @hamadmarri,

Unfortunately this patch is very laggy on my laptop when compiling at background (both w/ and w/o IL). I will try to recompile the kernel with CONFIG_NO_HZ_FULL and/or CONFIG_PREEMPT disabled and see if there is any difference.

Please try without IL. I am trying to run away from it.

Thank you

[raykzhao]

Hi @hamadmarri,

It's still very laggy even after disabling CONFIG_NO_HZ_FULL and commenting out skipping ticks. I also tried disabling full preemption i.e. preempt=voluntary but it doesn't seem helpful. Not sure if the performance would be better with higher CONFIG_HZ.

[hamadmarri]

Hi @raykzhao

I have issued a new testing patch for RDB
Please see this
#33

Thank you