Thanks all for visiting the articles. Long time HN reader, never expected to appear on HN! WOW!

Great article. Are those diagrams from Omnigraffle?

Obviously you are focusing on x86, but it might be worth mentioning where many of these ideas developed. For example, SGI Origin hardware (http://www.sgidepot.co.uk/origin/isca.pdf).

I was expecting more Dirk Pitt in this post

I was going to ask that - but assumed I'd get down voted into the err.. Pacific Vortex. I'll down vote myself for that one.

Nicely written article with good detail about the problems in the older architectures. One thing that I'm still a little fuzzy about is on the Linux kernel side of things in the context of multi core multi socket. Which CPU does the kernel actually run on and how does it dispatch the process/threads on the different cores/sockets?

Yes, Frank has a very nice series going there. More parts are expected to follow.

As to your question on what CPU the kernel normally runs on? AFAIK, it normally starts at core 0, processor 0. You can see for yourself where processes run.

For example using the ps command.

It will print kernel threads in square brackets and it can also list cpu and core number behind a /.

Eg, example output of a linux VM with 2 processors.

  $ ps -ax
   PID TTY      STAT   TIME COMMAND
     1 ?        Ss     0:03 /sbin/init splash
     2 ?        S      0:00 [kthreadd]
     3 ?        S      0:00 [ksoftirqd/0]
     5 ?        S<     0:00 [kworker/0:0H]
     7 ?        S      0:15 [rcu_sched]
     8 ?        S      0:00 [rcu_bh]
     9 ?        S      0:00 [migration/0]
    10 ?        S      0:00 [watchdog/0]
    11 ?        S      0:00 [watchdog/1]
    etcetera.. removed the rest for readability

The CPU scheduler then handles the rest to see on which thread/core to run a new process.

This post might also help:

http://superuser.com/questions/389161/what-do-the-mean-in-ps...

The simple answer is that the kernel runs on all cores. But the answer is really that it depends which part of the kernel you are talking about.

Whenever user mode does a trap to supervisor mode it is running the top half of the kernel. An interrupt (device or timer) causes the core to run the bottom half of the kernel. Many systems used to only send these to cpu0, and it is typical that lots of network I/O would occur on cpu0.

In the PIII era, with a wide bus shared by all cores, most kernel data structures were not multithreaded, i.e. single run queue, great big kernel locks. There are now many per-core data structures, which can be in local memory. See http://www.tldp.org/LDP/tlk/kernel/kernel.html and for a recent discussion of scheduling: http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf

Updated the series with ESX NUMA constructs: http://frankdenneman.nl/2016/08/22/numa-deep-dive-part-5-esx...

I'll have to devote some time to reading these. I'm running some 8-socket Oracle boxes and I've got NUMA disabled due to the sheer complexity of maintaining/understanding the configuration. Processor pinning, and so on.

I need to educate myself to understand the consequences of this decision, what the payoff would be to enabling NUMA, what my configs should look like, if there's an "easy way out", etc.

When you say 'NUMA disabled' do you mean NUMA aware scheduling and memory allocation is disabled? Because it is going to be NUMA no matter what you do.

Yes. Supposedly Oracle recommends such a configuration, but I haven't dug into it much beyond that.

What a fantastic series. The author really put a lot of work into these. Thank you Frank.

My pleasure! Im aiming to release some of the articles focussing on ESXi level scheduling this week. Stay tuned!

I wonder how NUMA compares to MAYA HEAP as far as memory performance goes?

What is MAYA HEAP?