Linux

Linux Master Application

A completely free application for testing your knowledge on Linux. Disclaimer: developed by repository owner

Linux

Linux Exercises

Basics

Name	Topic	Objective & Instructions	Solution
Navigation	cd, pwd	Exercise	Solution
Create and Destroy	touch, rm, mkdir	Exercise	Solution
Copy Time	touch, cp, ls	Exercise	Solution

Misc

Name	Topic	Objective & Instructions	Solution	Comments
Unique Count		Exercise	Solution

Linux Questions

Linux 101

What is Linux?

Wikipedia: "Linux is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged in a Linux distribution."

Red Hat: "Linux® is an open source operating system (OS). An operating system is the software that directly manages a system’s hardware and resources, like CPU, memory, and storage. The OS sits between applications and hardware and makes the connections between all of your software and the physical resources that do the work."

Explain what each of the following commands does and give an example on how to use it:

touch
ls
rm
cat
cp
mkdir
pwd
cd

touch - update file's timestamp. More commonly used for creating files

ls - listing files and directories

rm - remove files and directories

cat - create, view and concatenate files

cp - copy files and directories

mkdir - create directories

pwd - print current working directory (= at what path the user currently located)

cd - change directory

What each of the following commands does?

cd /
cd ~
cd
cd ..
cd .
cd -

cd / -> change to the root directory

cd ~ -> change to your home directory

cd -> change to your home directory

cd .. -> change to the directory above your current i.e parent directory

cd . -> change to the directory you currently in

cd - -> change to the last visited path

Some of the commands in the previous question can be run with the -r/-R flag. What does it do? Give an example to when you would use it

The -r (or -R in some commands) flag allows the user to run a certain command recursively. For example, listing all the files under the following tree is possible when done recursively (ls -R):

/dir1/ dir2/ file1 file2 dir3/ file3

To list all the files, one can run ls -R /dir1

Explain each field in the output of `ls -l` command

It shows a detailed list of files in a long format. From the left:

file permissions, number of links, owner name, owner group, file size, timestamp of last modification and directory/file name

What are hidden files/directories? How to list them?

These are files directly not displayed after performing a standard ls direct listing. An example of these files are .bashrc which are used to execute some scripts. Some also store configuration about services on your host like .KUBECONFIG. The command used to list them is, ls -a

What do > and < do in terms of input and output for programs?

They take in input (<) and output for a given file (>) using stdin and stdout.

myProgram < input.txt > executionOutput.txt

Explain what each of the following commands does and give an example on how to use it:

sed
grep
cut
awk

sed: a stream editor. Can be used for various purposes like replacing a word in a file: sed -i s/salad/burger/g

grep: a search tool. Used to search, count or match a text in a file:

searching for any line that contains a word in a file: grep 'word' file.md

or displaying the total number of times a string appears in a file: grep -c 'This is a string' file.md

cut: a tool for cutting out selected portions of each line of a file:

syntax: cut OPTION [FILE]

cutting first two bytes from a word in a file: cut -b 1-2 file.md, output: wo

awk: a programming language that is mainly used for text processing and data extraction. It can be used to manipulate and modify text in a file:
- syntax: awk [OPTIONS] [FILTER] [FILE] extracting a specific field from a CSV file: awk -F ',' '{print $1}' file.csv, output: first field of each line in the file

How to rename the name of a file or a directory?

Using the mv command.

Specify which command would you use (and how) for each of the following scenarios

Remove a directory with files
Display the content of a file
Provides access to the file /tmp/x for everyone
Change working directory to user home directory
Replace every occurrence of the word "good" with "great" in the file /tmp/y

rm -rf dir

cat or less

chmod 777 /tmp/x

cd ~

sed -i s/good/great/g /tmp/y

How can you check what is the path of a certain command?

whereis

which

What is the difference between these two commands? Will it result in the same output?

echo hello world
echo "hello world"

The echo command receives two separate arguments in the first execution and in the second execution it gets one argument which is the string "hello world". The output will be the same.

Explain piping. How do you perform piping?

Using a pipe in Linux, allows you to send the output of one command to the input of another command. For example: cat /etc/services | wc -l

Fix the following commands:

sed "s/1/2/g' /tmp/myFile
find . -iname *.yaml -exec sed -i "s/1/2/g" {} ;

sed 's/1/2/g' /tmp/myFile # sed "s/1/2/g" is also fine find . -iname "*.yaml" -exec sed -i "s/1/2/g" {} \;

How to check which commands you executed in the past?

history command or .bash_history file

also can use up arrow key to access or to show the recent commands you type

Running the command df you get "command not found". What could be wrong and how to fix it?

Most likely the default/generated $PATH was somehow modified or overridden thus not containing /bin/ where df would normally go. This issue could also happen if bash_profile or any configuration file of your interpreter was wrongly modified, causing erratics behaviours. You would solve this by fixing your $PATH variable:

As to fix it there are several options:

Manually adding what you need to your $PATH PATH="$PATH":/user/bin:/..etc

You have your weird env variables backed up.

You would look for your distro default $PATH variable, copy paste using method #1

Note: There are many ways of getting errors like this: if bash_profile or any configuration file of your interpreter was wrongly modified; causing erratics behaviours, permissions issues, bad compiled software (if you compiled it by yourself)... there is no answer that will be true 100% of the time.

How do you schedule tasks periodically?

You can use the commands cron and at. With cron, tasks are scheduled using the following format:

*/30 * * * * bash myscript.sh Executes the script every 30 minutes.

The tasks are stored in a cron file, you can write in it using crontab -e

Alternatively if you are using a distro with systemd it's recommended to use systemd timers.

I/O Redirection

Explain Linux I/O redirection

In Linux, IO redirection is a way of changing the default input/output behavior of a command or program. It allows you to redirect input and output from/to different sources/destinations, such as files, devices, and other commands.
Here are some common examples of IO redirection:

Redirecting Standard Output (stdout): ls > filelist.txt

Redirecting Standard Error (stderr): ls /some/nonexistent/directory 2> error.txt

Appending to a file: echo "hello" >> myfile.txt

Redirecting Input (stdin): sort < unsorted.txt

Using Pipes: Pipes ("|"): ls | grep ".txt$"

Demonstrate Linux output redirection

ls > ls_output.txt

Demonstrate Linux stderr output redirection

yippiekaiyay 2> ls_output.txt

Demonstrate Linux stderr to stdout redirection

yippiekaiyay &> file

What is the result of running the following command? yippiekaiyay 1>&2 die_hard

An output similar to: yippikaiyay: command not found...
The file die_hard will not be created

Filesystem Hierarchy Standard

In Linux FHS (Filesystem Hierarchy Standard) what is the /?

The root of the filesystem. The beginning of the tree.

What is stored in each of the following paths?

/bin, /sbin, /usr/bin and /usr/sbin
/etc
/home
/var
/tmp

binaries

configuration files

home directories of the different users

files that tend to change and be modified like logs

temporary files

What is special about the /tmp directory when compared to other directories?

/tmp folder is cleaned automatically, usually upon reboot.

What kind of information one can find in /proc?

It contains useful information about the processes that are currently running, it is regarded as control and information center for kernel.

What makes /proc different from other filesystems?

/proc is a special virtual filesystem in Unix-like operating systems, including Linux, that provides information about processes and system resources.

True or False? only root can create files in /proc

False. No one can create file in /proc directly (certain operations can lead to files being created in /proc by the kernel).

What can be found in /proc/cmdline?

The command passed to the boot loader to run the kernel

In which path can you find the system devices (e.g. block storage)?

/dev

Permissions

How to change the permissions of a file?

Using the chmod command.

What does the following permissions mean?:

777
644
750

777 - You give the owner, group and other: Execute (1), Write (2) and Read (4); 4+2+1 = 7. 644 - Owner has Read (4), Write (2), 4+2 = 6; Group and Other have Read (4). 750 - Owner has x+r+w, Group has Read (4) and Execute (1); 4+1 = 5. Other have no permissions.

What this command does? chmod +x some_file

It adds execute permissions to all sets i.e user, group and others

Explain what is setgid and setuid

setuid is a linux file permission that permits a user to run a file or program with the permissions of the owner of that file. This is possible by elevation of current user privileges.

setgid is a process when executed will run as the group that owns the file.

What is the purpose of sticky bit?

Its a bit that only allows the owner or the root user to delete or modify the file.

What the following commands do?

chmod
chown
chgrp

chmod - changes access permissions to files system objects

chown - changes the owner of file system files and directories

chgrp - changes the group associated with a file system object

What is sudo? How do you set it up?

sudo is a command-line utility in Unix-like operating systems that allows users to run programs with the privileges of another user, usually the superuser (root). It stands for "superuser do.
The sudo program is installed by default in almost all Linux distributions. If you need to install sudo in Debian/Ubuntu, use the command apt-get install sudo

True or False? In order to install packages on the system one must be the root user or use the sudo command

True

Explain what are ACLs. For what use cases would you recommend to use them?

You try to create a file but it fails. Name at least three different reason as to why it could happen

No more disk space

No more inodes

No permissions

A user accidentally executed the following chmod -x $(which chmod). How to fix it?

Using sudo setfacl -m u::rx /usr/bin/chmod will set the execute permissions on chmod for all the users. Post this, the chmod binary can be used as usual.

Scenarios

You would like to copy a file to a remote Linux host. How would you do?

There are multiple ways to transfer files between hosts. Personal opinion: use rsync

How to generate a random string?

One way is to run the following: cat /proc/sys/kernel/random/uuid

How to generate a random string of 7 characters?

mkpasswd -l 7

Systemd

What is systemd?

Systemd is a daemon (System 'd', d stands for daemon).
A daemon is a program that runs in the background without direct control of the user, although the user can at any time talk to the daemon.

systemd has many features such as user processes control/tracking, snapshot support, inhibitor locks..

If we visualize the unix/linux system in layers, systemd would fall directly after the linux kernel.
Hardware -> Kernel -> Daemons, System Libraries, Server Display.

How to start or stop a service?

To start a service: systemctl start <service name> To stop a service: systemctl stop <service name>

How to check the status of a service?

systemctl status <service name>

On a system which uses systemd, how would you display the logs?

journalctl

Describe how to make a certain process/app a service

Troubleshooting and Debugging

Where system logs are located?

/var/log

How to follow file's content as it being appended without opening the file every time?

tail -f <file_name>

What are you using for troubleshooting and debugging network issues?

dstat -t is great for identifying network and disk issues. netstat -tnlaup can be used to see which processes are running on which ports. lsof -i -P can be used for the same purpose as netstat. ngrep -d any metafilter for matching regex against payloads of packets. tcpdump for capturing packets wireshark same concept as tcpdump but with GUI (optional).

What are you using for troubleshooting and debugging disk & file system issues?

dstat -t is great for identifying network and disk issues. opensnoop can be used to see which files are being opened on the system (in real time).

What are you using for troubleshooting and debugging process issues?

strace is great for understanding what your program does. It prints every system call your program executed.

What are you using for debugging CPU related issues?

top will show you how much CPU percentage each process consumes perf is a great choice for sampling profiler and in general, figuring out what your CPU cycles are "wasted" on flamegraphs is great for CPU consumption visualization (http://www.brendangregg.com/flamegraphs.html)

You get a call from someone claiming "my system is SLOW". What do you do?

Check with top for anything unusual

Run dstat -t to check if it's related to disk or network.

Check if it's network related with sar

Check I/O stats with iostat

Explain iostat output

How to debug binaries?

What is the difference between CPU load and utilization?

How you measure time execution of a program?

Scenarios

You have a process writing to a file. You don't know which process exactly, you just know the path of the file. You would like to kill the process as it's no longer needed. How would you achieve it?

Run lsof <FILE_PATH>

Use the pid (process ID) from the lsof command and run kill <PID>

Kernel

What is a kernel, and what does it do?

The kernel is part of the operating system and is responsible for tasks like:

Allocating memory

Schedule processes

Control CPU

How do you find out which Kernel version your system is using?

uname -a command

What is a Linux kernel module and how do you load a new module?

A Linux kernel module is a piece of code that can be dynamically loaded into the kernel to extend its functionality. These modules are typically used to add support for hardware devices, filesystems, or system calls. The kernel itself is monolithic, but with modules, its capabilities can be extended without having to reboot the system or recompile the entire kernel.

Explain user space vs. kernel space

The operating system executes the kernel in protected memory to prevent anyone from changing (and risking it crashing). This is what is known as "Kernel space". "User space" is where users executes their commands or applications. It's important to create this separation since we can't rely on user applications to not tamper with the kernel, causing it to crash.

Applications can access system resources and indirectly the kernel space by making what is called "system calls".

In what phases of kernel lifecycle, can you change its configuration?

Build time (when it's compiled)

Boot time (when it starts)

Runtime (once it's already running)

Where can you find kernel's configuration?

Usually it will reside in /boot/config-<kernel version>.<os release>.<arch>

Where can you find the file that contains the command passed to the boot loader to run the kernel?

/proc/cmdline

How to list kernel's runtime parameters?

sysctl -a

Will running sysctl -a as a regular user vs. root, produce different result?

Yes, you might notice that in most systems, when running systctl -a with root, you'll get more runtime parameters compared to executing the same command with a regular user.

You would like to enable IPv4 forwarding in the kernel, how would you do it?

sudo sysctl net.ipv4.ip_forward=1

To make it persistent (applied after reboot for example): insert net.ipv4.ip_forward = 1 into /etc/sysctl.conf

Another way to is to run echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

How sysctl applies the changes to kernel's runtime parameters the moment you run sysctl command?

If you strace the sysctl command you can see it does it by changing the file under /proc/sys/...

In the past it was done with sysctl system call, but it was deprecated at some point.

How changes to kernel runtime parameters persist? (applied even after reboot to the system for example)

There is a service called systemd-sysctl that takes the content of /etc/sysctl.conf and applies it. This is how changes persist, even after reboot, when they are written in /etc/sysctl.conf

Are the changes you make to kernel parameters in a container, affects also the kernel parameters of the host on which the container runs?

No. Containers have their own /proc filesystem so any change to kernel parameters inside a container, are not affecting the host or other containers running on that host.

SSH

What is SSH? How to check if a Linux server is running SSH?

Wikipedia Definition: "SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network."

Hostinger.com Definition: "SSH, or Secure Shell, is a remote administration protocol that allows users to control and modify their remote servers over the Internet."

An SSH server will have SSH daemon running. Depends on the distribution, you should be able to check whether the service is running (e.g. systemctl status sshd).

Why SSH is considered better than telnet?

Telnet also allows you to connect to a remote host but as opposed to SSH where the communication is encrypted, in telnet, the data is sent in clear text, so it doesn't considered to be secured because anyone on the network can see what exactly is sent, including passwords.

What is stored in ~/.ssh/known_hosts?

The file stores the key fingerprints for the clients connecting to the SSH server. This fingerprint creates a trust between the client and the server for future SSH connections.

You try to ssh to a server and you get "Host key verification failed". What does it mean?

It means that the key of the remote host was changed and doesn't match the one that stored on the machine (in ~/.ssh/known_hosts).

What is the difference between SSH and SSL?

What ssh-keygen is used for?

ssh-keygen is a tool to generate an authentication key pair for SSH, that consists of a private and a public key. It supports a number of algorithms to generate authentication keys :

dsa

ecdsa

ecdsa-sk

ed25519

ed25519-sk

rsa (default)

One can also specify number of bits in key. Command below generates an SSH key pair with RSA 4096-bits :

$ ssh-keygen -t rsa -b 4096

The output looks like this:

Generating public/private rsa key pair. Enter file in which to save the key (/home/user/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/user/.ssh/id_rsa Your public key has been saved in /home/user/.ssh/id_rsa.pub The key fingerprint is: SHA256:f5MOGnhzYfC0ZCHvbSXXiRiNVYETjxpHcXD5xSojx+M user@mac-book-pro The key's randomart image is: +---[RSA 4096]----+ | . ..+***o| | o o++*o+| | . =+.++++| | B.oX+. .| | S *=o+ | | . o oE. | | . + + + | | . = + . | | . . | +----[SHA256]-----+

One can check how many bits an SSH key has with :

$ ssh-keygen -l -f /home/user/.ssh/id_rsa

Output should look like this :

4096 SHA256:f5MOGnhzYfC0ZCHvbSXXiRiNVYETjxpHcXD5xSojx+M user@mac-book-pro (RSA)

It shows the key is RSA 4096-bits.

-l and -f parameters usage explanation :

-l Show the fingerprint of the key file. -f filename Filename of the key file.

Learn more : How can I tell how many bits my ssh key is? - Superuser

What is SSH port forwarding?

Globbing & Wildcards

What is Globbing?

What are wildcards? Can you give an example of how to use them?

Explain what will ls [XYZ] match

Explain what will ls [^XYZ] match

Explain what will ls [0-5] match

What each of the following matches

?
*

The ? matches any single character

The * matches zero or more characters

What do we grep for in each of the following commands?:

grep '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' some_file
grep -E "error|failure" some_file
grep '[0-9]$' some_file

An IP address

The word "error" or "failure"

Lines which end with a number

Which line numbers will be printed when running `grep '\baaa\b'` on the following content:

aaa bbb ccc.aaa aaaaaa

lines 1 and 3.

What is the difference single and double quotes?

What is escaping? What escape character is used for escaping?

What is an exit code? What exit codes are you familiar with?

An exit code (or return code) represents the code returned by a child process to its parent process.

0 is an exit code which represents success while anything higher than 1 represents error. Each number has different meaning, based on how the application was developed.

I consider this as a good blog post to read more about it: https://shapeshed.com/unix-exit-codes

Boot Process

Tell me everything you know about the Linux boot process

Another way to ask this: what happens from the moment you turned on the server until you get a prompt

What is GRUB2?

What is Secure Boot?

What can you find in /boot?

Disk and Filesystem

What's an inode?

For each file (and directory) in Linux there is an inode, a data structure which stores meta data related to the file like its size, owner, permissions, etc.

Which of the following is not included in inode:

Link count
File size
File name
File timestamp

File name (it's part of the directory file)

How to check which disks are currently mounted?

Run mount

You run the mount command but you get no output. How would you check what mounts you have on your system?

cat /proc/mounts

What is the difference between a soft link and hard link?

Hard link is the same file, using the same inode. Soft link is a shortcut to another file, using a different inode.

True or False? You can create an hard link for a directory

False

True or False? You can create a soft link between different filesystems

True

True or False? Directories always have by minimum 2 links

True.

What happens when you delete the original file in case of soft link and hard link?

Can you check what type of filesystem is used in /home?

There are many answers for this question. One way is running df -T

What is a swap partition? What is it used for?

How to create a

new empty file
a file with text (without using text editor)
a file with given size

touch new_file.txt

cat > new_file [enter] submit text; ctrl + d to exit insert mode

truncate -s new_file.txt

You are trying to create a new file but you get "File system is full". You check with df for free space and you see you used only 20% of the space. What could be the problem?

How would you check what is the size of a certain directory?

du -sh

What is LVM?

Explain the following in regards to LVM:

PV
VG
LV

What is NFS? What is it used for?

What RAID is used for? Can you explain the differences between RAID 0, 1, 5 and 10?

Describe the process of extending a filesystem disk space

What is lazy umount?

What is tmpfs?

What is stored in each of the following logs?

/var/log/messages
/var/log/boot.log

True or False? both /tmp and /var/tmp cleared upon system boot

False. /tmp is cleared upon system boot while /var/tmp is cleared every a couple of days or not cleared at all (depends on distro).

Performance Analysis

How to check what is the current load average?

One can use uptime or top

You know how to see the load average, great. but what each part of it means? for example 1.43, 2.34, 2.78

This article summarizes the load average topic in a great way

How to check process usage?

pidstat

How to check disk I/O?

iostat -xz 1

How to check how much free memory a system has? How to check memory consumption by each process?

You can use the commands top and free

How to check TCP stats?

sar -n TCP,ETCP 1

Processes

how to list all the processes running in your system?

The "ps" command can be used to list all the processes running in a system. The "ps aux" command provides a detailed list of all the processes, including the ones running in the background.

How to run a process in the background and why to do that in the first place?

You can achieve that by specifying & at the end of the command. As to why, since some commands/processes can take a lot of time to finish execution or run forever, you may want to run them in the background instead of waiting for them to finish before gaining control again in current session.

How can you find how much memory a specific process consumes?

mem() { ps -eo rss,pid,euser,args:100 --sort %mem | grep -v grep | grep -i $@ | awk '{printf $1/1024 "MB"; $1=""; print }' } [Source](https://stackoverflow.com/questions/3853655/in-linux-how-to-tell-how-much-memory-processes-are-using)

What signal is used by default when you run 'kill *process id*'?

The default signal is SIGTERM (15). This signal kills process gracefully which means it allows it to save current state configuration.

What signals are you familiar with?

SIGTERM - default signal for terminating a process SIGHUP - common usage is for reloading configuration SIGKILL - a signal which cannot caught or ignored

To view all available signals run kill -l

What kill 0 does?

"kill 0" sends a signal to all processes in the current process group. It is used to check if the processes exist or not

What kill -0 does?

"kill -0" checks if a process with a given process ID exists or not. It does not actually send any signal to the process.

What is a trap?

A trap is a mechanism that allows the shell to intercept signals sent to a process and perform a specific action, such as handling errors or cleaning up resources before terminating the process.

Every couple of days, a certain process stops running. How can you look into why it's happening?

One way to investigate why a process stops running is to check the system logs, such as the messages in /var/log/messages or journalctl. Additionally, checking the process's resource usage and system load may provide clues as to what caused the process to stop

What happens when you press ctrl + c?

When you press "Ctrl+C," it sends the SIGINT signal to the foreground process, asking it to terminate gracefully.

What is a Daemon in Linux?

A background process. Most of these processes are waiting for requests or set of conditions to be met before actually running anything. Some examples: sshd, crond, rpcbind.

What are the possible states of a process in Linux?

Running (R) Uninterruptible Sleep (D) - The process is waiting for I/O Interruptible Sleep (S) Stopped (T) Dead (x) Zombie (z)

How do you kill a process in D state?

A process in D state (also known as "uninterruptible sleep") cannot be killed using the "kill" command. The only way to terminate it is to reboot the system.

What is a zombie process?

A process which has finished to run but has not exited.

One reason it happens is when a parent process is programmed incorrectly. Every parent process should execute wait() to get the exit code from the child process which finished to run. But when the parent isn't checking for the child exit code, the child process can still exists although it finished to run.

How to get rid of zombie processes?

You can't kill a zombie process the regular way with kill -9 for example as it's already dead.

One way to kill zombie process is by sending SIGCHLD to the parent process telling it to terminate its child processes. This might not work if the parent process wasn't programmed properly. The invocation is kill -s SIGCHLD [parent_pid]

You can also try closing/terminating the parent process. This will make the zombie process a child of init (1) which does periodic cleanups and will at some point clean up the zombie process.

How to find all the

Processes executed/owned by a certain user
Process which are Java processes
Zombie Processes

If you mention at any point ps command with arguments, be familiar with what these arguments does exactly.

What is the init process?

It is the first process executed by the kernel during the booting of a system. It is a daemon process which runs till the system is shutdown. That is why, it is the parent of all the processes

Can you describe how processes are being created?

How to change the priority of a process? Why would you want to do that?

To change the priority of a process, you can use the nice command in Linux. The nice command allows you to specify the priority of a process by assigning a priority value ranging from -20 to 19. A higher value of priority means lower priority for the process, and vice versa.

You may want to change the priority of a process to adjust the amount of CPU time it is allocated by the system scheduler. For example, if you have a CPU-intensive process running on your system that is slowing down other processes, you can lower its priority to give more CPU time to other processes.

Can you explain how network process/connection is established and how it's terminated?>

When a client process on one system wants to establish a connection with a server process on another system, it first creates a socket using the socket system call. The client then calls the connect system call, passing the address of the server as an argument. This causes a three-way handshake to occur between the client and server, where the two systems exchange information to establish a connection.

Once the connection is established, the client and server can exchange data using the read and write system calls. When the connection is no longer needed, the client or server can terminate the connection by calling the close system call on the socket.

What strace does? What about ltrace?

Strace is a debugging tool that is used to monitor the system calls made by a process. It allows you to trace the execution of a process and see the system calls it makes, as well as the signals it receives. This can be useful for diagnosing issues with a process, such as identifying why it is hanging or crashing.
Ltrace, on the other hand, is a similar tool that is used to trace the library calls made by a process. It allows you to see the function calls made by a process to shared libraries, as well as the arguments passed to those functions. This can be useful for diagnosing issues with a process that involve library calls, such as identifying why a particular library is causing a problem.

Find all the files which end with '.yml' and replace the number 1 in 2 in each file

find /some_dir -iname *.yml -print0 | xargs -0 -r sed -i "s/1/2/g"

You run ls and you get "/lib/ld-linux-armhf.so.3 no such file or directory". What is the problem?

The ls executable is built for an incompatible architecture.

How would you split a 50 lines file into 2 files of 25 lines each?

You can use the split command this way: split -l 25 some_file

What is a file descriptor? What file descriptors are you familiar with?

Kerberos File descriptor, also known as file handler, is a unique number which identifies an open file in the operating system.
In Linux (and Unix) the first three file descriptors are:

0 - the default data stream for input

1 - the default data stream for output

2 - the default data stream for output related to errors

This is a great article on the topic: https://www.computerhope.com/jargon/f/file-descriptor.htm

What is NTP? What is it used for?

Explain Kernel OOM

Security

What is chroot? In what scenarios would you consider using it?

What is SELiunx?

What is Kerberos?

What is nftables?

What firewalld daemon is responsible for?

Do you have experience with hardening servers? Can you describe the process?

How do you create a private key for a CA (certificate authority)?

One way is using openssl this way:

openssl genrsa -aes256 -out ca-private-key.pem 4096

How do you create a public key for a CA (certificate authority)?

openssl req -new -x509 -days 730 -key [private key file name] -sha256 -out ca.pem

If using the private key from the previous question then the command would be:

openssl req -new -x509 -days 730 -key ca-private-key.pem -sha256 -out ca.pem

Demonstrate one way to encode and decode data in Linux

Encode: echo -n "some password" | base64 Decode: echo -n "allE19remO91" | base64

Networking

How to list all the interfaces?

ip link show

What is the loopback (lo) interface?

The loopback interface is a special, virtual network interface that your computer uses to communicate with itself. It is used mainly for diagnostics and troubleshooting, and to connect to servers running on the local machine.

What the following commands are used for?

ip addr
ip route
ip link
ping
netstat
traceroute

What is a network namespace? What is it used for?

How to check if a certain port is being used?

One of the following would work:

netstat -tnlp | grep <port_number> lsof -i -n -P | grep <port_number>

How can you turn your Linux server into a router?

What is a virtual IP? In what situation would you use it?

True or False? The MAC address of an interface is assigned/set by the OS

False

Can you have more than one default gateway in a given system?

Technically, yes.

What is telnet and why is it a bad idea to use it in production? (or at all)

Telnet is a type of client-server protocol that can be used to open a command line on a remote computer, typically a server. By default, all the data sent and received via telnet is transmitted in clear/plain text, therefore it should not be used as it does not encrypt any data between the client and the server.

What is the routing table? How do you view it?

How can you send an HTTP request from your shell?

Using nc is one way

What are packet sniffers? Have you used one in the past? If yes, which packet sniffers have you used and for what purpose?

It is a network utility that analyses and may inject tasks into the data-stream travelling over the targeted network.

How to list active connections?

How to trigger neighbor discovery in IPv6?

One way would be ping6 ff02::1

What is network interface bonding and do you know how it's performed in Linux?

What network bonding modes are there?

There a couple of modes:

balance-rr: round robing bonding

active-backup: a fault tolerance mode where only one is active

balance-tlb: Adaptive transmit load balancing

balance-alb: Adaptive load balancing

What is a bridge? How it's added in Linux OS?

DNS

How to check what is the hostname of the system?

cat /etc/hostname

You can also run hostnamectl or hostname but that might print only a temporary hostname. The one in the file is the permanent one.

What the file /etc/resolv.conf is used for? What does it include?

What commands are you using for performing DNS queries (or troubleshoot DNS related issues)?

You can specify one or more of the following:

dig

host

nslookup

You run dig codingshell.com and get the following result:

ANSWER SECTION:
codingshell.com.	3515	IN	A	185.199.109.153

What is the meaning of the number 3515?

This is the TTL. When you lookup for an address using a domain/host name, your OS is performing DNS resolution by contacting DNS name servers to get the IP address of the host/domain you are looking for.
When you get a reply, this reply in cached in your OS for a certain period of time. This is period of time is also known as TTL and this is the meaning of 3515 number - it will be cached for 3515 seconds before removed from the cache and during that period of time, you'll get the value from the cache instead of asking DNS name servers for the address again.

How can we modify the network connection via `nmcli` command, to use `8.8.8.8` as a DNS server?

Find the connection name:

# nmcli con show NAME UUID TYPE DEVICE System ens5 8126c120-a964-e959-ff98-ac4973344505 ethernet ens5 System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet --

Here the connection name is "System ens5". Let's say we want to modify settings for this connection.

Modify the connection to use 8.8.8.8 as DNS server:

# nmcli con mod "System ens5" ipv4.dns "8.8.8.8"

We need to reactivate the connection for the change to take effect:

nmcli con up "System ens5"

Verify our settings once more:

cat /etc/resolv.conf nmcli -f ipv4.dns con show "System ens5"

Packaging

Do you have experience with packaging? (as in building packages) Can you explain how does it works?

How packages installation/removal is performed on the distribution you are using?

The answer depends on the distribution being used.

In Fedora/CentOS/RHEL/Rocky it can be done with rpm or dnf commands. In Ubuntu it can be done with the apt command.

RPM: explain the spec format (what it should and can include)

How do you list the content of a package without actually installing it?

How to know to which package a file on the system belongs to? Is it a problem if it doesn't belongs to any package?

Where repositories are stored? (based on the distribution you are using)

What is an archive? How do you create one in Linux?

How to extract the content of an archive?

Why do we need package managers? Why not simply creating archives and publish them?

Package managers allow you to manage packages lifecycle as in installing, removing and updating the packages.
In addition, you can specify in a spec how a certain package will be installed - where to copy the files, which commands to run prior to the installation, post the installation, etc.

DNF

What is DNF?

From the repo:

"Dandified YUM (DNF) is the next upcoming major version of YUM. It does package management using RPM, libsolv and hawkey libraries."

Official docs

How to look for a package that provides the command /usr/bin/git? (the package isn't necessarily installed)

dnf provides /usr/bin/git

Applications and Services

What can you find in /etc/services?

How to make sure a Service starts automatically after a reboot or crash?

Depends on the init system.

Systemd: systemctl enable [service_name] System V: update-rc.d [service_name] and add this line id:5678:respawn:/bin/sh /path/to/app to /etc/inittab Upstart: add Upstart init script at /etc/init/service.conf

You run ssh 127.0.0.1 but it fails with "connection refused". What could be the problem?

SSH server is not installed

SSH server is not running

How to print the shared libraries required by a certain program? What is it useful for?

What is CUPS?

What types of web servers are you familiar with?

Nginx, Apache httpd.

Users and Groups

What is a "superuser" (or root user)? How is it different from regular users?

How do you create users? Where user information is stored?

Command to create users is useradd

Syntax: useradd [options] Username

There are 2 configuration files, which stores users information

/etc/passwd - Users information like, username, shell etc is stored in this file
/etc/shadow - Users password is stored in encrypted format

Which file stores information about groups?

/etc/groups file stores the group name, group ID, usernames which are in secondary group.

How do you change/set the password of a user?

passwd <username> is the command to set/change password of a user.

Which file stores users passwords? Is it visible for everyone?

/etc/shadow file holds the passwords of the users in encrypted format. NO, it is only visible to the root user

Do you know how to create a new user without using adduser/useradd command?

YES, we can create new user by manually adding an entry in the /etc/passwd file.

For example, if we need to create a user called john.

Step 1: Add an entry to /etc/passwd file, so user gets created.

echo "john:x:2001:2001::/home/john:/bin/bash" >> /etc/passwd

Step 2: Add an entry to /etc/group file, because every user belong to the primary group that has same name as the username.

echo "john:x:2001:" >> /etc/group

Step 3: Verify if the user got created

id john

What information is stored in /etc/passwd? explain each field

/etc/passwd is a configuration file, which contains users information. Each entry in this file has, 7 fields,

username:password:UID:GID:Comment:home directory:shell

username - The name of the user.

password - This field is actually a placeholder of the password field. Due to security concerns, this field does not contain the password, just a placeholder (x) to the encrypted password stored in /etc/shadow file.

UID - User ID of the user.

GID - Group ID

Comment - This field is to provide description about the user.

home directory - Abousulte path of the user's home directory. This directory gets created once the user is added.

shell - This field contains the absolute path of the shell that will be used by the respective user.

How to add a new user to the system without providing him the ability to log-in into the system?

adduser user_name --shell=/bin/false --no-create-home You can also add a user and then edit /etc/passwd.

How to switch to another user? How to switch to the root user?

su command. Use su - to switch to root

What is the UID the root user? What about a regular user?

UID of root user is 0

Default values of UID_MIN and UID_MAX in /etc/login.defs UID_MIN is 1000 UID_MAX is 60000

Actually, we can change this value. But UID < 1000 are reserved for system accounts. Therefore, as per the default configuration, for regular user UID starts from 1000.

What can you do if you lost/forogt the root password?

Re-install the OS IS NOT the right answer :)

What is /etc/skel?

/etc/skel is a directory, that contains files or directories, so when a new user is created, these files/directories created under /etc/skel will be copied to user's home directory.

How to see a list of who logged-in to the system?

Using the last command.

Explain what each of the following commands does:

useradd
usermod
whoami
id

useradd - Command for creating new users usermod - Modify the users setting whoami - Outputs, the username that we are currently logged in id - Prints the

You run grep $(whoami) /etc/passwd but the output is empty. What might be a possible reason for that?

The user you are using isn't defined locally but originates from services like LDAP.
You can verify with: getent passwd

Hardware

Where can you find information on the processor (like number of CPUs)?

/proc/cpuinfo

You can also use nproc for number of processors

How can you print information on the BIOS, motherboard, processor and RAM?

dmidecoode

How can you print all the information on connected block devices in your system?

lsblk

True or False? In user space, applications don't have full access to hardware resources

True. Only in kernel space they have full access to hardware resources.

Namespaces

What types of namespaces are there in Linux?

Process ID namespaces: these namespaces include independent set of process IDs

Mount namespaces: Isolation and control of mountpoints

Network namespaces: Isolates system networking resources such as routing table, interfaces, ARP table, etc.

UTS namespaces: Isolate host and domains

IPC namespaces: Isolates interprocess communications

User namespaces: Isolate user and group IDs

Time namespaces: Isolates time machine

True or False? In every PID (Process ID) namespace the first process assigned with the process id number 1

True. Inside the namespace it's PID 1 while to the parent namespace the PID is a different one.

True or False? In a child PID namespace all processes are aware of parent PID namespace and processes and the parent PID namespace has no visibility of child PID namespace processes

False. The opposite is true. Parent PID namespace is aware and has visibility of processes in child PID namespace and child PID namespace has no visibility as to what is going on in the parent PID namespace.

True or False? By default, when creating two separate network namespaces, a ping from one namespace to another will work fine

False. Network namespace has its own interfaces and routing table. There is no way (without creating a bridge for example) for one network namespace to reach another.

True or False? With UTS namespaces, processes may appear as if they run on different hosts and domains while running on the same host

True

True or False? It's not possible to have a root user with ID 0 in child user namespaces

False. In every child user namespace, it's possible to have a separate root user with uid of 0.

What time namespaces are used for?

In time namespaces processes can use different system time.

Virtualization

What virtualization solutions are available for Linux?

KVM

XEN

VirtualBox

Linux-VServer

User-mode Linux

...

What is KVM?

Is an open source virtualization technology used to operate on x86 hardware.

From the official docs Recommended read:

Red Hat Article - What is KVM?

What is Libvirt?

It's an open source collection of software used to manage virtual machines. It can be used with: KVM, Xen, LXC and others. It's also called Libvirt Virtualization API.

From the official docs Hypervisor supported docs

AWK

What the awk command does? Have you used it? What for?

From Wikipedia: "AWK is domain-specific language designed for text processing and typically used as a data extraction and reporting tool"

How to print the 4th column in a file?

awk '{print $4}' file

How to print every line that is longer than 79 characters?

awk 'length($0) > 79' file

What the lsof command does? Have you used it? What for?

What is the difference between find and locate?

How a user process performs a privileged operation, such as reading from the disk?

Using system calls

System Calls

What is a system call? What system calls are you familiar with?

How a program executes a system call?

A program executes a trap instruction. The instruction jump into the kernel while raising the privileged level to kernel space.

Once in kernel space, it can perform any privileged operation

Once it's finished, it calls a "return-from-trap" instruction which returns to user space while reducing back the privilege level to user space.

Explain the fork() system call

fork() is used for creating a new process. It does so by cloning the calling process but the child process has its own PID and any memory locks, I/O operations and semaphores are not inherited.

What is the return value of fork()?

On success, the PID of the child process in parent and 0 in child process

On error, -1 in the parent

Name one reason for fork() to fail

Not enough memory to create a new process

Why do we need the wait() system call?

wait() is used by a parent process to wait for the child process to finish execution. If wait is not used by a parent process then a child process might become a zombie process.

How the kernel notifies the parent process about child process termination?

The kernel notifies the parent by sending the SIGCHLD to the parent.

How the waitpid() is different from wait()?

The waitpid() is a non-blocking version of the wait() function.
It also supports using library routine (e.g. system()) to wait a child process without messing up with other children processes for which the process has not waited.

True or False? The wait() system call won't return until the child process has run and exited

True in most cases though there are cases where wait() returns before the child exits.

Explain the exec() system call

It transforms the current running program into another program.
Given the name of an executable and some arguments, it loads the code and static data from the specified executable and overwrites its current code segment and current static code data. After initializing its memory space (like stack and heap) the OS runs the program passing any arguments as the argv of that process.

True or False? A successful call to exec() never returns

True
Since a successful exec replace the current process, it can't return anything to the process that made the call.

What system call is used for listing files?

What system calls are used for creating a new process?

fork(), exec() and the wait() system call is also included in this workflow.

What execve() does?

Executes a program. The program is passed as a filename (or path) and must be a binary executable or a script.

What is the return value of malloc?

Explain the pipe() system call. What does it used for?

Unix pipe implementation

"Pipes provide a unidirectional interprocess communication channel. A pipe has a read end and a write end. Data written to the write end of a pipe can be read from the read end of the pipe. A pipe is created using pipe(2), which returns two file descriptors, one referring to the read end of the pipe, the other referring to the write end."

What happens when you execute ls -l?

Shell reads the input using getline() which reads the input file stream and stores into a buffer as a string

The buffer is broken down into tokens and stored in an array this way: {"ls", "-l", "NULL"}

Shell checks if an expansion is required (in case of ls *.c)

Once the program in memory, its execution starts. First by calling readdir()

Notes:

getline() originates in GNU C library and used to read lines from input stream and stores those lines in the buffer

What happens when you execute ls -l *.log?

What readdir() system call does?

What exactly the command alias x=y does?

Why running a new program is done using the fork() and exec() system calls? why a different API wasn't developed where there is one call to run a new program?

This way provides a lot of flexibility. It allows the shell for example, to run code after the call to fork() but before the call to exec(). Such code can be used to alter the environment of the program it about to run.

Describe shortly what happens when you execute a command in the shell

The shell figures out, using the PATH variable, where the executable of the command resides in the filesystem. It then calls fork() to create a new child process for running the command. Once the fork was executed successfully, it calls a variant of exec() to execute the command and finally, waits the command to finish using wait(). When the child completes, the shell returns from wait() and prints out the prompt again.

Filesystem & Files

How to create a file of a certain size?

There are a couple of ways to do that:

dd if=/dev/urandom of=new_file.txt bs=2MB count=1

truncate -s 2M new_file.txt

fallocate -l 2097152 new_file.txt

What does the following block do?:

open("/my/file") = 5
read(5, "file content")

These system calls are reading the file /my/file and 5 is the file descriptor number.

Describe three different ways to remove a file (or its content)

What is the difference between a process and a thread?

What is context switch?

From wikipedia: a context switch is the process of storing the state of a process or thread, so that it can be restored and resume execution at a later point

You found there is a server with high CPU load but you didn't find a process with high CPU. How is that possible?

Advanced Networking

When you run ip a you see there is a device called 'lo'. What is it and why do we need it?

What the traceroute command does? How does it works?

Another common way to task this questions is "what part of the tcp header does traceroute modify?"

What is network bonding? What types are you familiar with?

How to link two separate network namespaces so you can ping an interface on one namespace from the second one?

What are cgroups?

Explain Process Descriptor and Task Structure

What are the differences between threads and processes?

Explain Kernel Threads

What happens when socket system call is used?

This is a good article about the topic: https://ops.tips/blog/how-linux-creates-sockets

You executed a script and while still running, it got accidentally removed. Is it possible to restore the script while it's still running?

It is possible to restore a script while it's still running if it has been accidentally removed. The running script process still has the code in memory. You can use the /proc filesystem to retrieve the content of the running script. 1.Find the Process ID by running ``` ps aux | grep yourscriptname.sh ``` Replace yourscriptname.sh with your script name. 2.Once you have the PID, you can access the script's memory through the /proc filesystem. The script will be available at /proc//fd/, where is the process ID of the running script. Typically, the script's file descriptor is 0 or 1.
You can copy the script content to a new file using the cp command:

cp /proc/<PID>/fd/0 /path_to_restore_your_file/yourscriptname.sh

Replace with the actual PID of the script and /path_to_restore_your_file/yourscriptname.sh with the path where you want to restore the script.

Memory

What is the difference between MemFree and MemAvailable in /proc/meminfo?

MemFree - The amount of unused physical RAM in your system MemAvailable - The amount of available memory for new workloads (without pushing system to use swap) based on MemFree, Active(file), Inactive(file), and SReclaimable.

What is the difference between paging and swapping?

Explain what is OOM killer

Distributions

What is a Linux distribution?

What Linux distributions are you familiar with?

What are the components of a Linux distribution?

Kernel

Utilities

Services

Software/Packages Management

Sed

Using sed, extract the date from the following line: 201.7.19.90 - - [05/Jun/1985:13:42:99 +0000] "GET /site HTTP/1.1" 200 32421

echo $line | sed 's/.*\[//g;s/].*//g;s/:.*//g'

Misc

What is a Linux distribution?

A collection of packages - kernel, GNU, third party apps, ...

Sometimes distributions store some information on the distribution in /etc/*-release file
- lsb_release is a common command you can use in multiple different distributions

Name 5 commands which are two letters long

ls, wc, dd, df, du, ps, ip, cp, cd ...

What ways are there for creating a new empty file?

touch new_file

echo "" > new_file

How `cd -` works? How does it knows the previous location?

$OLDPWD

List three ways to print all the files in the current directory

ls

find .

echo *

How to count the number of lines in a file? What about words?

You define x=2 in /etc/bashrc and x=6 ~/.bashrc you then login to the system. What would be the value of x?

What is the difference between man and info?

A good answer can be found here

Explain "environment variables". How do you list all environment variables?

What is a TTY device?

How to create your own environment variables?

X=2 for example. But this will persist to new shells. To have it in new shells as well, use export X=2

What a double dash (--) mean?

It's used in commands to mark the end of commands options. One common example is when used with git to discard local changes: git checkout -- some_file

Wildcards are implemented on user or kernel space?

If I plug a new device into a Linux machine, where on the system, a new device entry/file will be created?

/dev

Why there are different sections in man? What is the difference between the sections?

What is User-mode Linux?

In Linux, user mode is a restricted operating mode in which a user's application or process runs. User mode is a non-privileged mode that prevents user-level processes from accessing sensitive system resources directly.
In user mode, an application can only access hardware resources indirectly, by calling system services or functions provided by the operating system. This ensures that the system's security and stability are maintained by preventing user processes from interfering with or damaging system resources.

Additionally, user mode also provides memory protection to prevent applications from accessing unauthorized memory locations. This is done by assigning each process its own virtual memory space, which is isolated from other processes.

In contrast to user mode, kernel mode is a privileged operating mode in which the operating system's kernel has full access to system resources, and can perform low-level operations, such as accessing hardware devices and managing system resources directly.

Under which license Linux is distributed?

GPL v2

Files

README.md

Latest commit

History

README.md

File metadata and controls

Linux

Linux Master Application

Linux Exercises

Basics

Misc

Linux Questions

Linux 101

I/O Redirection

Filesystem Hierarchy Standard

Permissions

Scenarios

Systemd

Troubleshooting and Debugging

Scenarios

Kernel

SSH

Globbing & Wildcards

Boot Process

Disk and Filesystem

Performance Analysis

Processes

Security

Networking

DNS

Packaging

DNF

Applications and Services

Users and Groups

Hardware

Namespaces

Virtualization

AWK

System Calls

Filesystem & Files

Advanced Networking

Memory

Distributions

Sed

Misc