Linux and where am I

Intention:

The purpose of this page is to present the history of OS and roots of all modern systems.

History:

The first atemption is starting in 1991: "The history of Linux began in 1991 with the commencement of a personal project by Finnish student Linus Torvalds to create a new free operating system kernel. Since then, the resulting Linux kernel has been marked by constant growth throughout its history. Since the initial release of its source code in 1991" [1]

Personaly i started to use linux in second year of uni around 2006 and since then i never ever found  better solution for home/profesioanl usages. 


Linux roots
Linux family tree [1] 

   The success of Linux versus other operation systems is mainaly because of the decision to make it open for everyone while other company decided to keep it proprietary and hidden, this decision was the crucial in later success of Linux.  It can also be considered monolithic because it joint all of the basic services into the kernel. This differs from a microkernel architecture where the kernel provides basic services such as communication, I/O, and memory and process management, and more specific services are plugged in to the microkernel layer. That has been one of the main point of criticism and debates between Torvalds and Tanenbaum "LINUX is obsolete".

Architecture: 

The architecure of Linux operating is splited in 2 main levels. User space represents ther spcase where the user applications are executed. And kernel space where, the Linux kernel exists.
Linux architecture [2]

Also in the user space is located the GNU C Library (glibc/libc). This provides the system call interface that connects to the kernel and provides the mechanism to transition between the user-space application and the kernel. This is important because the kernel and user application occupy different protected address spaces. And while each user-space process occupies its own virtual address space, the kernel occupies a single address space.[3]

The Linux kernel can be further divided into three gross levels. At the top is the system call interface, which implements the basic functions such as read and write. Below the system call interface is the kernel code, which can be more accurately defined as the architecture-independent kernel code. This code is common to all of the processor architectures supported by Linux. Below this is the architecture-dependent code, which forms what is more commonly called a BSP (Board Support Package). This code serves as the processor and platform-specific code for the given architecture.

System Call Interface
The SCI is a thin layer that provides the means to perform function calls from user space into the kernel. The SCI is actually an interesting function-call multiplexing and demultiplexing service. The implementation is located in ./linux/kernel, as well as architecture-dependent portions in ./linux/arch.

Process management
Process management is focused on the execution of processes. In the kernel, these are called threads and represent an individual virtualization of the processor (thread code, data, stack, and CPU registers).
The kernel provides an API through the SCI to create a new process (fork, exec, or Portable Operating System Interface [POSIX] functions), stop a process (kill, exit), and communicate and synchronize between them (signal, or POSIX mechanisms).
Also in process management is the need to share the CPU between the active threads. The kernel implements a novel scheduling algorithm that operates in constant time, regardless of the number of threads vying for the CPU. This is called the O(1) scheduler, denoting that the same amount of time is taken to schedule one thread as it is to schedule many. The O(1) scheduler also supports multiple processors (called Symmetric MultiProcessing, or SMP). Process management sources is located in ./linux/kernel and architecture-dependent sources in ./linux/arch).

Memory management
Another important resource that's managed by the kernel is memory. For efficiency, given the way that the hardware manages virtual memory, memory is managed in what are called pages (4KB in size for most architectures). Linux provides abstractions over 4KB buffers, such as the slab allocator. This memory management scheme uses 4KB buffers as its base, but then allocates structures from within, keeping track of which pages are full, partially used, and empty. This allows the scheme to dynamically grow and shrink based on the needs of the greater system.

Supporting multiple users of memory, there are times when the available memory can be exhausted. For this reason, pages can be moved out of memory and onto the disk. This process is called swapping because the pages are swapped from memory onto the hard disk. You can find the memory management sources in ./linux/mm.

Virtual file system
The virtual file system (VFS) is an interesting aspect of the Linux kernel because it provides a common interface abstraction for file systems. The VFS provides a switching layer between the SCI and the file systems supported by the kernel see below image.
 
 The VFS provides a switching fabric between users and file systems [3]
The file system sources is locater under ./linux/fs

Network stack
TCP is the sockets layer, which is invoked through the SCI.
The sockets layer is the standard API to the networking subsystem and provides a user interface to a variety of networking protocols. From raw frame access to IP protocol data units (PDUs) and up to TCP and the User Datagram Protocol (UDP), the sockets layer provides a standardized way to manage connections and move data between endpoints. The networking sources is located in  ./linux/net.

Device drivers
The Linux source tree provides a drivers subdirectory that is further divided by the various devices that are supported, such as Bluetooth, I2C, serial, and so on. The device driver sources is located under ./linux/drivers.

Architecture-dependent code
While much of Linux is independent of the architecture on which it runs, there are elements that must consider the architecture for normal operation and for efficiency. The ./linux/arch subdirectory defines the architecture-dependent portion of the kernel source contained in a number of subdirectories that are specific to the architecture (collectively forming the BSP). Each architecture subdirectory contains a number of other subdirectories that focus on a particular aspect of the kernel, such as boot, kernel, memory management, and others.

More detailed Ubuntu architecture Architecture:



 System lifecycle:

The lifecycle of OS from power on to Power off is prity simple and obvious, but that only from initial point of view, so the lifecicle or state machine is:
 
Power On -> Load(from ROM to RAM) -> Bootloader/Bootstrap(load kernel) -> BootPROM/ PROM Monitor/BIOS -> Boot Manager(e.g. GRUB) -> Boot LiLo(Select kernel load configs) -> Kernel Init(start_kernel) ->Os Init(/sbin/init) ->Run(process tree)->Stop(halt/reboot/poweroff)->Shutdown(/bin/shutdown inhibit login)-> SIGTERM(send message to all precess for persisting current state) -> SIGKILL(kill process)->PowerOff

If we try to look dipper how state machine for every process is implemented than we can more states and transitions.
Process state diagram[5]

To see entire running system and hierarchy of processes try to run pstree.

Tools for system analyse 

To monitor process activity use one of: top/atop/htop/glances/dstat/saidar 
all of them provides a dynamic real-time view of a running system i.e. actual process activity

System Activity, Hardware and System Information memory, paging, block IO, traps, and cpu activity:
# vmstat 1 -a -m 
Who 
Find out who is logged on and what they are doing w/who command displays information about the users currently on the machine, and their processes.
# w
# who 


Process management:
Print all process 
# ps ax
# ps axu
Print Security Information: 
# ps -eo euser,ruser,suser,fuser,f,comm,label
# ps axZ
# ps -eM
Print A Process Tree
# ps -ejH
# ps axjf
# pstree
See Every Process Running As User eca
# ps -U eca -u eca u
Set Output In a User-Defined Format
# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
# ps -eopid,tt,user,fname,tmout,f,wchan
Find Out The Top 10 Memory Consuming Process
# ps -auxf | sort -nr -k 4 | head -10
Find Out top 10 CPU Consuming Process
# ps -auxf | sort -nr -k 3 | head -10
Average CPU Load, Disk Activity

The command iostat report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).
# iostat

System activity repport:

The sar command is used to collect, report, and save system activity information. To see network counter, enter:
# sar -n DEV | more
To display the network counters from the 24th:
# sar -n DEV -f /var/log/sa/sa24 | more
You can also display real time usage using sar:
# sar 4 5

Multiprocessor Usage
The mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:
# mpstat -P ALL
Process Memory Usage
The command pmap report memory map of a process. Use this command to find out causes of memory bottlenecks.
# pmap -d PID

To display process memory information for pid # 123 enter:
# pmap -d 123

Network Statistics
# netstat
# nmap 
# ss -lp
Real-time Network Statistics
# iptraf
Detailed Network Traffic Analysis
# tcpdump -i eth1 'udp port 53'
To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:
# tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2 -="" amp="" tcp="" xf0="">>2)) != 0)'
To display all FTP session to 202.54.1.5, enter:
# tcpdump -i eth1 'dst 202.54.1.5 and (port 21 or 20'
To display all HTTP session to 192.168.1.5:
# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'
Use wireshark to view detailed information about files, enter:
# tcpdump -n -i eth1 -s 0 -w output.txt src or dst port 80

System Calls
Trace system calls and signals. This is useful for debugging webserver and other server problems. See how to use to trace the process and see What it is doing.

#  strace -v stat


Kernel Statistics
/proc file system provides detailed information about various hardware devices and other Linux kernel information. See Linux kernel /proc documentations for further details. Common /proc examples:
# cat /proc/cpuinfo
# cat /proc/meminfo
# cat /proc/zoneinfo
# cat /proc/mounts
KDE System Guard – Real-time Systems Reporting and Graphing
KSysguard is a network enabled task and system monitor application for KDE desktop. This tool can be run over ssh session. It provides lots of features such as a client/server architecture that enables monitoring of local and remote hosts.

lsof – list open files, network connections and much more.

ntop web based tool – ntop is the best tool to see network usage in a way similar to what top command does for processes i.e. it is network traffic monitoring software. You can see network status, protocol wise distribution of traffic for UDP, TCP, DNS, HTTP and other protocols.
Conky – Another good monitoring tool for the X Window System. It is highly configurable and is able to monitor many system variables including the status of the CPU, memory, swap space, disk storage, temperatures, processes, network interfaces, battery power, system messages, e-mail inboxes etc.

GKrellM – It can be used to monitor the status of CPUs, main memory, hard disks, network interfaces, local and remote mailboxes, and many other things.

vnstat – vnStat is a console-based network traffic monitor. It keeps a log of hourly, daily and monthly network traffic for the selected interface(s).

mtr – mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.

Display Bandwidth Usage on Network Interface 
Type iftop command at the shell prompt to display traffic:
# iftop -i eth0


File system:

/bin: 
Contains the executable programs that are part of the Linux operating system.
Many Linux commands such as cat, cp, ls, more, and tar are located in /bin.
Example ls, cat, cp.

/dev:
All the devices like input devices, sound card, modems are stored.
It is a virtual directory that contains devices files.
Example : /dev/udp, /dev/urandom, /dev/sda1

/etc
Contains config folder of entire operating system.
All the global setting like ssh, telnet, and smtp/pop3 mail servers.
Also contains system’s password file like group lists, user skeletons, and cron jobs.
Example: /etc/resolv.conf, /etc/logrotate.conf

/home
Default directory for users to store the personal files.
Example /home/saugat, /home/sachit

/sbin
Contains binary executables typtically used by system admnistrator only available to root.
Mostly used for system maintenance purpose
Commands such as mount, shutdown, umount, reside here
Example: /sbin/halt/ /sbin/ip6tables

/usr
Contains shareable and read only datacontains binaries, libraries, documentation and source code for second level program
/usr/bin : Contains executable files for many Linux commands. It is not part of the core Linux operating system.
/usr/include : Contains header files for C and C++ programming languages
/usr/lib : Contains libraries for C and C++ programming languages.
/usr/local : Contains local files. It has a similar directories as /usr contains.
/usr/sbin : Contains administrative commands.
/usr/share : Contains files that are shared, like, default configuration files, images, documentation, etc.
/usr/src : Contains the source code for the Linux kernel.

/var
Includes user specific files such as mail message, database of installed programs, log files etc.

/var/cache: Storage area for cached data for applications.
/var/lib: Contains information related to the current state of applications. Programs modify this when they run.
/var/lock: Contains lock files which are checked by applications so that a resource can be used by one application only.
/var/log: Contains log files for different applications.
/var/mail: Contains users emails.
/var/opt: Contains variable data for packages stored in /opt directory.
/var/run: Contains data describing the system since it was booted.
/var/spool: Contains data that is waiting for some kind of processing.
/var/tmp: Contains temporary files preserved between system reboots

/tmp
All the temporary files are store here.
The files under this directory are deleted when system is rebooted.
For example: when new program is installed it use /tmp/ to put files during installation that won’t be
needed after the program is installed.

/mnt
Default location for mouting devices like cdrooms, floppy disk dries, USB memory sticks etc.
Example : /mnt/cdroom

/proc
contains information about system process
virtual file system that contains information about file system.
Example /proc/cpuinfo, /proc/swaps

/lib
share libraries are stored(perl, python, C, etc.)
/lib/ are also a kernel modules
Example: ld-2.11.1.so, libncurses.so.5.7

/opt
Config file for add on Application software are found here.
Third party application should be installed in this directory.

/root
Home directory of system administrator.’root’.
Root user has write privilege under this directory

/boot
Contains everything required for boot process.
Stores data that is used before the kernel begins executing user-mode program.
Example: /boot/boot.b, /boot/chain.b, /boot/config-kernel-version

References:
[1] https://en.wikipedia.org/wiki/List_of_Linux_distributions
[2] http://www.slideshare.net/quontrasolutionsusa/introduction-to-linux-kernel-by-quontra-solutions
[3] https://www.ibm.com/developerworks/library/l-linux-kernel
[4] https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_debate
[5] http://www.ksvali.com/wp-content/uploads/2009/09/processstate.gif
[6] http://www.cyberciti.biz/tips/linux-resource-utilization-to-detect-system-bottlenecks.html
[7] http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html
[8] http://www.cyberciti.biz/tips/linux-investigate-sockets-network-connections.html
[9] http://www.iptables.info/en/iptables-matches.html
[10] https://blog.mypapit.net/upload/files/linux_file_structure.jpg

No comments:

Post a Comment