This Linux guide is my own notes on Linux, for stuff that I did not like to documents that I have found on the internet. It is RedHat centric.
CentOS is supposed to be a "rebadged/un-commercialized" version of RHEL, keeping binary-level compatibilities at major release level, thus the discussion should apply equally to CentOS and Scientifi Linux.
Ubuntu is conquering the world now, so there are some Debian things here and there. What is Linux anyway? :)

funny/ugly/stupid chars when pasting? See bracketed_paste_mode

HOW-TO

 

RHEL vs CentOS vs Scientific Linux

Linux Admin 101

Linux init mode are not "cumulative". So, if default init is 5, script in run level 3 would not be run. If adding a Start script, may want to add them in both in rc3.d and rc5.d, unless it is really only wished to be available at one of the run level and not the other. Kill script should be placed in rc6.d, which shutdown/reboot scan.

RHEL3 - binary compatible with FC2 (but FC2 is EOL)
RHEL4 - binary compatible with FC3 (and maybe FC4)

Single User Mode RHEL 6

To boot into single user mode for maintenance,
enter "linux single" or "linux emergency" at the LILO boot: prompt.

If using GRUB, then, at the menu for choosing kernel, do:
 hit 'e'
 then arrow down to the kernel line
 hit 'e' again
 add 'single' to the end of the line
 hit enter
 hit 'b' to boot
 lather, rinse, repeat

Single User Mode RHEL 7

https://docs.fedoraproject.org/en-US/Fedora/19/html/Installation_Guide/s1-grub-targets.html
linux16 /vmlinux...  ro                        rd.lvm.lv=... 
linux16 /vmlinux...  rw init=/sysboot/bin/bash 


linux /vmlinux... systemd.unit=multi-user.target    

remove rhgb quiet
rhgh is redhat splash
quiet remove most verbose msg


/etc/default/grub
GRUB_TIMEOUT=60
GRUB_CMDLINE_Linux=... (remove rhgb quiet)
grub2-mkconfig

Single User Mode RHEL 7 with pxe boot with chrooted env

ln -sf /lib/systemd/system/rescue.target /etc/systemd/system/default.target

Old days Core dump

Linux kernel panic does not core dump to a file.
it will print output of the kernel core dump result to the stdout device:
a vga screen or a serial console.  
To setup serial console, do:
console=ttyS0,115200 console=tty0
but this would mean console boot/shutdown/panic message will no longer
be send to the VGA, but to serial port.  VGA may get a spawned
login prompt after OS is up.

kdump

yum install kexec-tools
chkconfig kdump on 
/etc/kdump.conf ::
core_collector makedumpfile -c --message-level 1 -d 31
# -d specify the dump filter (stuff to exclude)
# 1 	Zero pages
# 2 	Cache pages
# 4 	Cache private
# 8 	User pages
# 16 	Free pages 
# -c to enable compression
#
net sa@debug-svr.nx.net
sshkey /etc/acs/kdump/id_rsa-sa_kdump
path /data/kdump/
blacklist pvscsi
blacklist vmmemctl
/boot/grub/grub.conf : add crashkernel clause
	        kernel /boot/vmlinuz-2.6.18-371.11.1.el5 ro root=LABEL=/ crashkernel=768M@32M

test.  this WILL CRASH the machine:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
To look at vmcore file:
yum install crash
debuginfo-install kernel	# to install necessary "decoder" to analyze the crash file

crash  /var/crash/.../vmcore   /usr/lib/debug/lib/modules/.../vmlinux    # need matching kernel file
  log
  bt       	# for backtrace
  ps		# display processes
  vm
  files		# lsof
  help
somewhat helpful shell function to see if there are recent core dumps
lsdump () {
    for entry in $(ls /kdump|grep ^[1-9]); do
        ip=$(echo $entry|cut -d- -f1)
        hostname=$(dig +short -x $ip)
        date=$(echo $entry|cut -d- -f2-)
        echo "$date    $hostname ($ip)"
    done | sort
}

console in serial port

--kargs="nopti console=tty0 console=ttyS1,115200n8"

Linux "deviation" from Solaris

Some of the most notable things that Linux does rather differently than Solaris or traditional Unix.

RHEL/CentOS/Scientific Linux vs Ubuntu

Ubuntu is Debian-based. Some of the more often run-into diff for sys admin:

RHEL Ubuntu
nic /etc/sysconfig/network-scripts/ifcfg-NIC /etc/network/interfaces
Firewall systemctl enable/start/status iptables
systemctl enable/start/status firewalld
ufw enable/start/status

Most of this doc is CentOS centric.

64-bit Platforms

RHEL 3.0 and 4.0 maintains simultaneous relese/update levels for 64-bit platforms as their popular 32-bit x86 OS. Just need to get the "special" distro ISO to install on the 64-bit hardware and it will be good to go. Not all the AS/ES/WS flavor maybe available to all platform.
RHEL 4.0   	

rhel-ia64-as-4		Red Hat Enterprise Linux AS (v. 4 for 64-bit Intel Itanium)
rhel-x86_64-as-4	Red Hat Enterprise Linux AS (v. 4 for 64-bit AMD64/Intel EM64T)
rhel-ppc-as-4		Red Hat Enterprise Linux AS (v. 4 for 64-bit IBM POWER)
rhel-s390-as-4		Red Hat Enterprise Linux AS (v. 4 for 31-bit IBM S/390)
rhel-s390x-as-4		Red Hat Enterprise Linux AS (v. 4 for 64-bit IBM zSeries)
rhel-i386-as-4		Red Hat Enterprise Linux AS (v. 4 for 32-bit x86)   [typical intel pentium 4/80x86 release]
rhel-i386-ws-4		Red Hat Enterprise Linux WS (v. 4 for 32-bit x86)

RHEL 3.0 	

rhel-ia64-as-3		Red Hat Enterprise Linux AS (v. 3 for Itanium)
rhel-x86_64-as-3 	Red Hat Enterprise Linux AS (v. 3 for AMD64/Intel EM64T)   
rhel-ppc-as-3		Red Hat Enterprise Linux AS (v. 3 for iSeries and pSeries)
Commands for 64-bit info:
ld -V		# shows supported emulation for a given machine	
		# eg: elf_x86_64 elf_i386 i386linux

64-bit and rpm

AMD64 RedHat Linux utilize lots of rpm that has ARCH set to x86_64 (instead of the traditional i386). rpm -q by default won't tell you. Use something like

rpm -q --qf '%{NAME} %{VERSION} %{RELEASE} (%{ARCH})\n' Package_Name

to see which one you have installed. In 64-bit machines, most of the packages are x86_64. But quite often you will have a matching package of the same name, but is i386 ARCH. Some apps don't understand 64bit libs, so these older libs are sometime needed. There are also some compat libs. My friend Vic says:

* Compatibility Arch Support
Crapload of i386 binaries and libraries that run on x86_64 RedHat systems

* Compatibility Arch Development Support
You'll need some packages in Compatibility Arch Development Support to build
i386 RPMs or compile 32 bit binaries on an x86_64 system.

I find it annoying to install the Compatibility Arch Support group, though.

For instance, let's say you have an x86_64 arch server. You want some i386
compatibility for some random library.  Let's say zlib.i386 for whatever
reason.  Maybe some 3rd party application needs to run as a 32 bit binary and
requires zlib's /usr/lib/libz.so.1 instead of zlib.x86_64's
/usr/lib64/libz.so.1

You can just install Compatibility Arch Support and not worry about it, or go
back an install zlib.i386 specifically.  If you install the whole
Compatibility Arch Support group, you are going to end up with all the
freakin' i386 packages designed to run on x86_64 systems.  Which means, you'll
get a lot of crap you don't want to be living on a light weight server.  For
instance, kdebase.i386, gnome-blah.i386 and such.

One thing that is very useful to have is firefox.i386 or mozilla.i386 on an
x86_64 system. 
Most plugins don't play well with 64-bit browsers.

So, in short -- you can remove what you don't need :)


Endianness


network byte order: big-endian
intel: little-endian    (just think of intel being an @$$)
most RISC were big-endian?
new POWER is configurable.  
arm is said to be either, but not sure if it was a silicon level or firmware level change.

amazon linux using aarch64 is apparently little endian

To determine byte order, use one of:

python:
from sys import byteorder
print(byteorder)

perl:
perl -e 'use Config; print $Config{byteorder}'


for the purist, there is a C/C++ code in this stack overflow post. I was just too lazy to compile things. Perl one liner was just perfect for this :D

Compatibility

Torvals ensure linux kernel update does not break user space. glibc, which make lots of system calls, has to be kept very stable. Let the distro manage upgrade of kernel and glibc.
gcc does NOT come with glibc! :)
gcc and g++ come and use libstdc++ , which provide fn like new and delete, which maps to C API system calls like malloc(), thus only glibc need to be very stable, whereas compiler and libstdc++ can upgrade and have multiple versions.
Useful info in this reddit post


System Release                  kernel          glibc           gcc                    compat-glibc
----------------------------    -------------   -------------	-----------------      -------------
SuSE SLUS  9 (patch  3)		2.6.11.7	2.3.3-98.94	3.3.3-43.54		
SuSE SLUS 10 (patch 10)		2.6.16.27-0.9	2.4-31.5	4.1.2_20070115-0.11	

RHEL 3				2.4.21		   		3.2
RHEL 4.5 (WS)			2.6.9-55	2.3.4-2.36	3.4.6-8			2.3.2-95.30
RHEL 5				2.6.18		2.5		4.1			

RHEL 5.9                        2.6.18-348      2.5-107         4.1.2-54
Fedora Core 3                   2.6.9
Fedora Core 6			2.6.19-1	2.5-10.fc6	4.1.1-51.fc6		
Fedora 8                        2.6.23

RHEL 7.9                        3.10.0-1160.49  2.17-325        4.8.5                  n/a?
Amazon Linux 2.3                4.14.281-212    2.26-60		7.3.1
RHEL 8.1                        4.18.0-147.3.1  2.28-72		8.3.1-4.5



RHEL 2.1, 3, 4.  Increasing support cost: WS,                    WS HPC, 		ES (2 socket max),      AS
             5.                           Desktop + workstation, HPC Compute Nodes, 	Server, 		Advance Platform

For RH, each category has separation of basic, std, premium. HPC has a 4 socket version. See comparison chat.
Compatibility table of GCC vs binutils (but NOT glibc) from osdev

Initial Setup

Things to keep in mind when installing Linux.

For RHEL 3 and 4, once the OS is installed, additional international language support cannot be added (easily), RH recommends a reinstall. So, if you might need to support any given natural language, install it when you do the system install!
Starting with Fedora Core 4, additional language can be added from system-configure-packages.

CentOS Network Install CD
Use Web as source of rpm.  
Server: mirror.stanford.edu  			(no http:// prefix)
Directory: yum/pub/centos/5.1/os/x86_64/  	(leading slash should not matter)
or
Server: mirror.centos.org
Directory: /centos/5.2/os/x85_64
But unfortunately don't support proxy, so likely have to setup something locally.
The dir containing images sub dir is the right one, it need to have the rpm available rather than just .iso
Essentially, same as internal network install, but the web server is over the WAN, and not using a pre-defined kickstart.cfg file :)
Other mirror can be used, but some of them only offer ISO files.

Kickstart

A step-by-step command listing for setting up kickstart, (hopefully easier to read than Red Hat kickstart instructions, once it is working, tweaking it is much easier to comprehend). ...
Here is a rough outline:


  1. iso loopback mount DVD or copy all content of CD/DVD to a dir
  2. cretae a dir on web server to host the ks.cfg file, make file accessible as http://apache/ks/ks.cfg
  3. ks.cfg uses http or nfs install, indicating path where rpm can be retrieved. eg:
    nfs --server=10.140.91.44 --dir=/mnt/loopback/rhel-5.1-server-x86_64-dvd
    or
    url --url http://10.140.91.44/ks/serv51-64/dvd1
  4. client, use cd 1, at boot prompt, enter:
    linux ks=http://10.140.91.44/ks/serv51-64/ks.cfg
running commands after kickstart rpm are installed, everything is masked as ran from /:
%post 
#!/bin/bash
LOGFILE=/var/log/my-kickstart.log
echo "manual log to execution ran on post section of kickstart "  > ${LOGFILE}
pwd                                                              >> ${LOGFILE}

# Add yum repos (rpm from original dvd for easy install via yum)
echo "[rhel5]
name=RHEL5 Kickstart Server 
baseurl=file:///net/apache/mnt/loopback/rhel-5.3-server-x86_64-dvd/Server/
enabled=1
gpgcheck=0" > /etc/yum.repos.d/rhel5.repo

# install additional packages 
yum -y install j2re AdobeReader_enu

# Create symlink for java browser plugin
ln -s /usr/java/j2re1.4.2_07/plugin/i386/ns610-gcc32/libjavaplugin_oji.so /usr/lib/mozilla/plugins/

# Local services
chkconfig ntpd on



echo "nfserver:/export/home    /nfshome    nfs    rw,soft,intr,tcp,rsize=32768,wsize=32768,vers=3 0 0
" >> /etc/fstab

mkdir /nfshome/
touch /nfshome/MOUNT+PENDING
mount /nfshome

ln -s /bin/csh  /usr/bin/csh
ln -s /bin/tcsh /usr/bin/tcsh

echo "sn     ALL=(ALL) ALL" >> /etc/sudoers

mv /etc/yp.conf /etc/yp.conf.orig
wget http://apache/conf/yp.conf -O /etc/yp.conf

# Modify SSHD allowed protocols to use only ssh v2:
# sed -i.bak -c "s/#Protocol 2,1/Protocol 2/" /etc/ssh/sshd_config


Updating OS

RHEL 4.x  - Use up2date
RHEL 5.x  - Use yum
	    rhn_register # automatically prompt for root password when run as user, X-based.
RHEL 8.x  - Use did not finish, i mean dnf (yum alias to it)
SuSE      - Use rug

Version numbers with dots matter for kernel, glibc, etc. Only version number after dash are bug fixes and don't change behaviour. See compatibility section for more details.

Quick Ref

Hardware related command

see also
  • performance section
  • bios.html
    
    lspci			# list pci info
    lscpu
    
    
    lsmod			# display all kernel loaded modules.  
    modprobe -l		# display all kernel loaded modules.  now lsmod.
    
    hwbrowser		# GUI hardware browser, in /usr/bin  (RH9, RHEL 4)
    
    /dev/mic*		# many integrated core, for Phi cpu when setup as co-processor, accelerator (NOT the Dell C6320p which has Phi as main cpu)
    			# https://portal.tacc.utexas.edu/documents/13601/1041435/23-Intel_Xeon_Phi_Intro_Part_1.pdf/eeaf5f85-93d3-4de3-b523-b5f9ec906299
    
    udev			# some hw persisten naming thing, eg for hd that may change name due to san multipath, etc.
    
    

    Random Tidbits

    Timezone
    Changing Timezone
    logrotate
    /etc/cron.daily/logrotate	# cronjob that runs daily
    /etc/logrotate.d/syslog		# where /var/log/messages is stated to get rotated, compression, etc.
    /var/lib/logrotate/logrotate.status	# when a given log was last rotated.  new entry may have to wait eg 1 week before it get rotated
    
    /usr/sbin/logrotate -v /etc/logrotate.conf	# manually invoke logrotate with verbose output, stating why file are rotated or not.
    

    XDMCP

    enabling XDMCP will allow program like X Manager to use browser and login via virtual screen, 
    fully X Manager, Remote Display Manager style, rather than having to login 
    using ssh and start VNC first.  However, VNC is nicer in that 
    the session stays on the server, XDMCP, if client is a laptop and disconnect, 
    session will be reset.
    Note that botyh XDMCP and VNC are insecure by default!
    
    vi /etc/X11/fs/config
    # don't listen to TCP ports by default for security reasons
    ###no-listen = tcp
    ### commented out line above to enable XDMCP
    
    service xfs restart
    
    vi /etc/X11/xdm/xdm-config
    ! SECURITY: do not listen for XDMCP or Chooser requests
    ! Comment out this line if you want to manage X terminals with xdm
    !!!DisplayManager.requestPort:  0
    !!!Line above commented out to enable XDMCP
    
    
    vi /etc/X11/xdm/Xaccess
    # *                                     #any host can get a login window
    *       # have start by itself will allow all host to get login window
    
    
    vi /etc/X11/gdm/gdm.conf
    [xdmcp]
    ###Enable=false
    Enable=true
    ### XDMCP is enabled using the above clause
    ### this file need to be updated when GNOME is default windows manager/Display Manager
    
    vi /etc/kde/kdm/kderc
    [Xdmcp]
    ###Enable=false
    Enable=true
    ### XDMCP is enabled using the above clause
    ### this file need to be updated when KDE is default windows manager/Display Manager
    
    chmod 444 /etc/X11/xdm/Xservers                 # probably correct already
    chmod 755 /etc/X11/xdm/Xsetup_0
    
    
    older config need to update /etc/X11/XF86Config, but circa 2003
    most distro use Xorg,
    
    check /etc/X11/xorg.conf
    and ensure FontPath uses Xserver:
            FontPath     "unix/:7100"
    
    restart X:
    - killall gdm-binary 	# if running GNOME by default
    - ctrl+alt+bacckspace	# if in front of maching/keyboard
    - init 3; init 5	# This may work
    - reboot		# :)
    
    
    xorg pci BusID when multiple GPU is present and want to specify which one to use to accelearate video (virtual, VNC)
    nvidia-xconfig --query-gpu-info 
    returned pci BusID for each GPU card.
    


    GNOME

    gnome-terminal, configure to source .login/.cshrc when launched:
    
    Edit menu, Current Profile... Titles and Command tab, 
    check "Run command as a login shell".
    This should be saved in the user's config file somewhere under the user's home dir.  
    
    
    # to view user's setting.
    gconftool-2 --get /apps/gnome-terminal/profiles/Default/login_shell	
    
    # command line to set to source login shell, per user.
    gconftool-2 --type boolean --set /apps/gnome-terminal/profiles/Default/login_shell true			
    
    
    
    # read system wide setting
    gconftool-2 --direct --config-source xml:read:/etc/gconf/gconf.xml.defaults --get /apps/gnome-terminal/profiles/Default/login_shell		
    
    
    # set global settings as root, to source login when term opens.
    gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type boolean --set /apps/gnome-terminal/profiles/Default/login_shell true		
    
    

    pam

    Ref: Check to allow user login is really a convoluted matrix of many different combinations. All the different "if ssh, if user is foo, login in from bar, etc etc" is typically some sort of modules and config in pam.
    authconfig, authselect and other tools make changes to pam.
    /etc/nsswitch.conf is high level director
    /etc/sssd/sssd.conf is backend for the "sss" clause in nsswitch.conf, it is a cache daemon.
    /etc/security

    Whether ssh keys are allowed as authentication source is set in:
    - sshd.conf (if certain users are allowed to use ssh key, others are not)
    - pam sshd config (forcing interactive login?) (if config is same for all users) ??

    /etc/pam.d/ Files

    pam.d/ config really governs how things are done.
    Each file correspond to configuration used for that specific service.
    sshd	# used when ssh in to the machine
    login	# used to /bin/login (eg, local console login as dumb terminal?)
            # when physically in front of machine (? ie GUI, text login, and also su once logged in)
    
    gdm-password  # gdm based Display Manager use this pam.d config ?
    
    su	# when running su command
    sudo 	# when running sudo command
    
    system-auth	# often refered by other config file to use it.  think of this as C's #include.
    password-auth
    
    config-util	# refered by many system config tools, such as system-config-network
    
    clause:
    pam_stack.so service=system-auth   # this is like "import" system-auth.
    
    
    ??
    vnc	# most vnc use a local .vncpasswd, not relying on amp
    realvnc	# 
    screensaver lock?
    
    
    

    AAA

    
    aaa = 
    authentication  -- you are who you claim you are
    authorization   -- you are in the allowed-list for access
    accounting      -- log that you have accessed the system, duration, etc.
    
    pam.d/ files have these configurable controls 
    
    auth      - verify user credentials.  ? so here talk to ldap, nis, or sssd/local passwd 
    account   - user allowed to connect, password not expired?
    password  - ? write to shadow file about last login info
    session   - long duration session info, update wtmp, write when logout too
    
    

    Levels

    In increasing mandatory necessity:
    
    optional ?
    
    sufficient - eg login or su, allow use of radius credentials for login in console
    	auth       sufficient   pam_radius_auth.so conf=/etc/raddb/server
    
    substack ? like import?
    	auth       substack     password-auth
    
    include
    	use content of another config file (eg system-auth, config-util)
    	think of #include of C
    
    required
    	Failure also results in denial of authentication, 
    	although PAM will still call all the other modules listed for this service 
    	before denying authentication. 
    	*** think of it as mandatory but keep checking anyway ??!! ***
    
    requisite
    	Failure to authenticate via this module results in immediate denial of authentication. 
    	*** think of as mandatory, waste no more time, fail right away ***
    
    

    modules

    # Use a radius authentication server listed in a config file
    auth       required     pam_radius_auth.so conf=/etc/raddb/server	
    	
    
    # When SELinux operates in Permissive mode, 
    # disable linux user login for users listed in /etc/security/sepermit.conf
    # If user not listed, would still be allowed to login when selinux is disabled or in permissive mode
    auth       required     pam_sepermit.so
    
    
    # if file /etc/nologin exist, user not allowed to login
    account    required     pam_nologin.so
    
    
    # use/allow for system password auth  ??
    account    include      password-auth
    
    
    # traditional unix auth, usually /etc/passwd, /etc/shadow
    pam_unix.so
    
    
    

    authconfig -- RHEL 7 authentication configuration tool

    authconfig is the tool to use to configure authentication services in RHEL7/CentOS7. Manually changing nsswtich.conf isn't enough. There are pam.d, selinux, firewall updates.
    authconfig-tui is being deprecated, so for interactive config would need to rely on authconfig-gtk or system-config-authentication.
    See also ipa-client-install and realmd

    sssd is the daemon for System Security Services. It provides a unified interface to manage cache daamon, etc. nsswitch.conf refer this as sss.
    /etc/sysconfig/authconfig	# config file (multiple server use space as delimiter)
    authconfig --test		# show/test current auth service
    authconfig ... --update		# for any changes, always end with --update!
    
    authconfig --enablenis --nisdomain=research --nisserver=nis1,nis2 --disableldap --update	
    	# change authentication method of system to use nis, not ldap
    	# this thing may fail silently... 
    	# or other config in file not expressed in the cli doesn't get changed 
    	# and affect final behavior :(
    	# run test a/o check the resulting config file to be sure.
    
    authselect 	# replaced authconfig in rhel8
    authselect list
    authselect show sssd	
    
    sss_cache -E		# flush all cache
    sss_cache -u bob	# flush entry for specific user
    
    
    Ref:

    authselect -- RHEL 8 authentication configuration

    authselect show		# show current profile
    /etc/authselect		# config files loc      (/etc/sssd/ is largely empty now)
    
    Ref:

    LDAP

    See the LDAP doc for more details.

    authconfig 		# TUI, select to use ldap, etc.
    			# upon completion, ldap should work for user login.
    			# multiple server can be listed using comma separation
    			# or space separated in /etc/sysconfig/authconfig
    
    /etc/sysconfig/autofs	# update BASEDN to define where autofs will look for maps.  eg
    			# BASEDN="ou=us,dc=unixville,dc=com"
    service autofs restart	# restart autofs to effect the changes
    
    /usr/lib/autofs/autofs-ldap-auto-master		# list all known automount maps from ldap
    
    


    NIS

    ypcat (group, passwd)
    ypstart		# linux
    ypbind
    ypinit 		# client, specify machines to bind to
    		# server, specify whether master or slave
    		# Often messed up client, "up2date yp" to ensure running latest code with bug fixes.
    ypwhich		# tell what NIS server machine is bound to.
    passwd
    yppasswd (obsolete, for combatibility only)
    
    ypcat -k auto.master	(-k to display the key name too!)
    ypcat -k auto.direct	(the direct map, use /-)
    
    
    /etc/init.d/autofs	# most standard automount from auto.master, etc.  
    			# /net need to be enabled by uncommenting entry in /etc/auto.master .
    			# Some version of autofs (eg RH 4.0) have nasty bug that stale NFS mount
    			# may result in autofs daemon wanting to erase all files on server,
    			# at the very least, seen it cause lot of errors in syslog.
    
    /etc/init.d/amd		# an alternate for automount for /net/HOST/... paths
    			# install as up2date am-utils, and enable always start via ntsysv
    			# amd maybe a bit older than autofs and performance may not be as good.
    

    Network Connectivity



    mii-tool -vv		# determine NIC speed, duplex.
    ethtool -i eth0 	# list ethernet driver info
    
    Spawn login session on serial port:
    echo "7:2345:respawn:/sbin/agetty -h 9600 ttyS0" >> /etc/inittab
    echo "8:2345:respawn:/sbin/agetty -h 9600 ttyS1" >> /etc/inittab
    
    above does NOT redirect VGA to console, to do that, use
    (nothing will show on VGA output at boot):
    	[check zambeel notes, TBA]
    
    
    
    
    
    adding route:
    
    route add default gw 192.168.1.1
    
    sudo route add    -net 10.8.0.0/16  gw 10.8.21.1  # this worked, no need to add metric or dev, since machine is in local network of 10.8.21.x
    
    sudo route add    -net 10.0.0.0/16 dev eno1   # add local broadcast domain eg dual home machine w/ priv net
    sudo route add    -net 10.8.0.0/16  gw 10.8.17.1 metric 1 dev enp0s9
    sudo route delete -net 10.8.0.0/16  gw 10.8.17.1              enp0s9
    
    
    could try these if error above
    route add      -net 172.16.16.0 netmask 255.255.255.0 gw 172.16.0.1 eth2  # (but going out from first interface)
    route add      -net 172.16.16.0 netmask 255.255.255.0 gw 172.16.0.1 metric 1 dev eth2
    route add      -net 172.16.16.0 netmask 255.255.255.0 dev eth3
    route delete   -net 172.16.16.0 netmask 255.255.255.0 eth3
    
    
    dhclient	= get dhcp address from server for current client.
    
    dhcpcd		= slackware, start the dhcp client daemon, then interfaces 
    		  will get dhcp address.
    ifconfig hme0 dhcp		see purple book.  (Solaris?)
    
    
    irouted
    
    
    persistent static route, from: RH Doc permanent static route config for RedHat:
    
    /etc/sysconfig/network-scripts/route-ens3f0 ::
    
    10.8.0.0/16 via 10.8.17.1 dev ens3f0
    
    
    or /etc/sysconfig/network
    but NO mention of /etc/sysconfig/static-routes in the RH doc. (it was only mentioned here https://webhostinggeeks.com/howto/three-ways-to-configure-persistent-static-routes-on-linux-centos/ but unverified)

    Virtual Interface vs VLAN Tag
    
    eth0  is plain old default interface
    eth0:1 eth0:2 are virtual interfaces, with "traditional" single layer 2 eth config (ie no VLAN TAG)
    eth0.20 eth0.911 are interfaces with VLAN TAQ as required by switch.
    
    (switch could have "native vlan" that allow for default untagged traffic (eth frame is different between tag vs non tag eth frames.)
    
    
    adding virtual interface:
    ifcfg-eth0:1 
    
    
    adding interface with vlan tag 
    ifcfg-eth0.20
    ifcfg-eth0.911
    (no need to make this virtual, it automatically allow for multiple tagged interface per physical NIC.
    ie, there is never a need for eth0:1.911 or eth0.911:1, eth0.911 will be an additional interface to eth0 with the vlan id set)
    
    systemctl restart network
    -or-
    systemctl restart NetworkManger
    
    manual commands:
    modprobe --first-time 8021q
    # modinfo 8021q
    ip link add link enp0s3 name enp0s3.20 type vlan id 20
    ifconfig enp0s3.20 10.8.20.22 netmask 255.255.255.0 up
    
    ifcfg-enp0s3.20 , no need to be both virtual interface and vlan tag. it is already virtual.
    
    No need for "systemctl restart NetworkManger", though it doesn't hurt
    
    
    # ref: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configure_802_1q_vlan_tagging_using_the_command_line
    
    
    ipmi tag to vlan 20 (default is untagged):
    ipmitool lan set 1 vlan id 20
    
    
    Link Aggregation
    See http://www.cyberciti.biz/tips/linux-bond-or-team-multiple-network-interfaces-nic-into-single-interface.html
    Here is a script to setup bonding:
    #!/bin/bash
    
    # script to change machine with single eth0 to use bond0 (eth0+eth1)
    # completely automatic, generic for all hosts.
    # no testing of conditions, assume only 1 nic with 1 ip.
    # For RHEL 5.3 with bonding modules for kernel already pre-installed.
    
    # copy script to local drive so that it doesn't depend on network avail !!
    
    
    
    ModConfFile=/etc/modprobe.conf
    
    echo "alias bond0 bonding"                              >> $ModConfFile
    echo "options bond0 mode=balance-rr miimon=100"         >> $ModConfFile
    
    echo "#"                                                                                                >> $ModConfFile
    echo "# see http://www.cyberciti.biz/howto/question/static/linux-ethernet-bonding-driver-howto.php"     >> $ModConfFile
    echo "# Section 1 and 6 Switch config for more info of different modes."                                >> $ModConfFile
    echo "# balance-rr(0), balance-xor, boradcast: works with Cisco EtherChannel "                             >> $ModConfFile
    echo "# balance-xor is safer but less optimal than balance-rr"                                          >> $ModConfFile
    echo "# 802.3ad is the IEEE standard, aka lacp mode"                                                    >> $ModConfFile
    echo "# active-backup(1), balance-tlb and balance-alb modes do not require any specific configuration of the switch" >> $ModConfFile
    echo "#"                                                                                                >> $ModConfFile
    echo "# sed -i 's/^options bond0 mode=balance-alb/options bond0 mode=balance-rr/' $ModConfFile "        >> $ModConfFile
    
    
    # create a bond  interface
    echo '
    DEVICE=bond0
    NETMASK=255.255.255.0
    USERCTL=no
    BOOTPROTO=none
    ONBOOT=yes
    ' > /etc/sysconfig/network-scripts/ifcfg-bond0
    
    cat /etc/sysconfig/network-scripts/ifcfg-eth0 | grep ^IPADDR=10.140    >> /etc/sysconfig/network-scripts/ifcfg-bond0
    
    
    # update eth1 config
    echo '
    MASTER=bond0
    SLAVE=yes
    BOOTPROTO=none
    USERCTL=no
    ' >> /etc/sysconfig/network-scripts/ifcfg-eth1
    
    sed -i 's/ONBOOT=no/ONBOOT=yes/'        /etc/sysconfig/network-scripts/ifcfg-eth1
    
    
    
    # update eth0 config
    sed -i 's/^IPADDR=/#IPADDR=/'                           /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i 's/^NETMASK=/#NETMASK=/'                         /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i 's/^BOOTPROTO=static/BOOTPROTO=none/'            /etc/sysconfig/network-scripts/ifcfg-eth0
    echo '
    MASTER=bond0
    SLAVE=yes
    USERCTL=no
    ' >> /etc/sysconfig/network-scripts/ifcfg-eth0
    
    
    # next two lines need to run in sequence, network connectivity lost after modprobe
    /sbin/modprobe bonding ; /etc/init.d/network reload
    
    
    /etc/init.d/network status
    /sbin/ifconfig bond0
    /sbin/ifconfig eth0
    /sbin/ifconfig eth1
    cat /proc/net/bonding/bond0
    
    


    Firewall

    iptables (RHEL)

    For more details, see firewall.html and net.html#firewall
    
    /etc/sysconfig/iptables 	# firewall rule config file
    
    system-config-firewall		# GUI tool to set iptables firewall rules
    firewall-config			# use this instead of above in RHEL7 (since it use shorewalls)
    
    

    SAN Connectivity

    Basic SAN Commands

    Utility/Function Description
    fdisk Command used to create and manipulate partition tables.
    fsck Command used to check and repair a Linux filesystem.
    mkfs Command used to create a Linux filesystem on a device partition.
    insmod Utility used to dynamically load a single module into a running kernel.
    rmmod Utility used to unload loadable modules from the running kernel 
           if they are not in use and if other modules are not dependent upon those being removed.
    modprobe Utility used to load or remove a set of modules that can be either a single module or a stack of dependent modules.
    lsmod Utility used to list the currently loaded modules.
    lspci Utility used to display information about all of the PCI buses in the system and all of the devices connected to those buses.
    scsiinfo Utility to query information from a scsi device.
    
    Ref: Emulex/EMC Linux Guide.

    HBA - Emulex

    
    HBAnyware is an optional add-on utility, but it is not a driver req for FC functionality.
    Emulex HBA Driver is included in RHEL kernel since 4.1.  To verify:
    
    lsmod | grep lpfc
    modprobe -l | grep lpfc
    modinfo lpfc | fgrep version
    	# if installing HBAnyware + lputl, must find exact version matching lpfc driver !!
    
    
    CNA (convergent net+hba) driver on the other hand is not in kernel of std os distro.
    
    
    hbanyware		# Java GUI tool for emulex HBA, in /usr/sbin/hbanyware
    hbacmd			# cli
    
    sudo /usr/sbin/hbanyware/hbacmd listhbas	# show hbas on the host
    
    /usr/sbin/lpfc/lputil	# util to check config and coutner for Emulex HBA
    /usr/sbin/lpfc/lun_scan all	# check for new LUN, delete dead ones
    /usr/sbin/lpfc/lun_scan -r all	# supposed to check for size change of LUN, don't work in RHEL 5.2
    
    
    


    Restarting HBA driver
    Ref: p58 Unix user manual (by EMC) http://www.emulex.com/downloads/dell/drivers/linux-blade.html
    Run the following commands to stop the Emulex application daemons:
    cd /usr/sbin/hbanyware
    ./stop-hbanyware
    
    Run the following command to remove the Emulex Application
    Helper Module (lpfcdfc driver):
    /etc/init.d/elxlpfc stop
    
    Finally, unload the Emulex driver:
    modprobe -r lpfc
    To reload the Emulex driver, the Application Helper Module (lpfcdfc
    driver), and the Application Kit use the following steps.
    
    Load the Emulex driver:
    modprobe lpfc lpfc_log_verbose=0x10
    
    Reload the Emulex Application Helper Module (lpfcdfc driver):
    /etc/init.d/elxlpfc start
    
    Start the Emulex Application daemons:
    cd /usr/sbin/hbanyware
    ./start_elxdiscovery
    ./start_myserver
    
    

    HBA - Qlogic

    Driver typically install some command tools in /opt/QLogic_Corporation
    eg 
    scli	- san surfer cli
    qaucli	- convered network cli
    
    these can be run in a menu prompted interface.  see HBA info, WWN, BIOS version, etc.
    Can flash HBA using this command in linux and providing a .BIN file.
    
    getting driver to work in upgraded kernel was quite painful.
    Maybe should have avoided building the initrd image, which clobber with subsequent 
    kernel compilation...?
    
    The DMKS driver didn't work.  
    the Converged Network Driver in RPM worked out okay.  Had to hand edit modprobe.conf that 
    added module but didn't work in new kernel... then things worked.
    
    BIOS is not explicitly listed, but part of some package like converged console cli package.
    (and there there is a zip file that contain a .BIN (and DOS utility to install flash, but can be done in linux)).
    
    
    

    parallel scsi

    new LUN from SAN to a physical host, hba card can scan for new FC disk:
          /usr/sbin/lpfc/lun_scan all
    
    ESX direct raw lun mapping provides good old fashion scsi disk, not FC.  Command to use for scanning scsi bus is:
    	  echo "- - -" > /sys/class/scsi_host/host0/scan			# scan for new device, change host0 to appropriate controller
          echo 1 > /sys/block/sda/device/rescan					    # rescan for changed size, change sda to appropriate drive
    
    
    the hostX dir has to do with scsi id, but not necessarily assigned in same numbring.
    default 0:x device are in host0, but when i added scsi 2:10, 
    it was placed in host1.  I had skipped 1:x, presumably later on when this is added, 
    this would be assigned the next hostX dir.
    
    
    but /dev/sdc and /dev/sdd are not assigned as per scsi id number, but sequentially
    so sdc was 2:10, and sdd was 1:2   :(
    
    the scsi target id number can be found by looking at
    	ls -ld /sys/block/sd*/device
    
    
    Overall, in linux, scsi device in /dev/sdb, sdc, etc are not necessary static.  
    changes to scsi bus and boot order may make them scan at different order and 
    thus assigned different device path.  
    fs labeling and fstab mounting using such label circumvent around the problem.
    but no such labeling in raw scsi devices.
    
    emc powerpath provides a uniq /dev/emcpower* device path for the LUN.
    in its absence, can use udev naming rules.
    
    
    
    scsi_id : retrieve and generate a unique SCSI identifier
    	  the id presented should be consistent when presented to multiple
    	  host in a cluster environment.
    
    udev    : dynamic dev management  - seems to allow for mapping scsi_id 
    	  to specific path user desire, and make such path persistent 
    	  to the same physical disk independent of machine path, 
    	  since it uses id.
    
    linux machine path eg: /sys/block/sde 
    scsi id for the device can be retrieved by:
    	scsi_id -g -u -s /block/sde
    
    /etc/udev/rules.d/20-names.rules
    KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601607ba02900b63b5cf7c4b3e011", SYMLINK+="data10_d1_p%n"
    
    To test the rule w/o reboot, use the command "udevtest  /block/sde"
    
    Oracle ASM can use /dev/emcpowerX1 for data storage.  (need to have a partition in it).
    or use the symbolic name created by udev, like /dev/data10_d1_p1
    (udev create sym link at boot like /dev/data10_d1_p --> sde, /dev/data10_d1_p1 --> sde1)
    
    
    
    /etc/udev/rules.d/60-raw.rules   
    maps emcpowerX to /dev/raw/rawX so that scsi id reservation for voting can work.
    if no power path, then use scsi_id 
    eg:
    ACTION=="add", KERNEL=="emcpoweri", RUN+="/bin/raw /dev/raw/raw1 %N"
    ACTION=="add", KERNEL=="sd*[!0-9]", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601607ba0290056a27f08c6b3e011", RUN+="/bin/raw /dev/raw/raw1 %N"
    
    
    when the above rule runs, it will create "character special devices" path in 
    /dev/raw/raw1
    
    
    Oracle RAC uses /dev/raw/raw1 thru 5 to run its voting and quorum selection to power the cluster membership algorithm.
    No partitions are needed in such disk.  only scsi reservation commands are issued to it.
    
    
    
    

    Infiniband

    Info moved to infiniband.html

    RAID

    cat /sys/block/sda/queue/rotational
    0 = ssd
    1 = hdd
    

    mdRAID

    
    mdadm 
    
    cat /proc/mdstat 
    mdadm --detail /dev/mdNNN
    
    mdadm --detail --scan 			# could redirect output to like /etc/mdadm.conf to save config
    
    mdadm --create --verbose /dev/md127 --level=linear --raid-devices=3  /dev/sda1 /dev/sdb1 /dev/sdc1 
    
    
    have to assemble the device each time when running:
    mdadm --assemble /dev/md127 /dev/sda1 /dev/sdb1 /dev/sdc1 
    mdadm --assemble --scan
    
    mdadm --stop /dev/md127
    
    
    Ref: Linux RAID

    StorCLI

    storcli is the replacement for megacli, should be easier to use and script.
    eg use for LSI Logic / Symbios Logic MegaRAID SAS-3 3108

    For Dell PERC H730 controllers , their perclci seems very much like the storcli, but they did some crap in there so storcli binary won't work, have to use the perccli tool from dell. But same options.

    Virtual Disk should be the center of gravity for work to present a "LUN" from the raid controller to the host OS.
    
    /c0 		# is controller 0
    /c0/v0		# is controller 0, virtual disk 0, eg RAID 1 for OS
    /c0/v1		# is controller 0, virtual disk 1, eg RAID 6 for data
    /c0/eall	# eall = all enclosure.  e252, e8
    /c0/eall/sall	# sall = all slot.  
    /c0/e252/s1	# is controller 0, enclosure 252 (really just channel number), slot 1 (really hd on the channel).  ie hd 252:1 EID:Slot
    /c0/e8/s7	# is controller 0, hd at EID:Slt of 8:7 (eg RAID 6 volume for data)
    
    
    storcli64 /c0/eall/sall | grep Model\ Num	# get all hard drives model on controller 0
    
    storcli64 /c0 add vd raid6 drives=8:0-3		# create a LUN with disk 8:0 thru 8:3 (SID:Slt)
    
    storcli64 /c0/v1 start migrate type=raid6 option=add drives=8:4-7	# add disks 8:4 thru 8:7 to existing /c0/v1 VD (via migrate process)
    									# raid type remains the same
    
    storcli64 /c0/v1 start migrate type=raid5	# change from prev raid6 to raid5.  no disk addition needed.
    
    storcli64 /c0/vall show migrate			# show add progress (eg take 3 days to add 4 disks, 18 days to restripe from raid6 to raid5)
    
    
    storcli64 /c0 show				# constroller info, new disks show as UGood.
    storcli64 /c0/v1 show all			# display VD/LUN info
    storcli64 /c0/vall show init			# init is not always req.  will be done automatically in big for VD with ~5+ disks
    						# but before creating LVM/FS, may want to force it to start on small VD, just to be sure
    storcli64 /c0/vall show cc			# consistency check, seems to req that init have been done.  recommend 1+/month
    
    storcli64 /c0/v1 start cc			# cc (or init) can run in bg or fg, pause, resumed, etc.
    storcli64 /c0/v1 pause cc
    storcli64 /c0/v1 resume cc
    storcli64 /c0/v1 stop cc
    
    storcli64 /c0/v2 set name=dev_sdc		# set name of VD, best to put linux path on it for easy mapping.
    						# hopefully os don't remap it after reboot...
    
    # lsblk
    
    # lsscsi: 								# storcli64: 
    [0:0:8:0]    enclosu SMC      SC846P           0c1f  -        
    [0:2:0:0]    disk    AVAGO    MR9361-8i        4.68  /dev/sda 		# /c0/v0
    [0:2:1:0]    disk    AVAGO    MR9361-8i        4.68  /dev/sdb 		# /c0/v1
    [0:2:2:0]    disk    AVAGO    MR9361-8i        4.68  /dev/sdc 		# /c0/v2
     c:  vd 
    
        scsi id/path map to raid controller path ::
    ie, first number in lsscsi is likely the LSI/Avago controller number c0
        third number in lsscsi is likely the LSI/Avago controller Virtual Disk number eg VD1
    
    
    lsscsi -d	# device major and minor number
    lsscsi -w -s	# wwn, size
    lsscsi -i 	# -i = udev derived scsi_id, probably picked by zfs
    lsscsi -w -i -d	# 
    
                         -w                                   -d        -i                                 -s
    [0:0:11:0]   disk    0x5000c5009443358f          /dev/sda [8:0]     35000c5009443358f                  10.0TB
    [0:2:0:0]    disk    AVAGO    MR9361-8i    4.68  /dev/sdy [65:128]  3600605b00cd891a021fd7ddc04f32b71   1.02TB
    
    
    
    Bluemoon settings
    
    /opt/storcli64 /call show bootdrive	# see which VD is used to boot the machine (OS drive)
    
    
    /opt/storcli64 /c0 show migraterate      # check migration rebuild %, default is 30
    /opt/storcli64 /c0 set  migraterate=90   # upping to 90% on idle drive doesn't improve progress speed.
    
    /opt/storcli64 /c0/e8/sall show rebuild
    
    /opt/storcli64 /c0/v1 set wrcache=wt    # set write cache ## p221 of pdf has cmd for other settings
    
    
    ### p190 has controller properties.
    /opt/storcli64 /c0 show alarm 		# whether controller alarm is enabled (probably whether alarm is usable, not whether it is beeping) [only for 9750 and 9690a)
    
    storcli64 /call show event		# display controller events, presumably fail drive, error status, etc would be listed here
    
    
    p210 drive locate command
    /opt/storcli64 /c0/e8/s1 start locate
    /opt/storcli64 /c0/e8/s1 stop  locate
    
    
    storcli64 /c0/cv show all   		# check battery backup
    storcli64 /call/dall show all 		# get disk detail
    
    
    ## example commands and output in
    ## http://fibrevillage.com/storage/700-storcli-useful-commands-with-examples
    
    
    
    Ref:

    MegaRAID

    megaRAID controller is often used in white box low cost "home made" storage arrays.
    megacli is the main command to manipulate such raid.
    there is a gui called raid manager.
    /usr/local/MegaRAID\ Storage\ Manager/./startupui.sh    # GUI
    
    /opt/MegaRAID/MegaCli/MegaCli64
    
    # ?? not sure what these params are exactly:
    # -a    = array controller number.  
    # -a0   = array controller 0
    # -aAll = all array controllers (if there were multiple)
    # -l    = ??
    # -L    = Logical Disk?
    
    MegaCli64 -LDInfo -lall -aall	# array info and state
    MegaCli64 -LDInfo -lall -aall | egrep --color "Adapter|^Virtual Drive|Name|^RAID Level|^Size|State|Number Of Drives"
    
    MegaCli64 -AdpAllInfo -aALL	# HW info, whether BBU (Battery Backup Unit) is present, error counters 
    MegaCli64 -AdpAllInfo -aALL	| egrep --color "^Adapter|Product Name|Serial No|FW Version|Mfg. Date"  # controller hw info
    MegaCli64 -AdpAllInfo -aALL     | egrep --color "^Adapter|Virtual Drives|Degraded|  Offline"		# virtual  drive state
    MegaCli64 -AdpAllInfo -aALL	| egrep --color "^Adapter|Physical Devices|Disks|Critical|  Failed"	# physical drive stte
    
    
    MegaCli64 -LDInfo -Lall -aALL | egrep  --color -i "virtual|state|adapter"
    
    
    MegaCli64 -PDInfo -PhysDrv [1:2] -aALL | more
    MegaCli64 -Pdgetmissing -a2		# see if there are missing drive in adapter 2
    
    MegaCli64 -PdList -aall | grep --color -i firmware	# get firmware info
    MegaCli64 -LdPdInfo -aAll # logicak disk info
    
    lsblk
    
    ## how to find out which linux /dev/sdX maps to which VirtualDisk (LUN) on the RAID controller ??
    
    
    MegaCli64 -ShowSummary -aAll
    MegaCli64 -EncInfo -aAll						# enclosure info
    
    MegaCli64 -AdpEventLog -GetEventLogInfo -aAll	# event log
    MegaCli64 -AdpEventLog -GetEvents
    
    MegaCli64 -AdpBbuCmd -aAll 						# Backup Battery Unit (BBU) state
    MegaCli64 -AdpBbuCmd -GetBbuStatus -a0
    MegaCli64 -AdpBbuCmd -GetBbuDesignInfo -a1
    
    
    
    useful megacli cmd blog by Frederik :
    
    # eg of converting from RAID6 to RAID5 to get more space! 
    /usr/local/sbin/MegaCli64 -LDRecon -Start -r5 -L0 -a0
    echo "- - -" > /sys/class/scsi_host/host0/scan			# scan for new device, change host0 to appropriate controller
    echo 1 > /sys/block/sda/device/rescan				# rescan for changed size, change sda to appropriate drive
    
    
    MegaCli64 -CfgLdAdd -r0'[32:3]' -a0     # create new config for Logical Disk with raid0 (with disk 32:3 ie one plain disk)
    										# -a0 = on array controller #0
    MegaCli64 -CfgLdAdd -r5'[252:5,252:6,252:7]' -a0		# create new raid5 LUN
    
    MegaCli64 -LDRecon -Start -r5 -Add -PhysDrv[32:3] -L0 -a0	# extend existing LUN with 1 new Physical Drive
    MegaCli64 -LDRecon ShowProg L0 -a0				# view reconstruction progress (when replacing bad drive?, or extending VD?)
    MegaCli -LDGetProp -DskCache -LAll -aALL			# check physical HD cache, recommend enable iff have UPS
    MegaCli64 -PDList -aALL  					# find out about physical HD info
    
    MegaCli64 -PDList -aALL | egrep --color "Adapter|Enclosure|^Coerced Size|Device Id"
    MegaCli64 -PDList -aALL | egrep --color "Adapter|Enclosure|^Coerced Size|WWN|Inquiry Data|Device Id"
    
    # SAS Addresses are subsequent to WWN, eg:
    #            WWN:  5000C500631B3D3C
    # SAS Address(0): 0x5000c500631b3d3d
    # SAS Address(1): 0x5000c500631b3d3e
    #                                  ^
    
    
    MegaRAID level explanation" server fault:
    
    /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL | grep RAID
    
    ['Primary-0, Secondary-0, RAID Level Qualifier-0'] = RAID-0
    ['Primary-1, Secondary-0, RAID Level Qualifier-0'] = RAID-1
    ['Primary-5, Secondary-0, RAID Level Qualifier-3'] = RAID-5
    ['Primary-6, Secondary-0, RAID Level Qualifier-3'] = RAID-6
    ['Primary-1, Secondary-3, RAID Level Qualifier-0'] = RAID-10
    
    Primary-1   = "Mirrored array"
    Secondary-3 = "Spanned 0x03 A combination of stripping and concatenations involving Basic VDs of different sizes."
    RAID Level Qualifier-0 = "Mirroring across two extents"
    
    
    Additional info at
  • megaraid.html
  • MegaCLI cheetsheet
  • #hp

    HD Partitions

    From DOS days, disks are typically partitioned. Linux typically have up to 4 primary partions, and extended partitions if necessary. Technically, ext2/3 can be created on the whole disk without partition, ie, one can do
    mkfs -t ext3 /dev/sda
    instead of defining /dev/sda1, /dev/sda2, etc.
    But this is not recommended, another sys admin may think the disk is not in use. Partition also help add flags to indicate what fs is in a given partition.
    fdisk is the traditional command for manipulating partitions. It is reasonably user friendly, but can't handle large drive (2+ TB? or 8+ TB?). For really large drive, gnu parted will need to be used. parted can manipulate some fs, should be able to resize FAT fs/partitions.

    
    sudo parted /dev/sdb print		# show partition table
    sudo parted /dev/sdb mklabel gpt	# initialize disk as gparted, erase all existing partition!  then add label
    sudo parted /dev/sdb mklabel msdos	# initialize partition using msdos mbr
    
    sudo parted /dev/sdb mkpart primary ext4 5 263092224cyl 
    
    sudo parted /dev/sdb mkpart primary ext3 1 2048		# shoudl create a 2G partition as ext3 start at 1 (MB) and end in 2048 (MB, the default unit).  cannot specify partition number, they will be sequentially created.
    sudo parted /dev/sdb mkpart primary amufs 1 100%	# create a primary part
    	# instead of 100%, can use:
    	# cyl eg 123456000cyl
    	# mb     123456000    (ie, default is mb, finding disk size is tricky)
    	##   amufs cuz parted don't have lvm and don't want ext2/3
    sudo parted /dev/sdb set 1 LVM		# optional
    	## the LVM flag isn't likely relevant, but will be printed in output
    	## to help remind sys admin it is LVM and not say ext3 fs.
    
    
    parted /dev/sdz mkpart primary linux-swap 32GiB 40GiB # type id 82 # use GiB instead of MB so that it align correctly
    parted /dev/sdz mkpart primary ext2 40GiB 52GiB       # type id 83
    parted /dev/sdz mkpart primary ext2 52GiB 98%         # 98% of remaining space.  leave a bit at the end for possible conversion to gpt
    parted /dev/sdz align-check optimal 1
    parted /dev/sdz align-check optimal 2	# need to check alignment of each partition one at a time
    
    
    # repartitioning disk carefully could leave data seems untouched :)
    # need partition number and end size.  
    # if omitted will prompt, can use size in GB or % in addition to sector.
    # -1 is last block on disk.  
    sudo help parted # display help.  
    
    # eg resize partition 1, use 100% of avail space (eg after VM hd grew):
    sudo parted /dev/sda resizepart 1 100%  
    
    
    sudo parted /dev/sdb rm 1		# remove partition 1
    
    
    Careful if using fdisk on big drive where parted was needed to create partition table.
    old versions of fdisk had screwed things up on gtp-partitioned disks.
    
    fdisk -l		# see list of drives
    fdisk /dev/sda		# modify a specific drive
    			# m for help
    
    example fdisk vs parted output below.
    fdisk allowd a gap in partition number, probably not doable with parted.
    Linux RAID is type "29"
    
    #         Start          End    Size  Type            Name
     1         2048       475135    231M  Linux RAID      
     2       475136    253263871  120.6G  Linux RAID      
     3    253263872    256542719    1.6G  Linux RAID      
     5    263092224    468862094   98.1G  Linux RAID     
    
    Number  Start   End    Size    File system  Name  Flags
     1      1049kB  243MB  242MB   fat16              raid
     2      243MB   130GB  129GB                      raid
     3      130GB   131GB  1679MB                     raid
     5      135GB   240GB  105GB                      raid
    
    
    

    LVM

    
    File System:	FS: one ext3 fs on top of a given LV.
    Logical:	LV: many virtual/logical partitioins on top of a single VG.
    Aggregation:	VG: volume group, encapsulate many PVs. 
    Physical:	PV: place extends "markings" on a partition or whole disk 
    partition	sda1 or sdd
    kernel          -
    device		/dev/sda
    
    
    In VM environment, best to add a separate virtual disk, 
    and add the new disk to LVM using pvcreate+vgextend.
    Can use VM's feature to extend the existing LUN, 
    but would still need to use fdisk to create a new partition and then
    add this partition to LVM using pvcreate+vgextend.  
    
    (RHEL 5.4 need reboot to see expanded LUN, LVM partition can
    be an extended partition, but after fdisk creation of volume, reboot needed
    before pvcreate can mark the partition for use with vgextend.
    Newer Linux kernel can scan for new space without reboot, 
    
    echo "- - -" > /sys/class/scsi_host/host0/scan		# scan for new device, change host0 to appropriate controller
    echo 1       > /sys/block/sda/device/rescan		# rescan for changed size, change sda to appropriate drive
    
    pvs
    pvs -o +pe_start		# show offset, new lvm default to 1m and usually good enough to get correct alignment
    
    lvs --segment
    lvdisplay -vm 			# show segemts and write policy (linear vs stripped)
    
    iotop
    iostat 5 
    iostat -x -k 1
    
    
    
  • Percona alignment theory
  • SysAdmin Cast linear vs stripped LVM (but not sure if it is necessary or modern lvm takes care of it automatically?)
    PV on whole disk or partition?
    I used to like partition, just so that fdisk -l give more useful info.
    Now I am all for LVM using the whole disk without partition markings.
    This is because in a RAID controller environment (and lesser extend SAN), 
    the VirtualDisk (LUN) could grow (cuz added disks to the Raid Group).
    pvresize can then handle expanding the PV (it adds Extends), but:
    - if no partition, then pvresize will just expand to the new end.  not too painful.
    - if PV on partition, fdisk will see more space.  but partition is static.  
      Probably just add a new partition, then pvcreate on the new partition.
      (not sure if there are any way to enlarge existing partition via fdisk, maybe via gparted)
      Overall, having partition for LVM that wants to grow add an layer of complication.
    After pvexnted (or pvcreate), use the regular vgextend, lvresize, xfs_grow, etc to get more space.
    
    https://www.linuxquestions.org/questions/linux-software-2/adding-disk-space-to-root-logical-volume-with-pvresize-and-lvextend-762333/
    has a thread where pvresize on partitioned raid disk was done on LVM2 back in 2009 
    for more info, search on ServerFault for "lvm pvresize"...
    
    Separately, whole disk without partition does not create possible alignment problem.
    Modern fdisk would create partition that is correctly alignment with disk sector/block size (not so with older fdisk).
    Not using partition avoid this problem.
    ZFS does create partitions, and ZFS require a database of disks actual geometry (instead of "lie" it reports to windows-minded stack).  Lack of such correct definition could/would result in misalignment and reduces performance.
    
    
    LVM ccommand
    
    system-config-lvm			# GUI for LVM.  Pretty neat.
    lvm					# interactive shell for LVM, all cmd avail as stand alone.
    
    pvdisplay				# display info such as PV size, PE size, avail PE.
    pvcreate /dev/sda1			# create pv on partition, def RHEL 4 setup  
    pvcreate /dev/sdd			# whole disk allocated to LVM, can't hold swap or /boot, but typically ok for data volume
    
    pvresize /dev/sdd			# change block device , while LVM is live
    
    vgdisplay				# display vol grp info such as lvm/lvm2.
               -v				# list all LV on the vol group, which PV it uses.
    vgcreate VolGroup00 /dev/sda2		# create a new VG, this is the def in RHEL 4 setup.
    vgextend  VolGroup00 /dev/sda5
    vgreduce VG PV				# reduce number of disks in vol grp 
    vgreduce DataVG00 /dev/xvdh		# need to lvreduce to free up space till PV is free no extent alocated on it before reducing VG
    					# need to pvremove dev before removing it from say AWS
    
    vgremove  VolGroup00			# remove the whole volume group
    
    lvscan						# brief list of LV and their size.
    lvdisplay					# show info of all LV: path, LV size, Status, which VG it resides in.
    lvdisplay /dev/VolGroup00/LogVol01		# display specific info of a single LV (instead of all LV)
    lvcreate -n LogVol00 --size 500M VolGroup00	# create a new LV called LogVol00 of size 100 MB
    
    lvextend -L+100M /dev/VolGroup00/LV01		# extend existing LV (hosting swap of fs) by given size
    						# -L20G would set exact size of 20G
    lvremove /dev/mapper/VolGroup00/LV01		# remove unused LV
    
    mkfs -j /dev/VolGroup00/LogVol02	# create fs on the LV (-j will use jounal, ie ext3)
    					# NOTE: old kernel may give error.  2.6.16-1.2069_FC4 works.
    mkswap /dev/VolGroup00/LogVol02		# create swap partition on the LV 
    					# Doesn't seems to work, actually confuses kernel to the point that
    					# all subsequent mkfs command will also be refused.  BUG??!!
    
    
    e2fsadm					# extend or reduce fs and LVM1 at the same time 
    					# but not avail with LVM2 (eg FC4, RHEL 4)
    
    ext2online /dev/VolGroup00/LogVol00	# extend ext3 FS online (for LVM 1 or 2)  (RH4)
    ext2online /				# can use mount point instead of dev path
    ext2resize				# change ext2 FS after it has been UNMOUNTED.  ?? does not exist??
    resize2fs /dev/VolGroup00/LogVol00 	# RH 5, extend the FS to max space that the LVM has allocated.
    					# op done online in RH5, no umount needed.
    
    resize2fs /dev/VolGroup00/lv101 1000000	# reduce ext2/ext3 FS to size to exactly 1,000,000 bytes. 
    resize2fs /dev/VolGroup00/lv101  90G    # can use G M k prefix for size
    					# UMOUNT+FSCK FIRST
    
    xfs_growfs /mount/point 		# expand xfs, to max space of eg LV
    xfs_growfs /mount/point -D size
    
    lvreduce -L1G /dev/VolGroup00/LogVol00	# reduce the LV the FS is on 
    					# (Don't make it smaller than FS size!)
    					# After these procedure, there maybe some left over space,
    					# if using ext3, can use ext2online to reclaim the space.
    					# ext2 somehow can't be re-expanded, use ext3 !!
    					# use tunefs -j to convert to ext3 :)
    	
    
    /etc/fstab				# typically list which LV dev used by which FS
    ??	 				# no cmd to display what LV is being used by which FS :(
    
    
    Importing a new hard drive that has LVM:
    
    vgexport VolGrpName			# export a volume so that it can be imported (work on already "migrated" hd)
    vgimport VolGrpName			# import for use
    lvscan					# newly imported volume group will be listed as inactive
    vgchange -ay VolGrpName			# activate volume group
    mount -t ext3 /dev/to/fs /mnt/myfs	# mount the file system on the newly imported volume
    
    
    Moving LVM volume/disk group to another system
    See: tldp. Essentially:
    umount   /mount/to/FS
    vgchange -an VGNAME
    vgexport     VGNAME
    
    # move disk
    
    pvscan
    
    vgimport     VGNAME
    vgchange -ay VGNAME	# activate the Volume Group
    
    mount ... 
    
    
    Removing LVM stuff
    sudo umount /data
    sudo lvremove /dev/mapper/VolGroup02-LV_data
    
    sudo vgremove VolGroup02
    sudo pvremove /dev/sdb1
    
    
    Ref: http://www.redhat.com/magazine/009jul05/features/lvm2/

    Software Mirror Boot and/or OS volume

    There are several ways to setup software mirror for the OS volume (eg /) and/or the /boot volume.
    For simple server just needing mirror across two disks, best to let installer do the mirror during initial setup. Then the fact that it is not using RAID controller is largely transparent in daily sys admin task. Maybe RAID controller is warranted only when doing storage server with many disks, but then ZFS maybe a better choice there. RAID is also useful if want to keep CPU free. Also see discussion in hardware RAID versus mdadm versus LVM-striping
    summary:
  • sw raid tends to be more flexible and easier for most sys admin to manage.
  • hw raid offload load and let cpu do other work.
  • hybrid hw + sw may give convinient large fs, but then painful in recovery situations. should avoid.

    Installer setup mdadm mirror for /boot FS and LVM volume for OS FS
    This is the easiest method, use when setting the machine from scratch using the RedHat installer (anaconda).
    See CentOS 7 Installation with LVM RAID 1 – Mirroring by tuxfixer, with screenshots of interactive GUI install.
    or How to create LVM on Software RAID (mirroring) during installation of Red Hat Enterprise Linux using kickstart file?(RedHat, login req)

    Overall, installer wiil:
    1. select two disks for mirror during disk selection step of install process.
    2. /boot is setup as RAID device (mdadm)
    3. swap is a LVM volume group of RAID 1 - create new "centos" VG -- LV is centos-swap
    4. / is a LVM volume group of RAID 1 - re-utilize "centos" VG -- LV is centos-root

    OS update don't need special steps. /boot is mirrored constantly by mdadm.
    Check on grub.conf though.
    So other than initial setup, normal operation is oblivious whether a HW RAID controller doing the mirror or SW mdadm. Disk recovery would be different, and some pre-emptive health check using smartctl, lmsensors and disk scrubbing.

    Verify mirror setup after machine is build:
    cat /proc/mdstat 	# should see two mdNNN device
    blkid | grep boot 	# see 2 /dev/sd* disk and 1 /dev/mdNNN mdadm dev
    pvdisplay		# see single PV on the /dev/mdNN  mdadm dev, so LVM don't need to know about mirroring.
    vgdisplay		# see single VG called "centos"
    lvdisplay  		# see typical LV setup for a RHEL machine.  again, these don't know they are mirrored by mdadm
    
    
    mdadm mirror whole disk - manual config after OS had been setup
    See How do I convert my root disk to RAID1 after installation of Red Hat Enterprise Linux 7? (redhat, login req)
    The target result is essentially same as above done by the anaconda installer. However, since exinsting partition/volume/fs exist, some data movement dance is required, thus taking many tedious steps to do. Characterisitcs
    1. This method use mdadm and mirror the whole disk (actually, multiple partitions on each disk that are raid devices)
    2. quite a number of steps, to be done after initial OS is setup
    3. mdadm mirror for /boot using /dev/md* device
    4. Then run LVM on top of the /dev/md* mirrored disk. so LVM doesn't know that it is mirrored. /boot and whatever volume that is placed inside the LVM get the underlaying mirroring done transparently.
    5. still need to update grub manually
    Commands
    mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1 --metadata=1.0
    ...
    pvmove /dev/sda2 /dev/md1   # move VG from regular disk to md device (mirrored)
    ...
    dracut -f --mdadmconf
    



    LVM mirror only OS volume(s)
    See How do I convert my LVM root disk to LVM mirror after installation of Red Hat Enterprise Linux 7? (redhat, login required)
    The mdadm methods above are likely better. But this does not need to use mdadm if doesn't want to deal with that. But leave /boot non mirrored and need manual sync each time there is an update.
    Also, if doing mirror after OS has been setup, LVM-only approach req less acrobatic on the existing data thus a bit easier to carry out.


    Characterisitcs
    1. It uses mirror at the LV level (lvconvert).
    2. Creates mirror (legacy) or raid1 (rhel6,7) segment types to hold /, swap, etc.
    3. /boot is left to be rsync manually.
    4. grub need to be setup for each boot disk/partition
    5. Long term require more manual maintenance
    6. Maybe easier to boot up in disaster scenario, assuming homework was done to mirror /boot and had grub up to date.
    7. overall, probably not the best solution.
    Terminology
    1. linear volume (single lvm with NO RAID?)
    2. segment type (aka LV type) --type
    3. mirrored volume (used up to RHEL 6, considered legacy in 2018)
    4. raid1 (rhel7 default)
    Commands
    vgextend system_vg /dev/vdb2     # add normal disk device into LVM purview
    
    lvconvert -m1 --type raid1 /dev/system_vg/root /dev/sdb     # create mirror at LV level, using specific disk that LVM has control of.
    
    
    lvconvert -m1 datavg/testlv
    # converts from linear to mirrored volume 
    
    
    lvconvert -m1 datavg/testlv /dev/sdc
    # remove mirror, /dev/sdc is the device that is DETACHED
    
    
    lvs -a -o+devices
    
    dmsetup table
    
    
    work be for RHEL7
    
    convert single disk EFI boot to sw raid AFTER OS install
    If using EFI, may have to check into this: < A HREF="https://access.redhat.com/solutions/4194011">How to convert single disk EFI boot to software RAID after installation (redhat "solution in progress").

    File System Related Command

    See also: fs for loopback mount, samba, etc.
    mkswap  /dev/hdd3			# def RHEL 2.1 swap dev
    mkswap  /dev/VolGroup00/LogVol01	# def RHEL 4   swap dev
    swapon  -s 				# list swap dev (eg, which LV is swap partion hosted on)
    swapon  -a 				# turn on  all swap partitions/files
    swapoff -a 				# turn off all swap partitions/files
    
    
    
    mkfs -t ext3 /dev/hdd2	# create a new fs on ide drive secondary slave.
    			# probably only create as ext2;  then:
    tune2fs -j /dev/hda3 	# add journaling to it via tune2fs  (manual for ext2 in 2.1)
    			# (probably automatic/default in ext3/as 3.0)
    
    e2label /dev/hda3 [Partition-Label]
    		view/change label of a partition
    		used in /etc/fstab LABEL=/label-name for mounting
    		so that mapping does not have to be maintained in fstab
    findfs LABEL=/boot	# determine which disk host a given FS with the assigned label.
    
    raven:/usr/local/bin/wshaper  traffic shaping and throttle specific connection bandwidth usage.
    
    du -kxS * 	= display space usage Summary for only (x) one file sys (ie local)
    				linux only
    du -kl		= local disk in solaris?
    
    df -kl 		= report fs space usage of local fs, solaris and linux
    
    ncdu		= enhanced du, but  like du, still doesn't store/cache result, so take long time to scan.  but at least interactive drill down helps while program is not closed.
    
    
    
    

    Undelete and data recovery tools

    There is no undelete in XFS or Ext or most unix FS.
    However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.
    Umount the file system as soon as realize needing to do data recovery. This help preserve as many inodes and data blocks before they get overwritten by new files.
    If hardware is failing, dd the disk/partition image to a new device and recover from that.
    If accidentally deleted lot of files from rm -r, proceed carefully, keeping the FS mounted as RO is okay by most tool. Restore to new location so as not to overwrite any block.
    ext4magic
    ext4magic is the best undelete tool for ext I have used so far (had more success undeleting files from ext4 fs than extundelete).
    apt install ext4magic # avail in ubuntu
    ext4magic /dev/sda3 -L # list recoverable files
    ext4magic /dev/sda3 -r
    ext4magic /dev/sda3 -m # works when FS still mounted RO, find more files than shown by -L
    recovered data in ./RECOVERDIR
    Was not successful in defining syntax to undelete only one specific file
    extundelete
    ext4 is journaled 
    
    extundelete will scan the journal, inode, and retrieve files.
    But FS should be unmounted ASAP so that data are preserved as much as possible.
    eg run:
    mount /dev/sdb1 /RESTORE	# eg new external hd to store restored files
    cd /RESTORE
    extundelete /dev/sda3 --restore-all 	
    would retrieve all files it can undelete, and save in current dir, 
    under a dir called RESTORED_FILES
    it will preserve original directory structure and filename.
    
    
    EPEL have this package for CentOS7:
    extundelete-0.2.4-6.el7.x86_64
    
    if compiling from source, ensure these packages are installed:
    
    e2fsprogs-libs-1.42.9-10.el7.x86_64
    e2fsprogs-devel-1.42.9-10.el7.x86_64
    e2fsprogs-1.42.9-10.el7.x86_64
    
    
    photorec
    photorec is a signature based recovery tool that work largely independent of FS (thus work for FAT, NTFS, ext2, etc, but not RaiserFS because it store small amount of data in the b-tree itself for better performance).
    It does more than AV datas. Can recover zip, txt, mathlab mdl, CAD, pdf, ps, etc. It handles some 300 file families
    Unfortunately it does *NOT* recover file name or directory structure. it will determine extension from the file magic/signature. Scripts exist that help sort files by type into directories, recover filename from EXIF, MP3 tags, etc.
    qPhotoRec is a GUI.
    Static link binary of TUI for both PhotoRec and TestDisk avail from single tar.bz2 at cgsecurity download.
    TestDisk
    TestDisk use a TUI to recover recover parition table and undelete file.
    It also undelete files, presumably scanning FS table to recover deleted file, can provide name and directory structure.
    For Ext, may work similarly to extundelete, ext4magic.
    Handles other FS type like FAT, NTFS, HFS+.
    Note that EXT is listed under "MS DATA" partition type.
    It recommends using PhotoRec if desired file is not undeleted.
    Commercial tools
  • SysDev lab sale two product lines. I wonder if they are really just different in the UI skin and menus. Both use a GUI, and recover XFS, EXT2/3/4, and other.
    They were the only tool I found (July 2018) that preserve filename and directory structure for XFS.
    The base version likely suffice for recovery from accidental rm on files. RAID version could be used if destroyed raid volume? and the Pro version has (maybe) gimmicky viewer tools or on-the-fly edit of FS? Trial version works, but restrict recovery size to 700K per file.
    There is a IntelliRAW feature, that just scan for say picture or video, file will have no name and could be truncated. Useful only if it portion recovery of content is useful. For structured data file this won't be all that useful.
    On retrospect, maybe similar to PhotoRec, but need to define rules, so not played with.

  • R-Linux Free undelete tool for ext2/3/4. /usr/bin/rlinux. GUI. Has ability to create an image of the partition needing data recovery. For my ext4 thumbdrive test this tool worked horribly. scanned and found old FAT files and recovered none of ext4 files. Didn't produce any filename or directory structure.
  • R-studio Data Recovery. Same GUI as r-linux, with support for additional FS types: Ext2/3/4, HFS, UFS, NTFS, exFAT, but NO XFS!!. Demo version limit to 256KB per file.
    Fedora rpm worked in centos7. run: /usr/bin/rstudio
    Probably less useful tools
    Ref: XFS FAQ
    stack exchange


    XFS

    
    system boot up doesn't do any of the check.
    only do mount...
    only run the below if have suspect of fs consistency problem.
    
    
    mount and umount the system cleanly before running the check, 
    so that xfs log is in proper state
    
    
    xfs is a parallel fs.
    parallelism is governed by agcount (aggregate group count).
    each ag is a parallel thread.
    But don't set agcount per cpu thread, but on number of spinning disk.
    
    agcount of 4 to 16 are common.  really large array could have larger number.
    don't have agcount in thousands, it will have detrimental impact on performance.
    
    # use xfs_info to see agcount.  if FS grew from GB to TB, it may have too many ag for efficient work.
    # when too many ag (eg agcount=5295839, agsize=6336 blks), kworker spends many minutes scanning thru them before actual file operation can take place.
    # recreate the fs  with eg agcount=125, agsize=268435328 blks and things will be much faster
    # xfs_repair is equally faster and more "responsive" with a more normal agcount (hundres to thousands).
    # agcount * agsize = FS size.  largest agsize is 1 TiB, thus agcount on reallly large fs would be in hundreds.
    # xfs tuning per XFS fag is said to be largely not necessary, as default works well most of the time.
    # http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
    # agcount = thousands means write may have to consider all such AG to find optimal place to write.
    # when not catched, this result in lots of disk head seek.  
    # see http://xfs.9218.n7.nabble.com/Re-xfs-very-slow-after-mount-very-slow-at-umount-td424.html
    # ideally, agcount ~ number of disk strips (cpu core should not matter, as the slow component is disk)
    # thus, agcount of 4 to 16 are common number.
    
    # small agsize also means large files will be divided into many ag... thus creating fragmentation
    # (and lots of search for ag to use)
    
    # xfs was orig written by SGI for supercomputer, which operate on large files
    # so lots of small file access may not be xfs' strength.
    
    
    xfs_info				# see fs info, like agcount, agsize, sunit, swidth
    
    xfs_repair			    	# fs must be unmounted, or resulting fs maybe inconsistent (per RHEL7.4 man page)
    xfs_repair -n /dev/datavg/datavol	# dry run method, but (skip?) extended attributes
    
    xfs_check				# deprecated and not avail in rhel7
    both of these commands use xfs_db to do its work.
    
    
    xfsmisc/xfsstats.pl -f			# stats, not avail for rhel7
    
    
    # http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide//tmp/en-US/html/ch02s07.html
    Use the -p option to xfs_bmap to view unwritten extents.
    # xfs_io -f -c 'resvsp 0 10m' -c 'bmap -vp' /tmp/foo
    
    
    
    xfs_metadump
    xfs_mdrestore
    these commands can be used to make copy of the xfs metadta (which is also journaled) and then can then use xfs_repair -L on such backup image to see effect and decide whether to proceed with the repair.
    serverfault Ref:
    XFS vs EXT3


    ZFS

    ZFS is a combination of LVM and file system into single fused storage function.
    At the same time, it also bring many enterprise storage system features to the open source world. Originally developed by Sun for Solaris, Oracle can't backtrack it. CDDL stopping many distribution from including it in their .ISO. Cannonical still ship it with Ubuntu and provide support for it. Has yet to be sued by Oracle :) Centos can get a zfs.repo and get it. May still need some kernel modif (DMKS? no need to manually compile kernel?).
    ZFS basic cmd
    
    zfs list		# 
    zfs mount		# display currently mounted zfs fs.  may as well use zfs list
    
    zfs share -a		# share all zfs, done automatically when zfs "boot"
    	sharesmb=on 
    	sharenfs=on
    
    
    zpool list
    zpool status		# list all disks that make up a pool and their status.  how "raid group" is defined.
    
    
    ZFS pool creation
    Many example from https://calomel.org/zfs_raid_speed_capacity.html
    
    zpool create POOLNAME drive		# single drive, so raid0 :)
    zpool create storage  /dev/sdc
    
    
    zpool create POOLNAME drives		# multiple drives, raid0 (no redundancy)
    zpool create storage  da0 da1 da2
    
    
    zpool create POOLNAME raidz drives... 	# raidz (raid5) with 3 drives (loose 1 drive for redundancy)
    zpool create storage  raidz da0 da1 da2
    
    zpool create POOLNAME raidz2 drives...	# raidz2  (raid6, loose 2 drives for redundancy)
    zpool create storage  raidz2 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23
    
    
    zpool create POOLNAME raidz3 drives...	# raidz3  (raid7, loose 3 drives for redundancy, min 5 drives)
    zpool create storage  raidz3 da0 da1 da2 da3 da4
    
    
    # 3 mirror-pair, concatenated together.  use 6 drives, loose space of 3
    zpool create POOLNAME VVVVVV   ...    VVVVVV  ...     VVVVVV ...
    zpool create storage  mirror da0 da1  mirror da2 da3  mirror da4 da5
    
            NAME        STATE     READ WRITE CKSUM
            storage     ONLINE       0     0     0
              mirror-0  ONLINE       0     0     0
                da0     ONLINE       0     0     0
                da1     ONLINE       0     0     0
              mirror-1  ONLINE       0     0     0
                da2     ONLINE       0     0     0
                da3     ONLINE       0     0     0
              mirror-2  ONLINE       0     0     0
                da4     ONLINE       0     0     0
                da5     ONLINE       0     0     0
    
    
    
    
    # 2 raid6 stripes concatenated together.  each strip loose 2 drives to redundancy, total 4 drives overhead
    # 4TB drives, 12 of them.  each stripe has 16TB usable, totalling ~30 TB.
    zpool create POOLNAME VVVVV ...		             VVVVV ... 
    zpool create storage  raidz2 da0 da1 da2 da3 da4 da5 raidz2 da6 da7 da8 da9 da10 da11
    
            NAME        STATE     READ WRITE CKSUM
            storage     ONLINE       0     0     0
              raidz2-0  ONLINE       0     0     0
                da0     ONLINE       0     0     0
                da1     ONLINE       0     0     0
                da2     ONLINE       0     0     0
                da3     ONLINE       0     0     0
                da4     ONLINE       0     0     0
                da5     ONLINE       0     0     0
              raidz2-1  ONLINE       0     0     0
                da6     ONLINE       0     0     0
                da7     ONLINE       0     0     0
                da8     ONLINE       0     0     0
                da9     ONLINE       0     0     0
                da10    ONLINE       0     0     0
                da11    ONLINE       0     0     0
    
    
    # once a pool is created, a FS is automatically created with the same name??
    
    zpool add -f geek1 raidz /dev/sde /dev/sdf 	# add new drives to existing zpool caked "geek1"
    
    zpool get mountpoint
    zpool set mountoint ...
    
    
    zpool offline POOLANEM drive			# offline a drive
    zpool replace POOLNAME ...			# replace drive (when there are unused disks from a different slot)
    
    zfs get quota
    zfs set quota=...
    
    zfs snapshot ...				# take snapshot.  avail in .zfs/snapshot of top of FS
    
    zfs sharenfs ...				# build-in nfs export if desired to use it
    
    
    
    ZFS Alignment, Performance
    Automatic alignment may not work, cuz HD lies about their geometry. Linux may need -o ashift. see ZFS Perf Tuning
    Top-level vdevs contain an internal property called ashift, which stands for alignment shift. It is set at vdev creation and it is immutable. It can be read using the zdb command.
    zfs iostat -v
    zpool status
    
    
    ZFS fs, dataset
    zfs create POOLNAME/volname1		# create a file system called "volume1" under the zpool POOLNAME  ??
    zfs create 
    
    There is a concept of dataset (like qtree in netapp) below the file system, and itseems like it can be nested. see Unix Area for example.
    ZFS other
    /usr/sbin/smcwebserver start # start a admin web portal, at “https://system-name:6789/zfs”
    ZFS Ref

    Package/Software management


    rpm is a basic command to manage software packages, available in just about every linux distro. up2date is a Red Hat Network moneyware to automatically download updates and install them. YUM is the freeware version, used by Fedora (from Yellow Dog distro) and actually better than up2date. But for those paying money to RH for support, keep to their tool maybe advisible.
    rpm -ivh [path/file.rpm]     # install rpm, vh = verbose hash for progress monitor
        --nodeps                 # install pkg even if fail dependencies check
        --noorder                # 
    for File in `ls -1`; do rpm -ivh --nodeps  $File; done
                                 # install all rpm in a dir eg cdrom
                                 # it seems that already installed packages will be overwritten by default
                                 # there is no "force" options needed to overwrite package to redo bad install.
    
    rpm -qa                      # query for all installed rpm packages in the machine
    rpm -qa arch=i386	     # same as above, specifying 32-bit binaries
    rpm -qa arch=x86_64	     # same as above, specifying 64-bit binaries for AMD 64 / Intel EM64T
    
    rpm -qa | grep glibc         # query for glibc version
    rpm -qpl [rpmfile]           # list where are the files installed to.
    rpm -ql  [pck_name]          # query installed package to see where files were send to
                                 #  (partial package name will work). 
    rpm -qi                      # query info of a specified package.
    rpm -qf /path/file           # query see which package installed the given file 
    rpm -e [pck_name]            # erase pcka_name as listed in rpm -qa from computer.  stop if it break deps
    rpm -e [pck_name] --nodeps   # forceful erase, even if it breaks dependencies.
    -Uvh [pkgname]               # Upgrade (erasing old packages, which install don't do)
                                 #  h= print hash sign as it goes, 
                                 #  v = nicely formated output
    rpm -Fvh *.rpm               # Freshen/update all rpm if an older version is installed
                                 # eg.  Refresh from AS 4.0 Update 1 to Update 2 via -F of all the rpm from cd.
    -V [pkgname]                 # Verify integrity of installed package.
    
    
    rpm -qa --qf '%{NAME} \t\t %{VERSION} \t %{RELEASE} \t %{ARCH}\n' 	
    	# display whether a given package is 32-bit i386 or x86_64.
    echo "%_query_all_fmt     %%{name}-%%{version}-%%{release}.%%{arch}" >> ~/.rpmmacros 
    	# to set the rpm query arch stuff permanently in the rpm query macro
    	# fedora does something like this by default.
    

    RedHat's up2date

    up2date				# GUI/CLI program to download patches, need RH registration.
    up2date ypbind			# ypdate ypbind client package to latest version
    #up2date nis			# update the nis package to latest version (??)
    up2date libstdc++-3.2.3-42	# update a specific rpm
    up2date -l			# list available downloads
    up2date -d			# download patches (as per config)
    up2date -i 			# install downloaded patches
    up2date --config		# run configuration (GUI or TUI), register machine, set exclusion, etc
    

    DNF (Did Not Finish-- Fedora, centos/redhat 8.x )

    yum is alias to dnf.
    dnf list --showduplicates perl-interpreter
    

    YUM (yellow dog, fedora, centos, redhat)

    yum			# yellow dog update modified, better rpm updated than up2date
    			# run cmd by itself will do all update automatically.
    			# not installed by default in RHEL 4, only in FC.
    yum update		# can be added to cronjob for auto update
    yum update  --security	# RHEL only, update rpm only if they are security related.
    yum update --cves  CVE-2021-4034	# update only rpm that fix the listed vulnerability (RHEL only)
    yum update polkit			# update specific rpm
    
    yum list		# list avail pkg and version number
    yum list  autofs	# list installed and avail pkg/ver for the pkg autofs
    yum info [pkgname]	# only display info of avail packages, opt followed by pkg name.
    		
    yum install yumex	# donwload and install the Yum Extender (GUI) rpm
    
    yum search wireshark	# search for package... give more info than list | grep...
    
    yum --enablerepo=epel --enablerepo=dag ...	# tmp enable named repos for this run (they have to be in /etc/yum.repos.d but disabled)
    
    repoquery --whatprovides '*bin/yes'   # coreutils
    yum       --whatprovides '*bin/yes'   # coreutils
    
    yum --setopt=tsflags=noscripts -y install boinc-client  # pass --noscripts to rpm when installing the boinc-client*rpm
    # it is like setting yum.conf tsflags option that make rpm install package without triggering execution of install script"
    
    
    # yum package bundles/collections (meta package?)
    yum grouplist
    yum groupinfo "Development Tools" 		# cmake, cvs, and many other stuff is in the optional part :(
    yum --setopt=group_package_types=optional groupinstall "@Development tools"
    
    
    Creating YUM Repository
    For RHEL 5.2 and older, 
    dvd dir has some media id in it.
    can't have yum and kickstart use same dir list :(
    It creates a series of xml files under the dir repodata/  (where all the RPMs are)
    Therefore, one need to copy all RPM from CD/DVD (RPMS or Server dir) to NFS or HTTP server location.
    cd /path/to/rpm-repository ; createrepo .
    createrepo is an rpm in RHEL5 not installed by default.
    (make sure .discinfo and .treeinfo are copied from root of DVD before running createrepo)
    
    For RHEL 5.3, such restrictions doesn't apply anymore, and 
    yum repo can point to the same dir tree that kickstart use,
    both can be a loopback mount to a DVD iso.
    
    
    Client side config: 
    add file to /etc/yum.repos.d/
    eg call it rhel5.repo
    contents:
    
    [rhel5]
    name=RHEL 5.1 Server
    ##baseurl=http://apache/ks/serv51-64/dvd1/Server/
    ##baseurl=file:///unixhome/sa/repo/RHEL/5Server/x86_64/Server
    baseurl=http://apache/loopback/rhel-5.3-server-x86_64-dvd/Server/
    baseurl=file:///net/apache/mnt/loopback/rhel-5.3-server-x86_64-dvd/Server/
    enabled=1
    gpgcheck=0
    
    

    apt (debian, centos, redhat)

    http://dag.wieers.com recommends use of apt and rpmsource for getting packages automatically,
    including dependencies.  However, it doesn't seems to like yum, and recommends command for its 
    removal (apt-get --fix-broken install).
    Sample package that is good to use are Apache mod_perl modules RH doesn't package but DAG does.  
    eg perl-Apache-DBI (Apache::DBI mod_perl),  perl-Compress-Zlib 
    
    install apt...rpm and rpmforge...rpm
    
    
    apt-get update				# update repository db
    apt-get -d install perl-Compress-Zlib	# install a specific package
    					# -d = download only, stored in /var/cache/apt/archives
    
    
    apt-get -y --quiet install DEBname   # answer yes.  DEBIAN_FRONTEND=noninteractive dont always stop the questioning :/
    
    aptitude search foo			# kinda equiv to yum list | grep foo
    aptitude install foo			# equiv to yum install foo
    aptitude show genome-music		# brief info about pkg
    					# info stored in /var/lib/dpkg/available
    cat/var/lib/dpkg/info/genome-music.list # list files installed by a given package
    
    dselect
    dpkg -l					# list installed packages (.dpkg) = rpm -qa
    dpkg -L	pkgname				# list files installed by named package
    dpkg -i file.deb			# rpm -ivh
    apt-get install ./file.deb		# probably better, solve dependencies
    
    dpkg  --search file			# find which package has file
    rpm    -qif    file
    
    
    note that sometime the dependencies of a package is to be satisfied by OS distro rpm, 
    in such case, apt-get would fail saying broken package.  Just install the necessary rpm (eg perl-Digest-HMAC) 
    from CD and re-run apt-get.
    
    


    SuSE Enterprise (SLUS/SLED)

    rug se [PKG-NAME]	# find rpm package from pre-configured repository.
    
    

    Open SuSE

    Open SuSE is the name for the former commercial SuSE that Novell ditched after it publised Enterprise SUSE.
    yast			# TUI far-style
    
    

    Snap, Flatpack, AppImages

    AppImages is not same as Aptainer (formerly Singularity)
    
    Snap is popularized by Ubuntu.  It is also a distro build system.
    
    Flatpack is popularized by Fedora
    
    Ponder about using them rather than building software via EasyBuild, Spack, and the like.
    
    

    System Services

    chkconfig                       # Easiest way to manipulate rc services, 
                                    # bettter than ntsysv or mangling xinetd file manually!
    				# for up to rhel6
    chkconfig --list xinetd         # see if xinetd service is enabled
    chkconfig --list rsh            # see if specific xinetd service is running, eg rsh
    chkconfig --list telnet         # see if specific xinetd service is running, eg telnet
    chkconfig --list autofs         # see if specific xinetd service is running, eg autofs automount
    chkconfig --list amd            # see if specific xinetd service is running, eg amd automount
    chkconfig --add ypbind		# register ypbind into service db
    chkconfig --level 345 ypbind on	# enable ypbind on run level 3,4,5
    chkconfig --level 345 rsh    on # enable rsh on run level 3,4,5, starting the service now also.
    chkconfig --level 35  httpd  on # enable httpd in init 3 and 5
    chkconfig vsftpd on             # enable  ftp  as xinetd controlled service (level 2,3,4,5)
    chkconfig talk   off            # disable talk as xinetd controlled service (all level)
    
    
    service iptables off            	# temporary stop firewall service (till reboot)
    service iptables status         	# check running status 
    chkconfig --level 345 iptables stop 	# disable firewall service from starting on run level 3,4,5
    					# RHEL 2.1 use ipchains.
    
    
    for SVC in `ls -1 /etc/xinetd.d`; do chkconfig --list $SVC; done
                                    # see which xinetd service is on or off.
    for SVC in `ls -1 /etc/init.d`; do chkconfig --list $SVC; done
                                    # see which init service run when
    				# but some file are not actually service
    
    serviceconf			# GUI, see what service run at what run level.
    redhat-config-services 		# same as above
    ntsysv 				# TUI of serviceconf (vsftp/telnet/rlogin, etc)
    				# Need to have ntsysv*rpm
    
    
    systemd
    RHEL7, some newer ubuntu?
    systemctl status firewalld
    SYSTEMD_LOG_LEVEL=debug systemctl start lightdm   # as close as "verbose" mode for systemctl
    

    SuSE

    sudo service SuSEfirewall2_setup status		# firewall service  
    sudo /sbin/SuSEfirewall2 status			# alt script to check status  
    chkconfig --level 345 SuSEfirewall2_setup  off	# turn off firewall
    
    
    

    Perforamnce

    see also hw cmd
    see also cpu_tuning
    
    numactl -H
    numactl ... cmd  # run command with specific memory config info (eg, restrict malloc to Phi MCDRAM?) 
    
    lstopo
    lstopo-no-graphics aka hwloc-ls
    
    
    lscpu
    
    turbostat 	# cpu current speed in Bzy_MHz column
    		# https://github.com/torvalds/linux/blob/master/tools/power/x86/turbostat/turbostat.c
    
    
    cpupower frequency-info
    cpupower frequency-set -d 1000000
    		
    
    cat /proc/cpuinfo | grep MHz		# current core speed in ubuntu, max core speed for rhel7.2 (7.4 may work better)
    
    sudo cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq  # ubuntu, rhel 7.4+ 
    sudo cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq	# ubuntu, fedora?
    
    sudo cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor  # rhel, but not present if no governor avail
    
    
    RHEL7 may need CPUfreq driver to be installed 
    https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/power_management_guide/cpufreq_governors
    or cpupower from kernel-tools
    https://access.redhat.com/solutions/1287663
    can have this run via singularity
    
    possible kargs for governors:
    https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html
    intel_pstate=disable # this maybe bad, no lower MHz in idle, no turbo?   was tried in specific cascadelake checking on IB osu_latency
    intel_pstate=passive
    intel_pstate=force # don't use acpi-cpufreq even if preferred on a given system, careful: thermal implication.  ok on skylake?
    no_hwp, hwp_only, support_acpi_ppc, per_cpu_perf_limits
    
    pstate avail for most Intel SandyBridge and later CPU.
    pstate with hwp (hardware managed pstate) is recommended and default policy.  cpu decide on pstate, os can give it hints.  ?hwp avail to skylake and later?
    
    
    
    sudo dmidecode -t processor | grep Speed	# but "Current Speed" is from ACPI and maybe stale
    						# Max Speed 
    						# don't seems good on ubuntu laptop
    						# show max speed for SL7 (ie match /proc/cpuinfo MHz info) 
    
    
    ulimit
    
    /etc/security/limits.conf ::
    *             -   memlock        unlimited
    *          soft   memlock        unlimited
    *          hard   memlock        unlimited
    
    or
    
    *      hard memlock unlimited
    oracle hard nproc 16384
    oracle soft stack 10240
    oracle soft core unlimited
    oracle soft nproc 16384
    oracle soft nofile 131072
    oracle hard nofile 131072
    *      hard core 0
    oracle hard core unlimited
    *      soft memlock unlimited
    oracle hard stack 10240
    
    
    
    
    /etc/sysctl.conf ::
    
    net.ipv4.ip_forward = 0
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.accept_source_route = 0
    kernel.sysrq = 0
    kernel.core_uses_pid = 1
    net.ipv4.tcp_syncookies = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    
    kernel.sem = 250 32000 100 128
    net.core.rmem_default = 4194304
    net.core.rmem_max = 16777216
    net.core.wmem_default = 262144
    net.core.wmem_max = 16777216
    net.ipv4.tcp_wmem = 4096 262144 16777216
    net.ipv4.tcp_rmem = 4096 4194304 16777216
    
    fs.file-max = 6815744
    net.ipv4.ip_local_port_range = 9000 65500
    fs.aio-max-nr = 1048576
    kernel.randomize_va_space=0
    kernel.exec-shield=0
    vm.panic_on_oom=1
    vm.swappiness = 0
    
    
    ulimit/sysctl
    ref: https://www.linuxtechi.com/set-ulimit-file-descriptors-limit-linux-servers/
    
    ulimit -a 		# show all limits
    ulimit -n 		# open file limit
    ulimit -n 2048 		# set file limit to 2048, above that need provision in /etc/security/limits.conf
    
    
    sysctl -a | grep file   # system-wide limit
    sysctl -p ... 		# make changes live
    typically source /etc/sysctl.conf  # and sysctl.d/ 
    
    user or @group would still be limited by 
    /etc/security/limits.conf ::
    
    @crd             soft    nofile           804200
    @crd             hard    nofile           804270 # this is still ok.  1204270 caused user unable to login or run process!
    oracle           soft    nofile           131072 # dont remember why oracle has such number
    oracle           hard    nofile           131072
    
    
    kargs
    
    cat /proc/cmdline	# list of kargs passed to kernel at boot
    			# also reported in the first few lines of dmesg
    
    kargs are to be space separated.
    the comma used in some args are specific to that one arg, eg "console=ttyS1,115200n8"  are options for console, not kargs proper.
    
    "iommu=pt net.ifnames=0 biosdevname=0 acpi_irq_nobalance console=tty0 console=ttyS1,115200n8"
    
    iommu=pt is used by AMD for nic 
    net.ifnames=0 biosdevname=0 	# tell rhel 7 to use eth1, eth2 etc nic name rather than enp196s0f1 
    
    

    sensors

    lm-sensors
    $ sensors
    acpitz-virtual-0						[1]
    Adapter: Virtual device						[2]
    temp1:        +42.0°C  (crit = +104.0°C)			[3]
    
    thinkpad-isa-0000						[4]
    Adapter: ISA adapter						[5]
    fan1:        3773 RPM						[6]
    
    coretemp-isa-0000						[7]
    Adapter: ISA adapter						[8]
    Physical id 0:  +42.0°C  (high = +87.0°C, crit = +105.0°C)	[ 9l  9i  9m  9c]
    Core 0:         +42.0°C  (high = +87.0°C, crit = +105.0°C)	[10l 10i 10m 10c]
    Core 1:         +33.0°C  (high = +87.0°C, crit = +105.0°C)	[11l 11i 11m 11c]
    
    # ubuntu cpu temperature (readable by user)
    
     9l 	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_label
     9i	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_input   #  42000
     9m	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_max	#  87000
     9c	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_crit	# 105000
       	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_crit_alarm #   0 (times reached alarm level?)
    
    
    10l 	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp2_label	# Core 0
    10i	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp2_input	#  42000
    
    11i	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp3_input	#  31000
    
    # sl7 cpu temperature 
    
    /sys/bus/platform/devices/coretemp.0/temp2_label
    /sys/bus/platform/devices/coretemp.0/temp2_input
    /sys/bus/platform/devices/coretemp.1/temp5_input
    
    grep [0-9] /sys/bus/platform/devices/coretemp.*/temp*_input
    
    /sys/class/thermal
    
    

    Kernel


    grub

    To disable nouveau driver from loading (so that commercial nvidia driver can be installed) (newer linux use this driver before even X, thus blacklist in modprobe don't cut it) ref:
  • https://linuxconfig.org/how-to-install-the-nvidia-drivers-on-centos-7-linux
  • https://forums.fedoraforum.org/showthread.php?310858-How-can-i-remove-disable-nouveau-in-grub
  • https://www.redhat.com/archives/rhl-list/2004-May/msg07775.html
    append to boot argument to kernel: rd.driver.blacklist=nouveau 	# don't remember if this really worked, maybe not
    append to boot argument to kernel: nouveau.modeset=0 		# this worked on exalearn 2018.11
    
    edit /etc/default/grub :: 
    GRUB_CMDLINE_LINUX="[ . . . ] rhgb quiet rd.driver.blacklist=nouveau"  		# **ml4hep1 2020.1215**
    append to boot argument to kernel: nouveau.modeset=0 				# this worked on exalearn 2018.11
    (remove rhgb quiet for verbose text boot)
    grub2-mkconfig -o /boot/grub2/grub.cfg      # BIOS mode, UEFI need a diff config
    
    check with lsmod | grep nouveau
    cat /proc/cmdline
    
    
    sudo update-grub
    sudo update-grub2
    
    Ubuntu/Mint seems to use quiet splash for GUI boot. Mint added nomdmonddf nomdmonism
    nouveau.blacklist=1 in grub.conf didn't work in Mint.
    Driver Manager was the right tool to change from nouveau to nvidia driver in Mint.

    Kernel Compilation

    
    compiling linux kernel  
    (similar procedure for old 2.4, new 2.4, and 2.6)
    
    download kernel source (srpm, rpm -ivh ...)
    Get the version applicaple to the base OS (eg RH 8.0) and kernel similar to the one installed 
    (latest maybe okay, but then other support may not be available).
    
    [
    For ES2.1/RH7.2, 
    building custom kernel just req getting RPMS packages for:
    kernel-headers-2.4.9-e.12	(disk1)
    kernel-source-2.4.9-e.12 	(disk2)
    SRPMS are not needed!  Though there are dependencies such as gcc, glibc-dev, 
    kernel dev, etc, which are RPMS, not Source RPMS!
    Files from SRPMS/ are not really needed unless really do heavy dev, 
    and they do not show up in rpm -qa!
    Note that RH does not provide support for customized kernel.
    ]
    
    
    cd /usr/src/linux-2.4* [ use /usr/src/kernels/2.6.11-* for FC4 ]

    make menuconfig
    this lauches a text menu program to createa config file.
    * indicates compiled into kernel
    M indicates loadable modules.

    make xconfig
    # GUI, an alnternate to the TUI menuconfig.
    # RHEL 2.1 (RH 7.2) GUI is a big grid
    # 2.6 kernel is a tree based GUI with lot of explanations; auto compile when done.

    make dep; make bzImage; make modules
    make modules_install
    
    #make install		# may not be avail.
    cp vmlinux... ...
    
    For old machines runing LiLo (RHEL 2.1 and older):
    
    cp /etc/lilo.conf.anaconda /etc/lilo.conf
    vi /etc/lilo.conf
    /sbin/lilo -v -C  /etc/lilo.conf
    to install a new boot loader w/ new config into MBR.
    
    
    For newer machines running Grub (RHEL 3.0/RH 8.0/Fedora):
    grub
    /etc/grub.conf (link to /boot/grub/grub.conf).
    edit of such file doesn't require lilo install, but then need to reactivate grub...
    
    
    ---
    
    From README
    
     - make config : standard kernel compiling config.
     - Alternate configuration commands are:
            "make menuconfig"  Text based color menus, radiolists & dialogs.
            "make xconfig"     X windows based configuration tool.
            "make oldconfig"   Default all questions based on the contents of
                               your existing ./.config file.
       
    Then do:
    
    make bzImage	# create compressed kernel image
    		# files are placed in .../linux/arch/i386/boot/bzImage
    make bzdisk	# optional, make boot floppy disk.
    
    make modules 	# needed if anything is in loadable module
    make modules_install	# suggested to do backup, but not sure where
    
    
    

    Upgrading Kernel

    
    Upgrading RH kernel, just need to get the later RPMS, package format kernel-VER.rpm, and install that.  
    No other packages are needed, and no recompiling needed.   It insall a stock RH tested general kernel.
    
    rpm -Uvh new-kernel-ref.rpm
    Install stuff like /boot/vmlinux-... vmlinuz-... System.map-... initrd-... 
    plus lot of files with version number on filename into /lib, etc
    
    The rpm has to be from RH.  Changes to /boot/grub/grub.conf may be needed.
    
    Easiest is just to use up2date and let it update kernel also, 
    then just reboot, it will do all the necesary update and changes.
    
    OS upgrade typically update grub.conf so that old kernel can be booted.
    Especially useful when certain drivers are tied to the kernel and don't work after kernel upgrade.
    eg. HBA driver from QLogic, PowerPath.
    DKMS would be really helpful as system is patched to new kernel.
    
    

    Kernel Tuning

    For general performance tuning, see the tool page
    ipcs -l 	# display interprocess communication parameters
    
    sysctl 		main tool to viewing kernel parameters
    	-a	display parameter and its setting.
    	-w	write new value for a given param
    	-n 	supress printing of key, just print value.
    	-p 	re-read /etc/sysctl.conf and make changes effective immediately
    
    eg:
    $ sysctl kernel.threads-max
    kernel.threads-max = 20479
    $ sysctl -n kernel.shmmax
    33554432
    $ sysctl -a
    ...
    kernel.tainted = 0
    kernel.core_name_format = core
    kernel.core_uses_pid = 0
    kernel.child-runs-first = 1
    kernel.panic = 0
    kernel.domainname = brio.com
    kernel.hostname = lancer
    kernel.version = #1 SMP Tue Feb 11 02:24:10 EST 2003
    kernel.osrelease = 2.4.9-e.12smp
    kernel.ostype = Linux
    fs.aio-max-pinned = 163836
    fs.aio-max-size = 131072
    ...
    
    
    Alternatively, the kernel params are listed in /proc/sys
    and they can be viewed or changed like all the other /proc parameters.
    eg:
    
    $ cat /proc/sys/kernel/sem 	# semaphore :: semmsl semmns semopm semmni
    250     32000   32     128
    $ cat /proc/sys/kernel/shmmax
    33554432
    $ cat /proc/sys/kernel/threads-max
    20479
    $
    
    Changing kernel parameters:
    
    Edit /etc/sysctl.conf for permanent changes that persist thru reboot.  
    Run sysctl -p to read this file and make changes effective immediately
    eg, for ora10g R2 install, add the following to /etc/sysctl.conf
    to modify equiv params in /proc/sys/kernel/shmmax and sem:
    kernel.shmmax=1036870912
    kernel.sem=250 32000 100 128
    
    Or, for one time change:
    echo VALUE > /proc/sys/kernel/file
    eg:
    echo "250     32000 100 128" > /proc/sys/kernel/sem
    
    
    smartctl	# tool to read hd SMART data.  gnome-disks is a GUI front end of this.
    		# ubuntu# apt-get install smartmontools
    
    

    Kernel Tuning Links

    http://www-106.ibm.com/developerworks/linux/library/l-adfly.html?ca=den-wud describe the major /proc entries in Linux, what they do and stuff.

    Oracle 10g install on linux params are described in: http://www.puschitz.com/InstallingOracle10g.shtml

    http://groups.google.com/groups?q=increase+thread+linux&start=10&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=jw98.77j.1%40gated-at.bofh.it&rnum=20 A thread where it seems linux kernel only support 1024 threads. Changes would be in the config file and recompile kernel. Info on 2.4

    http://www.volano.com/linux.html Older info, 2.2 kernel. Info about source file to change to alter kernel param.

    Kernel Modules

    
    modprobe -c		# list all modules
    insmod			# simple, no dependencies check, not too user friendly
    lsmod
    
    rpm -q --scripts kernel		# find out what kernel script runs and what modules are loaded.
    
    mkinitrd -v -f initrd-2.6.18-53.1.13.el5 2.6.18-53.1.13.el5	# create new initrd image
    
    
    
    
    
    
    


    DKMS

    Dynamic Kernel Module System
    Source code living outside kernel source, get rebuild automatically when new kernel is installed. Used by eg Dell PERC RAID controller, which add the mpt2sas driver (but not always newer than what comes with the OS distribution).
    rpm -ivh dkms-...rpm 		# install the dkms package/feature
    rpm -ivh driver...dkms.rpm	# these would actually be source codes 
    
    dkms status			# see if dkms is installed/configured (/etc/dkms)
    modinfo mpt2sas			# could be from native os or dkms
    

    SELinux - Security Enhanced Linux

    Ref: Overall system state:
    
    SELinux in RHEL 7 has 2 states
    - disabled
    - enabled        
    
    and 2 mode while enabled:
    - permissive
    - enforcing
    
    sestatus      # check status, 
    setenforce 0  # go into permissive mode
    setenforce 1  # go into enforcing  mode
    
    
    
    SE Linux has quite a number of places where permissions are set...
    
    useradd -Z user_u 	# when adding new user, also give it a default context. 
    
    
    sudo semanage login -l 	# list defined context for all users 
    			# (but not for user not added with useradd?)
    
    sudo semanage login -a -s user_u tin	
    			# hopefully will rid "Unable to get valid context for [USER]" in Mint when Gnome starts.
    
    
    semanage port -l | grep 514	# see if syslogd can use that port (@@ vs @ for tcp vs udp in rsyslog.conf)
    ps -eZ | grep syslog		# see process selinux context
    
    id -Z	# context of current user
    	# but not getting the info from $( semanage login -l ) ??!!
    	# user need to be in sysadm_t to be able to run sudo, su
    
    by default, unconfined_u user run in the unconfined_t domain
    
    
    seinfo			# summary info of context
    seinfo -t		# list all context types
    semanage fcontext -l 	# list all file path and context applied to them
    ls -Z file		# show context of a specific file
    
    cp --preserve=context foo bar	# copy file, preserving its context
    
    
    sesearch ...
    
    chcon --type var_t /var/www/html/index.html
    findcon ...
    restorecon ...
    matchpathcon  path
    sesearch ...
    
    
    sudo semanage fcontext -a -t git_user_content_t  "/export/tin/context_managed_dir(/.*)?"
    	
    (?) add file entry to context db , but context is not applied to file, not visible in ls -Z 
    
    /etc/selinux/targeted/contexts/files/file_contexts.local	local context db
    
    
    getsebool -a
    getsebool    use_nfs_home_dirs=1	# get specific setting
    setsebool -P use_nfs_home_dirs=1	# allow sshd to read authorizedkeys in NFS mounted home dir 
    
    Ansible module:
    
    - seboolean:
        name: use_nfs_home_dirs
        state: yes
        persistent: yes
    
    
    Controlling file contexts yourself

    Allow binary to open tcp port
    eg: change ssh to use port 2222 in rhel7
    denials noted in /varlog/audit/audit.log
    
    semanage port -a -t ssh_portt -p tcp 2222
    
    SSHD to read sym link
    Say home dir got filled, and whatever reason LVM etc not used, and home dir was moved and sym linked, users who ssh in may get a permission error bout not being able to cd to their home dir user get logged into / but yet they can just run `cd` and get back to their home dir.
    It is definately a SELinux problem if `setenforce 0` to go to permissive mode and problem goes away.
    Fix:
    semanage fcontext -a -t ssh_home_t '/global/home/users'
    restorecon -v '/global/home/users'
    Relabeled /global/home/users from system_u:object_r:default_t:s0 to system_u:object_r:ssh_home_t:s0
    
    PS.
    chcon --type home_root_t /home/tin or the like wasn't likely needed
    drwxr-xr-x. 35 root root  system_u:object_r:home_root_t:s0         4096 Nov  1 17:18 /home
    drwx------. 16 bofh users unconfined_u:object_r:user_home_dir_t:s0 4096 May  6 10:52 /home/bofh
    
    
    Ref:
  • https://linuxfreelancer.com/could-not-chdir-to-home-directory-permission-denied
  • CF_BK/sw/selinux.rst
    SELinux for RHEL 7
    Can only be fully disabled by editing config file and reboot.
    ping in rhel7 does not rely on setuid bit, but on selinux context to gain elevated priv to do the ping.
    Seems like RHEL7 has SELinux enable by default, and it is very usable, don't tend to run into problem/restrictions with most generic sys admin config as desktop machine.

    SELinux for Ubuntu 14
    Ubuntu by and large don't talk about SELinux and is disabled by default. Older distro don't even have it installed by default. It is installed on Ubuntu 14.04/Mint 17.2.
    Documentation is refered to the Debian page at: https://wiki.debian.org/SELinux/Setup
    
    apt-get install selinux-basics selinux-policy-default auditd 
    apt-get install setools	# may have lots of dependencies in older debians
    
    download script from https://wiki.debian.org/SELinux/Setup?action=AttachFile&do=view&target=_load_selinux_policy
    
    cp -p ~/Downlaods/_load_selinux_policy   /usr/share/initramfs-tools/scripts/init-bottom/
    
    update-initramfs -u
    selinux-activate
    ls -la /.autorelabel
    
    
    Old info for SELinux from RHEL 4 days
    Starting from Red Hat Enterprise Linux 4.0 ships with an implementation of Security Enhanced Linux. They use "targeted policy" which only limits functionality to a pre-defined set of deamon processes. The defined set will then have to obey the Mandatory Access Control (MAC) and become more secure even when exploited. The rest of the process are monitored but are freely allowed to change state, and thus they will operate more like the traditional Discretionary Access Control (DAC).

    Overall, enabling SELinux in RHEL 4 is largely safe and don't cause too much headache. A few known places are known to break, such as chroot, dhcp. The old RHEL4 settings is largely what is the "discretionary", ie monitoring/logging SELinux mode.


    Some places it may cause lots of logging. Specific daemons (eg SNMPD) can be excempt from the SELinux targeted policy via:
    A1.  Modify the /etc/selinux/targeted/booleans file.  
    A2.  Change snmpd_disable_trans=0 to snmpd_disable_trans=1 
    A3.  reboot your server
    
    -or-
    
    B1.  echo "1 1" > /selinux/booleans/snmpd_disable_trans
    B2.  echo "1"   > /selinux/commit_pending_bools
    B3.  Check that the value in /etc/selinux/targeted/booleans is changed to
         snmpd_disable_trans=1
    
    
    http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/selinux-guide/rhlcommon-section-0010.html
    
    
    
    Starting with RHEL6 (or 5?), there is an enforcing mode. This will add an extra layer of restrictions. eg, httpd that got hacked, but if running in SELinux environment, the kernel will still block access to things beyond its context (eg access to user's home dir).

    sudo chcon -v  --type=httpd_sys_content_t /var/www/html/index.html	# change SELinux context for a file, 
    									# this eg allow apache httpd to be allowed access to the file 
    									# (SELinux restrictions will show up in error_log).
    sudo chcon -vR --type=httpd_sys_content_t /var/www/html			# change SELinux context for a dir, Recursively.
    
    sestatus					# check status and mode of SELinux, such as enforcing or monitoring
    ls -Z /var/www/html/index.html			# -Z works for many commands, show SELinux context
    ps -efZ
    
    
    

    Video/Graphics

    Ubuntu can use x-swap PPA driver for nvidia, much easier than downloading drivers directly from nVidia and trying to install it for ubuntu. ref: https://launchpad.net/~ubuntu-x-swat/+archive/ubuntu/updates
    Zorin used ppa to install nvidia-390 for Quadro 4000 card.
    Centos 7
    
    blacklist nouveau kernel driver...
    
    systemctl  enable multi-user.target --force	# not necessary?
    systemctl  set-default multi-user.target    # no GUI, good for server, esp those with GPU cards!
    systemctl  start graphical.target
    systemctl  set-default graphical.target
    
    systemctl  get-default 	# see what's the default target
    
    https://www.google.com/search?q=disable+graphical+login+at+boot+rhel7&rlz=1C1GCEA_enUS814US814&oq=disable+graphical+login+at+boot+rhel7&aqs=chrome..69i57.14145j0j7&sourceid=chrome&ie=UTF-8
    
    
    ligthadm stop
    
    cp /etc/X11/xorg.conf /etc/X11/xorg.conf.bak
    nvidia-xconfig    
    nvidia-settings    
    
    system-configure-display	# rhel6
    
    
    Installing nvidia if nouveau had been installed could be rather painful. see devtalk.nvidia on steps to rid old nvidia.ko

    ANSI escapes

    Ansi is the most common escape sequence to draw colors on screen.
    more colorfile.txt will parse them correctly.
    less colorfile.txt does not (yet) display file with color.  to rid them:
    
    cat colorfile.txt | strip-ansi | less
    
    npm install --global strip-ansi-cli
    
    sed can likely remove them, but not as easy as above.
    
    vt100 dont support color, but still support ncurses/readline to draw character on screen  (?)
    
    docker build, 
    Rscript install
    apt-get install
    are tools that tend to draw on screen and make for very annoying logs :/
    
    
    


    Vendor Specific Tools/Packages

    HP (Intel Hardware)

    HP Smart array rpm eg: hpacucli-7.60-18
    part of HP/Compaq ProLiant Support Pack (PSP)
    
    More info:
    http://www1.jpn.hp.com/products/software/oe/linux/mainstream/bin/support/doc/general/mgmt/acuxe/v731-1cli/hpacucli-7.31-1.linux.txt
    
    
    /usr/sbin/hpacucli
    
    => ctrl all show 			# list all controller
    => ctrl all show status			# show controller info
    => ctrl slot=0 array all show 		# list all array in a given controller
    => ctrl slot=0 array all show status	# array status, an array is a mirror set.
    => ctrl slot=0 array A   show		# give info and config of a specific array
    
    
    To find RAID config, simply:
    cat /proc/driver/cciss/cciss0
    
    
    
    HP System Health Application and Isight Management Agent - hpasm
    
    For Proliant servers eg DL 585
    
    rpm -ivh hpasm-7.6.0-111.rhel3.i386.rpm
    
    hpasm activate			# activate and configure SNMP strings, etc
    /etc/init.d/hpasm  reconfigure 	# change settings
    
    it create a bunch of cma* process
    
    For command usage, see http://gentoo-wiki.com/HOWTO_Use_Hpasm
    
    
    

    Dell

    Collect log for support using
    racadm
    
    techsupreport   collect
    jobqueue view
    techsupreport   export -f tsr_report.zip
    
    
    jobqueue delete --all
    

    VMware ESX



    Upstart

    upstart replaces /etc/inittab in rhel6, ubuntu 14, allow specifying user, etc.
    but abandoned in rhel7, ubuntu 15 in favor of systemd

    For pure upstart process (and not just link to init/rc script), To enable/disable a service, one have to vi the upstart config file, eg /etc/init/smbd.conf and comment out the "start on..." line
    
    service smbd start
    service smbd stop
    
    
    initctl connects/controls init daemon using D-Bus (Ubuntu 14)
    
    initctl --system list			# similar to chkconfig --list, but with status
    initctl --system list | grep ufw	# (ubuntu) uncomplicated firewall
    initctl list-env
    


    Ref: upstart doc from Ubuntu

    /etc/init/couchdb.conf ::
    
    # couchdb - a RESTful document oriented database
    
    description "Start the system-wide CouchDB instance"
    author "tin ho"
    
    # upstart will start couchdb automatically on boot based on this config file
    # for manual control, run these as root:
    # initctl start couchdb
    # initctl stop couchdb
    # initctl status couchdb
    
    #start on filesystem and static-network-up
    #stop on deconfiguring-networking
    #start on runlevel [345]
    start on stopped rc RUNLEVEL=[345]
    stop  on runlevel [!345]
    
    
    env COUCHDB=/usr/local/bin/couchdb
    env COUCHDB_USER=couchdb
    env COUCHDB_STDOUT_FILE=/var/run/couchdb/stdout.log
    env COUCHDB_STDERR_FILE=/var/run/couchdb/stderr.log
    
    
    respawn
    
    pre-start script
        mkdir -p /var/run/couchdb
        chown couchdb:couchdb /var/run/couchdb
        logger -i -s -t upstart_couchy -p local6.info 'initctl couch reached pre-start script section'
    end script
    
    
    script
            # remember, for su, username is at the end!
            #exec su -c "logger -i -s -t upstart_couchy -p local7.info 'this is a test msg from initctl couchy'" couchdb
            logger -i -s -t upstart_couchy -p local6.info 'initctl couch starting couchdb'
            exec su -c "$COUCHDB -a /usr/local/etc/couchdb/default.ini -a /usr/local/etc/couchdb/local.ini"  $COUCHDB_USER
    
    end script
    
    post-stop script
        logger -i -s -t upstart_couchy -p local6.info 'initctl couch reached post-stop script section'
    end script
    
    
    
    


    RHEL7 new stuff

    
    
    
    systemd
    systemctl list-unit-files			# replaces chkconfig --list
    systemctl list-dependencies 			# show service dependency tree
    systemctl list-dependencies multi-user.target 	# show service dependency tree for a specific boot target
    
    
    systemctl show bruker_lmgr.service		# get details of the bruker lmgrd service
    						# init script fragments consolidated into /etc/systemd/system
    systemctl daemon-reload bruker_lmgr		# reload service after changes to init script fragment
    
    /etc/systemd/system				# dir where startup files are stored
    
    /etc/systemd/system/multi-user.target.wants	# expect sym link to /usr/lib/systemd/system
    /etc/systemd/system/                    	# can drop a file here and allow systemctl enable SVC_NAME ## preferred
    /etc/systemd/system/default.target.wants	# can drop a file here (uniq name that does not conflict with other service in /usr/lib/systemd/system) and it will be run on boot
    
    systemd-analyze blame						# time line of service starting, bottom = first service, top = last service
    
    
    Ref:
     
    systemctl enable  vncserver0			# add sym link to the target dir to start service on "runlevel" change
    systemctl start   vncserver0
    systemctl status  vncserver0			# look at output/error message from the start up process
    systemctl disable vncserver0			# remove sym link, but seems like service still listed...
    
    # /etc/systemd/system/vncserver0.service  # ensure executable
    # systemctl daemon-reload
    # systemctl enable vncserver0			# add sym link to multi-user.target.wants to start the service
    [Unit]
    Description=Remote desktop service (VNC)
    After=syslog.target network.target
    
    [Service]
    #Type=forking   # for process that deamonize itself and return to the command prompt within ~120sec
    Type=simple
    
    ExecStart=/usr/sbin/runuser -l sn -c "/usr/bin/x0vncserver -PasswordFile=/home/sn/.vnc/passwd -AlwaysShared=1 -display=:0"
    PIDFile=/home/sn/.vnc/%H%i.pid
    
    [Install]
    WantedBy=multi-user.target
    

    Network Manager

    CentOS7 install and use Network Manager if it is adds a Desktop or Developer sw group.
    Server incidentally would still use the traditional network stack.

    If not changing location and don't need DHCP, really better off disabling network manager!
    ypbind would NOT bind if NetworkManager is running (even when ifcfg-* has NM_CONTROLLED=no)!! NetworkManager is really evil in the server world! But if so much as include some developer or gui package in SL7, NM will be installed and enabled!!

    Ref: https://superuser.com/questions/397588/nis-failing-on-centos-6
    Mostly Disabling Network Manager in centos7
    vi /etc/NetworkManager/NetworkManager.conf 
    	plugins=ifcfg-rh
    
    vi /etc/sysconfig/network-scripts/ifcfg-WhatEver
    	NM_CONTROLLED=no
    	ONBOOT=yes
    	BOOTPROTO=none
    	IPADDR=...
    
    nmcli device 			# should show the interface no longer managed by Network Manager
    
    #systemctl enable network ??	# not enabled on system that use Network Manager exclusively?
    systemctl start  network 
    systemctl enable network       # redirect to something else...
    								# machine with Network Manager does not have this enabled (centos 7 default!)
    
    
    Completelyt Disabling Network Manager in centos7
    
    systemctl disable NetworkManager.service;
    systemctl stop NetworkManager.service;
    
    systemctl enable network 
    ##systemctl start  network 
    
    /usr/sbin/service   network start 
    /usr/sbin/chkconfig network on			# still present in rhel7 !
    
    
    Network Manager surprises
    Cannot remove /home?  can't rename it?  Get "Device or resource busy"?
    lsof, fuser don't report any process actually hogging /home ? 
    
    Have a look at 
    https://centosfaq.org/centos/cant-delete-or-move-home-on-73-install/
    
    Network Manager could have a `ProtectHome=read-only` which would prevent changes to HOME!
    
    Also check ProtectSystem=true
    
    Actually, lots of things in SystemD is doing this, eg:
    
    grep -r  ProtectHome /usr/lib/systemd
    /usr/lib/systemd/system/systemd-timedated.service:ProtectHome=yes
    /usr/lib/systemd/system/systemd-hostnamed.service:ProtectHome=yes
    /usr/lib/systemd/system/systemd-importd.service:ProtectHome=yes
    /usr/lib/systemd/system/systemd-localed.service:ProtectHome=yes
    /usr/lib/systemd/system/systemd-machined.service:ProtectHome=yes
    /usr/lib/systemd/system/chronyd.service:ProtectHome=yes
    /usr/lib/systemd/system/NetworkManager.service:ProtectHome=read-only
    /usr/lib/systemd/system/bluetooth.service:ProtectHome=true
    
    
    
    
    Network Devices
    nic has new naming convention (in rhel7 and ubuntu 15) 
    ref: http://blog.learningtree.com/rhel-7-changes-where-did-my-network-interface-go/
    
    eno1, eno2, ...	on-board ethernet nic.  index provided by firmware.
    ens1, ens2, ... PCIe hotplug nic
    enp2s0       	eth nic on PCI bus address 02:0:0 , ie bus #2, slot #0, function 0.
    wlp0s3f1u4	wifi. p0s3 is USB controller's PCI address.  f1u4 is USB addressing.
    enx112233445566	mac-based naming (but would need to change it when NIC is replaced).
    
    
    
    #nm* are NetworkManager commands (which tends to be GUI in laptop)
    
    nmtui					# Text UI
    nmtui-connect				# select wifi, or activate LAN port
    
    nmcli h					# cmd help
    nmcli n					# whether nm is enabled
    nmcli nm ?				# list interfaces managed by network-manager
    nmcli d					# show devices
    nmcli c					# show connections
    nmcli connection up ifname wlp3s0	# actiate a specific connection	
    nm-connection-editor			# nm GUI app, not the dock panel thingy
    
    /etc/NetworkManager/system-connections  # location for Debian NM interfaces info (?)
    
    systemctl restart NetworkManager	# restart the network stack, rhel7
    systemctl restart network		# restart the network stack, even if running NM
    service network-manager restart		# ubuntu 14
    
    
    iproute2 (ip cmd)	# replaces net-tools (ifconfig, route, netstat, arp)
    			# for the subcommands of ip, can use a single or first two letters, so long as it is non ambiguous.
    
    ip [opt] [sub-cmd]
    
    ip addr			# show ip address, aka ip a, replaces ifconfig.  will show NO-CARRIER.  UP just indicate driver is loaded?
    ip ad  show wlp3s0	# show info for a specif ineterface only
    
    ip route		# see routing table, replaces netstat -r
    ip route ...		# replaces route...
    
    ip link			# cumulative stat (subset of ip addr output)
    ip -s link		# -s provides stats info for all links
    
    ip neigh		# mac to ip mapping, replaces arp -a
    ip -4 neigh
    
    ip -4 addr		# show ip address, all nics, IPv4 only
    ip -6 addr show ens1
    
    ifstat			# LAN stats, replaces netstat -i
    
    ss -a			# replaces netstat -a, same output format.
    ss -au 			# udp only, 
    ss -at4			# tcp only, IPv4 only
    
       -r 			# resolve ip to hostname 
       -l			# show listening service only
    
    
    ethtool eno1		# set duplex, etc of an interface
    
    
    DNS settings under evil watch of Network Manager
    
    /etc/NetworkManager/NetworkManager.conf		# NM config
    	dns=none				# tell it not to mangle /etc/resolv.conf
    
    DNS server change should not go to /etc/resolv.conf anymore if managed by NetworkManger
    instead, edit /etc/sysconfig/network-scripts/ifcfg* 
    DNS1=8.8.8.8
    
    
    -or-
    
    /etc/sysconfig/network-scripts/ifcfg-eno1
    DNS1=8.8.8.8
    DOMAIN="lbl.gov als.lbl.gov berkeley.edu"
    ONBOOT=yes
    NM_CONTROLLED=yes				# set to no and the ifcfg will be on traditional "networking" but not NetworkManager will not change it IP, do dhcp with it, etc.
    
    
    Ref:
  • Debian Admin Handbook (also as .epub in google book)
  • http://blog.learningtree.com/rhel-7-new-features-linux-network-commands-with-iproute2/

    authentication in rhel7

    authconfig is the tool to use to configure authentication services in RHEL7/CentOS7. Manually changing nsswtich.conf isn't enough. There are pam.d, selinux, firewall updates.
    See also ipa-client-install and realmd

    firewall in rhel7

    firewalld is default firewall, does NOT write to /etc/sysconfig/iptables
    
    iptables		# rhel 6 default, boot up executes /etc/sysconfig/iptables ...
    
    ??
    firewall-config		# GUI tool.  DONT use system-config-firewall
    			# shorewall, not default, but supported as os package.
    

    RHEL 8 New STuff

    default, no /usr/bin/python , which may break things like ansible (unless otherwise configured to explicityly use python3)
    alternatives --set python /usr/bin/python3
    
    journalctl -no-pager -f -u postfix    	# see systemctl output for (service) unit file "postfix"
    					# -no-pager is to rid the annoying def of no word wrapping 
    					# -f is follow, like tail -f
    journalctl -xe ...
    
    
    

    Linux Desktop Progies


    sublime		text editor (not IDE, so light) good for python programming .  avail in Linux, Win, Mac.
    
    
    mtv  		mpegtv ($10 shareware req reg) for viewing mpeg video.   mpegtv.com
    mvtp		cli of mtv, free.  
    		-z = 2x2 zoom.  
    		-G+10+10 = place window in specified +x+y geometry 
    mpeg_play	bmrc.berkeley.edu mpeg video player.  in old a.out (ZMAGIC) binary though.
    		avail for solaris, etc.
    mpegplay	mpeg video player found in Knoopix distro.
    
    xine		free video player (vcd, dvd, avi, but no mpeg?).  http://xinehq.de.
    totem		alternate front end UI for xine.
    
    mpg123		cli for playing mp3 audio
    mpg321		drop in replacement for mpg123
    xmms		winamp like mp3 player (GUI for mpg123?)
    
    kate		KDE editor w/ many extras, like CLI window, mgnt tabs, somewhat like Visual Studio editor
    
    lynx		text based web browser
    elinks		a much better text based browser
    		set proxy server http"//address:port thru env var http_proxy
    BitchX		a text based irc client 
    irssi		a newer text based irc client, some cmd same as BitchX
    
    
    iEvince/GPdf 	GNOME pdf viewer
    Okular/KPDF	KDE pdf viewer
    evince		Document viewer, PDF, PS, XPS, TIFF
    xpdf		X Window pdf viewer
    PDFedit		GPL pdf internal structure editor
    CUPS-PDF	PDF generator extension to CUPS printing system, generate pdf file
    		in /var/spool/cups-pdf, but somehow doesn't work in x86_64 
    
    		http://localhost:631/	- cups web interface
    ---
    
    ssh-agent | xclip		# send output to xclip , require working X 
    
    xclip -o				# paste output from X clipboard , just echo
    xclip -o | bash			# pipe to bash to execute 
    
    
    clip 					# part of geomview  ?? 
    readreg works with screen clipboard
    pbcopy for OS X
    cygwin, windows has /dev/clipboard
    
    ref: https://stackoverflow.com/questions/749544/pipe-to-from-the-clipboard-in-bash-script
    
    
    
    fsview			# A GUI to see directory/file space usage, kinda interesting.
    redhat-logiviewer	# GUI tool to parse many diff logs
    
    gqview			# acdsee like image viewer
    eog			# eyes of genome, graphics file viewer
    eom			# eyes of mate
    eeyes			# electric eyes
    gimp			# "photoshop"
    
    Oothers to ponder:
    sudo snap install irfanview # via wine
    gthumb			# thumbview, crop, resize (probably not resample)
    nomacs
    XnViewMP
    GwenView
    
    
    ethereal is replaced by wireshark
    but that only install text-based snooper called /usr/sbin/tshark
     
    older ethereal comes with a gtk+, gnome, kde package.
    But those are somehow no longer shipped (by up2date, rpmfind).
    
    
    xxd -r 			# reverse hex to ascii conversion, from vim-common rpm
    			# xxd /etc/passwd | xxd -r 
    od			# octal dump
    hexdump			# hex dump
    
    xterm, use rxvt color scheme, grey on black. MiscFixed font, 8pt.
    gnome-terminal
    kterm ??
    
    
    iptraf 	- ip trafic monitor
    ss	- socket usage (related to netstat)
    pmap	- process memory usgae
    mpstat	- multiprocessor usage
    KSysGuard - kde system guard - real time reporting and graphing 
    	    similar one on Gnome
    
    gnome-system-monitor	# windows-style performance meter
    
    ntop 	- network top (need kernel tapping?)
    etherApe - bubble graphics of where traffic is going, but some bubbls become too big.
    Conky
    GKrellM
    vnstat	- network traffic monitor
    htop	- better top, 
    mtr	- ping and traceroute fused
    
    http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html has good overview.
    
    
    diff tools
    
    vimdiff    file1.txt file2.txt          # character-based side-by-side diff, handle up to 4 files.
    					# always default to side by side diff.
                                            # ^w J = vertical split   (toggle)
                                            # ^w H = horizontal split 
    
    
    vimdiff -o A.txt B.txt C.txt will open three horizontal splits for each file.
    vimdiff -O A.txt B.txt C.txt will open three vertical splits for each file
    
    
    - fldiff # X, side by side, opt to ignore space, best diff so far ## avail in  wsl, zorin, not in centos7
    - meld # lots of deps on KDE or Qt?  but new version is very good.  fairly close to github web diff.  only side by side diff. 
    - icdiff # text color sdiff by char, very nice (but no ignore space) ## not in zorin, ubuntu?
    
    
    - dwdiff # doesnt seems to be in centos7
    - colordiff # -yw for side and ignore space, but not too good
    
    - kdiff3,  GUI, up to 3 files diff? can't handle large diff.  depends/install kde-runtime 
    - kdiff3-qt # lots of qt stuff needed
    
    - mgdiff # clone of xdiff, old school X, no ignore space option
    - xxdiff # directory diff and merge tool
    - wdiff  # gnu word diff
    - docdiff
    
    
    
    minicom
    See also apple#serial
    minicom		# telix like program in Linux (a bit thicker than Solaris tip)
    
    minicom -o -m	# -o = no modem init
    		# -m = use meta key (instead of ctrl-a + key, can use ALT+key
     		#      or ESC, key.  eg ALT-O for options)
    		#      useful when running minicom inside "screen", 
    		#      which also use ^A for escape, thus req ^a, a, key combo.
    minicom -o -m  -D /dev/tty.usbserial -b 9600 --8bit 
    
    one anoying thing seems that minicom param change in ALT-O is not fully
    effective right away, especially on change of serial port.  Need to save it
    eg as default .dlf, then restart minicom for it to become effective!
    
    
    screen /dev/tty.usbserial 9600   # worked in mint 19
    	# ^a k 		: kill the connection (terminate screen and its use of /dev/tty...)
    	# ^a ^b    	: send break (for sun's go prompt)   (don't need ^B (upper case B))
    	# ^a b    	: send break (for sun's go prompt)   (after ^a, it is ^b OR just b, not a triplet of key seq to send the break)
    	# ^a H    	: beging/end loging to file screen.n  ie toggle capture to file.  capture output, non print password won't be captured
    	
    
    serial speed to try:
    9600
    19200
    38400
    115200
    Solaris, use 9600, 8n1, no flow control (hardware of software).
    Acopia seems to have required Flow Control DTS/DTR.
    NetApp now use 115200 ?
    
    dmesg | egrep --color -i serial\|tty # should determine serial port, eg ttyS0 ttyUSB0
    /dev/cu* are said to be obsolete circal kernel 2.x days.
    kernel 3.x and 4.x has build in driver for most of the usb-serial adapter?
    mint and wall-e has driver for the start-tech (blue) and belkin.  
    if usb dev removed, /dev/ttyUSB0 will be removed.
    
    

    iso2usb - making bootable usb thumbdrive from .iso

    
    CentOS 8, 7, 6.5 and newer
    --------------------------
    
    Need dd, their iso format has a portion that windows cannot copy correctly.
    dd if=centos.iso of=/dev/sdz
    
    must write to the whole usb, not a partition in it!  ie, can't use /dev/sdz1
    so will need to dedicate a whole usb key to it.
    
    see https://wiki.centos.org/HowTos/InstallFromUSBkey
    
    Resulting image looks like (partition is read only, and extra space on usb is wasted):
    
        Disk /dev/sdc: 15.1 GiB, 16170196480 bytes, 31582415 sectors
        /dev/sdc1  *        0 15335423 15335424  7.3G  0 Empty
        /dev/sdc2       23772    44151    20380   10M ef EFI (FAT-12/16/32)
    
        /dev/sdc1 on /media/tin/RHEL-8-1-0-BaseOS-x86_64 type iso9660 (ro,nosuid,nodev,relatime,nojoliet,check=s,map=n,blocksize=2048,uid=43413,gid=100,dmode=500,fmode=400,uhelper=udisks2)
    
    
    Fedora Media Writer for windows work, easy, but overwrite whole USB and create read-only FS (or is it ext2?).
      It likely use dd underneath.
    
    
    
    Linux
    =====
    
    * UNetbootin ? works for ubuntu?  
    
    * livecd-iso-to-disk said can preserve files on usb 
        - https://docs.fedoraproject.org/en-US/quick-docs/creating-and-using-a-live-installation-image/index.htm]
    
    * etcher for ubuntu... work for fedora.iso ? ... 
        - https://itsfoss.com/create-fedora-live-usb-ubuntu/
        - appImage (multiplatform), GUI.
        - on Ubuntu 18.04, wiped the whole USB and created an iso9660 RO FS
          resulting usb has single partition, of type 17: Hidden HPFS/NTFS
          /dev/sdd1  *        0 3710975 3710976  1.8G 17 Hidden HPFS/NTFS
          may as well just use dd?  it does "label" the disk, eg: /media/tin/SL-75-x86_64-LiveDVDgnome
    
    
    
    
    * live-iso
    
    
    SL7 could use dd, but also have other options listed in  http://www.livecd.ethz.ch/usbdisk.html
    
    * live usb creator gui -
      yum --enablerepo=sl-addons install liveusb-creator
    
    
    livecd-tools (cli)
    ------------------
    
    for SL7 can be found here: http://www.livecd.ethz.ch/download/RPMS/7x/x86_64/
    
    livecd-iso-to-disk SL-64-i386-2013-04-17-LiveCD.iso /dev/sdb1
    
    To reserve space for data persistence use option --overlay-size-mb: 
    livecd-iso-to-disk --overlay-size-mb 1024 SL-64-i386-2013-04-17-LiveCD.iso /dev/sdb1
    
    
    
    windoze tools
  • LiLi, was old favorite, but stopped working for CentOS 7 or 8 iso. maybe okay for ubuntu/debian. lili writes to existing FAT32 partition, so could use the usb for regular storage as well.
  • UNetbootin (win/linux/mac). Write to existing FAT32, make it bootable.
    Seems a bit slow?
    Support "preserve files across reboot, for Ubuntu only.

    Download and run, no install needed. http://unetbootin.github.io/ Cant work for Centos 7 iso.
  • Rufus, referred by Ubuntu. Actually has more features than UNetbootin. Can format thumbdirve to FAT32 (harder to get these days in Win10).
    Also allow for persistent partition.
    MBR, UEFI, BIOS. It download Syslinux files during the burn process as needed.
    Download and run, no install needed. https://rufus.ie/en/
  • PS. DVD sizes are: 4.7, 8.5, 9.4, 17.08 GB. So, maybe create a partition of 18GB for hosting such boot media.

    Links

    Stress test program, to help diag nasty hardware problems:


    TBD

    Locale/Language
    Old system use:
    export LANG=C
    export LANGUAGE=C
    export LC_ALL=C
    
    
    Newer system may or may not set correctly, try:
    LANG=C.utf8
    LANGUAGE=C.UTF-8
    
    export LANG=en_US.UTF-8   # root@beagle, vnc :3 works, but still ugly
    
    LOCALE="en_US.utf8"
    
    LC_ALL
    
    LC_CTYPE
    LC_MESSAGES
    LC_COLLATE
    
    
    
    regenerate locale, arch only?
    edit /etc/locale.gen
    sudo locale-gen   
    
    
    
    
    TBD
    
    
    # cmd.linux.ref
    
    # some content formerly adapted from cmd.admin.ref.
    
    ************************************************************
    Red Hat 4.0   (and not available in 3.0)
    ************************************************************
    /usr/sbin/system-config-kickstart	kickstart configurator.
    system-config-netboot			pxe, tftp boot setup
    system-config-language			GUI language config tool for the machine
    
    many of these system-config-* had been named redhat-config-* in RH 3.0
    
    
    ************************************************************
    Red Hat 9.0    2003/07
    Also for RH 3.0 (and presumably available in RH 4.0)
    ************************************************************
    
    
    
    /etc/X11/XF86Config	# config file read by X server.
    redhat-config-xfree86	# config tool to create XF86Config, try system-config-display in ES 4.0 and later.
    redhat-config-*		# lot of red hat gui config tool.
    nautilus		# file explorer
    gqview			# acdsee like image viewer
    
    
    
    
    ************************************************************
    Red Hat ES 2.1  (7.2 base)
    ************************************************************
    
    /usr/sbin/setup		# rh version of aix smitty, not all that.
    
    gnorpm				# GUI rpm manager, not in 3.0 or 4.0 (/usr/bin)
    
    ************************************************************
    
    /etc/sendmail.cf
    edit DS line to read
    DSsv-smtp.hybridauto.com
    and sv-smtp will become the smarthost that will be used to realy all the outbound mail.
    
    /etc/rc.d/init.d/ 
    /etc/rc[35].d/		# newer RH system, note that links need exist in 3 and 5
    
    runlevel		# what init level machine currently in
    who -r			# RH 9 and AS 3.0 (cmd exist in Solaris)
    
    
    /etc/inetd.conf
    /etc/xinetd.conf, /etc/xinetd.d/
    
    /etc/printcap  or /etc/printer.conf
    /etc/motd
    /etc/release		# see which os cd was used eg sol 8 01/00 for Jan 2000 build. 
    
    /etc/resolv.conf	# dns setup
    /etc/nsswitch.conf	# order of search for files/dns/nis
    
    
    
    printtool  (red hat print config)
    
    e-conf		enlightment configuration
    
    linux-conf	most essential config of linux in here
    control-panel	old red hat panel, configure network, printer, etc
    
    

    Keyboard shortcuts

    Zorin

    ctrl + shf + PrtScr ==> clipboard (draw rectangle)
           shf + PrtScr ==> File in Pictures folder
    
    ctrl + alt + PrtScr ==> clipboard (current window)
           alt + PrtScr ==> File in Pictures folder.
    

    Know Your Penguins - From @Linux Tweeter feed
    Credit: @Linux


    [Doc URL: http://tin6150.github.io/psg/linux.html]
    (cc) Tin Ho. See main page for copyright info.
    Last updated: 2020-12-18

    sn50
    hoti1
    bofh1