Linux Commands

Examples of frequently used Linux shell commands for quick reference

Note: all commands were tested on Ubuntu Linux

Compression & Extraction

tar -cvf output.tar FOLDER

Tar compress folder [tar]

tar -xvf output.tar

Tar DEcompress folder [tar]

tar -jcvf output.bz2 --exclude=data FOLDER

Compress folder using bzip2 while excluding "data" subfolder

"-c" means compress

zip output.zip FOLDER/*

Zip folder [zip]

Sorting

sort textfile.txt | uniq

Sort and Remove duplicate lines from text file [sort, uniq]

"uniq -u" will print unique lines only

Word counting

wc -w textfile.txt

Count number of words in textfile.txt [wc]

"wc -l" will print number of lines in file
"wc -c" will print number of characters in file

Search in files (Grep)

grep "the" textfile.txt

Find and print occurrences of "the" in textfile.txt [grep]

grep -P "[0-9]+" yearly.stats.txt

Find and print all lines containing numbers in textfile.txt using perl Regular Expression "[0-9]+" [grep]

adding "-o" will print only the numbers not the full lines

Search for files

find . -name "*hadoop*"

Find and print all files names containing the word "hadoop" in current directory and all sub directories [find]

"." means current directory
Omitting first "*" will make the search in the current directory only

find . -type f -name 'CERTIFICATE*.pdf' -execdir cp {} /PATH/printing/{} \;

Find all files containing the word "CERTIFICATE" and copy them into the following path "/PATH/printing/"

"{}" refers to any file found

File head and tail

head textfile.txt tail textfile.txt

Show first (head) and last (tail) 10 lines in textfile.txt [head]

Set environment variables

export PATH=$PATH:/NEW_PATH

Append a new folder to "PATH" environment variable [export]

Restart Linux service


	sudo service network-manager restart

Restart network-manager service [service]

Add user to group


	sudo usermod -g vboxusers karim

Add user "karim" to "vboxusers" group [usermod]

Make files executable (Runnable)


		sudo chmod +x file.sh

Make file.sh executable [chmod]

Change file ownership


		sudo chown karim:karimgroup file.sh

Change ownsership of file.sh to user "karim" and group "karimgroup" [chown]

Access remote Linux machine using SSH


		ssh karim@machine_host.com

Show total memory used


		free -m

[free]

Join two files horizontally using [paste]


		paste -d "|" FILE1.txt  FILE2.txt > FILE3.txt

Write lines consisting of the sequentially corresponding lines from each FILE

"-d" means delimiter

Join two files vertically using [cat]


		cat FILE1.txt  FILE2.txt > FILE3.txt

Compare two sorted files and show the differences using [comm]


		comm -3 FILE1.txt FILE2.txt

"-3" means suppress common lines

Convert text file to lowercase using [tr]


		cat FILE.txt |  tr '[:upper:]' '[:lower:]

Get execution time of a Linux command using [time]


		time GET http://online-toolz.com > /dev/null

"> /dev/null" is used to suppress GET command output

List contents of jar file


		jar tf  FILE.jar

Split large files [split]


		split -l 100 INPUT_FILE.txt OUTPUT_FILE_PREFIX.txt

Split INPUT_FILE to multiple files 100 lines each

Remove first or last line from file [tail]


		head -n NUM_OF_LINES_MINUS_ONE INPUT.TXT  >  OUTPUT.TXT 

		tail -n NUM_OF_LINES_MINUS_ONE INPUT.TXT  >  OUTPUT.TXT

First command will remove last line, Second command will remove first line

"NUM_OF_LINES_MINUS_ONE" should be the number of lines in the file minus one - use wc -l to get it

Show disk usage of first level sub-directories [du]


		du -h -d 1 .

"-h" means human redable (ex: GB instead of long number of bytes)
"-d" is the depth of folders to show

TASKS: Sending HTTP Form post request using [curl]

curl -F "account_login=6pd9vkgk" -H "Content-Type: application/x-www-form-urlencoded" https://URL

"application/x-www-form-urlencoded" is the header for HTTP form
"account_login=6pd9vkgk" is the form payload

TASKS: Sniff your web traffic using tcpdump [tcpdump]

sudo tcpdump -i wlan0 -A port 80 -s 0

"wlan0" means your wifi card, change it to eth0 or other for your LAN
"80" is the HTTP port for web traffic

TASKS: PORT forwarding from port 8889 to port 8888 [nc]

mkfifo httpPIPE 

		  nc -q -1 -l 8889 0<httpPIPE | nc -q -1 localhost 8888 1>httpPIPE

put httpPIPE in standard input "0" and put standard output "1" in httpPIPE
"-q" will prevent quiting after EOF, will wait forever

TASKS: Rename all files in current dir, add ".postfix" at the end [rename]

rename 's/(.*)/$1.postfix/' *

"$1" refers to the original filename

TASKS: Sending email from command line [sendmail]

echo "YOUR_MESSAGE" | mail -s "SUBJECT" TARGET_EMAIL@gmail.com

Make sure you have sendmail installed "sudo apt-get install sendmail"

TASKS: Download full website [wget]

wget --recursive --wait=2 -U "Mozilla" -p -H http://twitter.com

--wait will wait for 2 seconds between page downloads
-U for user agent
-p and -H are needed to download all pages linked to index page even if in different subdomain

TASKS: Run a specific file every 30 minutes [crontab]

 
	crontab -e  

	0,30 * * * * /home/FILE_PATH 

	CTRL O 

	CTRL X

run "crontab -l " to make sure the cronjob is created correctly
0,30 means minute 0 and minute 30

TASKS: Disable word wrapping in terminal [less]

 
		cat index.html | less -S

-S did the job

TASKS: Count lines of code in a directory [less]

 
		rm -rf temptotal.txt; find . -regex ".*.\(php\|html\|css\|js\)"  -not  -path "./libs/*" | xargs  -I FILE  sh -c "echo FILE; cat FILE | tr -d '\t' | tr -s '\n\n' '\n' | wc -l   " | xargs -I COUNT sh -c "echo COUNT;echo COUNT >> temptotal.txt"   ; echo "Total:" ;  cat temptotal.txt | awk '{ sum+=$1} END {print sum}'; rm -rf temptotal.txt;

Count lines of code in files with php,html,css,js extentions excluding "libs" folder"

The code removes double end lines "\n\n" before counting

TASKS: Delete files that have more than 2MB of data [rm,xargs]

 
		sudo find /YOUR/FOLDER_PATH -size +2M | xargs rm -rf

First find files with 2MB+ size and then pass each one to "rm" command to be deleted

xargs is used to pass each file to rm as a parameter
WARNING: make sure you are in the right folder since "-rf" will delete files without confirmation

TASKS: Show memory usage by process [ps]

 
		ps -e -orss=,args=,command,user,pid | sort -b -k1n | awk '{ hr=($1/1024)/1024 ; printf("%13.2f GB ",hr) } { for ( x=2 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }'

TASKS: Print second line from CSV file [ps]

 
		cat file.csv | awk  -F ","   '{ print $2 }'

For this example columns are assumed to be separated by ","

TASKS: Mail forwarding from root local mail to external email address using sendmail

 
		echo "root: you@yourmail.com" >> /etc/aliases && newaliases

"newaliases" is used to rebuild alias DB

TASKS: Convert PDF file pages to JPG images [convert]

 
		convert -quality 100 -density 122x122 -antialias FILE.pdf FILE_PREFIX-%d.jpg

"%d" will be replaced by page numbers

TASKS: Create video(Timelapse) from multiple images using [ffmpeg]

 
		 ffmpeg  -r 15  -start_number 756 -i 'DSC00%3d.JPG'  VIDEO.mp4

"-r 15" is the frame rate per second
"%3d" will be replaced by image number starting from "756" as indictaed by -start_number

TASKS: Count and sort duplicates in file [sort]

 
		 cat FILE.txt  | sort | uniq -c | sort -nr

"-nr" means reverse numeric sorting

TASKS: Record screen session (Screencasting) Audio/Video [ffmpeg]

 
		ffmpeg -video_size 1024x768 -framerate 25 -f x11grab -i :0.0 -f pulse -ac 2 -i default ~/output.mkv

TASKS: convert comma separated file into rows using [sed]

 
		cat INPUT.txt | sed 's/,/\r\n/g'  > OUTPUT.txt

sed will replace all "," with "\r\n" (endlines)

sed pattern is 's/SEARCH_FOR/REPLACE_WITH/g'
sed can also be used to clean text files

TASKS: Scrape web pages using [wget]

 
		 wget -q http://online-toolz.com -O - | tr -d '\r\n|\n' |  grep -o "<title>[^<]*</title>" | sed  -r "s|</?title>||g"

Get the title of online-toolz.com website

"-O -" is used to redirect output to stdout instead of file
"tr -d" will delete endlines to make searching and replacing easier
grep finds the "title" tag fragment and "sed" will remove the "title" tags keeping tag content

Keywords: Linux, Shell, Script, Commands, Examples, Bash

Share