Showing posts with label Bash. Show all posts
Showing posts with label Bash. Show all posts

Friday, March 29, 2013

Sort,uniq (string, int) columns in csv files


cat file

text1,5
text4,3
text3,1
text1,4
text7,2
text3,7
text2,9


1. less file | sort -u -t, -k1,1 - sort the first column and get unique texts
text1,5
text2,9
text3,1
text4,3
text7,2

2. less file | sort -u -t, -k1,1 | sort -t, -k2 -nr - get unique texts on first column and reverse order for the second column using -r option
text2,9
text1,5
text4,3
text7,2
text3,1

Saturday, September 15, 2012

Grep in pdf file for a certain word

find . -name '*.pdf' -exec pdftotext {} - \; | grep -rn "test"

find . -name "*.pdf" -print0| while read -d $'' file; do co=$(pdftotext -q "$file" - |grep -crn "test"); if [ $co -ne 0 ]; then echo $co - "$file" ; fi ; done

Tuesday, September 11, 2012

Download a web site using wget

wget -r --level=0 -convert-links --page-requisites --no-parent <URL>
httrack <URL>

Friday, May 4, 2012

How to get the last modified item using ll command

touch file
echo "test" > file
cat `ll -ltr | tail -1 | awk '{print $8}'`

Result:
test

Wednesday, February 15, 2012

screenshot of a webpage

catycapt --url="www.google.com" --out=/home/test/screenshot.png

Thursday, February 9, 2012

Download pictures using lynx and wget

test@test:~$ lynx --source "http://www.funnyjokes.org/" | grep gif
    <td><a href="/"><img src="images/logo.gif" border="0" /></a></td>
<a href="funny_cartoons.aspx"><img src="http://www.martybucella.com/toon.gif" alt="cartoon" width=150 /></a>

test@test:~$ lynx --source "http://www.funnyjokes.org/" | grep http | grep gif | cut -d\" -f4
http://www.martybucella.com/toon.gif
test@test:~$ wget `lynx --source "http://www.funnyjokes.org/" | grep http | grep gif | cut -d\" -f4`
--2012-02-09 16:31:46--  http://www.martybucella.com/toon.gif
Resolving www.martybucella.com... 72.167.131.126
Connecting to www.martybucella.com|72.167.131.126|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14076 (14K) [image/gif]
Saving to: `toon.gif'

100%[====================================================================================================================================================================================================>] 14,076      33.8K/s   in 0.4s  

2012-02-09 16:31:47 (33.8 KB/s) - `toon.gif' saved [14076/14076]
test@test:~$ display toon.gif

Wednesday, February 8, 2012

Use bittorrent from command line

sudo apt-get install bittorrent
btdownloadcurses "http://name.torrent"

 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| file:     test                                                                                                                                                                                                       |
| size:     1,192,962,793 (1.1  G)                                                                                                                                                                                                          |
| dest:     /home/work/Downloads/test                                                                                                                                                                                  |
| progress: ############################################################################################################################################################################################################################### |
| status:   download succeeded!                                                                                                                                                                                                             |
| speed:      0    B/s down -   0    B/s up                                                                                                                                                                                                 |
| totals:     1.1  G   down -  30.7  M   up                                                                                                                                                                                                 |
| error(s):

Friday, February 3, 2012

Print PS output using AWK

ps aux | grep -v grep | grep apache2 | awk '{printf $1","$2","$3","$4","$5","$6","$7","$8","$9","$10"," ; for(i=11;i<=17;i++){printf "%s%s",sep,$i;sep=FS} print ""}'
root,2284,0.0,0.1,203544,9176,?,Ss,Feb02,0:00,/usr/sbin/apache2 -k start  
www-data,2398,0.0,0.1,205544,8236,?,S,Feb02,0:00, /usr/sbin/apache2 -k start  
www-data,2399,0.0,0.1,205544,8236,?,S,Feb02,0:00, /usr/sbin/apache2 -k start  
www-data,2400,0.0,0.1,205544,8236,?,S,Feb02,0:00, /usr/sbin/apache2 -k start  
www-data,2401,0.0,0.1,205544,8236,?,S,Feb02,0:00, /usr/sbin/apache2 -k start  
www-data,2402,0.0,0.1,205544,8236,?,S,Feb02,0:00, /usr/sbin/apache2 -k start

TIME,USER,PID,%CPU,%MEM,VSZ,RSS,TTY,STAT,START,TIME,COMMAND

Thursday, February 2, 2012

ps aux output


$ ps aux  
USER       PID  %CPU %MEM  VSZ RSS     TTY   STAT START   TIME COMMAND
root     29505  0.0  0.0 38196 2728 ?        Ss   Mar07   0:00 sshd: can [priv] 
USER = user owning the process
PID = process ID of the process
%CPU = It is the CPU time used divided by the time the process has been running.
%MEM = ratio of the process’s resident set size to the physical memory on the machine
VSZ = virtual memory usage of entire process
RSS = resident set size, the non-swapped physical memory that a task has used
TTY = controlling tty (terminal)
STAT = multi-character process state
START = starting time or date of the process
TIME = cumulative CPU time
COMMAND = command with all its arguments

Saturday, December 10, 2011

Remove blank lines using grep/sed

$ grep -v '^$' input.txt > output.txt
$ sed '/^$/d' input.txt > output.txt

Friday, December 9, 2011

Tuesday, December 6, 2011

Record terminal actions to a file and also see the results on terminal

bash |tee record.txt

Brace Expansion

ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car, Phone, Pull}
{Car, Phone, Pull}
ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car,Phone,Pull}
Car Phone Pull
ubuntu1004@ubuntu1004:/tmp/folder$ echo O{Car,Phone,Pull}O
OCarO OPhoneO OPullO
ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car,Phone,Pull}\'s
Car's Phone's Pull's
ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car,Phone,Pull}\'s END
Car's Phone's Pull's END
ubuntu1004@ubuntu1004:/tmp/folder$ echo {1..10}
1 2 3 4 5 6 7 8 9 10
ubuntu1004@ubuntu1004:/tmp/folder$ echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z
ubuntu1004@ubuntu1004:/tmp/folder$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
ubuntu1004@ubuntu1004:/tmp/folder$ echo {A..M}
A B C D E F G H I J K L M
ubuntu1004@ubuntu1004:/tmp/folder$ echo {A..z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [  ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z
ubuntu1004@ubuntu1004:/tmp/folder$ echo {{A..Z},{a..z}}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z
ubuntu1004@ubuntu1004:/tmp/folder$ echo {A..Z},{a..z}
A,a A,b A,c A,d A,e A,f A,g A,h A,i A,j A,k A,l A,m A,n A,o A,p A,q A,r A,s A,t A,u A,v A,w A,x A,y A,z B,a B,b B,c B,d B,e B,f B,g B,h B,i B,j B,k B,l B,m B,n B,o B,p B,q B,r B,s B,t B,u B,v B,w B,x B,y B,z C,a C,b C,c C,d C,e C,f C,g C,h C,i C,j C,k C,l C,m C,n C,o C,p C,q C,r C,s C,t C,u C,v C,w C,x C,y C,z D,a D,b D,c D,d D,e D,f D,g D,h D,i D,j D,k D,l D,m D,n D,o D,p D,q D,r D,s D,t D,u D,v D,w D,x D,y D,z E,a E,b E,c E,d E,e E,f E,g E,h E,i E,j E,k E,l E,m E,n E,o E,p E,q E,r E,s E,t E,u E,v E,w E,x E,y E,z F,a F,b F,c F,d F,e F,f F,g F,h F,i F,j F,k F,l F,m F,n F,o F,p F,q F,r F,s F,t F,u F,v F,w F,x F,y F,z G,a G,b G,c G,d G,e G,f G,g G,h G,i G,j G,k G,l G,m G,n G,o G,p G,q G,r G,s G,t G,u G,v G,w G,x G,y G,z H,a H,b H,c H,d H,e H,f H,g H,h H,i H,j H,k H,l H,m H,n H,o H,p H,q H,r H,s H,t H,u H,v H,w H,x H,y H,z I,a I,b I,c I,d I,e I,f I,g I,h I,i I,j I,k I,l I,m I,n I,o I,p I,q I,r I,s I,t I,u I,v I,w I,x I,y I,z J,a J,b J,c J,d J,e J,f J,g J,h J,i J,j J,k J,l J,m J,n J,o J,p J,q J,r J,s J,t J,u J,v J,w J,x J,y J,z K,a K,b K,c K,d K,e K,f K,g K,h K,i K,j K,k K,l K,m K,n K,o K,p K,q K,r K,s K,t K,u K,v K,w K,x K,y K,z L,a L,b L,c L,d L,e L,f L,g L,h L,i L,j L,k L,l L,m L,n L,o L,p L,q L,r L,s L,t L,u L,v L,w L,x L,y L,z M,a M,b M,c M,d M,e M,f M,g M,h M,i M,j M,k M,l M,m M,n M,o M,p M,q M,r M,s M,t M,u M,v M,w M,x M,y M,z N,a N,b N,c N,d N,e N,f N,g N,h N,i N,j N,k N,l N,m N,n N,o N,p N,q N,r N,s N,t N,u N,v N,w N,x N,y N,z O,a O,b O,c O,d O,e O,f O,g O,h O,i O,j O,k O,l O,m O,n O,o O,p O,q O,r O,s O,t O,u O,v O,w O,x O,y O,z P,a P,b P,c P,d P,e P,f P,g P,h P,i P,j P,k P,l P,m P,n P,o P,p P,q P,r P,s P,t P,u P,v P,w P,x P,y P,z Q,a Q,b Q,c Q,d Q,e Q,f Q,g Q,h Q,i Q,j Q,k Q,l Q,m Q,n Q,o Q,p Q,q Q,r Q,s Q,t Q,u Q,v Q,w Q,x Q,y Q,z R,a R,b R,c R,d R,e R,f R,g R,h R,i R,j R,k R,l R,m R,n R,o R,p R,q R,r R,s R,t R,u R,v R,w R,x R,y R,z S,a S,b S,c S,d S,e S,f S,g S,h S,i S,j S,k S,l S,m S,n S,o S,p S,q S,r S,s S,t S,u S,v S,w S,x S,y S,z T,a T,b T,c T,d T,e T,f T,g T,h T,i T,j T,k T,l T,m T,n T,o T,p T,q T,r T,s T,t T,u T,v T,w T,x T,y T,z U,a U,b U,c U,d U,e U,f U,g U,h U,i U,j U,k U,l U,m U,n U,o U,p U,q U,r U,s U,t U,u U,v U,w U,x U,y U,z V,a V,b V,c V,d V,e V,f V,g V,h V,i V,j V,k V,l V,m V,n V,o V,p V,q V,r V,s V,t V,u V,v V,w V,x V,y V,z W,a W,b W,c W,d W,e W,f W,g W,h W,i W,j W,k W,l W,m W,n W,o W,p W,q W,r W,s W,t W,u W,v W,w W,x W,y W,z X,a X,b X,c X,d X,e X,f X,g X,h X,i X,j X,k X,l X,m X,n X,o X,p X,q X,r X,s X,t X,u X,v X,w X,x X,y X,z Y,a Y,b Y,c Y,d Y,e Y,f Y,g Y,h Y,i Y,j Y,k Y,l Y,m Y,n Y,o Y,p Y,q Y,r Y,s Y,t Y,u Y,v Y,w Y,x Y,y Y,z Z,a Z,b Z,c Z,d Z,e Z,f Z,g Z,h Z,i Z,j Z,k Z,l Z,m Z,n Z,o Z,p Z,q Z,r Z,s Z,t Z,u Z,v Z,w Z,x Z,y Z,z
ubuntu1004@ubuntu1004:/tmp/folder$ echo {z..a}
z y x w v u t s r q p o n m l k j i h g f e d c b a
ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car,Phone,Pull}\'s
Car's Phone's Pull's
ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car,Phone,Pull}\'s END
Car's Phone's Pull's END
ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car,Phone,Pull}\'s\ END
Car's END Phone's END Pull's END
ubuntu1004@ubuntu1004:/tmp/folder$ echo {Car,Phone,Pull}\'s\ .
Car's . Phone's . Pull's .
ubuntu1004@ubuntu1004:/tmp/folder$ echo -e {Car,Phone,Pull}\'s\ END."\n"
Car's END.
 Phone's END.
 Pull's END.

ubuntu1004@ubuntu1004:/tmp/folder$ echo -e "\b"{Car,Phone,Pull}\'s\ END."\n"
Car's END.
Phone's END.
Pull's END.


ubuntu1004@ubuntu1004:/tmp/folder$ echo {0..100..10}
0 10 20 30 40 50 60 70 80 90 100
ubuntu1004@ubuntu1004:/tmp/folder$ echo {50..100..5}
50 55 60 65 70 75 80 85 90 95 100
ubuntu1004@ubuntu1004:/tmp/folder$ echo {01..10}
01 02 03 04 05 06 07 08 09 10
ubuntu1004@ubuntu1004:/tmp/folder$ echo {01..100}
001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100

Remove files and folders eficiently

ubuntu1004@ubuntu1004:/tmp/folder$ ll
total 16
drwxr-xr-x  4 ubuntu1004 ubuntu1004 4096 2011-12-06 22:51 ./
drwxrwxrwt 14 root       root       4096 2011-12-06 22:51 ../
-rw-r--r--  1 ubuntu1004 ubuntu1004    0 2011-12-06 22:51 file1
-rw-r--r--  1 ubuntu1004 ubuntu1004    0 2011-12-06 22:51 file2
-rw-r--r--  1 ubuntu1004 ubuntu1004    0 2011-12-06 22:51 file3
drwxr-xr-x  2 ubuntu1004 ubuntu1004 4096 2011-12-06 22:51 folder1/
drwxr-xr-x  2 ubuntu1004 ubuntu1004 4096 2011-12-06 22:51 folder2/
ubuntu1004@ubuntu1004:/tmp/folder$ find . -type f
./file2
./file1
./file3
ubuntu1004@ubuntu1004:/tmp/folder$ find . -type f | xargs
./file2 ./file1 ./file3
ubuntu1004@ubuntu1004:/tmp/folder$ find . -type f | xargs ls -lh
-rw-r--r-- 1 ubuntu1004 ubuntu1004 0 2011-12-06 22:51 ./file1
-rw-r--r-- 1 ubuntu1004 ubuntu1004 0 2011-12-06 22:51 ./file2
-rw-r--r-- 1 ubuntu1004 ubuntu1004 0 2011-12-06 22:51 ./file3
ubuntu1004@ubuntu1004:/tmp/folder$ find . -type f | xargs rm -f
ubuntu1004@ubuntu1004:/tmp/folder$ ll
total 16
drwxr-xr-x  4 ubuntu1004 ubuntu1004 4096 2011-12-06 23:13 ./
drwxrwxrwt 14 root       root       4096 2011-12-06 22:51 ../
drwxr-xr-x  2 ubuntu1004 ubuntu1004 4096 2011-12-06 22:51 folder1/
drwxr-xr-x  2 ubuntu1004 ubuntu1004 4096 2011-12-06 22:51 folder2/
ubuntu1004@ubuntu1004:/tmp/folder$ find . -type d | xargs rm -rf
rm: cannot remove directory `.'
ubuntu1004@ubuntu1004:/tmp/folder$ ll
total 8
drwxr-xr-x  2 ubuntu1004 ubuntu1004 4096 2011-12-06 23:14 ./
drwxrwxrwt 14 root       root       4096 2011-12-06 22:51 ../

Grep for multiple words

grep 'A\|B' file
A B C D
B C D E
ubuntu1004@ubuntu1004:~$ cat -n file
     1 A B C D
     2 B C D E
     3 F G H I
     4 G I R T
ubuntu1004@ubuntu1004:~$ grep -rn 'A\|B' file
1:A B C D
2:B C D E

Get current ubuntu version

ubuntu1004@ubuntu1004:~$cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=natty
DISTRIB_DESCRIPTION="Ubuntu 11.04"

ubuntu1004@ubuntu1004:~$cat /etc/lsb-release | grep DISTRIB_CODENAME | cut -d\= -f2
natty
ubuntu1004@ubuntu1004:~$grep DISTRIB_CODENAME /etc/lsb-release | cut -d\= -f2
natty

Netstat cmd

ubuntu1004@ubuntu1004:~$ netstat --tcp --numeric
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 192.168.1.136:53499     74.125.232.196:80       ESTABLISHED
tcp        0      0 192.168.1.136:46617     199.7.48.190:80         TIME_WAIT
tcp        0      0 192.168.1.136:45177     74.125.79.138:80        ESTABLISHED
ubuntu1004@ubuntu1004:~$ netstat --tcp --listening
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 *:ssh                   *:*                     LISTEN    
tcp        0      0 localhost:ipp           *:*                     LISTEN    
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN    
tcp6       0      0 localhost:ipp           [::]:*                  LISTEN    
ubuntu1004@ubuntu1004:~$ netstat --tcp --programs
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
ubuntu1004@ubuntu1004:~$ netstat --route
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.1.0     *               255.255.255.0   U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth0
default         DD-WRT          0.0.0.0         UG        0 0          0 eth0
ubuntu1004@ubuntu1004:~$ netstat --statistics
Ip:
    1819 total packets received
    0 forwarded
    0 incoming packets discarded
    1817 incoming packets delivered
    1308 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    38 active connections openings
    0 passive connection openings
    6 failed connection attempts
    0 connection resets received
    0 connections established
    1513 segments received
    1216 segments send out
    2 segments retransmited
    0 bad segments received.
    41 resets sent
Udp:
    94 packets received
    0 packets to unknown port received.
    0 packet receive errors
    94 packets sent
UdpLite:
TcpExt:
    7 TCP sockets finished time wait in fast timer
    17 delayed acks sent
    1013 packet headers predicted
    127 acknowledgments not containing data payload received
    3 predicted acknowledgments
    2 congestion windows recovered without slow start after partial ack
    2 other TCP timeouts
    11 connections reset due to unexpected data
IpExt:
    InMcastPkts: 15
    OutMcastPkts: 17
    InBcastPkts: 216
    InOctets: 1911507
    OutOctets: 132874
    InMcastOctets: 2879
    OutMcastOctets: 2959
    InBcastOctets: 43434
ubuntu1004@ubuntu1004:~$ netstat --tcp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    

column command for formatting data

ubuntu1004@ubuntu1004:~$ cat -n file.cvs
     1 Cars $5000 10
     2 Beer $5 1000
     3 Computers $500 100
ubuntu1004@ubuntu1004:~$ column -t file.cvs
Cars       $5000  10
Beer       $5     1000
Computers  $500   100
ubuntu1004@ubuntu1004:~$ cat file.cvs |column -t
Cars       $5000  10
Beer       $5     1000
Computers  $500   100
ubuntu1004@ubuntu1004:~$ pico file.cvs
ubuntu1004@ubuntu1004:~$ cat -n file.cvs
     1 Old Cars $5000 10
     2 Beer $5 1000
     3 Computers $500 100
ubuntu1004@ubuntu1004:~$ cat file.cvs |column -t
Old        Cars  $5000  10
Beer       $5    1000
Computers  $500  100
ubuntu1004@ubuntu1004:~$ pico !!:1
pico file.cvs
ubuntu1004@ubuntu1004:~$ cat -n !!:1
cat -n file.cvs
     1 Cars|$5000|10
     2 Beer|$5|1000
     3 Computers|$500|100
ubuntu1004@ubuntu1004:~$ column -t -s "|" file.cvs
Cars       $5000  10
Beer       $5     1000
Computers  $500   100
ubuntu1004@ubuntu1004:~$ pico !-2:2
pico file.cvs
ubuntu1004@ubuntu1004:~$ !-3
cat -n file.cvs
     1 Old Cars|$5000|10
     2 Beer|$5|1000
     3 Computers|$500|100
ubuntu1004@ubuntu1004:~$ !-3
column -t -s "|" file.cvs
Old Cars   $5000  10
Beer       $5     1000
Computers  $500   100