Bash oneliners
Check if input is integer
re='^[0-9]+$'
[[ $input =~ $re ]] && echo integer || echo "NOT" >&2
Concatenate list of lines to single line
cat <( egrep -o '^[^ ]+ ' /etc/passwd )\
| while read line; do echo -n "$line "; done
Convert float to integer
printf -v int %.0f "$float"
or
float=1.23
int=${float%.*}
Convert lines to space separated
i.e. items separated by linebreaks to space separated
For a file:
cat tmp.txt | sed ':a;N;$!ba;s/\n/ /g' > tmp2.txt
For a variable:
var=`echo -e "something\nover\nthe\nrainbow"`
echo "$var"; echo; sed ':a;N;$!ba;s/\n/ /g' <<<"${var}"
outputs
something
over
the
rainbow
something over the rainbow
Copy a whole directory preserving permissions
This is my favourite bash one-liner of all time:
(cd /source/dir && tar cpf - . ) | (cd /target/directory && tar xpf -)
Create empty image
convert -size 1x1 xc:transparent pixel.png
Date from epoch seconds
date -d @1234567890
Sat 14 Feb 2009 12:31:30 AM CET
DEC-HEX conversions
From decimal to hexadecimal
echo "obase=16;4095" | bc
printf "%x\n" 4095
From hexadecimal to decimal
echo "ibase=16;FFF" | bc
printf "%d\n" 0xfff
Depth limited recursive grep search
grep axios `find . -maxdepth 2 -type f`
Disable history (temporarily)
set +o history # turn off
set -o history # turn on
Display weather from console
According to this posting, simply by using
curl wttr.in
e.g.
curl wttr.in/offenbach
it is possible to display the current weather and forecast:
Weather report: offenbach
\ / Sunny
.-. 15 °C
― ( ) ― ↙ 2 km/h
`-’ 10 km
/ \ 0.0 mm
┌─────────────┐
┌──────────────────────────────┬───────────────────────┤ Mon 18 May ├───────────────────────┬──────────────────────────────┐
│ Morning │ Noon └──────┬──────┘ Evening │ Night │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │
│ _ /"".-. 0..17 °C │ _ /"".-. 0..21 °C │ _ /"".-. 0..21 °C │ _ /"".-. 0..17 °C │
│ \_( ). ↖ 4-5 km/h │ \_( ). ↗ 6-7 km/h │ \_( ). ↘ 10-13 km/h │ \_( ). ↘ 8-14 km/h │
│ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │
│ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
┌─────────────┐
┌──────────────────────────────┬───────────────────────┤ Tue 19 May ├───────────────────────┬──────────────────────────────┐
│ Morning │ Noon └──────┬──────┘ Evening │ Night │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │
│ _ /"".-. 0..19 °C │ _ /"".-. 0..23 °C │ _ /"".-. 0..21 °C │ _ /"".-. 0..18 °C │
│ \_( ). ↗ 3-4 km/h │ \_( ). ↘ 4 km/h │ \_( ). ↘ 13-18 km/h │ \_( ). ↖ 12-17 km/h │
│ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │
│ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
┌─────────────┐
┌──────────────────────────────┬───────────────────────┤ Wed 20 May ├───────────────────────┬──────────────────────────────┐
│ Morning │ Noon └──────┬──────┘ Evening │ Night │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │
│ _ /"".-. 0..18 °C │ _ /"".-. 0..21 °C │ _ /"".-. 0..22 °C │ _ /"".-. 0..19 °C │
│ \_( ). ↓ 10-12 km/h │ \_( ). ↓ 12-13 km/h │ \_( ). ↓ 11-15 km/h │ \_( ). ↓ 8-13 km/h │
│ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │
│ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
Location: Offenbach am Main, Regierungsbezirk Darmstadt, Hessen, Deutschland [50.088766,8.766286]
Follow @igor_chubin for wttr.in updates
Expand IP CIDR block
The following lists all IP addresses
IP="87.251.74.0/24"
nmap -sL $IP | awk '/Nmap scan report/{print $NF}' | tr -d '()' | more
outputs
87.251.74.0
87.251.74.251
87.251.74.252
...
87.251.74.253
87.251.74.254
87.251.74.255
Find all text (ascii) files
find /var/www/html/ -type f \
-exec grep -Iq . {} \; \
-and -print
According to the reference, the -I option to grep tells it to immediately ignore binary files.
Find files with illegal characters
E.g. those not permitted in a Windows in filename
find . -name "*[<>:\\|?*]*"
Find newest file
This may take long!
find . -type f -exec stat --format '%Y :%y %n' "{}" \; \
| sort -nr | cut -d: -f2- | head
Generate a series of zero padded integers
for i in $(seq -f "%05g" 10 15); do echo $i; done
OR also
printf "%05d\n" $i
produces
00099
Using the -v flag stores the output in another variable:
i=99
printf -v j "%05d" $i
echo $j
See this resource.
Get lastmod of file in epoch seconds using stat
stat -c %y FILE
Grep out comments and blank lines
egrep -v "^#|^$" FILE | less
# OR
egrep -v "^[#;]+|^$" FILE | less
Handling files/directories with spaces using find
find . -print0 | while IFS= read -d $'\0' file; do echo $file; done
Notes:
$'\0'is an ANSI C-quoted string that technically creates a NUL (0x0 byte) but effectively results in the empty (null) string (same as'').IFS= readis required to suppress trimming of leading and trailing whitespace.
See here for more detailed explanation.
Increment a variable
In bash there are several ways to increment a variable:
var=$((var+1))
((var=var+1))
((var+=1))
((var++))
((++var))
let "var+=1"
let "var++"
let "++var"
IP address oneliners
Resolve hostname to IP address
getent hosts unix.stackexchange.com | \
awk '{ print $1 ; exit }'
exit causes only the first IP to be returned
Get the external IP address
curl -s checkip.dyndns.org | \
sed -e 's/.*Current IP Address: //' -e 's/<.*$//'
or shorter
wget http://ipinfo.io/ip -qO -
Extract IP address from some text
echo $TEXT | grep -E -o '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)'
JSON inplace update using jq
Given a file package.json containing JSON code, the following adds a new key and value to “scripts”:
json="$(jq '.scripts.dev="netlify dev"' package.json)" && \
echo "$json" > package.json
such that, e.g.
{
"name": "test_app",
"version": "1.0.0",
"main": "app.js",
"license": "MIT",
"dependencies": {
"faunadb": "^4.4.1"
}
}
becomes
{
"name": "test_app",
"version": "1.0.0",
"main": "app.js",
"license": "MIT",
"dependencies": {
"faunadb": "^4.4.1"
},
"scripts": {
"dev": "netlify dev"
}
}
List the subdirectories in a tarball
tar tfz tarball.tgz --exclude '*/*'
The --exclude option to excludes contents within a subdirectory.
Match string
string='Some not very long string';
[[ $string == *"very long"* ]] && echo match || echo NOT # match
[[ $string == *"not long"* ]] && echo match || echo NOT # NOT
Also using REGEX match function with group capture:
line="## 13. Check if an object is a Promise"
[[ "$line" =~ ^##' '+([0-9]+)\.' '+(.+) ]] && { \
echo $line; \
echo ${BASH_REMATCH[1]}; echo ${BASH_REMATCH[2]}; \
}
outputs
## 13. Check if an object is a Promise
13
Check if an object is a Promise
Mount partition from commandline
LABEL=TV_ARC
udisksctl mount -b $(findfs LABEL=$LABEL)
List all known partitions with their respective labels
lsblk --fs
Output history without line numbers
i.e. select only characters from pos 8
history | cut -c 8-
Output whitepace as control characters
E.g. to show the contents of the special variable IFS
printf "%q" "$IFS"
displays
$' \t\n'
Pipe a list of files to be added to a tarball
find /path/to/files -name \*.txt | tar -cvf allfiles.tar -T -
Pipe a webcam shot to Imagemagick display
Requires that fswebcam is installed:
fswebcam --device=/dev/video1 --png=0 - | display
Print PDF file from command line
https://makandracards.com/makandra/24202-linux-how-to-print-pdf-files-from-the-command-line
lp example.pdf
Recursively find the newest file in a subdirectory
find . -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" "
Remove leading zeros from a variable
# Removes 1 or more longest occurrences from the beginning of the string
epoch="1593151200"
hour=`date +%H --date=@${epoch}` # "08"
hour=${hour##+(0)} # "8"
Ref: http://tldp.org/LDP/abs/html/string-manipulation.html
Replace forward with back slash in filepath
echo "$filepath" | sed -E 's/\//\\/g'; done
Save output to an array using IFS
IFS=$'\n'
arr=($(grep -n "search term" file.txt | sed 's/:.*//'))
unset IFS
Show CPU/memory intensive processes
# top 10 memory consuming processes
ps -auxf | sort -nr -k 4 | head -10
# top 10 CPU consuming processes
ps -auxf | sort -nr -k 3 | head -10
Show currently installed packages on Debian/Ubuntu system
Complete list
/usr/bin/dpkg-query -Wf '${Package}\t${Version}\n'
List packages without reverse dependencies
for target in `dpkg -l | grep '^ii' | awk '{ print \$2 }'`; do
if [ "`apt-cache rdepends --installed $target | wc -l`" = "2" ]; then
echo "$target"
fi
done
Source: https://www.client9.com/debian-find-packages-without-reverse-dependencies/
Show seconds remaining in the minute
echo $(( 61-$(echo $(date +%S | sed -e 's/^0//')) ))
Function or one-liner to slugify a text string
slugify() { echo "$1" | iconv -t ascii//TRANSLIT | sed -r s/[^a-zA-Z0-9]+/-/g | sed -r s/^-+\|-+$//g | tr A-Z a-z; }
Sort output by column
e.g. 2nd column
du . -hk --max-depth=2 | sort -k2
Trim leading or trailing whitespace using sed
# leading space
echo ' 30 01' | sed -e 's/^[[:space:]]*//'
# trailing spaces
echo " test test test " | sed 's/ *$//'
Use bc to do calculations
echo "469958656/39.905" | bc -l
11776936.62448314747525372760
Use sed to insert lines before or after a match
Given a file, FILE, containing this content:
# Header 1 - Primary
Some detail
then this sed command will append (in place) AFTER the line containing # Header 1 - Primary the text \n## Section Header (note the leading linefeed which needs to be escaped):
sed -i '/# Header 1.*/a \\n## Section Header' FILE
resulting in
# Header 1 - Primary
## Section Header
Some detail
To insert BEFORE the matched text, use the switch /i instead.
To do only ONE replacement,
sed -i '0,/# Header 1.*/a \\n## Section Header' FILE
See here for more detail and examples using sed.
Useful applications of netcat
Check if a computer is reachable via SSH with netcat
A non-zero result means not available:
nc -z HOST PORT > /dev/null
echo $?
or
nc -z HOST PORT && echo available || echo UNAVAILABLE
Ref: https://unix.stackexchange.com/a/6811
Check for an open port with netcat
nc -zv HOST PORT1 PORT2 PORT3
e.g.
nc -zv 7.19.8.89 8222 8223
results in
Connection to 7.19.8.89 8222 port [tcp/*] succeeded!
nc: connect to 7.19.8.89 port 8223 (tcp) failed: No route to host
indicating first port is open while second is not.
Using netcat to transfer files
On the SENDER (with IP 192.168.178.4):
tail -f /var/log/pihole.log | nc -vv -l -p 4567
outputs
Listening on [0.0.0.0] (family 2, port 4567)
Then on the RECEIVER
nc -vv 192.168.178.4 4567 # output on STDOUT
# OR to save to a local file
nc -vv 192.168.178.4 4567 > saveToFile
outputs
Connection to 192.168.178.41 4567 port [tcp/*] succeeded!
Aug 9 14:33:26 dnsmasq[10974]: reply mesu-cdn.origin-apple.com.akadns.net is <CNAME>
Aug 9 14:33:26 dnsmasq[10974]: reply mesu.apple.com.edgekey.net is <CNAME>
...
Once the client disconnects, so does the server (SENDER).
SSH port 22 honeypot using netcat
sudo -- bash -c 'while true; do echo started; \
netcat -l -p 22 >> /tmp/netcat.log; done'
Simple webserver with netcat
nc -vv -l 127.0.0.1 -p 4567 < index.html
To serve a JSON file as a simple RESTful API server:
FILE="content.json"
PORT=3333
while :; do \
echo -en "HTTP/1.1 200 OK\r\nConnection: keep-alive\r\nContent-Type: application/json; charset=utf-8\r\n\r\n$(<$FILE)\r\n\r\n" \
| nc -q 1 -k -l -p $PORT; \
done
Transfer whole directory using netcat
On the SENDER (with IP 192.168.178.4):
tar -cvf - ~/some_dir | nc -vv -l -p 9876
Then on the RECEIVER:
cd /tmp; nc 192.168.178.4 9876 | tar -xvf -
Testing network speed with netcat
This may require installation of netcat-traditional package on Ubuntu.
On the SENDER (with IP 192.168.178.4):
time nc -vv -n -l -p 7845 >/dev/null
Then on the RECEIVER:
time yes | nc.traditional -vv -n 192.168.178.4 7845 >/dev/null
Then after some time press Ctrl+C to end the connection. Something like the following will be displayed:
(UNKNOWN) [192.168.178.41] 7845 (?) open
^C sent 469958656, rcvd 0
real 0m39,905s
user 0m0,373s
sys 0m1,771s
i.e. ~ (469.958.656 * 8)/1000/1000 ~= 3760 Mb/sec
Using cut and handling multiple spaces
ps aux | grep -v grep | tr -s ' ' | cut -d' ' -f3,11
^^^^^^^^^^^^ ^^^^^^^^^ ^^^^^^^^^^^^^
| | |
| | get 3rd & 11th fields
| |
| squeeze spaces
|
grep out 'grep'
View image via SSH
This requires that the program eog is installed locally:
cd /tmp && FIFO=$(mktemp -u); \
ssh remote_host "cat /remote/path/to/image" > "$FIFO"; \
eog "$FIFO" && rm "$FIFO";
Website spider using wget
url="https://marcgreyling.com"
wget --spider --force-html -r -l2 $url 2>&1 | grep '^--' | awk '{ print $3 }' | sort -u > urls_`date +%Y%m%d_%Hh%M`.txt