Useful Ubuntu Commands

This repo contains some useful and repetitive ubuntu commands (especially for data scientists).

Find total number of files in a folder and its subfolders:

find . -type f | wc -l
ls /folder_address/ | wc -l
--> Reference1
--> Reference2

Find and count total number of files in particular directory with specific extension and (ends with specific string):

find ./ -mindepth 1 -type f -name "*_mono.mp3" -printf x | wc -c

Get list of all files in a folder, and put the output in a file:

ls -R > output_list_all_files.txt
--> Reference1

Get total length of mp3 files in a folder:

for file in *.mp3; do mp3info -p "%S\n" "$file"; done | paste -sd+ | sed 's+\(.*\)+(\1)/60+' | bc
--> above command prints result in minutes
--> Reference1
--> Reference2

Get total length of wav files in a folder:

soxi -D * | awk '{SUM += $1} END { printf "%d:%d:%d\n",SUM/3600,SUM%3600/60,SUM%60}'
or
ls|grep ".wav"|parallel -j$(nproc) soxi -D {}|awk '{SUM += $1} END { printf "%d:%d:%d\n",SUM/3600,SUM%3600/60,SUM%60}'
--> Reference1
--> Reference2

Merge and concat all mp3 & wav files in a folder into a single file via ffmpeg:

find *.mp3 | sed 's:\ :\\\ :g'| sed 's/^/file /' > fl.txt; ffmpeg -f concat -i fl.txt -c copy output.mp3; rm fl.txt
or
ffmpeg -f concat -safe 0 -i <( for f in *.wav; do echo "file '$(pwd)/$f'"; done ) output.wav
--> Reference1
--> Reference2
--> Reference3
--> Reference4

Split an audio file into some equal multiple parts:

ffmpeg -i input_file.mp3 -f segment -segment_time 3 -c copy out%03d.mp3
--> Reference1

Split/Segment an audio file from start_time to end_time:

ffmpeg -i input.mp3 -acodec copy -ss START_TIME -to END_TIME output.mp3
or
ffmpeg -i input.mp3 -ss 1.9 -to 3.5 -c copy output.mp3
or for folder based usage of above command, one can use below command:
for i in *.mp3; do ffmpeg -i "$i" -ss 0.0 -to 20.0 -c copy "../mp3_files_first_20_seconds/$i"; done
--> Reference1
--> Reference2

Convert multiple audio files from .mp3 to .wav:

for i in *.mp3; do ffmpeg -i "$i" -f wav "${i%}.wav"; done
--> Reference1

Convert multiple audio files from .wav to .mp3:

for i in *.wav; do ffmpeg -i "$i" -f mp3 "${i%}.mp3"; done
--> Reference1

Move (mv) milions of files of folder with extension .wav to another folder:

find . -name '*.wav' | xargs mv --target-directory=/path/to/dest_dir/
--> Reference1

Find & count one word or phrase in a txt/csv file ubuntu command:

grep "hello" train_data.csv --> will just show the results
grep -c "hello" train_data.csv --> will count number of results

List just 10 first files in a folder/directory:

ls | head -10

--> Reference1 [great refrence for ls command]

List just 10 last files in a folder/directory:

ls | tail -10

Find and replace text within a file using commands:

--> Save in-place and replace all the word "original" to "new" word in "predictions_all.json" file.
awk '{gsub(/original/,"new phrase")}1' ./input_manifest.json > ./output_manifest.json
or
sed -i 's/original/new/g' predictions_all.json
--> Reference1

Show specific line of file:

head -13 file_name | tail +13
or
sed -n '13p' file.txt
or
sed -n '20,25p' lines.txt
or
awk 'NR==5' lines.txt
or
awk 'NR>=20 && NR<=25' lines.txt

--> Reference1
--> Reference2

Using ffmpeg to cut audio file from start to stop time:

ffmpeg -ss 1 -i input.wav -to 20 -c copy output.wav
--> above command, will cut from second 1 to 20.

--> Reference1
--> Reference2

Using ffmpeg to convert stereo to mono audio file:

ffmpeg -i stereo_input.wav -ac 1 mono_output.wav

Using ffmpeg to convert stereo to mono audio files (convert multiple files in a folder):

find . -name '*.mp3' -exec ffmpeg -i '{}' -ac 1 'mono_{}' \; for i in *.mp3; do ffmpeg -i "$i" -ac 1 "./output_folder/$i"; done
or
find . -name '*.mp3' -exec ffmpeg -i '{}' -ac 1 '{}_mono.mp3' \; for i in *.wav; do ffmpeg -i "$i" -ac 1 "${i%.*}.wav"; done

--> Reference1
--> Reference2
--> Reference3

Using ffmpeg to convert stereo to mono audio files (extract just Right Channel):

ffmpeg -i stereo.wav -filter_complex "[0:a]channelsplit=channel_layout=stereo:channels=FR[right]" -map "[right]" front_right.wav
or
for i in *.mp3; do ffmpeg -i "$i" -filter_complex "[0:a]channelsplit=channel_layout=stereo:channels=FR[right]" -map "[right]" "./output_folder/$i"; done

--> Reference1

Using ffmpeg to convert stereo to mono audio files (extract just Left Channel):

ffmpeg -i stereo.wav -filter_complex "[0:a]channelsplit=channel_layout=stereo:channels=FL[left]" -map "[left]" front_left.wav
or
for i in *.mp3; do ffmpeg -i "$i" -filter_complex "[0:a]channelsplit=channel_layout=stereo:channels=FL[left]" -map "[left]" "./output_folder/$i"; done

--> Reference1

Get date and time using command line:

date "+%H:%M:%S %d/%m/%y"

--> Reference1

Schedule run ubuntu command at specific time:

at now +8 hours
at> python3 run_test.py # type your desired command
--> press ctrl+D to save the commands

--> Reference1
--> Reference2
--> Reference3

Do for example one command for each file in folder (ubuntu command):

for i in *.mp3; do ffmpeg -i "$i" -ac 1 "./output_folder/$i"; done
or
for i in *.wav; do python run_code.py "$i"; done
--> "$i" is the name of input file
--> Reference1

Get output in Ubuntu:

ls > output_directory/ls_output.txt

--> To redirect the output of a command to a file, type the command, specify the > or the >> operator, and then provide the path to a file you want to the output redirected to.
--> Reference1

Split/segment audio file from start_time to end_time:

ffmpeg -i input.mp3 -acodec copy -ss START_TIME -to END_TIME output.mp3
ffmpeg -i input.mp3 -ss 1.9 -to 3.5 -c copy output.mp3

Split/segment audio files of a folder:

for i in *.mp3; do ffmpeg -i "$i" -ss 0.0 -to 20.0 -c copy "../mp3_files_20_first_seconds/$i"; done

--> Reference1
--> Reference2

Convert for example 8Khz audio file to 16Khz via ffmpeg:

ffmpeg -i input.mp3 -acodec pcm_s16le -ac 1 -ar 16000 output.wav
for i in ./*.wav; do ffmpeg -i "$i" -acodec pcm_s16le -ac 1 -ar 16000 "output_folder/$i"; done

--> Reference1

Change sudo or user password in ubuntu:

sudo passwd tom
sudo passwd root

--> Reference1
--> Reference2

List all local users:

cut -d: -f1 /etc/passwd

--> Reference1

Move all files (but not folders) out of nested sub-directories into another folder (e.g., from Downloads folder to Videos folder):

find ~/Downloads/ -type f -print0 | xargs -0 mv -t ~/Videos

--> Reference1

Silence Detection in Audio File via ffmpeg:

ffmpeg -i input.wav -af silencedetect=n=-40dB:d=1.0 -f null -
ffmpeg -i input.wav -af silencedetect=n=-40dB:d=1.0 -f null - 2> ./silence.txt

--> Reference1
--> Reference2
--> Reference3
--> Reference4: Remove the Silent Parts of a Video Using FFmpeg and Python [Important & Useful]

Extract audio from video via ffmpeg:

ffmpeg -i sample.avi -q:a 0 -map a sample.mp3

--> Reference1

Get output of ffmpeg in python:

--> Reference1
--> Reference2
--> Reference3
--> Reference4
--> Reference5

Increase Audio Volume via ffmpeg:

--> Reference1
--> Reference2
--> Reference3

Empty swap in Ubuntu:

sudo swapoff -a; sudo swapon -a

--> Reference1

How to Clear RAM Memory Cache, Buffer and Swap Space on Linux:

--> Reference1
--> Reference2

Kill all processes owned by a user "username":

pkill -U username

--> Reference1

Address problem of low memory of Sysytem:

sudo echo 1 > /proc/sys/vm/overcommit_memory

--> Reference1
--> Reference2

Convert Ubuntu Shell to Bash (that e.g., ctrl + l works correctly & clear screen):

/bin/bash or chsh -s /bin/bash
--> Reference1
--> Reference2

Most used Ubuntu terminal commands for manipulating CSV and text files:

• Most used Ubuntu terminal commands for manipulating CSV and text files