Jamie Thomson

Thoughts, about stuff

Use awk to split a text file into multiple files based on some criteria

with 4 comments

In the spirit of “when you learn something stick it in a blog post so you can find it later” here’s a solution to splitting a file into multiple files using the Unix/Linux command awk.

Here’s a demo script that dumps some sample data into a file and then uses awk to split it into multiple output streams:

echo "John,Bananas,London,10" > /tmp/input
echo "John,Apples,London,5" >> /tmp/input
echo "Jane,Bananas,London,2" >> /tmp/input
echo "John,Apples,Leeds,3" >> /tmp/input
echo "Jane,Bananas,Leeds,8" >> /tmp/input
echo Jane buying Bananas
echo ===================
cat /tmp/input | awk -F, '$1=="Jane" && $2=="Bananas"'
echo John buying in London
echo =====================
cat /tmp/input | awk -F, '$1=="John" && $3=="London"'
echo Buying in Leeds
echo ===============
cat /tmp/input | awk -F, '$3=="Leeds"'

which when run returns this output:

$ . /tmp/demoawk.sh

Jane buying Bananas

John buying in London

Buying in Leeds

Like I said, I wrote this for my own reference but if someone else finds it useful, so much the better! Thank you to my colleague Marcin Kulisz who helped me out with this.




Written by Jamiet

July 24, 2017 at 8:05 am

Posted in Uncategorized

4 Responses

Subscribe to comments with RSS.

  1. One of my first ever jobs was writing shell scripts that used awk and sed to process files. I used to sysadmin DEC VAX/VMS/OpenVMS and Unix systems. Probably one of my earliest introductions to programming and regular expressions. I still have the O’Reilly sed and awk UNIX Powertools book from 1990. Happy days 🙂

    Bert Craven

    July 24, 2017 at 8:46 am

    • Nice. That’s the way to learn, get thrown in at the deep end. What’s old is new again! After years of living in MSFT land with all their GUIs I’m really enjoying doing all the same stuff but on the command-line.


      July 24, 2017 at 8:48 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: