Nrthugu

I want to match strings from a pattern file to look into Source.txt file.

pattern_list.txt has 139k lines

Source.txt more than 5 millions lines

If I use grep like this it tooks 2 seconds to get the output.

grep -F -f pattern_list.txt Source.txt > Output.txt

But if I try with this AWK script it gets stuck and after 10 min I need to stop because nothing happens.

awk 'NR==FNR {a[$1]; next} {



for (i in a) if ($0 ~ i) print $0



 }' FS=, OFS=, pattern_list.txt Source.txt > Output.txt

pattern_list is like this

21051

99888

95746

and source.txt like this

72300,2,694

21051,1,694

63143,3,694

25223,2,694

99888,8,694

53919,2,694

51059,2,694

What it wrong with my AWK script?

I'm running on Cygwin in Windows.

if you are doing literal match this should be faster than your approach

$ awk -F, 'NR==FNR{a[$0]; next} $1 in a{print $1,$3,$8,$20}' pattern_list source > output

However, I think sort/join will still be faster than grep and awk.

If increasing performance is your goal, you'll need to multithread this (AWK is unlikely faster, perhaps slower?).

If I were you, I'd partition the source file, then search each part:

$ split -l 100000 src.txt src_part

$ ls src_part* | xargs -n1 -P4 fgrep -f pat.txt > matches.txt

$ rm src_part*