Nrthugu

guys, I have one file text size 38GB and my system ram = 64 gig
I run this code for remove duplicate but type MemoryError

lines = open('file.txt', 'r').readlines()



lines_set = set(lines)



out  = open('b.txt', 'w')



for line in lines_set:

    out.write(line)

Your code loads the whole file into memory:

lines = open('file.txt', 'r').readlines()

Then it allocates more memory, scaled to the size of the file:

lines_set = set(lines)

If you want to be able to operate on files of size approaching or exceeding the amount of memory you have, you need to avoid loading the whole thing into memory at once.

One option would be to write as you read, avoiding storing any line except the one you're operating on in memory, and perform deduplication using hashes instead of exact equality testing.

For example:

seen = set()

with open('file.txt', 'r') as infile:

    with open('b.txt', 'w') as outfile:

        for line in infile:

            h = sha256(line)

            if h in seen:

                continue

            seen.add(h)

            outfile.write(line)

This still requires that the hashes of all unique lines fit in memory - however this is closer to 32 bytes per line. Depending on the length of lines in your file, this may or may not be good enough. If it is not good enough, you can move the seen set to secondary storage - ie, disk. You may want to keep a preliminary filter in main memory (ie, RAM) for performance reasons. For example, keep a set of the first 4 or 8 bytes of the sha256 in memory. Only consult the on-disk seen set if you see the prefix is in the in-memory seen.