Python Very Large Set. How To Avoid Out Of Memory Exception?

I use a Python set collection to store unique objects. Every object has __hash__ and __eq__ overridden. The set contains near 200 000 objects. The set itself takes near 4 GB of mem

Solution 1:

The best approach would probably be to make the objects that you are storing in the set smaller. If they contain unnecessary fields, remove them.

To reduce the general object overhead you could also use __slots__ to declare the used fields:

classPerson(object):
   __slots__ = ['name', 'age']
   def__init__(self):
      self.name = 'jack'self.age = 99

Solution 2:

Sets indeed use a lot of memory, but lists don't.

>>>from sys import getsizeof>>>a = range(100)>>>b = set(a)>>>getsizeof(a)
872
>>>getsizeof(b)
8424
>>>

If the only reason why you use a set is to prevent duplicates, I would advise you to use a list instead. You can prevent duplicates by testing if objects are already in your list before adding them. It might be slower than using the built-in mechanics of sets, but it would surely use a lot less memory.

Solution 3:

Try using __slots__ to reduce your memory usage.

When I last had this problem with lots and lots of objects, using __slots__ reduces the memory usage to 1/3.

Here is a SO question about __slots__ you might find interesting.