What Is The Easiest Way To Load A Filtered .tda File Using Pandas?

Pandas has the excellent .read_table() function, but huge files result in a MemoryError. Since I only need to load the lines that satisfy a certain condition, I'm looking for a way

Solution 1:

you can use the chunksize parameter to return an iterator

see this: http://pandas.pydata.org/pandas-docs/stable/io.html#iterating-through-files-chunk-by-chunk

• filter the chunk frames however you want
• append the filtered to a list
• concat at the end

(alternatively you could write them out to new csvs or HDFStores or whatever)