So, today I had to write a data exporter. I approached the problem from different angles, but just to discover that it doesn't take that long to export a million rows to Excel, nor does it require complex engineering, like some might've thought. OK, now, if you want to download more than one million rows (and/or if the order of these doesn't really matter), then you'll have to rethink the Excel and adopt CSV. This because Excel doesn't really support more than 1,048,576 rows (by 16,384 columns).
With the right architecture you can export 5 million records from your database within microseconds to CSV. Just imagine having a few queue-listeners, and splitting the data, efficiently, through the different jobs. Lets say that it takes half a second to inject 10K records, then, with only one queue available we would have to wait 4 minutes and 10 seconds for the entire CSV to get assembled, nevertheless, with 10 queues it would only take us 25 seconds to generate the CSV.
Alright, if you need some advice on how to build something alike, jump toQuestions & Answers and let us know your challenges, and how can we help you.
