I am working on a financial simulation (financial_life), in which I would like to collect my timeseries data in a DataFrame during simulation. However, while looking through several posts on StackOverflow and experimenting with timeit, I noticed that appending data to a Dataframe (e.g. via append or via loc-expansion) is far more inefficient than the append-function for a simple list and then transforming everything to a dataframe, if requested. The latter one is, what I am doing now.
So I was wondering, whether there is any performant way of creating a DataFrame in an iterative manner?
Comment From: jorisvandenbossche
Exactly what you are doing, append to a list and then transforming to a dataframe at the end. Is this not working for you? Another option would be to allocate the DataFrame from the start and fill it, but then you need to know its size of course
Comment From: MartinPyka
Yes, it is working for me, but I was just wondering whether I could avoid kind of duplicating the Data in Order to create a DataFrame. My initial idea was to create a class that inherits from DataFrame, which I would extend by my own functions. But I think, I go now with the conversion of the data to DataFrame afterwords.