Feature Type
-
[ ] Adding new functionality to pandas
-
[X] Changing existing functionality in pandas
-
[ ] Removing existing functionality in pandas
Problem Description
It would be great if the pd.to_sql function would add new columns to an existing database table when appending. This would happen when the dataframe which is to be appended to the database table contains columns not currently present in the database table. When appending 2 pandas dataframe using pd.concat, this is handled automatically. However as far as I know, this is currently not possible with pd.to_sql.
Feature Description
An implementation could look like pd.to_sql(...., add_columns:bool=True) from the pandas user point of view.
Alternative Solutions
Alternative solutions I can think of and find online would be writing a custom function which detects whether columns in the dataframe are not present in the database table, and then either: 1) read the entire table to Python, concat with the new dataframe, and then write to the database table with if_exists='replace' 2) first send an 'ALTER TABLE' query adding the new columns to the table, and then use pd.to_sql
Solution number 2 would require figuring out the correct SQL data types for the new columns. It would be preferable to use existing pandas functionality for this instead of figuring it out myself, however it's not straightforward for me how to implement code from https://github.com/pandas-dev/pandas/blob/main/pandas/io/sql.py in a custom function.
Additional Context
No response
Comment From: phofl
I don't think this is in scope for us
Comment From: TeoSkondras
take
Comment From: phofl
Please wait till there is consensus
Comment From: TeoSkondras
of course!