Feature Type

  • [ ] Adding new functionality to pandas

  • [X] Changing existing functionality in pandas

  • [ ] Removing existing functionality in pandas

Problem Description

It would be great if the pd.to_sql function would add new columns to an existing database table when appending. This would happen when the dataframe which is to be appended to the database table contains columns not currently present in the database table. When appending 2 pandas dataframe using pd.concat, this is handled automatically. However as far as I know, this is currently not possible with pd.to_sql.

Feature Description

An implementation could look like pd.to_sql(...., add_columns:bool=True) from the pandas user point of view.

Alternative Solutions

Alternative solutions I can think of and find online would be writing a custom function which detects whether columns in the dataframe are not present in the database table, and then either: 1) read the entire table to Python, concat with the new dataframe, and then write to the database table with if_exists='replace' 2) first send an 'ALTER TABLE' query adding the new columns to the table, and then use pd.to_sql

Solution number 2 would require figuring out the correct SQL data types for the new columns. It would be preferable to use existing pandas functionality for this instead of figuring it out myself, however it's not straightforward for me how to implement code from https://github.com/pandas-dev/pandas/blob/main/pandas/io/sql.py in a custom function.

Additional Context

No response

Comment From: phofl

I don't think this is in scope for us

Comment From: TeoSkondras

take

Comment From: phofl

Please wait till there is consensus

Comment From: TeoSkondras

of course!