Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import psycopg2 #version 2.9.3
import pandas as pd #version 1.4.0
conn = psycopg2.connect(
host=XXX,
dbname=XXX,
user=XXX,
password=XXX,
)
df = pd.read_sql("select 1", conn)
ImportError: Missing optional dependency 'SQLAlchemy'. Use pip or conda to install SQLAlchemy.
Issue Description
There was no SQLAlchemy dependency in the previous version when using a psycopg2 connection object in read_sql pandas function
Even after installing SQLAlchemy, there is a unexpected warning :
UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy
Expected Behavior
No error neither warning using a psycopg2 connection object in read_sql pandas function
Installed Versions
Comment From: phofl
Can you try on master, this was (maybe) partly adressed in #45416
Comment From: abowden1989
I don't think this is addressed by the change in https://github.com/pandas-dev/pandas/commit/3418679ae6556b63966b0dfa67d6a0cc4a5f03e5. When trying to use a different connection object when sqlalchemy is not installed, the code will fail on the line
sqlalchemy = import_optional_dependency("sqlalchemy")
The change I would propose would be to pass errors="ignore" to the import_optional_dependency call, and then only do the isinstance checks relating to sqlalchemy if the return from import_optional_dependency is not None
This wouldn't address the issue that a warning is shown (which seems reasonable to me), but would mean that sqlalchemy doesn't need to be installed to use other dbapi2 connections (some of which may be compatible with sqlite3)
Comment From: jbrockmendel
pls give the issue an informative title
Comment From: sknutsonsf
I am getting the same warning messages after installing SQLAlchemy
Comment From: simonjayhawkins
Even after installing SQLAlchemy, there is a unexpected warning :
UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy
@fangchenli this is the intended behavior? this issue is closed off by #45679?
Comment From: fangchenli
Even after installing SQLAlchemy, there is a unexpected warning : UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy
@fangchenli this is the intended behavior? this issue is closed off by #45679?
Yes, this is the intended behavior. See #45416. We don't officially support raw dbapi2 objects other than sqlite3. But if people think we should support it, we can keep this open for discussion.
Comment From: simonjayhawkins
But if people think we should support it, we can keep this open for discussion.
I think maybe could be a separate discussion so closing this one. The ImportError issue is fixed.
Comment From: tuttoaposto
Sorry I'm a late add here. I got the same warning with cx_Oracle.connect and pyodbc.connect. That didn't happen before until I reinstalled all my python packages. Current using pandas 1.4.1. Also installed SQLAlchemy-1.4.31.
Anyway to include these connections? What should I do to make the warning go away?
Comment From: david-waterworth
@fangchenli so just to confirm, are you saying you never supported other connection types, but this warning has been added recently? I'm suddenly seeing the same message using psycopg2 on code that I've been running fine for 3+ years?
Also, the message itself has grammatical errors
only support should be only supports
ordatabase should be or database
Comment From: fangchenli
@fangchenli so just to confirm, are you saying you never supported other connection types, but this warning has been added recently? I'm suddenly seeing the same message using
psycopg2on code that I've been running fine for 3+ years?Also, the message itself has grammatical errors
only supportshould beonly supportsordatabaseshould be ordatabase
That is correct. Before the recent changes, all non-sqlalchemy connections were passed to a fallback class that was only tested against sqlite3. Surprisingly it has worked well with many different connection types for many years. We probably will address this issue and officially support other raw SQL connections soon.
And apologize for the grammatical errors lol...
Comment From: Sieboldianus
Same for me: psycopg2.connect() has worked flawlessly for 3 years and I am ignoring this warning for now:
import psycopg2
import warnings
import pandas as pd
def db_query(sql_query: str, db_conn: psycopg2.extensions.connection) -> pd.DataFrame:
"""Execute SQL Query and get Dataframe with pandas"""
with warnings.catch_warnings():
# ignore warning for non-SQLAlchemy Connecton
# see github.com/pandas-dev/pandas/issues/45660
warnings.simplefilter('ignore', UserWarning)
# create pandas DataFrame from database query
df = pd.read_sql_query(sql_query, db_conn)
return df
db_conn= psycopg2.connect(
host=DB_HOST,
port=DB_PORT ,
dbname=DB_NAME,
user=DB_USER,
password=DB_PASS
)
df_test = db_query("SELECT * FROM public.test LIMIT 1", db_conn)
Comment From: jodur
Same for me. I use the same for firebird with fbd without warnings for years, until i switched my development to newer python versions and latest panda release.
def ReadLandenTabelSyntess():
SELECT="Select at_land.gc_id AT_LAND_GC_ID,at_land.gc_code SYN_LAND_CODE,at_land.gc_omschrijving SYN_LAND from at_land"
try:
conn = fdb.connect(**DATABASE)
at_land=pd.read_sql_query(SELECT,con=conn,coerce_float=False)
except:
logger.exception('**STOP** Error accessing Atrium database')
sys.exit(1)
conn.close()
return at_land
````
**Comment From: gilesmca**
It was asked above how to make the warning go away, for psycopg2 connections (and I suspect most of the others) I found a fairly painless way of quickly building an sqlalchemy engine from an existing connection which should then stop those annoying warnings:
import pandas as pd import psycopg2 import sqlalchemy from sqlalchemy.pool import StaticPool … engine = sqlalchemy.create_engine('postgresql+psycopg2://', poolclass=StaticPool, creator= lambda: {psycopg2connection}) data = pd.read_sql_query("select something”, con=engine)
I hope this helps.
**Comment From: l736k**
I'm facing this problem too with a pyodbc connection to a SQL Server database
class db: def init(self): self._conn = pyodbc.connect( 'DRIVER={ODBC Driver 17 for SQL Server};' 'SERVER=' + CONFIGS['db']['server'] + ';' 'DATABSE=' + CONFIGS['db']['database'] + ';' 'UID=' + CONFIGS['db']['uid'] + ';' 'PWD=' + CONFIGS['db']['pwd'] + ';' ) self._cursor = self._conn.cursor()
[...]
def query(self, sql):
return pandas.read_sql_query(sql, self._conn)
**Comment From: arani-mohammad**
I found this link: https://stackoverflow.com/questions/71082494/getting-a-warning-when-using-a-pyodbc-connection-object-with-pandas
from sqlalchemy.engine import URL
connection_string = "DRIVER={ODBC Driver 17 for SQL Server};SERVER=dagger;DATABASE=test;UID=user;PWD=password"
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = create_engine(connection_url)
df = pd.sql_read(query, engine)
**Comment From: eabase**
@arani-mohammad Wrong imports, that snippet does not work with p3.10.3 + `MS SQL`, **and** the panda doesn't have an `sql_read()`, but a `read_sql()`.
**Comment From: sat-ch**
> > @fangchenli so just to confirm, are you saying you never supported other connection types, but this warning has been added recently? I'm suddenly seeing the same message using `psycopg2` on code that I've been running fine for 3+ years?
> > Also, the message itself has grammatical errors
> > `only support` should be `only supports` `ordatabase` should be or `database`
>
> That is correct. Before the recent changes, all non-sqlalchemy connections were passed to a fallback class that was only tested against sqlite3. Surprisingly it has worked well with many different connection types for many years. We probably will address this issue and officially support other raw SQL connections soon.
>
> And apologize for the grammatical errors lol...
@fangchenli
Could you please give us a tentative date on when this would be fixed?
Thanks
**Comment From: sat-ch**
`UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy`
Did anyone solve the warning issue for oracle database?
thanks
**Comment From: lonnylundsten**
> > @fangchenli so just to confirm, are you saying you never supported other connection types, but this warning has been added recently? I'm suddenly seeing the same message using `psycopg2` on code that I've been running fine for 3+ years?
> > Also, the message itself has grammatical errors
> > `only support` should be `only supports` `ordatabase` should be or `database`
>
> That is correct. Before the recent changes, all non-sqlalchemy connections were passed to a fallback class that was only tested against sqlite3. Surprisingly it has worked well with many different connection types for many years. We probably will address this issue and officially support other raw SQL connections soon.
I think supporting various database connections is the way to go. I've tested pretty much all connectors and they did not report an error until very recently. Even with an error reported, my code works fine. The error is just annoying.
In testing I've found that the DBAPI2 connectors are much, much faster than others, at least when connecting to SQL Server. CTDS (https://zillow.github.io/ctds/) is actually the fastest connector library that I've found for connecting to SQL Server, however, pandas does still give an error when using this library. It would be great if this and other connectors could get 'official' support.
**Comment From: rkechols**
Does anyone know a way to simply suppress the warning without installing sqlalchemy?
**Comment From: nsmcan**
> Does anyone know a way to simply suppress the warning without installing sqlalchemy?
[Approach of @Sieboldianus](https://github.com/pandas-dev/pandas/issues/45660#issuecomment-1077355514) works fine
**Comment From: holmpa3**
I am connecting to a MySQL database in the following code:
mydb = pymysql.connect( host="localhost", database=db_name, user="root", password="password", autocommit=True
)
mycursor = mydb.cursor()
engine = create_engine('mysql+pymysql://root:password@localhost/%s' % db_name )
And I get the same warning... I am using SQLAlchemy to create the engine, is the issue that I am using pymysql to create the connection?
Are there plans to support pymysql connections?
**Comment From: cdcadman**
@holmpa3, you can pass `engine` instead of `mydb` to avoid the warning. Pandas will create a sqlalchemy connection from the engine, so you don't need to create `mydb` for pandas to work.
**Comment From: holmpa3**
@cdcadman I think I was passing mydb so I could manually commit changes. I seem to remember trying to get by only using engine but had issues with changes not being commited. It is just a warning anyway so I think I should be fine, any thoughts?
**Comment From: cdcadman**
@holmpa3 if you are passing `engine` to `DataFrame.to_sql` and the inserted records are not being committed, this could be a bug in pandas - please send a code sample if this is the case. If you are trying to commit a statement that you are passing to `read_sql_query`, then it might not be committed when you pass an Engine. You could instead pass a sqlalchemy Connection and handle the transaction yourself:
engine = create_engine('mysql+pymysql://root:password@localhost/%s' % db_name ) with engine.connect() as conn: with conn.begin(): dtf = pd.read_sql_query(statements, conn) ```