Hi all,
I noticed this bug today when I was trying to cast a large table (~56,000 rows by 8500 columns) from np.float64
to np.float32
. Since not every column in the table is made of float.64
s I first selected columns that were.
Code Sample, a copy-pastable example if possible
float_cols = curtable.dtypes[curtable.dtypes=="float64"]
The output then looked something like this:
float_cols
['GTEX-1117F-0426-SM-5EGHI',
'GTEX-1117F-0626-SM-5N9CS',
'GTEX-1117F-0726-SM-5GIEN',
'GTEX-1117F-1326-SM-5EGHH',
'GTEX-1117F-2426-SM-5EGGH',
'GTEX-1117F-2526-SM-5GZY6',
'GTEX-1117F-2826-SM-5GZXL',
'GTEX-1117F-2926-SM-5GZYI',
'GTEX-1117F-3026-SM-5GZYU',
'GTEX-1117F-3226-SM-5N9CT',
'GTEX-111CU-0126-SM-5GZWZ',
'GTEX-111CU-0226-SM-5GZXC',
'GTEX-111CU-0326-SM-5GZXO',
'GTEX-111CU-0526-SM-5EGHK',
'GTEX-111CU-0626-SM-5EGHL',
'GTEX-111CU-0726-SM-5GZYD',
'GTEX-111CU-0826-SM-5EGIJ',
'GTEX-111CU-0926-SM-5EGIK',
'GTEX-111CU-1026-SM-5EGIL',
'GTEX-111CU-1126-SM-5EGIM',
'GTEX-111CU-1226-SM-5EGIN',
'GTEX-111CU-1326-SM-5NQ8L',
'GTEX-111CU-1426-SM-5GZYP',
'GTEX-111CU-1526-SM-5N9FS',
'GTEX-111CU-1726-SM-5EGHM',
'GTEX-111CU-1826-SM-5GZYN',
'GTEX-111CU-1926-SM-5GZYZ',
'GTEX-111CU-2026-SM-5GZZC',
'GTEX-111CU-2226-SM-5N9G5',
'GTEX-111FC-0126-SM-5N9DL',
'GTEX-111FC-0226-SM-5N9B8',
'GTEX-111FC-0326-SM-5GZZ1',
'GTEX-111FC-0426-SM-5N9CV',
'GTEX-111FC-0526-SM-5GZZ8',
'GTEX-111FC-0626-SM-5N9CU',
'GTEX-111FC-0826-SM-5GZWO',
'GTEX-111FC-1026-SM-5GZX1',
'GTEX-111FC-1126-SM-5GZWU',
'GTEX-111FC-1926-SM-5GZYC',
'GTEX-111FC-2026-SM-5GZYO',
'GTEX-111FC-2926-SM-5GZY7',
'GTEX-111FC-3126-SM-5GZZ2',
'GTEX-111FC-3326-SM-5GZYV',
'GTEX-111VG-0008-SM-5Q5BG',
'GTEX-111VG-0326-SM-5GZX7',
'GTEX-111VG-0526-SM-5N9BW',
'GTEX-111VG-0726-SM-5GIDC',
'GTEX-111VG-1626-SM-5EGIO',
'GTEX-111VG-1926-SM-5GIDO',
'GTEX-111VG-2226-SM-5N9DU',
'GTEX-111VG-2326-SM-5N9BK',
'GTEX-111VG-2426-SM-5GZXD',
'GTEX-111VG-2526-SM-5GZXP',
'GTEX-111VG-2626-SM-5GZY2',
'GTEX-111YS-0006-SM-5NQBE',
'GTEX-111YS-0008-SM-5Q5BH',
'GTEX-111YS-0126-SM-5987T',
'GTEX-111YS-0326-SM-5GZZ3',
'GTEX-111YS-0426-SM-5987O',
'GTEX-111YS-0526-SM-5GZXJ',
'GTEX-111YS-0626-SM-5GZXV',
'GTEX-111YS-0726-SM-5GZY8',
'GTEX-111YS-0826-SM-5GZYK',
'GTEX-111YS-0926-SM-5EGGI',
'GTEX-111YS-1026-SM-5GZYE',
'GTEX-111YS-1126-SM-5GZYQ',
'GTEX-111YS-1226-SM-5EGGJ',
'GTEX-111YS-1326-SM-5EGGK',
'GTEX-111YS-1426-SM-5GID8',
'GTEX-111YS-1526-SM-5GZYW',
'GTEX-111YS-1626-SM-5GZZ9',
'GTEX-111YS-1726-SM-5GIED',
'GTEX-111YS-1826-SM-5GIEP',
'GTEX-111YS-1926-SM-5GICC',
'GTEX-111YS-2026-SM-5EGGL',
'GTEX-111YS-2126-SM-5EGGM',
'GTEX-111YS-2226-SM-5987P',
'GTEX-111YS-2326-SM-5987L',
'GTEX-111YS-2426-SM-5GZZQ',
'GTEX-1122O-0003-SM-5Q5DL',
'GTEX-1122O-0005-SM-5O99J',
'GTEX-1122O-0008-SM-5QGR2',
'GTEX-1122O-0126-SM-5GICA',
'GTEX-1122O-0226-SM-5N9DA',
'GTEX-1122O-0326-SM-5H124',
'GTEX-1122O-0426-SM-5H12G',
'GTEX-1122O-0526-SM-5N9DM',
'GTEX-1122O-0626-SM-5N9B9',
'GTEX-1122O-0726-SM-5GIEV',
'GTEX-1122O-0826-SM-5GICV',
'GTEX-1122O-0926-SM-5N9C9',
'GTEX-1122O-1126-SM-5NQ8X',
'GTEX-1122O-1226-SM-5H113',
'GTEX-1122O-1326-SM-5H11F',
'GTEX-1122O-1426-SM-5H11R',
'GTEX-1122O-1526-SM-5N9CL',
'GTEX-1122O-1626-SM-5N9CX',
'GTEX-1122O-1726-SM-5GICM',
'GTEX-1122O-1826-SM-5EGIP',
'GTEX-1122O-1926-SM-5EGIQ',
'GTEX-1122O-2026-SM-5NQ91',
'GTEX-1122O-2126-SM-5EGIR',
'GTEX-1122O-2426-SM-5GIDN',
'GTEX-1128S-0005-SM-5P9HI',
'GTEX-1128S-0008-SM-5Q5DP',
'GTEX-1128S-0126-SM-5H12S',
'GTEX-1128S-0326-SM-5GZZF',
'GTEX-1128S-0726-SM-5N9D6',
'GTEX-1128S-0826-SM-5GZZI',
'GTEX-1128S-0926-SM-5GZZU',
'GTEX-1128S-1626-SM-5H12O',
'GTEX-1128S-2126-SM-5H12U',
'GTEX-1128S-2326-SM-5GZZY',
'GTEX-1128S-2426-SM-5H11B',
'GTEX-1128S-2526-SM-5H11N',
'GTEX-1128S-2726-SM-5H12C',
'GTEX-1128S-2826-SM-5N9DI',
'GTEX-113IC-0006-SM-5NQ9C',
'GTEX-113IC-0008-SM-5QGRF',
'GTEX-113IC-0226-SM-5HL5C',
'GTEX-113IC-0426-SM-5HL5O',
'GTEX-113IC-0526-SM-5HL76',
'GTEX-113JC-0006-SM-5O997',
'GTEX-113JC-0008-SM-5QGR6',
'GTEX-113JC-0126-SM-5EGJW',
'GTEX-113JC-0726-SM-5GZZR',
'GTEX-113JC-0926-SM-5H114',
'GTEX-113JC-1026-SM-5H117',
'GTEX-113JC-1926-SM-5N9GE',
'GTEX-113JC-2226-SM-5EGJG',
'GTEX-113JC-2326-SM-5EQ4E',
'GTEX-113JC-2726-SM-5EGIS',
'GTEX-113JC-2826-SM-5EGIT',
'GTEX-117XS-0005-SM-5PNU6',
'GTEX-117XS-0008-SM-5Q5DQ',
'GTEX-117XS-0426-SM-5GZZN',
'GTEX-117XS-0526-SM-5987Q',
'GTEX-117XS-0726-SM-5H131',
'GTEX-117XS-1926-SM-5GICO',
'GTEX-117XS-2026-SM-5GID1',
'GTEX-117XS-2226-SM-5GIE1',
'GTEX-117XS-2526-SM-5H11G',
'GTEX-117XS-2726-SM-5N9BL',
'GTEX-117XS-2926-SM-5N9BX',
'GTEX-117XS-3026-SM-5N9CA',
'GTEX-117XS-3126-SM-5GIDP',
'GTEX-117YW-0005-SM-5NQ8Z',
'GTEX-117YW-0008-SM-5QGRD',
'GTEX-117YW-0126-SM-5EGGN',
'GTEX-117YW-0226-SM-5N9CM',
'GTEX-117YW-0326-SM-5N9CY',
'GTEX-117YW-0526-SM-5H11C',
'GTEX-117YW-0826-SM-5H11O',
'GTEX-117YW-1426-SM-5EGGO',
'GTEX-117YW-1526-SM-5EGGP',
'GTEX-117YW-1826-SM-5PNY5',
'GTEX-117YW-1926-SM-5H121',
'GTEX-117YW-2026-SM-5N9BH',
'GTEX-117YW-2226-SM-5N9DB',
'GTEX-117YW-2426-SM-5Q5AE',
'GTEX-117YW-2626-SM-5GZZH',
'GTEX-117YW-2826-SM-5H116',
'GTEX-117YX-0126-SM-5EGH5',
'GTEX-117YX-0226-SM-5EGH6',
'GTEX-117YX-0326-SM-5GICL',
'GTEX-117YX-0526-SM-5EGJH',
'GTEX-117YX-0626-SM-5EGJI',
'GTEX-117YX-0726-SM-5GIET',
'GTEX-117YX-0826-SM-5H11J',
'GTEX-117YX-1026-SM-5H11V',
'GTEX-117YX-1126-SM-5H128',
'GTEX-117YX-1226-SM-5H11S',
'GTEX-117YX-1326-SM-5H125',
'GTEX-117YX-1426-SM-5H12H',
'GTEX-117YX-1526-SM-5H12T',
'GTEX-117YX-1626-SM-5GZZG',
'GTEX-117YX-1726-SM-5GZZS',
'GTEX-117YX-1826-SM-5H12K',
'GTEX-117YX-1926-SM-5GIEC',
'GTEX-117YX-2126-SM-5GIEL',
'GTEX-117YX-2226-SM-5EGJJ',
'GTEX-117YX-2326-SM-5H12W',
'GTEX-117YX-2426-SM-5GZZJ',
'GTEX-117YX-2526-SM-5EQ4Q',
'GTEX-117YX-2626-SM-5EQ53',
'GTEX-1192W-0005-SM-5NQBQ',
'GTEX-1192W-0008-SM-5QGRE',
'GTEX-1192W-0126-SM-5EGGS',
'GTEX-1192W-0226-SM-5EGGT',
'GTEX-1192W-2626-SM-5Q5AF',
'GTEX-1192W-3026-SM-5GZZV',
'GTEX-1192X-0008-SM-5Q5B7',
'GTEX-1192X-0226-SM-5H12D',
'GTEX-1192X-0326-SM-5H11U',
'GTEX-1192X-0426-SM-5GIEE',
'GTEX-1192X-0726-SM-5987R',
'GTEX-1192X-1126-SM-5EGGU',
'GTEX-1192X-1526-SM-5H11I',
'GTEX-1192X-1826-SM-5GIE2',
...]
However, when I use this column list to select these rows, the returned DataFrame is all zeros:
np.all(curtable.loc[:, float_cols] == 0)
True
But when I just use one column, seems like there is no issue.
np.all(curtable.loc[:, float_cols[0]] == 0)
False
Problem description
My understanding is that the .loc
indexer can be supplied with a list of values from the index/columns to select those cells within a table. Perhaps this is an issue when working with particularly large tables (in my case, 56000 by 8500)
Expected Output
Output of pd.show_versions()
Comment From: jreback
can you show a repro example. not even sure what the issue is.
Comment From: jreback
pls comment if you have a repo