Repository navigation
Keep SQL Server integer columns exact when they contain NULLs - #11
Open
maharanay22 wants to merge 1 commit into
Open
maharanay22 wants to merge 1 commit into
maharanay22 wants to merge 1 commit into
Conversation
The SQL Server handler built the result DataFrame without dtypes, so a NULL in an int, bigint or bit column made pandas fall back to float64 or object. Large IDs lost precision (9007199254740993 came back as 9007199254740992.0), small values turned into floats (3 -> 3.0) and the columns were reported as FLOAT or DOUBLE instead of INT. Rebuild integer and bit columns that contain NULLs with the pandas nullable Int64 and boolean dtypes. For ODBC connections, also take the column type from cursor.description, which pyodbc fills with the Python type of the column, so DECIMAL, DATE and TIME columns are no longer reported as TEXT or DATETIME.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The SQL Server handler builds the result DataFrame in
_make_table_response()without any dtypes and then guesses the column types from the pandas dtype. When anint,bigintorbitcolumn contains a NULL, pandas falls back tofloat64(orobjectforbit), and that causes three problems:bigintID of9007199254740993comes back as9007199254740992.0, which is a different ID3comes back as3.0FLOAT(orDOUBLEforbit) instead ofINT/TINYINTThis happens with both the native pymssql connection and the ODBC (pyodbc) connection. The ODBC path has a second problem: it ignores
cursor.description[1], which pyodbc fills with the exact Python type of each column. Because it only looks at the pandas dtype,DECIMAL/NUMERICcolumns are reported asTEXT,DATEcolumns asTEXT(orDATETIME, depending on whether the first row is NULL) andTIMEcolumns asTEXTThis change does two things:
Int64andbooleandtypes. This only happens when every non-NULL value in the column is anint(or every one is abool). Columns without NULLs keep their current dtypes, and float, decimal and other columns are left alone. The PostgreSQL handler already returns nullableInt64/booleancolumns, so this matches what MindsDB gets from itcursor.descriptionwhen pyodbc provides one (int,float,Decimal,date,time,datetime,str,bytes,bool). The existing dtype-based guess is still used when the type is missingFixes: no linked issue, because issues are turned off for this repository. The bug is described in full above
Type of change
Verification Process
To ensure the changes are working as expected:
tests/unit/handlers/test_mssql.py, new testsTestMSSQLHandler::test_types_casting_with_nulls(pymssql) andTestMSSQLHandlerODBC::test_odbc_types_from_cursor_description(ODBC)_make_table_response()with cursor descriptions and rows shaped like what pymssql and pyodbc return for abigint/intcolumn with a NULL, plus ODBCdecimal,date,bitandnvarcharcolumnsBefore (current main):
After:
Unit tests. The two new tests fail on main and pass with the fix, and the existing tests still pass:
I didn't have a SQL Server instance to test against, so the rows and descriptions in the tests follow the shapes pymssql and pyodbc return.
ruff checkandruff format --check(0.11.11) pass on the changed files. No dependencies changedAdditional Media:
I didn't record a video or take screenshots. The before and after terminal output above shows the change instead
Checklist:
No documentation change is needed
I used an AI assistant to help investigate this and draft the change and tests, and I reviewed and tested it myself