Python 3 support

[tfeldmann]

pysqlcipher cannot be installed with

sudo pip3 install pysqlcipher

because it fails with syntax errors for the python2 print statement.

[kalikaneko]

See some general comments on #11. Patches or reports beyond the print statement are welcome.

[rigglemania]

Because this was the only project preventing me from using Python 3 in one of my projects, I looked at what would be required to get this working in Python 3 and while it definitely required some work, I think I have a working port verified by being able to encrypt/decrypt a SQLite database in a test project. Due to the significance of the changes (including starting from scratch for the entire src directory), I think a new repo is the best approach (i.e. pysqlcipher3). I'll try to get a repo set up in the next few days with my changes, but here's an overview of what I had to do.

1. Fixed all print statements
2. Fixed some encoding issues in reading files due to changes in Python 3
3. Fixed issues with checking for the type of an instance (ListType and TupleType are no longer valid)
4. Grabbed the latest Python core source code and copied the files from /Modules/_sqlite/* into the /src directory of this project (deleting all existing files first). From the research I did, why the pysqlite repo seems to only support Python 2, pysqlite is the external name for the core sqlite3 library, which is why I grabbed the source files from there.
5. Merged changes from /Lib/sqlite3/* from the Python core source code into the /lib directory of this project
6. Renamed the extension in setup.py from pysqlcipher._sqlite to pysqlcipher._sqlite3 as in Python 3, naming conventions of the init method in module.c matter as the module names much match exactly.

Hopefully, I'm on the right track (I'm still relatively new to Python, but am an experienced software developer). Let me know if you have any questions or thoughts.

[tfeldmann]

@rigglemania This is the best thing I read all day. Thank you for your work.

[kalikaneko]

exciting! I wonder if the py3 version will be able to be run on python2.7 also.
waiting to see this new repo :)

[rigglemania]

No, unfortunately the python 3.4 version won't be able to be run on python 2.7. While some of the simple python 3 changes like the print statement could possibly be handled with the future module, the changes to the concrete types layer used in the .c and .h files seem like to big of changes. I'm making progress on the repo, but it's going a little bit slower because I'm actually trying to fully understand the setup.py file to clean it up where possible (I'm starting from scratch and including what is actually necessary). Unfortunately, the documentation on disttools is pretty weak so I'm finding I'm learning the most from actually digging through the source code.

[rigglemania]

@kalikaneko What's the purpose of the build_static command? Is this something that is really necessary and if so, how is the functionality used and how can it be tested. To me, it seems like we would be missing the static linking when compiling as described here: https://www.zetetic.net/sqlcipher/introduction/

[kalikaneko]

@rigglemania: it can be ignored, and I probably will remove it from setup. It was an attempt to compile the extension against a statically compiled sqlcipher, but that turned out to be a bad idea that ended nowhere (all the _build_extension method in MyBuildExt class can be ignored too). Currently, if you attempt to execute python setup.py build_static, you end up with a libsqlite.a artifact in the build folder, but I found that it cannot be loaded as an extension (which has to be dynamically linked against interpreter, is my reading. I think what would probably make sense is to first compile sqlcipher statically, and then link extension against that, but that's not what I ended up doing).

as I commented in other issue, I think the best option for a future release will be to follow upstream pysqlite steps and remove the amalgamation-as-default option, and to assume libsqlcipher is somewhere to be found in the system instead.

I'm a extension n00b myself, but I can try to help you with the porting.

Thinking about packaging this for debian/ubuntu and its eventual inclusion (with both py2 and py3 support) maybe one option will be to include two distinct source folders in the same repo, so both packages can be generated from the same tree, and then cleaning up setup.py so that it can be executed with both python versions. Will you be interested in co-maintaining the package?

[kalikaneko]

@rigglemania also, while cleaning setup.py it will make a lot of sense to rename classes so they are more meaningful.

One step on this direction is to clearly mark each of the two builders:
kalikaneko@c155277#diff-2eeaed663bd0d25b7e608891384b7298R378

[rigglemania]

I think we're on the same page as I've already done a lot of things, including some of the renames as well as switching the default to link against system lib rather than the amalgamation. I've got everything working in my local repo now (all unit tests passing) with the exception of the cross_bdist_wininst. Is this class currently working with Python 2.7?

Also, good idea on the different source folders idea to support both Python versions. I think I may be able to get that working without too much effort. I'm not sure about the tests files yet and if there are any Python 3 changes in them.

Finally, before I realized the static lib setup code wouldn't work, I actually had the same lib file being generated (at least the sizes were exactly the same) using the build_clib command with only around 10-20 lines. So if you do end up messing some more with creating the static lib, you should check out this command rather than copying/pasting for the _build_ext method.

[rigglemania]

So looking more at the cross_bdist_wininst file, I ended up deleting this file in my repo and removed the command from the setup.py file. Maybe it could work if building against the amalgamation, but since this isn't recommended for production, I don't see how building a Windows C extension on Linux linking against the Linux SqlCipher library would work.

I've got my repo up now: https://github.com/rigglemania/pysqlcipher3. It works against both Python 2 and Python 3. My testing consisted of both building linking against SQL Cipher (now the default behavior) and building the amalgamation (responsibility is now on user to provider source files similar to pysqlite). Then after installing against both Python versions, I ran all the unit tests, which passed (after a slight modification to a Python 2 test). I also tested creating a simple encrypted DB as described in the usage section. I still want to write a few unit tests for the libsqlcipher functionality, but other than that, I think the repo shouldn't need much more changes.

[molecular]

trying to get OpenBazaar to work which is using pysqlcipher. I'm running into this issue and am not sure how to get around it.

[kalikaneko]

@rigglemania this is awesome!
as you probably saw the cross_bdist_wininst file came from the pysqlite fork. I guess it might be useful to produce a self-contained extension for windows, but I agree that for now windows support might be a bit far.
I will set some time this week for testing your package, and if it looks so well as it seems we could release a 3.0 version in pypi, and the matching debian packages. I think we could have the 3.0 package published but not being the default relase for some time, so that users can adapt to the non-amalgamation by default.

@molecular the current status of pysqlcipher is only python 2.x support. Very likely there will a package available next week with py3 support thanks to rigglemania work!

[rigglemania]

@kalikaneko By removing cross_bdist_wininst.py, it doesn't mean that you can't create a self-contained extension for Windows. It just means that you have to build the extension on Windows and not Linux or Mac (which sounds unsafe to me anyways).

As for my repo, I published a package on PyPi as 'pysqlcipher3' (due to request to so). I was thinking that if you merged my changes into your repo, you could publish the same 'pysqlcipher' package on PyPi and it would work for both versions of python.

Finally, I just checked in some sqlcipher unit tests I wrote as well as a bug fix for the Python 2 version, so you'll want to grab the latest.

[yardenac]

Any updates on this? Should I just use @rigglemania's fork? https://github.com/rigglemania/pysqlcipher3