64 bit PyPy and Windows¶
Getting 64-bit windows to work with PyPy was a long-standing request that was
finally cracked in the otherwise cursed year of 2020. The problem is that we
assume that the integer type of RPython (rffi.Signed
) is
large enough to (occasionally) contain a pointer value cast to an
integer. On most platforms the corresponding C type of long
satisfies this
condition. But on 64-bit windows, a long
is 32-bits, while
sizeof(void*)
is 64-bits. The simplest fix is to make sure that
rffi.Signed
can hold a 64-bit integer, which resutls in a python2 with the
following incompatibility between CPython and PyPy on Win64:
CPython: sys.maxint == 2**31-1, sys.maxsize == 2**63-1
PyPy: sys.maxint == sys.maxsize == 2**63-1
…and, correspondingly, PyPy2 supports ints up to the larger value of
sys.maxint before they are converted to long
.
What we did¶
The first thing done was to do hack a CPython2
until it fits this model: replace the field in PyIntObject with a long
long
field, and change the value of sys.maxint
. This is available in
nulano’s branch of cpython (again: this is python2).
This hacked pyton was used in the next steps. We’ll call it CPython64/64.
First the tests in
rpython/translator/c/test/
, like test_standalone.py
and
test_newgc.py
were made to pass on top of CPython64/64.
This runs small translations, and some details were
wrong. The most obvious one is to make
the integer type Signed
equal to long
on every other platform, but on Win64 it
should be something like long long
.
Then a more generally review of all the C files in
rpython/translator/c/src
for the word long
, which means a
32-bit integer even on Win64, replaced it with Signed
.
Then, these two C types have corresponding RPython types: rffi.LONG
and lltype.Signed
respectively. The first should really correspond
to the C long
, as verified by the test_rffi_sizeof
test. The
size of the latter is verified in rpython/rlib/rarithmetic
.
Once these basic tests worked, we reviewed rpython/rlib/
for
uses of rffi.LONG
versus lltype.Signed
. The goal was to
fix some more LONG-versus-Signed
issues, by fixing the tests — as
always run on top of CPython64/64. Note that there was some early work
done in rarithmetic
with the goal of running all the
tests on Win64 on the regular CPython, but this early work was abandoned as a
bad idea. Look only at CPython64/64.
This was enough to get a translation of PyPy with -O2
with a minimal set of modules, starting with --no-allworkingmodules
;
using CPython64/64 to run this translation too. Careful checking of
the warnings of the C compiler at the end revealed more places that needed
work. By default, MSVC
reports a lot of mismatches of integer sizes as warnings instead of
errors.
Then we reviewed pypy/module/*/
for LONG-versus-Signed
issues. This got us a working translated
PyPy on Windows 64 that includes all --translationmodules
, i.e.
everything needed to run translations. Once we had that, the hacked
CPython64/64 becomes much less important, because we can run future
translations on top of this translated PyPy. This made it to the nightly
builds on the default branch, and needs to be used by anyone else who wants to
working on Win64. The whole process
ends up with a strange kind of dependency — we need a translated PyPy in
order to translate a PyPy —, but that’s ok here, as Windows executables are
supposed to never be broken by newer versions of Windows.
Happy hacking :-)