PyTest failure in Python 3.11 and above caused by calling a method that throws in a generated class #1178
Comments
|
This should be simple to fix. The problem should be in native/common/jp_exception.cpp where we set the exception information. tb_lineno is one of the fields we set. My guess it that we didn’t have a line number in the compiled code so we left it as NULL. The PyTest code may incorrect as it is blindly assuming that the result can’t be None, which I don’t see as a requirement in the spec. I will need to investigate further and read the some PEPs.
|
|
I was unable to replicate the issue from your example. I get |
|
At this point, I am almost certain this is a pytest bug, since this code still cause the internal error: def test_reproducer():
try:
code_that_cause_exception()
except:
raise Exception |
|
Forgot that Reproducer is https://github.com/Christopher-Chianelli/issue-reproducer/tree/jpype-1178 |
Christopher-Chianelli
changed the title
|
If the generated class has source information |
|
Unfortunately I got So I was unable to build the reproducer to verify the test. Sorry! I will try again next weekend to see if I can sort out what is going on. Unfortunately, as I don't use maven it doesn't make much sense to me. The specific call where we mess with linenum is in native/common/jp_exception.cpp As you can see we are calling the constructor with a valid traceback linenum. If there is a problem then most likely traceback changed its signature starting in 3.11 and we would need to adjust the constructor call to account for it. I scanned the Python code base and don't see an errors in the handoff (though admittedly the code is very ugly....) As you can see form this code it may be the same error as in the PR. There is a potential the lost exception is tripping a problem in the copied traceback. But if that was the case the traceback would have failed without producing, and not one with Py_None in the linenum slot. Thus I still don't have a good explanation for what could be happening here. |
|
I think you checked out the wrong branch; the branch should be |
|
Ah... will try again later them. |
|
I got Is this correct or incorrect? |
|
This is odd,
I don't know why it randomly worked then failed after a This is what it looks like when the internal error happens: mvn clean install
python -m venv venv
. venv/bin/activate
pip install JPype1 pytest
pytest
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: .../issue-reproducer
collected 2 items
tests/test_reproducer.py
INTERNALERROR> Traceback (most recent call last):
...Note the second test is not even run. |
* Python 3.12 changed the meaning of the arg for both COMPARE_OP
and JUMP instructions:
* For COMPARE_OP, the index now starts at the fifth-lowest bit.
* For JUMP instructions,
"The argument of a jump is the offset of the target instruction relative to the instruction that appears immediately after the jump instruction’s CACHE entries.
As a consequence, the presence of the CACHE instructions is transparent for forward jumps but needs to be taken into account when reasoning about backward jumps."
For both of these, we parse the argrepr. In particular:
* For COMPARE_OP, it be "<", ">", "==", "!=", ">=" or "<=".
* For JUMP instructions, it be "to BYTECODE_OFFSET", which is exactly
twice the instruction index.
* Python 3.12 changed the meaning of FOR_ITER to leave the iterator on
the stack (and push an undocumented object) when exhausted,
and added the instruction END_FOR to pop them both off. However, since
the target for FOR_ITER is always END_FOR (see https://github.com/python/cpython/blob/17a82a1d16a46b6c008240bcc698619419ce5554/Python/bytecodes.c#L2289-L2293),
we can "optimize" it by using the old implementation and making END_FOR a no-op.
* Python 3.12 merged LOAD_ATTR and LOAD_METHOD into one opcode.
* Python 3.12 added BINARY_SLICE and STORE_SLICE opcodes to implement
`a[start:end]` and `a[start:end] = iterable` (previously,
BUILD_SLICE was used followed by `GET_ITEM` or `SET_ITEM`).
* Python 3.12 added END_SEND for generator cleanup, which we
implement as a NOOP.
* Python 3.12 added CLEANUP_THROW and INTRINSIC_STOPITERATION_ERROR,
which replace TOS with its value if TOS a StopIteration,
else reraise the error on TOS. CLEANUP_THROW additionally pops
two additional values off the stack.
* Python 3.12 added LOAD_FAST_CHECK and LOAD_FAST_AND_CLEAR.
LOAD_FAST now will never raise unbound local error,
LOAD_FAST_CHECK might raise an unbound local error,
and LOAD_FAST_AND_CLEAR will load NONE when the local is unbound
(and additionally sets the local to null).
Since Java rightfully says "???" when it sees code that can access
an unbound local, we need to set locals used by
LOAD_FAST_AND_CLEAR to null before generating code
for opcodes.
* Python 3.12 added RETURN_CONST to return a constant.
* Python 3.12 changed UNARY_POS from a "fast" opcode to a
slow "intrinsic" opcode since "+a" is rarely used. This
does not affect us (other than having a new alternative
way of writing "UNARY_POS" as
"INTRINSIC_1_INTRINSIC_UNARY_POSITIVE".
* Simplified and fix some bugs in PythonSlice code
* Add a workaround for jpype-project/jpype#1178
|
any lead on a fix here? I can't reproduce this locally but fails on github actions |
|
I haven't touched it as I have no reproducing method. Could it be a particular version of Python 3.11. They backported some broken behavior in. It was the stuff of nightmares when I was dealing with 3.13 as what was ported in had wotk arounds only in later versions. Hence the comment... "Python 3.11 is dead to me." |
|
This problem has cropped up again in my Python 3.12 testing for release. Not sure yet how to deal with it. |
|
I have identified this issue. It is maddening. We are setting the line number to -1 on an integer field. Depending on the system and how it is called -1 will be changed to None. This is actually correct behavior because None means don't print a meaningless line number. But the pytest is trying to do math with it. We are once again the victim of random changes in Python code base. |
Has this been reported upstream? I suggest that this sounds like a bug with pytest doing math on something that is now optional (but which wasn't optional before) |
|
I am not sure on that one. I was able to replicate it locally, but I was focused on avoiding the bug rather the packaging it into a workable replicator. One would have to find out which versions of Python and which versions of pytest. My testing showed the Windows 3.12.6 was the one producing the None from -1. |
Currently, to see results from PyTest failures caused by an Exception with a cause in Python 3.11 and above, I need to have this
@JImplementationFor:To reproduce:
Python test:
Java class:
Exception from PyTest:
The text was updated successfully, but these errors were encountered: