E57_ERROR_INTERNAL exceptions when trying to read data3d with 0 points #262

nh2 · 2023-10-20T12:59:46Z

Hi, thanks for the library.

I observed that when creating a .reader() for a data3d entry that has 0 points, E57_ERROR_INTERNAL gets thrown, likely incorrectly.

The 2 throw locations are these (linking code from v2.2.0 but that code seems the same with master):

Case 1:

libE57Format/src/Packet.cpp

Lines 97 to 101 in 65c0d23

    
           /// Offset can't be 0 
        
           if ( packetLogicalOffset == 0 ) 
        
           { 
        
              throw E57_EXCEPTION2( E57_ERROR_INTERNAL, "packetLogicalOffset=" + toString( packetLogicalOffset ) ); 
        
           }

Case 2:

libE57Format/src/CompressedVectorReaderImpl.cpp

Lines 103 to 110 in 65c0d23

    
           if ( sectionLogicalStart == 0 ) 
        
           { 
        
              //??? should have caught this before got here, in XML read, get this if CV 
        
              // wasn't written to 
        
              // by writer. 
        
              throw E57_EXCEPTION2( E57_ERROR_INTERNAL, 
        
                                    "imageFileName=" + cVector_->imageFileName() + " cvPathName=" + cVector_->pathName() ); 
        
           }

With my real E57 file I encountered the issue on, the first one is thrown, and with a mini synthetic empy E57 point cloud I tried to craft with help of CloudCompare and Meshlab, the second one is thrown.

For the second one, here is a file to repro: empty-cloud.e57.zip

I believe that for such a 0-point point cloud, creating a .reader() from it should work, simply reading 0 points, and not give E57_ERROR_INTERNAL.

The text was updated successfully, but these errors were encountered:

asmaloney · 2023-10-20T13:53:51Z

Hi Niklas.

I'm trying to reproduce this.

Are you getting this with the latest release or an older version?

When you are talking about .reader() for a data3d what functions are you calling specifically? i.e. is it failing when creating the e57::Data3DPointsFloat, when calling SetUpData3DPointsData, or when calling .read()?

(Do you mind if I add this E57 file to the test suite?)

asmaloney · 2023-10-20T14:25:59Z

I was able to reproduce the second one (after removing a specific check for 1 or more points in E57SimpleData.cpp). I don't currently see a path to reproducing the first one.

Not quite sure how to fix it at the moment because the SetUpData3DPointsData function creates and returns the CompressedVectorReader (which is what's throwing), and the function signature does not allow for an error to be returned. I guess I could at least throw from there with a clearer error message and how to solve it?

The simple solution is to not call that if you have 0 points :-) You do have that information from the Data3D header before calling into SetUpData3DPointsData.

Here's a test that outlines it:

TEST( SimpleReaderData, NoPoints )
{
   e57::Reader *reader = nullptr;

   E57_ASSERT_NO_THROW( reader = new e57::Reader( TestData::Path() + "/self/no-points.e57", {} ) );

   ASSERT_TRUE( reader->IsOpen() );
   EXPECT_EQ( reader->GetImage2DCount(), 0 );
   EXPECT_EQ( reader->GetData3DCount(), 1 );

   e57::E57Root fileHeader;
   ASSERT_TRUE( reader->GetE57Root( fileHeader ) );

   CheckFileHeader( fileHeader );
   EXPECT_EQ( fileHeader.guid, "{EC1A0DE4-F76F-44CE-E527-789EEB818347}" );

   e57::Data3D data3DHeader;
   ASSERT_TRUE( reader->ReadData3D( 0, data3DHeader ) );

   ASSERT_EQ( data3DHeader.pointCount, 0 );

   const uint64_t cNumPoints = data3DHeader.pointCount;

   if ( cNumPoints > 0 )
   {
      e57::Data3DPointsFloat pointsData( data3DHeader );

      auto vectorReader = reader->SetUpData3DPointsData( 0, cNumPoints, pointsData );

      const uint64_t cNumRead = vectorReader.read();

      vectorReader.close();

      EXPECT_EQ( cNumRead, cNumPoints );
   }

   delete reader;
}

See ASTM standard Table 9: "recordCount" shall be in the interval [0, 2^63). Part of #262

nh2 · 2023-10-20T14:52:17Z

Are you getting this with the latest release or an older version?

I'm testing with v2.2.0.

When you are talking about .reader() for a data3d what functions are you calling specifically?

I'm calling from pye57 reader = points.reader(dest_buffers), where points is the field in a data3d vector entry.

I think this calls CompressedVectorNode::reader()

Here are the GDB stack traces for Case 1 from the issue description:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7cec7b1a in e57::PacketReadCache::lock(unsigned long, char*&) [clone .cold] ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7ceeacc4 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) () from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cee8231 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#5  0x00007fff7cefb66d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()

My debug print I added prints e.errorCode() 11 e.context() packetLogicalOffset=0.

I'll try to produce the one for Case 2 in a moment as well.

(Do you mind if I add this E57 file to the test suite?)

You can definitely use it. The XML looks a bit odd when viewed in less, but that might be because of embedded binary data?

Meshlab crashed in the middle of writing it (I filed cnr-isti-vclab/meshlab#1433 for that) but it seems to still have written it.

asmaloney · 2023-10-20T15:05:00Z

I'm testing with v2.2.0.

Ah. The "simple reader" stuff was significantly revamped for version 3, so it's possible nothing I do here will apply to 2.x.

The XML looks a bit odd...

The XML is OK. The E57 format stores binary data followed by XML and the whole file includes CRC checksums on each 1020 bytes, so that's why you get problems with less. Here's what it looks like when read properly:

<?xml version="1.0"?>
<e57Root type="Structure" xmlns="http://www.astm.org/COMMIT/E57/2010-e57-v1.0" xmlns:nor="http://www.libe57.org/E57_NOR_surface_normals.txt">
    <formatName type="String"><![CDATA[ASTM E57 3D Imaging Data File]]></formatName>
    <guid type="String"><![CDATA[{EC1A0DE4-F76F-44CE-E527-789EEB818347}]]></guid>
    <versionMajor type="Integer">1</versionMajor>
    <versionMinor type="Integer" />
    <e57LibraryVersion type="String"><![CDATA[E57Format-2.2.0-x86_64-linux-gcc122]]></e57LibraryVersion>
    <data3D type="Vector" allowHeterogeneousChildren="1">
        <vectorChild type="Structure">
            <guid type="String"><![CDATA[{50c4acef-3526-4a44-b6b8-2d32c8457814}]]></guid>
            <pose type="Structure">
                <rotation type="Structure">
                    <w type="Float">1.00000000000000000e+00</w>
                    <x type="Float" />
                    <y type="Float" />
                    <z type="Float" />
                </rotation>
                <translation type="Structure">
                    <x type="Float" />
                    <y type="Float" />
                    <z type="Float" />
                </translation>
            </pose>
            <points type="CompressedVector" fileOffset="0" recordCount="0">
                <prototype type="Structure">
                    <cartesianX type="Float" precision="single" />
                    <cartesianY type="Float" precision="single" />
                    <cartesianZ type="Float" precision="single" />
                    <nor:normalX type="Float" precision="single" minimum="-1.0000000e+00" maximum="1.0000000e+00" />
                    <nor:normalY type="Float" precision="single" minimum="-1.0000000e+00" maximum="1.0000000e+00" />
                    <nor:normalZ type="Float" precision="single" minimum="-1.0000000e+00" maximum="1.0000000e+00" />
                </prototype>
                <codecs type="Vector" allowHeterogeneousChildren="1" />
            </points>
        </vectorChild>
    </data3D>
    <images2D type="Vector" allowHeterogeneousChildren="1" />
</e57Root>

(Edit: Oh - and I'm afraid that stack trace doesn't really give any useful information about where it's coming from. I don't know the Python stuff, so it's possible it needs to add the check for 0 points before trying to read like I have in the test I posted above?)

nh2 · 2023-10-20T15:09:50Z

I'll try to produce the one for Case 2 in a moment as well.

Here we go, for the repro file attached above:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7ceb0ba7 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) [clone .cold] () from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7cee8341 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cefb77d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()

My debug print prints

e.errorCode() 11 e.context() imageFileName=/path/to/empty-cloud.e57 cvPathName=/data3D/0/points

nh2 · 2023-10-20T15:28:21Z

I don't currently see a path to reproducing the first one.

@asmaloney I have a real-world E57 file I have that causes it, which is probably the fastest way to repro Case 1.

But it's 38 GB and I can't post it publicly.
May I share it with you privately, and would you be up for sending me an email agreeing that you can use it only for the purpose of investigating this bug and to not distribute it further?

I also tried to downsample it with CloudCompare for easier sharing in the hope that this would preserve the 0-points data3d entry (of which there's only one in the file), but CloudCompare crashes on it with

[E57] Error: bad SourceDestBuffer (E57_ERROR_BAD_BUFFER)
    context: pathName=cartesianX
An error occurred while loading 'MainHouse': the third-party library in charge of saving/loading the file has thrown an exception
```, 

CloudCompare also uses `libE57Format` according to https://github.com/CloudCompare/CloudCompare/blob/e243efbb3e6ae65a96e42da7620fd46771f90fbe/CHANGELOG.md?plain=1#L972-L974

However, here the error is different, and this downsampling problem might be CloudCompare's own problem.

asmaloney · 2023-10-20T15:28:46Z

So it sounds like there are a couple of possible issues here:

you're using a Python wrapper
it's using an old version of the code

This library uses exceptions everywhere (certainly not my choice!), not error return. So the "right" way to use it is to wrap calls to the API with try/catch. This would have to be done in the Python wrapper.

Two solutions for the Python wrapper code:

Add try/catch to handle exceptions
Check the number of points before trying to read

asmaloney · 2023-10-20T15:34:00Z

But it's 38 GB

😆 Yikes. I don't think I can deal with that right now.

Can you post a full stacktrace? The frames in the one you posted don't really tell us much other than were the exception was thrown.

What software produced the file? (Hopefully the e57LibraryVersion in the XML in the file should tell us.)

asmaloney · 2023-10-20T16:13:04Z

(I sent an email to the address in your profile to followup.)

nh2 · 2023-10-21T13:35:33Z

Two solutions for the Python wrapper code:

Some clarifications:

The Python wrapper pye57 does correctly catch all exceptions. We see the gdb stack trace not because an exception bubbles up to the top-level, but because I used break _Unwind_RaiseException in gdb to get the trace to the two code locations I pointed out in the issue description, to support that those are really the right locations and which path is taken there.
I agree that checking the number of points before trying to read is a workaround, which I'm using currently.
It still seems worthwhile to figure out whether this error should be raised. From my understanding, E57_ERROR_INTERNAL should not surface if the caller does everything correctly, which it seems to do in my case.

Can you post a full stacktrace? The frames in the one you posted don't really tell us much other than were the exception was thrown.

Here are the full stack traces for the two cases, but I don't think it's too helpful because the immediate frame on top of what I posted is already pybind11, Python's C++ binding:

Case 1:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7cec7b1a in e57::PacketReadCache::lock(unsigned long, char*&) [clone .cold] ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7ceeacc4 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) () from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cee8231 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#5  0x00007fff7cefb66d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#6  0x00007fff7cfa2907 in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}, e57::CompressedVectorReader, e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}&&, e57::CompressedVectorReader (*)(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#7  0x00007fff7cf93979 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#8  0x00007ffff7c65103 in cfunction_call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#9  0x00007ffff7b1f930 in _PyObject_MakeTpCall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#10 0x00007ffff7b28592 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#11 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#12 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#13 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#14 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#15 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#16 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#17 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#18 0x00007ffff7a63221 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#19 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#20 0x00007ffff7aad76c in PyVectorcall_Call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#21 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#22 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#23 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#24 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#25 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#26 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#27 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#28 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#29 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#30 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#31 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#32 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#33 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#34 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#35 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#36 0x00007ffff7b285f4 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#37 0x00007ffff7c6b258 in thread_run () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#38 0x00007ffff7a749c2 in pythread_wrapper () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
--Type <RET> for more, q to quit, c to continue without paging--
#39 0x00007ffff789fe24 in start_thread () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6
#40 0x00007ffff79219b0 in clone3 () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6

Case 2:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7ceb0ba7 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) [clone .cold] () from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7cee8341 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cefb77d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#5  0x00007fff7cfa2b47 in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}, e57::CompressedVectorReader, e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}&&, e57::CompressedVectorReader (*)(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#6  0x00007fff7cf93bb9 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#7  0x00007ffff7c65103 in cfunction_call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#8  0x00007ffff7b1f930 in _PyObject_MakeTpCall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#9  0x00007ffff7b28592 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#10 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#11 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#12 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#13 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#14 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#15 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#16 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#17 0x00007ffff7a63221 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#18 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#19 0x00007ffff7aad76c in PyVectorcall_Call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#20 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#21 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#22 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#23 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#24 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#25 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#26 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#27 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#28 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#29 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#30 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#31 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#32 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#33 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#34 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#35 0x00007ffff7b285f4 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#36 0x00007ffff7c6b258 in thread_run () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#37 0x00007ffff7a749c2 in pythread_wrapper () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#38 0x00007ffff789fe24 in start_thread () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6
#39 0x00007ffff79219b0 in clone3 () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6

What software produced the file? (Hopefully the e57LibraryVersion in the XML in the file should tell us.)

The Faro SCENE proprietary laser scan processing software (e57LibraryVersion: SCENE_E57RefImpl-1.1.312).

asmaloney · 2023-10-21T14:01:48Z

OK - thanks. Yes those stack traces are not useful 😄

It still seems worthwhile to figure out whether this error should be raised. From my understanding, E57_ERROR_INTERNAL should not surface if the caller does everything correctly, which it seems to do in my case.

One of the design principles from the original lib was that all objects should be in a reasonable state and that no pointers are used for the objects in the API. That means no returning pointers & checking for nullptr on error for example.

So the problem we have in this case is that SetUpData3DPointsData returns a CompressedVectorReader, but a CompressedVectorReader is only valid when there are > 0 points. So we are kind of stuck because of the design.

One solution might be to change the function signature for SetUpData3DPointsData to do something different, but again the rest of the library doesn't return errors; it uses exceptions. Another might be to modify CompressedVectorReader to allow for 0 points. This is probably a better solution and I will look into this (though I'm hesitant to touch this code 😆 ).

For now, I agree about returning E57_ERROR_INTERNAL. I will change it to catch this issue earlier in SetUpData3DPointsData and throw a more useful error.

The Faro SCENE proprietary laser scan processing software (e57LibraryVersion: SCENE_E57RefImpl-1.1.312).

Ah interesting - that's a (possibly modified) earlier version of the library that E5Format was derived from (E57RefImpl 1.1.332). It will likely suffer from the some of the same things as this library. Even though it was called a reference library it didn't follow the standard in some cases...

nh2 · 2023-10-21T20:02:42Z

Copying here one part from our email exchange:

@asmaloney found that my SCENE-written file I provided certainly does not write the 0-point cloud spec-compliantly, because it does not write a data index, but the spec requires that:

The original E57RefImpl (and libE57Format because it comes from E57RefImpl) doesn't create/store any indices. It's required by the standard.

E57 Standard: 9.3.5:

A CompressedVector shall contain at least one index packet and at least one data packet.

This means that all E57-writing programs using E57RefImpl will likely write the 0-points pointcloud wrong.

We haven't figured out what the way to do it correctly is; quoting @asmaloney:

It's unclear what one is supposed to write when you have 0 points - both in the header and in the binary data. It's not specified in the standard. With E57 Standard: 9.3.5 (see next comment), an index and data packet are also required. Maybe it's possible to write each of those with zero entries, but now it's getting ridiculous...

I agree with that, it's definitely weird for the spec to require these to exist.

…ith zero points Instead of throwing ErrorInternal which is a bit misleading, throw a new exception ErrorData3DReadInvalidZeroRecords. It looks like this: trying to read an invalid Data3D with zero records - check for zero records before trying to read this Data3D section (ErrorInvalidZeroRecordsData3D): fileOffset cannot be 0; cvPathName=/data3D/0/points imageFileName=ZeroPointsInvalid.e57 Part of #262

…ith zero points (#264) Instead of throwing ErrorInternal which is a bit misleading, throw a new exception ErrorData3DReadInvalidZeroRecords. It looks like this: trying to read an invalid Data3D with zero records - check for zero records before trying to read this Data3D section (ErrorInvalidZeroRecordsData3D): fileOffset cannot be 0; cvPathName=/data3D/0/points imageFileName=ZeroPointsInvalid.e57 Part of #262

… packet Part of #262

… packet (#266) Part of #262

A compressed vector with 0 points must still have a data packet. Part of #262

asmaloney · 2023-10-26T14:18:00Z

OK - I'm going to call this "done".

Trying to read a Data3D with an ill-formed header will now give an ErrorData3DReadInvalidZeroRecords exception instead of ErrorInternal. (#264)
It looks like this:

trying to read an invalid Data3D with zero records - check for zero records before trying to read this Data3D section (ErrorInvalidZeroRecordsData3D): fileOffset cannot be 0; cvPathName=/data3D/0/points imageFileName=ZeroPointsInvalid.e57
Properly write a Data3D section with 0 points by including an empty data packet. (#266)
Properly read a Data3D section with 0 points if it includes an empty data packet. (#267)

Note that this doesn't address the missing (required) index packet. That is a failure to comply with the standard, but does not throw any exceptions.

nh2 · 2023-10-27T14:49:11Z

@asmaloney Thanks!

asmaloney added a commit that referenced this issue Oct 20, 2023

A Data3D section is allowed to have 0 points

1b0862e

See ASTM standard Table 9: "recordCount" shall be in the interval [0, 2^63). Part of #262

asmaloney added a commit that referenced this issue Oct 20, 2023

A Data3D section is allowed to have 0 points

8ee405e

See ASTM standard Table 9: "recordCount" shall be in the interval [0, 2^63). Part of #262

asmaloney mentioned this issue Oct 20, 2023

A Data3D section is allowed to have 0 points #263

Merged

asmaloney added a commit that referenced this issue Oct 20, 2023

A Data3D section is allowed to have 0 points (#263)

f1c3940

See ASTM standard Table 9: "recordCount" shall be in the interval [0, 2^63). Part of #262

asmaloney mentioned this issue Oct 22, 2023

Give better error information when trying to read an invalid Data3D with zero points #264

Merged

asmaloney added a commit that referenced this issue Oct 24, 2023

{standard} A compressed vector with 0 records must still write a data…

8fa34d3

… packet Part of #262

asmaloney added a commit that referenced this issue Oct 24, 2023

{standard} A compressed vector with 0 records must still write a data…

9fac5e1

… packet Part of #262

asmaloney mentioned this issue Oct 24, 2023

{standard} A compressed vector with 0 records must still write a data packet #266

Merged

asmaloney added a commit that referenced this issue Oct 26, 2023

{standard} A compressed vector with 0 records must still write a data…

d4d5496

… packet (#266) Part of #262

asmaloney added a commit that referenced this issue Oct 26, 2023

{standard} Read a properly-formed compressed vector with 0 points

43d19e2

A compressed vector with 0 points must still have a data packet. Part of #262

asmaloney added a commit that referenced this issue Oct 26, 2023

{standard} Read a properly-formed compressed vector with 0 points

9fd689d

A compressed vector with 0 points must still have a data packet. Part of #262

asmaloney mentioned this issue Oct 26, 2023

{standard} Read a properly-formed compressed vector with 0 points #267

Merged

asmaloney added a commit that referenced this issue Oct 26, 2023

{standard} Read a properly-formed compressed vector with 0 points (#267)

96ebf2b

A compressed vector with 0 points must still have a data packet. Part of #262

asmaloney closed this as completed Oct 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E57_ERROR_INTERNAL exceptions when trying to read data3d with 0 points #262

E57_ERROR_INTERNAL exceptions when trying to read data3d with 0 points #262

nh2 commented Oct 20, 2023 •

edited

Loading

asmaloney commented Oct 20, 2023

asmaloney commented Oct 20, 2023

nh2 commented Oct 20, 2023 •

edited

Loading

asmaloney commented Oct 20, 2023 •

edited

Loading

nh2 commented Oct 20, 2023

nh2 commented Oct 20, 2023

asmaloney commented Oct 20, 2023

asmaloney commented Oct 20, 2023

asmaloney commented Oct 20, 2023

nh2 commented Oct 21, 2023

asmaloney commented Oct 21, 2023

nh2 commented Oct 21, 2023

asmaloney commented Oct 26, 2023

nh2 commented Oct 27, 2023

E57_ERROR_INTERNAL exceptions when trying to read data3d with 0 points #262

E57_ERROR_INTERNAL exceptions when trying to read data3d with 0 points #262

Comments

nh2 commented Oct 20, 2023 • edited Loading

asmaloney commented Oct 20, 2023

asmaloney commented Oct 20, 2023

nh2 commented Oct 20, 2023 • edited Loading

asmaloney commented Oct 20, 2023 • edited Loading

nh2 commented Oct 20, 2023

nh2 commented Oct 20, 2023

asmaloney commented Oct 20, 2023

asmaloney commented Oct 20, 2023

asmaloney commented Oct 20, 2023

nh2 commented Oct 21, 2023

asmaloney commented Oct 21, 2023

nh2 commented Oct 21, 2023

asmaloney commented Oct 26, 2023

nh2 commented Oct 27, 2023

nh2 commented Oct 20, 2023 •

edited

Loading

nh2 commented Oct 20, 2023 •

edited

Loading

asmaloney commented Oct 20, 2023 •

edited

Loading