Implement fn:transform #4386

adamretter · 2022-05-12T18:53:48Z

Implements the fn:transform function specification

The transform function is complex, so we have chosen to structure it as a number of top level classes in their own
package, org.exist.xquery.functions.fn.transform which are invoked by the main org.exist.xquery.functions.fn.FnTransform class.

The implementation relies on the Saxon XSLT processing library to compile referenced stylesheet(s).

In highly simplified terms, the implementation:

Reads and checks all the parameters/options to the function
Invokes Saxon to compile the identified stylesheet
Retrieves a transformer from the compilation
Figures out which form of application of the stylesheet is required
Constructs (a) delivery object(s) to build output according to the requested delivery format
Calls Saxon again (the transformer this time) applying the stylesheet to the input
Builds the result map using the contents of the delivery objects, and returns it

Unit Tests

A number of unit tests have been added, driven by the an attempt to test the same functionality as some of the individual XQTS tests efficiently during development. These tests do not provide complete coverage of the features
of fn:transform, and the XQTS tests should be considered as the truth.

XQTS Tests

The goal of the development of fn:transform has been to pass all the tests in the fn-transform test set of XQTS tests for transformation.

The tests are run with the eXist XQTS Runner

The test fn-transform-68 currently fails. We are satisfied that the behaviour exhibited by our
code is correct, and that the test is deficient Issue
All other tests from the fn:transform test set pass, except for 4 tests which are not run because they require features that eXist-db does not support (e.g. custom schema types in XQuery) .

This open source contribution to the eXist-db project was commissioned by the Office of the Historian, U.S. Department of State, https://history.state.gov/.

brihaye · 2022-05-12T19:01:18Z

Be careful, you copied the W3C's "FOXT0001" error code too many times, I'm afraid ;-)

adamretter · 2022-05-12T19:30:45Z

Thanks @brihaye I have fixed that now.

exist-core/src/main/java/org/exist/xquery/functions/fn/FnTransform.java

exist-core/src/test/java/xquery/xquery3/XQuery3Tests.java

exist-core/src/main/java/org/exist/xquery/XQueryContext.java

exist-core/src/main/java/org/exist/xquery/functions/fn/FnTransform.java

sonarqubecloud · 2022-06-22T14:54:52Z

SonarCloud Quality Gate failed.

3 Bugs
1 Vulnerability
0 Security Hotspots
55 Code Smells

39.0% Coverage
0.0% Duplication

exist-core/src/main/java/org/exist/xquery/functions/fn/transform/Options.java

Set up some unit tests from equivalent XQTS tests. Wrap document returned in a map, per spec. Serialize output when requested, borrowing eXist serialization code. Use of stylesheet-params option working Rename unit tests to .xqm Improve transform option type error checking Make badly typed options throw XPTY0004 (type error XPath exception) instead of throwing directly from Java. Handle some of fn:transform “initial-template” param This is part of what is termed call-template invocation, and the transformer’s callTemplate() method is used. Reject call on non-existent source-node Add getStringValue() and toString() to XQueryContext for help in debugging Try harder to resolve stylesheet URIs (in retrospect probably not right yet).

and fix up the primary output key in the case where it should not be “output”

XQTS test fn-transform <test-case name="fn-transform-7b"> Uses a stylesheet of the form: <out xmlns:xsl='http://www.w3.org/1999/XSL/Transform' xsl:version='2.0'> <xsl:value-of select='.' /> </out> and we were not picking the stylesheet version from that. Burrow through the element and check for the appropriate NamespaceNode and xsl:version.

To set the base URI, we need to set the system id for the stylesheet. Then the stylesheet can resolve-uri successfully, and write secondary output documents. Add a result document handler to collect the output for secondary output documents. Result documents now returned to us and we can build tha map with URI and value. Test runner passing document URI where that exists, and we set it up as the system id to the Saxon transformer if it does exist.

Separate Options class within FnTransform where fully resolved values of options are fetched from the options map once. Called as the first step of eval(), and leaves doing evaluation work based on the options as a cleanly separated task. Raw output request is now handled, and rejected (pre XSLT 3.0) or produced (3.0+) per the spec. Handling raw output and checking for the raw option mutliple times was the indication that we needed to factor out the options processing.

The fn:transform implementation is large, and has accumulated many static and dynamic subclasses of FnTransform. This change creates its own package, and creates several top level classes in this package. The FnTransform module itself becomes a much smaller stub around a call into a Transform object which is part of the new package.

Unit tests for fn:transform are mostly composed of convenient UT re-interpretations of the XQTS test suite. There are a lot, so collect them in a subdirectory.

Broken in the refactor. The refactored code was fetching secondary result documents from the wrong delivery object, so the secondary results were non-existent. Fetch from the correct delivery object.

This seems to break a number of tests outside of fn:transform. Instead, adapt the option handling for fn:transform to do an explicit check on the case of requiring a string, and supply the StringValue of an atomic value if the value is not of string subtype.

fix flagged vulnerability

All XQTS tests work within the raw Saxon to eXist conversion code we have, so remove the fallback conversion-via-serialization, which incidentally helps coverage checking.

XQTS tests were intermittently getting the wrong error message becase we were pullling it as the last error from a shared global error listener (error logger). Make an individual listener for each compilation/request which also side-effects the write to the shared logger.

Cleaned up the ones which make sense. Not cleaned up - functions with high complexity where the complexity seems essential to the function. Not cleaned up - old code we have just brushed past lightly, and which sonarcloud has consequently highlighted.

joewiz · 2022-08-09T17:54:05Z

@alanpaxton Thank you for this PR and the truly Herculean effort that went into implementing this complex function!

exist-core/src/test/xquery/xquery3/transform/fnTransformErr16.xqm

exist-core/src/test/java/xquery/xquery3/XQuery3Tests.java

exist-core/src/main/java/org/exist/xquery/functions/fn/transform/Options.java

exist-core/src/main/java/org/exist/xquery/functions/fn/transform/Transform.java

dizzzz

Very impressed by the work, and the needed complexity and wrapping.

dizzzz · 2022-08-09T19:46:39Z

it would be nice if we could get of the old unmaintained xalan lib.....

Turn XSLT version number into an XSLTVersion class to be more explicit about precise versions, and to avoid potential float equality issues (2.99999999 != 3.0) Declaration order - member variables first Clarify a commented-out line

sonarqubecloud · 2022-08-10T10:56:06Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
21 Code Smells

75.7% Coverage
0.0% Duplication

adamretter added this to the eXist-6.1.0 milestone May 12, 2022

adamretter force-pushed the feature/fn-transform branch from 8a75c9d to 1a72a48 Compare May 12, 2022 19:29

alanpaxton force-pushed the feature/fn-transform branch 2 times, most recently from a53c578 to 13d3ce4 Compare May 24, 2022 07:56

joewiz added enhancement new features, suggestions, etc. xquery issue is related to xquery implementation labels May 25, 2022