[Bug]: Tests Passing Locally but Failing on GitHub Actions

[Ajai-Suvendran]

Description

I am encountering an issue where all my tests pass successfully when executed locally, but when I push the code to GitHub, the same tests consistently fail during the GitHub Actions workflow.
GitHub PR link:
ballerina-platform/module-ballerinax-financial.iso20022-to-swiftmt#5

Steps to Reproduce

Run the tests locally using the bal test command.
Push the code to GitHub.
Observe the test results during the GitHub Actions workflow execution.

Affected Version(s)

Ballerina 2210.10.1

OS, DB, other environment details and versions

Local OS - Windows 11

Related area

-> Test Framework

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[gayaldassanayake]

This is an assertion failure, where the expected output is different from the actual one.It looks like the error is not from the test framework side. At the point of assertion, the values are not similar.

My hunch is that this difference arises due to the carriage return differences in UNIX v windows platforms. You will have to check where this difference is arising from, in the source code.

I will close this issue since the root cause is not likely to be from the test framework side. We can re-open if necessary.

[github-actions]

This issue is NOT closed with a proper Reason/ label. Make sure to add proper reason label before closing. Please add or leave a comment with the proper reason label now.

      - Reason/EngineeringMistake - The issue occurred due to a mistake made in the past.
      - Reason/Regression - The issue has introduced a regression.
      - Reason/MultipleComponentInteraction - Issue occured due to interactions in multiple components.
      - Reason/Complex - Issue occurred due to complex scenario.
      - Reason/Invalid - Issue is invalid.
      - Reason/Other - None of the above cases.

[RivinduM]

Hi @gayaldassanayake,

If you inspect the test log at[1], it's clear that the expected and the actual strings are similar. Following is one example for your reference.

expected: '{1:F01CHASUS330000000000}{2:O200092710250101BKAUATWWN}{4:
 			:20:39857579
			:32A:09
 			05251000000,00
 			:53B:/34554-3049
 			:56A:CITIUS33
 			:57A:CITIUS33MIA
			-}'
actual	: '{1:F01CHASUS330000000000}{2:O200092710250101BKAUATWWN}{4:
 			:20:39857579
 			:32A:0905
 			251000000,00
 			:53B:/34554-3049
 			:56A:CITIUS33
 			:57A:CITIUS33MIA
 			-}'
 			 
Diff	:
			
--- actual
+++ expected 
	 
@@ -1,7 +1,7 @@ 

 -{1:F01CHASUS330000000000}{2:O200092710250101BKAUATWWN}{4:
-:20:39857579
-:32A:0905251000000,00
-:53B:/34554-3049
-:56A:CITIUS33
-:57A:CITIUS33MIA
+{1:F01CHASUS330000000000}{2:O200092710250101BKAUATWWN}{4:
+:20:39857579
+:32A:0905251000000,00
+:53B:/34554-3049
+:56A:CITIUS33
+:57A:CITIUS33MIA

From the source code, it's just a string assertion[2]. So ideally this should work irrespective of whether you are working on a Unix or a windows platform. This could be some issue how the assertion works in the Ballerina side and is something that needs to figured out from the Ballerina team not from the developer.

Please consider this as a high priority issue as this is impacting our release deadlines.

[1] https://github.com/ballerina-platform/module-ballerinax-financial.iso20022-to-swiftmt/actions/runs/12594006933/job/35100844445?pr=5
[2] https://github.com/ballerina-platform/module-ballerinax-financial.iso20022-to-swiftmt/blob/26fc8172ca89e1fa4d0546fce812853c23520f17/iso20022_to_switmt/tests/to_swift_message_test.bal#L268

[gayaldassanayake]

Hi @RivinduM,

I dabbled with test code of your project, and here are my observations.

The string equality between the expected and the actual strings also returns false (actual == expected => false). This means we either have a bug in the equality check or there are characters not visible to the naked eye in either of the strings.

What worked for me was normalizing the actual output for carriage return values. It turns out that swiftmt:getFinMessage output contains the carriage return (actual.includes("\r\n")=> true). You might need to look into why that is happening.

Replacing the carriage return with \n would be an adequate workaround until then.

string actual = check swiftmt:getFinMessage(check toSwiftMtMessage(inputXml, "101"));
regexp:RegExp regex = re `\r\n`;
string actualNormalized = regex.replaceAll(actual, "\n");
test:assertEquals(actualNormalized, expected, msg = "testToIso20022Xml result incorrect"); 

[Ajai-Suvendran]

The issue is not because of a bug in the Ballerina test framework but arises from the behavior of the getFin function in the Prowide library (external Java library used) , which appends line endings (\r\n) to the end of each line in the resulting SWIFT FIN message.

In Ballerina, when string templates are used, the line endings in the template are determined by the operating system:

On Windows: The line endings are interpreted as \r\n.
On Linux: The line endings are interpreted as \n.
As a result, when the output of the getFin function is processed in Ballerina:

On Windows, the additional \r is preserved in the string.
On Linux, only \n is appended to each line.
This inconsistency causes the output to differ between Windows and Linux environments, leading to potential mismatches or unexpected behavior, especially in tests or comparisons.

[github-actions]

This issue is NOT closed with a proper Reason/ label. Make sure to add proper reason label before closing. Please add or leave a comment with the proper reason label now.

      - Reason/EngineeringMistake - The issue occurred due to a mistake made in the past.
      - Reason/Regression - The issue has introduced a regression.
      - Reason/MultipleComponentInteraction - Issue occured due to interactions in multiple components.
      - Reason/Complex - Issue occurred due to complex scenario.
      - Reason/Invalid - Issue is invalid.
      - Reason/Other - None of the above cases.