samples(storage transfer): add samples and test cases for storage transfer services #2857

mahendra-google · 2024-12-04T10:22:38Z

Adding samples and tests cases for following storage transfer services:

1. storagetransfer_quickstart
2. storagetransfer_transfer_to_nearline
3. storagetransfer_get_latest_transfer_operation
4. storagetransfer_manifest_request
5. storagetransfer_transfer_from_posix
6. storagetransfer_download_to_posix
7. storagetransfer_transfer_posix_to_posix
8. storagetransfer_create_event_driven_gcs_transfer

…transfer_transfer_to_nearline.

…b.com/mahendra-google/dotnet-docs-samples into missing_.net_transfer_service_samples

…storagetransfer_transfer_to_nearline.

…or storagetransfer_transfer_to_nearline.

…b.com/mahendra-google/dotnet-docs-samples into missing_.net_transfer_service_samples

…utput message in sample output and related small changes.

…andard output of test log summary.

…test transfer operation.

…rvice_samples

…sfer to nearline and check latest transfer operation (#1)

…file

…ket using manifest file (#2)

…om posix (#3)

…vice.

…fer services. (#4)

… folder in posix file system (#5)

…xture changes

…ole , rewording , removing some unnecessary symbols, few test checks removed related to posix file transfer.

snippet-bot · 2024-12-04T10:22:44Z

Here is the summary of changes.

You are about to add 7 region tags.

storagetransfer/api/StorageTransfer.Samples/CheckLatestTransferOperationSample.cs:14, tag storagetransfer_get_latest_transfer_operation
storagetransfer/api/StorageTransfer.Samples/CreateEventDrivenGcsTransferSample.cs:16, tag storagetransfer_create_event_driven_gcs_transfer
storagetransfer/api/StorageTransfer.Samples/DownloadToPosixSample.cs:14, tag storagetransfer_download_to_posix
storagetransfer/api/StorageTransfer.Samples/TransferBetweenPosixSample.cs:14, tag storagetransfer_transfer_posix_to_posix
storagetransfer/api/StorageTransfer.Samples/TransferFromPosixSample.cs:14, tag storagetransfer_transfer_from_posix
storagetransfer/api/StorageTransfer.Samples/TransferToNearlineSample.cs:14, tag storagetransfer_transfer_to_nearline
storagetransfer/api/StorageTransfer.Samples/TransferUsingManifestSample.cs:15, tag storagetransfer_manifest_request

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

amanda-tarafa

I only reviewed this partially but there are plenty of change requests that will apply to all samples and tests in this PR.

Splitting this PR in several, each container 2 samples and their tests, will make things easier to review.

amanda-tarafa · 2024-12-11T08:46:15Z

storagetransfer/api/StorageTransfer.Samples/StorageTransfer.Samples.csproj

@@ -6,5 +6,6 @@

  <ItemGroup>
    <PackageReference Include="Google.Cloud.StorageTransfer.V1" Version="2.7.0" />
+    <PackageReference Include="xunit.abstractions" Version="2.0.3" />


Why would you need this on the samples project?

I forgot to remove that I added for the output helper. I will remove this.

storagetransfer/api/StorageTransfer.Samples/QuickstartSample.cs

storagetransfer/api/StorageTransfer.Samples.Tests/QuickstartTest.cs

amanda-tarafa · 2024-12-11T08:49:55Z

storagetransfer/api/StorageTransfer.Samples.Tests/QuickstartTest.cs

@@ -25,7 +25,6 @@ public class QuickstartTest : IDisposable
    {
        private readonly StorageFixture _fixture;
        private string _transferJobName;
-


amanda-tarafa · 2024-12-11T08:51:49Z

storagetransfer/api/StorageTransfer.Samples/QuickstartSample.cs

@@ -36,8 +35,8 @@ public TransferJob Quickstart(
                ProjectId = projectId,
                TransferSpec = new TransferSpec
                {
-                    GcsDataSink = new GcsData { BucketName = sourceBucket },
-                    GcsDataSource = new GcsData { BucketName = sinkBucket }
+                    GcsDataSink = new GcsData { BucketName = sinkBucket },


It's surprising that the test needed no changes. Are we testing for the right thing?

Yes its just checking transfer job created or not .

amanda-tarafa · 2024-12-11T09:10:40Z

storagetransfer/api/StorageTransfer.Samples.Tests/TransferUsingManifestTest.cs

+    public void TransferUsingManifest()
+    {
+        TransferUsingManifestSample transferUsingManifestSample = new TransferUsingManifestSample();
+        var storage = StorageClient.Create();


Don't you have one in the fixture?

storagetransfer/api/StorageTransfer.Samples.Tests/TransferUsingManifestTest.cs

amanda-tarafa · 2024-12-11T09:15:11Z

storagetransfer/api/StorageTransfer.Samples.Tests/TransferUsingManifestTest.cs

+        byte[] byteArray = Encoding.UTF8.GetBytes("flower.jpeg");
+        MemoryStream stream = new MemoryStream(byteArray);
+        storage.UploadObject(_fixture.BucketNameSource, _fixture.ManifestObjectName, "application/octet-stream", stream);


This seems like the kind of thing that you would add to the fixture, i.e. create s bucket with some data in it that you can use to test transfer jobs.

But, the quickstart was not actually transferring anything. We really don't test the service in samples, we test the samples themselves, so creating and running a transfer job from and empty source seems enough for that.

storagetransfer/api/StorageTransfer.Samples/QuickstartSample.cs

storagetransfer/api/StorageTransfer.Samples.Tests/StorageTransfer.Samples.Tests.csproj

amanda-tarafa

Thanks @JesseLovelace

storagetransfer/api/StorageTransfer.Samples.Tests/StorageTransfer.Samples.Tests.csproj

storagetransfer/api/StorageTransfer.Samples/QuickstartSample.cs

storagetransfer/api/StorageTransfer.Samples.Tests/TransferUsingManifestTest.cs

…estSample.cs Co-authored-by: Amanda Tarafa Mas <14878252+amanda-tarafa@users.noreply.github.com>

…gManifestTest.cs Co-authored-by: Amanda Tarafa Mas <14878252+amanda-tarafa@users.noreply.github.com>

…ithub.com/mahendra-google/dotnet-docs-samples into event_driven_gcs_transfer_service_samples

amanda-tarafa · 2024-12-19T17:00:46Z

@mahendra, there are still conflicts in your branch. Let me know when it's ready for review again, thanks!

mahendra · 2024-12-19T19:20:19Z

@mahendra, there are still conflicts in your branch. Let me know when it's ready for review again, thanks!

@amanda-tarafa - I think you meant @mahendra-google and not me.

amanda-tarafa · 2024-12-19T20:17:05Z

@mahendra, there are still conflicts in your branch. Let me know when it's ready for review again, thanks!

@amanda-tarafa - I think you meant @mahendra-google and not me.

Yes, you are right, sorry for the noise.

amanda-tarafa · 2024-12-20T06:33:31Z

Just as a heads up, it's unlikely that I can get to this before January.

I'm running the tests though so we at least know whether things are working or not.

amanda-tarafa · 2024-12-20T07:04:46Z

Tests are green, thanks. I'll re-review as soon as possible, but likely to be in January.

mahendra-google added 28 commits October 9, 2024 02:00

feat(Storage Transfer) : Addition of Sample and Test Case for storage…

ba3c128

…transfer_transfer_to_nearline.

Merge branch 'missing_.net_transfer_service_samples' of https://githu…

c2982ef

…b.com/mahendra-google/dotnet-docs-samples into missing_.net_transfer_service_samples

feat(Storage Transfer) : Addition of Schedule End Date in Sample for …

36706a4

…storagetransfer_transfer_to_nearline.

samples(storage transfer) : Addition of schedule end date in sample f…

afcba29

…or storagetransfer_transfer_to_nearline.

Merge branch 'missing_.net_transfer_service_samples' of https://githu…

5d43c18

…b.com/mahendra-google/dotnet-docs-samples into missing_.net_transfer_service_samples

chore(deps): add dependency xunit.abstractions version 2.0.3 to see o…

3efcf76

…utput message in sample output and related small changes.

code changes in transfer to nearline related to capture message in st…

35e18a3

…andard output of test log summary.

feat(storage transfer) : addition of sample and test case to check la…

0acb7fd

…test transfer operation.

Merge branch 'GoogleCloudPlatform:main' into missing_.net_transfer_se…

e3c0619

…rvice_samples

feat (storage transfer) : addition of samples and test cases for tran…

000b432

…sfer to nearline and check latest transfer operation (#1)

Merge branch 'GoogleCloudPlatform:main' into main

910ad26

Merge branch 'GoogleCloudPlatform:main' into main

398b3d5

storage transfer from POSIX file system to GCS bucket using manifest …

b5d9b10

…file

minor changes

3c9d243

samples(storage transfer): transfer from POSIX file system to GCS buc…

f4b73e0

…ket using manifest file (#2)

Merge branch 'GoogleCloudPlatform:main' into main

1acec24

addition of sample and test case for transfer from posix

375f1e0

samples(storage transfer): add samples and test cases for transfer fr…

81beca4

…om posix (#3)

Merge branch 'GoogleCloudPlatform:main' into main

ce4a089

added sample and test case for download to posix storage transfer ser…

139190a

…vice.

samples(storage transfer): add sample and test case for storage trans…

7357db6

…fer services. (#4)

code changes for creation of temp folder in posix file system

bb66a63

couple of test checks added in download to posix

526b445

samples(storage transfer): code changes in logic for creation of temp…

8a06504

… folder in posix file system (#5)

added sample and test case for posix to posix storage transfer

edc89e4

one test check added in posix to posix storage transfer

7d100d4

add sample and test case for event driven transfer and few storage fi…

4b88480

…xture changes

samples(storage transfer): code changes for printing messages to cons…

effd0c2

…ole , rewording , removing some unnecessary symbols, few test checks removed related to posix file transfer.

mahendra-google requested review from a team as code owners December 4, 2024 10:22

product-auto-label bot added api: storagetransfer Issues related to the Storage Transfer Service API. samples Issues that are directly related to samples. labels Dec 4, 2024

blunderbuss-gcf bot assigned frankyn and jskeet Dec 4, 2024

jskeet assigned amanda-tarafa and unassigned jskeet Dec 4, 2024

mahendra-google mentioned this pull request Dec 4, 2024

samples(storage transfer): add samples and test cases for storage transfer services #2824

Closed

8 tasks

amanda-tarafa added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 11, 2024

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 11, 2024

amanda-tarafa requested changes Dec 11, 2024

View reviewed changes

amanda-tarafa requested changes Dec 12, 2024

View reviewed changes

mahendra-google and others added 5 commits December 19, 2024 14:49

Update storagetransfer/api/StorageTransfer.Samples/TransferUsingManif…

f5d8252

…estSample.cs Co-authored-by: Amanda Tarafa Mas <14878252+amanda-tarafa@users.noreply.github.com>

Update storagetransfer/api/StorageTransfer.Samples.Tests/TransferUsin…

be9c4bb

…gManifestTest.cs Co-authored-by: Amanda Tarafa Mas <14878252+amanda-tarafa@users.noreply.github.com>

code refactoring changes

be74818

Merge branch 'event_driven_gcs_transfer_service_samples' of https://g…

a69aaa1

…ithub.com/mahendra-google/dotnet-docs-samples into event_driven_gcs_transfer_service_samples

code changes

4b5d2e5

Merge branch 'main' into event_driven_gcs_transfer_service_samples

52eb9de

mahendra-google requested a review from amanda-tarafa December 20, 2024 06:18

amanda-tarafa added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 20, 2024

kokoro-team removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Dec 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

samples(storage transfer): add samples and test cases for storage transfer services #2857

samples(storage transfer): add samples and test cases for storage transfer services #2857

mahendra-google commented Dec 4, 2024

snippet-bot bot commented Dec 4, 2024 •

edited

Loading

amanda-tarafa left a comment

amanda-tarafa Dec 11, 2024

mahendra-google Dec 16, 2024

amanda-tarafa Dec 11, 2024

amanda-tarafa Dec 11, 2024

mahendra-google Dec 16, 2024

amanda-tarafa Dec 11, 2024

amanda-tarafa Dec 11, 2024

amanda-tarafa left a comment

amanda-tarafa commented Dec 19, 2024

mahendra commented Dec 19, 2024

amanda-tarafa commented Dec 19, 2024

amanda-tarafa commented Dec 20, 2024 •

edited

Loading

amanda-tarafa commented Dec 20, 2024

samples(storage transfer): add samples and test cases for storage transfer services #2857

Are you sure you want to change the base?

samples(storage transfer): add samples and test cases for storage transfer services #2857

Conversation

mahendra-google commented Dec 4, 2024

Adding samples and tests cases for following storage transfer services:

snippet-bot bot commented Dec 4, 2024 • edited Loading

amanda-tarafa left a comment

Choose a reason for hiding this comment

amanda-tarafa Dec 11, 2024

Choose a reason for hiding this comment

mahendra-google Dec 16, 2024

Choose a reason for hiding this comment

amanda-tarafa Dec 11, 2024

Choose a reason for hiding this comment

amanda-tarafa Dec 11, 2024

Choose a reason for hiding this comment

mahendra-google Dec 16, 2024

Choose a reason for hiding this comment

amanda-tarafa Dec 11, 2024

Choose a reason for hiding this comment

amanda-tarafa Dec 11, 2024

Choose a reason for hiding this comment

amanda-tarafa left a comment

Choose a reason for hiding this comment

amanda-tarafa commented Dec 19, 2024

mahendra commented Dec 19, 2024

amanda-tarafa commented Dec 19, 2024

amanda-tarafa commented Dec 20, 2024 • edited Loading

amanda-tarafa commented Dec 20, 2024

snippet-bot bot commented Dec 4, 2024 •

edited

Loading

amanda-tarafa commented Dec 20, 2024 •

edited

Loading