[Documentation]Take your application to production #349

elvaliuliuliu · 2019-11-23T01:50:41Z

Currently, there are some questions asked by customers that how they can run spark dotnet application in different scenarios. This PR gathers most commonly asked scenarios and provides general instructions on how customer can package their applications and submit jobs in such scenarios.

I have closed the previous pr #345 (cause I couldn't edit there), I have moved all the previous comments here so that we can discuss here. Thanks for your understanding and sorry for any inconvenience.

elvaliuliuliu · 2019-11-23T01:59:20Z

docs/take-to-prod.md

@@ -0,0 +1,85 @@
+Taking your .NET for Apache Spark Application to Production


comments from @suhsteve "This title somehow makes me feel like after reading this I will know how to make my spark application production ready. Is this the purpose of this document ? Or just to outline different spark-submit scenarios."

I believe the purpose of this doc should tell users how to move their application to production. I am thinking of putting different scenarios along with the instruction on how to move it to production based on these different scenarios. Any suggestions to make this doc more precise and explicit would be really appreciate.

elvaliuliuliu · 2019-11-23T02:02:35Z

docs/take-to-prod.md

+#### 2. SparkSession code references a function from a Nuget package that has been installed in the csproj
+This would be the use case when `SparkSession` code references a function from a Nuget package in the same project (e.g. mySparkApp.csproj).
+#### 3. SparkSession code references a function from a DLL on the user's machine
+This would be the use case when `SparkSession` code reference business logic (UDFs) on the user's machine (e.g. `SparkSession` code in the mySparkApp.csproj and businessLogic.dll on a different machine). 


comments from @suhsteve :"Why does businessLogic.dll be from a different machine ?"

This would be one of the scenarios, wouldn't it?

It may make more sense if this scenario description was something like loading assemblies dynamically at runtime.

Got it, but I am wondering how can we structure the description of this scenario to be more explainable to users?

elvaliuliuliu · 2019-11-23T02:10:44Z

docs/take-to-prod.md

+#### 2. SparkSession code and business logic in the same project, but different .cs files
+This would be the use case when you have `SparkSession` code and business logic (UDFs) in the different .cs files but in the same project (e.g. SparkSession in Program.cs, business logic in BusinessLogic.cs and both are in mySparkApp.csproj).
+
+### Package your application


comments from @suhsteve "I don't know if this section is too useful."

I think it will be very helpful if we can put more detailed instruction here. Any suggestions?

elvaliuliuliu · 2019-11-23T02:11:19Z

docs/take-to-prod.md

+
+## How to deploy your application when you have a single dependency
+### Scenarios
+#### 1. SparkSession code and business logic in the same Program.cs file


@suhsteve I will keep the number here to make it more clear.

elvaliuliuliu · 2019-11-23T02:14:04Z

docs/take-to-prod.md

+===
+
+# Table of Contents
+This how-to provides general instructions on how to take your .NET for Apache Spark application to production.


Comments from @bamurtaugh : "What does it mean to take an app to production? Perhaps add a couple words/sentence defining that (does it just mean running on-prem? Deploying to cloud? Building and running spark-submit? CI/CD?)"

Great point! @rapoth Could you please help with elaborating this a little more?

elvaliuliuliu · 2019-11-23T02:15:00Z

docs/take-to-prod.md

+  - [Package your application](#package-your-application-1)
+  - [Launch your application](#launch-your-application-1)
+
+## How to deploy your application when you have a single dependency


comments from @bamurtaugh : "What does single dependency mean? I think it could help users to include a short explanation here or at the top of the document of what a dependency means in the .NET for Spark context."

Actually I am not so sure if we should use single dependency and multiple dependency to define and separate these scenarios. @rapoth and @imback82 any suggestions? Thanks.

docs/take-to-prod.md

suhsteve · 2019-12-09T19:20:09Z

docs/take-to-prod.md

+--deploy-mode cluster \
+--files <some dir>\<dotnet version>\mySparkApp.dll \
+<some dir>\<dotnet version>\microsoft-spark-<spark_majorversion.spark_minorversion.x>-<spark_dotnet_version>.jar \
+dotnet <some dir>\<dotnet version>\mySparkApp.dll <app arg 1> <app arg 2> ... <app arg n>


dotnet is used in this example, but won't this fail in user scenarios where dotnet may not be available on their cluster ?

Yes I agree, but I have put pre-requisites in this example.

Or we can also put an example like mySparkApp args or dotnet mySparkApp.dll args which gives both options depending on cluster environment.

docs/take-to-prod.md

suhsteve · 2019-12-09T19:24:23Z

docs/take-to-prod.md

+#### 2. SparkSession code references a function from a Nuget package that has been installed in the csproj
+This would be the use case when `SparkSession` code references a function from a Nuget package in the same project (e.g. mySparkApp.csproj).
+#### 3. SparkSession code references a function from a DLL on the user's machine
+This would be the use case when `SparkSession` code reference business logic (UDFs) on the user's machine (e.g. `SparkSession` code in the mySparkApp.csproj and businessLogic.dll on a different machine). 


It may make more sense if this scenario description was something like loading assemblies dynamically at runtime.

suhsteve · 2019-12-09T19:26:13Z

docs/take-to-prod.md

+    "archives": ["adl://<cluster name>.azuredatalakestore.net/<some dir>/businessLogics.zip#udfs”, "adl://<cluster name>.azuredatalakestore.net/<some dir>/myLibraries.zip”],
+    "args": ["adl://<cluster name>.azuredatalakestore.net/<some dir>/mySparkApp.zip","mySparkApp","<app arg 1>","<app arg 2>,"...","<app arg n>"]
+}
+```


You should say somewhere that scenarios 1 and 2 can also be submitted using these submission examples.

Do you mean scenarios 1 and 2 in single dependency session?
They can submit it using the example here, but they have to zip single dll first, would like be more work for user in such case? How about we specify the another example usage of mySparkApp args as I mentioned earlier?

If users are manually submitting then yes, but in "production" use, users would most likely automate the packaging and submission. It would be good to know that they don't have to take different steps if they have single vs multiple dependencies.

If we would like to unify these two sections, should we remove the command example in the first section?

I just feel like we need to add more contexts in this whole instruction in general, any suggestions would be really appreciated.

init

398810c

elvaliuliuliu commented Nov 23, 2019

View reviewed changes

elvaliuliuliu mentioned this pull request Nov 23, 2019

[Documentation]Instructions on how to take your application to production #345

Closed

elvaliuliuliu commented Nov 23, 2019

View reviewed changes

Merge branch 'master' into elva/taketoprod

3371ff3

suhsteve reviewed Dec 9, 2019

View reviewed changes

imback82 assigned elvaliuliuliu Dec 9, 2019

elvaliuliuliu added 3 commits December 9, 2019 15:18

Merge branch 'master' into elva/taketoprod

24142b4

resolve comments

2e254e8

Merge branch 'master' into elva/taketoprod

d1dc709

Base automatically changed from master to main March 18, 2021 16:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Documentation]Take your application to production #349

[Documentation]Take your application to production #349

elvaliuliuliu commented Nov 23, 2019 •

edited

Loading

elvaliuliuliu Nov 23, 2019

elvaliuliuliu Nov 23, 2019

suhsteve Dec 9, 2019 •

edited

Loading

elvaliuliuliu Dec 10, 2019

elvaliuliuliu Nov 23, 2019

elvaliuliuliu Nov 23, 2019

elvaliuliuliu Nov 23, 2019

elvaliuliuliu Nov 23, 2019

suhsteve Dec 9, 2019

elvaliuliuliu Dec 10, 2019

suhsteve Dec 9, 2019 •

edited

Loading

suhsteve Dec 9, 2019

elvaliuliuliu Dec 10, 2019

suhsteve Dec 10, 2019

elvaliuliuliu Dec 10, 2019

		@@ -0,0 +1,85 @@
		Taking your .NET for Apache Spark Application to Production

[Documentation]Take your application to production #349

Are you sure you want to change the base?

[Documentation]Take your application to production #349

Conversation

elvaliuliuliu commented Nov 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhsteve Dec 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhsteve Dec 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elvaliuliuliu commented Nov 23, 2019 •

edited

Loading

suhsteve Dec 9, 2019 •

edited

Loading

suhsteve Dec 9, 2019 •

edited

Loading