Propagating environment variables when using Spark’s internal REST API to submit jobs
In an earlier post I introduced my client that wraps the calls to Spark’s REST API to submit jobs, instead of using the spark-submit script. This REST API, while not officially in the Spark documentation, works just fine and almost the same way as the spark-submit script and we are even using it in production. Lately,…