Skip to content

Latest commit

 

History

History
110 lines (75 loc) · 3.25 KB

synapse-spark-readme.md

File metadata and controls

110 lines (75 loc) · 3.25 KB
title keywords author ms.author ms.date ms.topic ms.prod ms.technology ms.devlang ms.service
Azure Synapse Spark client library for JavaScript
Azure, javascript, SDK, API, @azure/synapse-spark, synapseanalytics
ramya-rao-a
ramyar
10/05/2021
reference
azure
azure
javascript
synapseanalytics

Azure Synapse Spark client library for JavaScript - version 1.0.0-beta.4

This package contains an isomorphic SDK for Spark.

Getting started

Install the package

npm install @azure/synapse-spark

Currently supported environments

See our support policy for more details.

Key concepts

How to use

Examples

Get Spark Jobs information

import { SparkClient } from "@azure/synapse-spark";
import { DefaultAzureCredential } from "@azure/identity";

export async function main(): Promise<void> {
  const credential = new DefaultAzureCredential();

  let client = new SparkClient(credential, "https://mysynapse.dev.azuresynapse.net", "mysparkpool");
  let output = await client.sparkBatch.getSparkBatchJobs();
  console.log("output:", output);
}

Submit a Spark job

import { DefaultAzureCredential } from "@azure/identity";
import { SparkClient, SparkBatchJobOptions } from "@azure/synapse-spark";

export async function main(): Promise<void> {
    const credential = new DefaultAzureCredential();
    const endpoint = "https://mysynapse.dev.azuresynapse.net";
    const poolName = "mysparkpool";
    const pysparkFile = "abfss://<filesystem>@<storage-account>.dfs.core.windows.net/<pyspark-script-path>";

    let client = new SparkClient(credential, endpoint, poolName);
    
    const sparkJobOptions: SparkBatchJobOptions = {
        name: 'MySparkJob',
        file: pysparkFile,
        driverMemory: "1g",
        driverCores: 1,
        executorMemory: "4g",
        executorCores: 2,
        executorCount: 2,
        arguments: [],
        configuration: {}
    };
    
    await client.sparkBatch.createSparkBatchJob(sparkJobOptions);
}

Related projects

Impressions

Troubleshooting

Enabling logging may help uncover useful information about failures. In order to see a log of HTTP requests and responses, set the AZURE_LOG_LEVEL environment variable to info. Alternatively, logging can be enabled at runtime by calling setLogLevel in the @azure/logger:

import { setLogLevel } from "@azure/logger";

setLogLevel("info");

Next steps

In the future, you'll find additional code samples here.

Contributing

If you'd like to contribute to this library, please read the contributing guide to learn more about how to build and test the code.

Impressions