Delta Sharing - Lobyco Help center

Intro

Delta Sharing is an open protocol for secure real-time exchange of large datasets, which enables secure data sharing across products for the first time. It is a simple REST protocol that securely shares access to part of a cloud dataset. It leverages modern cloud storage systems, such as S3, ADLS, or GCS, to reliably transfer large datasets. With Delta Sharing, the user accessing shared data can directly connect to it through Pandas, Tableau, or dozens of other systems that implement the open protocol, without having to deploy a specific platform first. This reduces their access time from months to minutes, and makes life dramatically simpler for data providers who want to reach as many users as possible. The Data Platform still supports the traditional data exchange with files and Storage Accounts, but we now also offer Delta Sharing, an open protocol for data sharing that makes it easier and more secure to share data between organizations.

Improved security and data governance
Full and incremental data loads
Support for larger data sets

More info:

API reference: the REST endpoints behind this protocol — shares, schemas, tables, queries and change feeds — are documented under Delta Sharing Protocol.

Data exploration

Before using Delta Sharing, contact Lobyco to obtain an activation link - a unique, one-time-use URL that self-destructs after being accessed. By clicking this URL, you will download a configuration file containing all the necessary credentials.

Important: Ensure you save this file in a secure location, as it is essential for accessing the shared data.

Python

After you save the profile file, you can use it in the connector to access shared tables.

List all shared tables

import delta_sharing

# Point to the profile file. It can be a file on the local file system or a file on a remote storage.
profile_file = "<profile-file-path>"

# Create a SharingClient.
client = delta_sharing.SharingClient(profile_file)

# List all shared tables.
tables = client.list_all_tables()

for table in tables:
  print(f"share = {table.share}, name = {table.name}, schema = {table.schema}")

List table metadata

import delta_sharing

profile_file = "<profile-file-path>"

table_url = profile_file + "#<share-name>.<schema-name>.<table-name>"

# Show table metadata
table_metadata = delta_sharing.get_table_metadata(table_url)

print(f"Description - {table_metadata.description}") # main table description
print(f"Format - {table_metadata.format}")
print(f"Schema String - {table_metadata.schema_string}")
print(f"Partition Columns - {table_metadata.partition_columns}")
print(f"Version - {table_metadata.version}")
print(f"Size - {table_metadata.size}")
print(f"Number of Files - {table_metadata.num_files}")
print(f"Created Time - {table_metadata.created_time}")

Query table data using pandas

# Create a url to access a shared table.
# A table path is the profile file path following with `#` and the fully qualified name of a table 
# (`<share-name>.<schema-name>.<table-name>`).
table_url = profile_file + "#<share-name>.<schema-name>.<table-name>"

# Fetch 10 rows from a table and convert it to a Pandas DataFrame. This can be used to read sample data 
# from a table that cannot fit in the memory.
delta_sharing.load_as_pandas(table_url, limit=10)

Query table changes using pandas

<starting-version>: optional. The starting version of the query, inclusive.
<ending-version>: optional. The ending version of the query, inclusive.
<starting-timestamp>: optional. The starting timestamp of the query. This is converted to a version created greater or equal to this timestamp. (format: yyyy-mm-dd hh:mm:ss[.fffffffff])
<ending-timestamp>: optional. The ending timestamp of the query. This is converted to a version created earlier or equal to this timestamp. (format: yyyy-mm-dd hh:mm:ss[.fffffffff])

delta_sharing.load_table_changes_as_pandas(
  table_url,
  starting_version=<starting-version>,
  ending_version=<starting-version>)

delta_sharing.load_table_changes_as_pandas(
  table_url,
  starting_timestamp=<starting-timestamp>,
  ending_timestamp=<ending-timestamp>)

​Intro

​Data exploration

​Python

​List all shared tables

​List table metadata

​Query table data using pandas

​Query table changes using pandas

Intro

Data exploration

Python

List all shared tables

List table metadata

Query table data using pandas

Query table changes using pandas