Feature Store APIs

Feature group

class sagemaker.feature_store.feature_group.FeatureGroup(name=NOTHING, sagemaker_session=<class 'sagemaker.session.Session'>, feature_definitions=NOTHING)

Bases: object

FeatureGroup definition.

This class instantiates a FeatureGroup object that comprises of a name for the FeatureGroup, session instance, and a list of feature definition objects i.e., FeatureDefinition.

Parameters
Return type

None

name

name of the FeatureGroup instance.

Type

str

sagemaker_session

session instance to perform boto calls.

Type

Session

feature_definitions

list of FeatureDefinitions.

Type

Sequence[FeatureDefinition]

Method generated by attrs for class FeatureGroup.

create(s3_uri, record_identifier_name, event_time_feature_name, role_arn, online_store_kms_key_id=None, enable_online_store=False, offline_store_kms_key_id=None, disable_glue_table_creation=False, data_catalog_config=None, description=None, tags=None, table_format=None)

Create a SageMaker FeatureStore FeatureGroup.

Parameters
  • s3_uri (Union[str, bool]) – S3 URI of the offline store, set to False to disable offline store.

  • record_identifier_name (str) – name of the record identifier feature.

  • event_time_feature_name (str) – name of the event time feature.

  • role_arn (str) – ARN of the role used to call CreateFeatureGroup.

  • online_store_kms_key_id (str) – KMS key id for online store (default: None).

  • enable_online_store (bool) – whether to enable online store or not (default: False).

  • offline_store_kms_key_id (str) – KMS key id for offline store (default: None). If a KMS encryption key is not specified, SageMaker encrypts all data at rest using the default AWS KMS key. By defining your bucket-level key for SSE, you can reduce the cost of AWS KMS requests. For more information, see Bucket Key in the Amazon S3 User Guide.

  • disable_glue_table_creation (bool) – whether to turn off Glue table creation or not (default: False).

  • data_catalog_config (DataCatalogConfig) – configuration for Metadata store (default: None).

  • description (str) – description of the FeatureGroup (default: None).

  • tags (List[Dict[str, str]]) – list of tags for labeling a FeatureGroup (default: None).

  • table_format (TableFormatEnum) – format of the offline store table (default: None).

Returns

Response dict from service.

Return type

Dict[str, Any]

delete()

Delete a FeatureGroup.

describe(next_token=None)

Describe a FeatureGroup.

Parameters

next_token (str) – next_token to get next page of features.

Returns

Response dict from the service.

Return type

Dict[str, Any]

update(feature_additions)

Update a FeatureGroup and add new features from the given feature definitions.

Parameters

feature_additions (Sequence[Dict[str, str]) – list of feature definitions to be updated.

Returns

Response dict from service.

Return type

Dict[str, Any]

update_feature_metadata(feature_name, description=None, parameter_additions=None, parameter_removals=None)

Update a feature metadata and add/remove metadata.

Parameters
  • feature_name (str) – name of the feature to update.

  • description (str) – description of the feature to update.

  • parameter_additions (Sequence[Dict[str, str]) – list of feature parameter to be added.

  • parameter_removals (Sequence[str]) – list of feature parameter key to be removed.

Returns

Response dict from service.

Return type

Dict[str, Any]

describe_feature_metadata(feature_name)

Describe feature metadata by feature name.

Parameters

feature_name (str) – name of the feature.

Returns

Response dict from service.

Return type

Dict[str, Any]

put_record(record)

Put a single record in the FeatureGroup.

Parameters

record (Sequence[FeatureValue]) – a list contains feature values.

ingest(data_frame, max_workers=1, max_processes=1, wait=True, timeout=None, profile_name=None)

Ingest the content of a pandas DataFrame to feature store.

max_worker number of thread will be created to work on different partitions of the data_frame in parallel.

max_processes number of processes will be created to work on different partitions of the data_frame in parallel, each with max_worker threads.

The ingest function will attempt to ingest all records in the data frame. If wait is True, then an exception is thrown after all records have been processed. If wait is False, then a later call to the returned instance IngestionManagerPandas’ wait() function will throw an exception.

Zero based indices of rows that failed to be ingested can be found in the exception. They can also be found from the IngestionManagerPandas’ failed_rows function after the exception is thrown.

profile_name argument is an optional one. It will use the default credential if None is passed. This profile_name is used in the sagemaker_featurestore_runtime client only. See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for more about the default credential.

Parameters
  • data_frame (DataFrame) – data_frame to be ingested to feature store.

  • max_workers (int) – number of threads to be created.

  • max_processes (int) – number of processes to be created. Each process spawns max_worker number of threads.

  • wait (bool) – whether to wait for the ingestion to finish or not.

  • timeout (Union[int, float]) – concurrent.futures.TimeoutError will be raised if timeout is reached.

  • profile_name (str) – the profile credential should be used for PutRecord (default: None).

Returns

An instance of IngestionManagerPandas.

Return type

sagemaker.feature_store.feature_group.IngestionManagerPandas

athena_query()

Create an AthenaQuery instance.

Returns

An instance of AthenaQuery initialized with data catalog configurations.

Return type

sagemaker.feature_store.feature_group.AthenaQuery

as_hive_ddl(database='sagemaker_featurestore', table_name=None)

Generate Hive DDL commands to define or change structure of tables or databases in Hive.

Schema of the table is generated based on the feature definitions. Columns are named after feature name and data-type are inferred based on feature type. Integral feature type is mapped to INT data-type. Fractional feature type is mapped to FLOAT data-type. String feature type is mapped to STRING data-type.

Parameters
  • database (str) – name of the database. If not set “sagemaker_featurestore” will be used.

  • table_name (str) – name of the table. If not set the name of this feature group will be used.

Returns

Generated create table DDL string.

Return type

str

class sagemaker.feature_store.feature_group.AthenaQuery(catalog, database, table_name, sagemaker_session)

Bases: object

Class to manage querying of feature store data with AWS Athena.

This class instantiates a AthenaQuery object that is used to retrieve data from feature store via standard SQL queries.

Parameters
Return type

None

catalog

name of the data catalog.

Type

str

database

name of the database.

Type

str

table_name

name of the table.

Type

str

sagemaker_session

instance of the Session class to perform boto calls.

Type

Session

Method generated by attrs for class AthenaQuery.

run(query_string, output_location, kms_key=None, workgroup=None)

Execute a SQL query given a query string, output location and kms key.

This method executes the SQL query using Athena and outputs the results to output_location and returns the execution id of the query.

Parameters
  • query_string (str) – SQL query string.

  • output_location (str) – S3 URI of the query result.

  • kms_key (str) – KMS key id. If set, will be used to encrypt the query result file.

  • workgroup (str) – The name of the workgroup in which the query is being started.

Returns

Execution id of the query.

Return type

str

wait()

Wait for the current query to finish.

get_query_execution()

Get execution status of the current query.

Returns

Response dict from Athena.

Return type

Dict[str, Any]

as_dataframe()

Download the result of the current query and load it into a DataFrame.

Returns

A pandas DataFrame contains the query result.

Return type

pandas.core.frame.DataFrame

class sagemaker.feature_store.feature_group.IngestionManagerPandas(feature_group_name, sagemaker_fs_runtime_client_config, max_workers=1, max_processes=1, profile_name=None, async_result=None, processing_pool=None, failed_indices=NOTHING)

Bases: object

Class to manage the multi-threaded data ingestion process.

This class will manage the data ingestion process which is multi-threaded.

Parameters
  • feature_group_name (str) –

  • sagemaker_fs_runtime_client_config (botocore.config.Config) –

  • max_workers (int) –

  • max_processes (int) –

  • profile_name (str) –

  • async_result (multiprocessing.pool.ApplyResult) –

  • processing_pool (pathos.multiprocessing.ProcessPool) –

  • failed_indices (List[int]) –

Return type

None

feature_group_name

name of the Feature Group.

Type

str

sagemaker_fs_runtime_client_config

instance of the Config class for boto calls.

Type

Config

data_frame

pandas DataFrame to be ingested to the given feature group.

Type

DataFrame

max_workers

number of threads to create.

Type

int

max_processes

number of processes to create. Each process spawns max_workers threads.

Type

int

profile_name

the profile credential should be used for PutRecord (default: None).

Type

str

Method generated by attrs for class IngestionManagerPandas.

property failed_rows

Get rows that failed to ingest.

Returns

List of row indices that failed to be ingested.

wait(timeout=None)

Wait for the ingestion process to finish.

Parameters

timeout (Union[int, float]) – concurrent.futures.TimeoutError will be raised if timeout is reached.

run(data_frame, wait=True, timeout=None)

Start the ingestion process.

Parameters
  • data_frame (DataFrame) – source DataFrame to be ingested.

  • wait (bool) – whether to wait for the ingestion to finish or not.

  • timeout (Union[int, float]) – concurrent.futures.TimeoutError will be raised if timeout is reached.

Feature definition

class sagemaker.feature_store.feature_definition.FeatureDefinition(feature_name, feature_type)

Bases: sagemaker.feature_store.inputs.Config

Feature definition.

This instantiates a Feature Definition object where FeatureDefinition is a subclass of Config.

Parameters
Return type

None

feature_name

The name of the feature

Type

str

feature_type

The type of the feature

Type

FeatureTypeEnum

Method generated by attrs for class FeatureDefinition.

to_dict()

Construct a dictionary based on each attribute.

Return type

Dict[str, Any]

class sagemaker.feature_store.feature_definition.FractionalFeatureDefinition(feature_name)

Bases: sagemaker.feature_store.feature_definition.FeatureDefinition

Fractional feature definition.

This class instantiates a FractionalFeatureDefinition object, a subclass of FeatureDefinition where the data type of the feature being defined is a Fractional.

feature_name

The name of the feature

Type

str

feature_type

A FeatureTypeEnum.FRACTIONAL type

Type

FeatureTypeEnum

Construct an instance of FractionalFeatureDefinition.

Parameters

feature_name (str) – the name of the feature.

class sagemaker.feature_store.feature_definition.IntegralFeatureDefinition(feature_name)

Bases: sagemaker.feature_store.feature_definition.FeatureDefinition

Fractional feature definition.

This class instantiates a IntegralFeatureDefinition object, a subclass of FeatureDefinition where the data type of the feature being defined is a Integral.

feature_name

the name of the feature.

Type

str

feature_type

a FeatureTypeEnum.INTEGRAL type.

Type

FeatureTypeEnum

Construct an instance of IntegralFeatureDefinition.

Parameters

feature_name (str) – the name of the feature.

class sagemaker.feature_store.feature_definition.StringFeatureDefinition(feature_name)

Bases: sagemaker.feature_store.feature_definition.FeatureDefinition

Fractional feature definition.

This class instantiates a StringFeatureDefinition object, a subclass of FeatureDefinition where the data type of the feature being defined is a String.

feature_name

the name of the feature.

Type

str

feature_type

a FeatureTypeEnum.STRING type.

Type

FeatureTypeEnum

Construct an instance of StringFeatureDefinition.

Parameters

feature_name (str) – the name of the feature.

class sagemaker.feature_store.feature_definition.FeatureTypeEnum(value)

Bases: enum.Enum

Enum of feature types.

The data type of a feature can be Fractional, Integral or String.

Inputs

class sagemaker.feature_store.inputs.Config

Bases: abc.ABC

Base config object for FeatureStore.

Configs must implement the to_dict method.

abstract to_dict()

Get the dictionary from attributes.

Returns

dict contains the attributes.

Return type

Dict[str, Any]

classmethod construct_dict(**kwargs)

Construct the dictionary based on the args.

Parameters

kwargs – args to be used to construct the dict.

Returns

dict represents the given kwargs.

Return type

Dict[str, Any]

class sagemaker.feature_store.inputs.DataCatalogConfig(table_name=NOTHING, catalog=NOTHING, database=NOTHING)

Bases: sagemaker.feature_store.inputs.Config

DataCatalogConfig for FeatureStore.

Parameters
  • table_name (str) –

  • catalog (str) –

  • database (str) –

Return type

None

table_name

name of the table.

Type

str

catalog

name of the catalog.

Type

str

database

name of the database.

Type

str

Method generated by attrs for class DataCatalogConfig.

to_dict()

Construct a dictionary based on the attributes provided.

Returns

dict represents the attributes.

Return type

Dict[str, Any]

class sagemaker.feature_store.inputs.OfflineStoreConfig(s3_storage_config, disable_glue_table_creation=False, data_catalog_config=None, table_format=None)

Bases: sagemaker.feature_store.inputs.Config

OfflineStoreConfig for FeatureStore.

Parameters
Return type

None

s3_storage_config

configuration of S3 storage.

Type

S3StorageConfig

disable_glue_table_creation

whether to disable the Glue table creation.

Type

bool

data_catalog_config

configuration of the data catalog.

Type

DataCatalogConfig

table_format

format of the offline store table.

Type

TableFormatEnum

Method generated by attrs for class OfflineStoreConfig.

to_dict()

Construct a dictionary based on the attributes.

Returns

dict represents the attributes.

Return type

Dict[str, Any]

class sagemaker.feature_store.inputs.OnlineStoreConfig(enable_online_store=True, online_store_security_config=None)

Bases: sagemaker.feature_store.inputs.Config

OnlineStoreConfig for FeatureStore.

Parameters
Return type

None

enable_online_store

whether to enable the online store.

Type

bool

online_store_security_config

configuration of security setting.

Type

OnlineStoreSecurityConfig

Method generated by attrs for class OnlineStoreConfig.

to_dict()

Construct a dictionary based on the attributes.

Returns

dict represents the attributes.

Return type

Dict[str, Any]

class sagemaker.feature_store.inputs.OnlineStoreSecurityConfig(kms_key_id=NOTHING)

Bases: sagemaker.feature_store.inputs.Config

OnlineStoreSecurityConfig for FeatureStore.

Parameters

kms_key_id (str) –

Return type

None

kms_key_id

KMS key id.

Type

str

Method generated by attrs for class OnlineStoreSecurityConfig.

to_dict()

Construct a dictionary based on the attributes.

Return type

Dict[str, Any]

class sagemaker.feature_store.inputs.S3StorageConfig(s3_uri, kms_key_id=None)

Bases: sagemaker.feature_store.inputs.Config

S3StorageConfig for FeatureStore.

Parameters
  • s3_uri (str) –

  • kms_key_id (str) –

Return type

None

s3_uri

S3 URI.

Type

str

kms_key_id

KMS key id.

Type

str

Method generated by attrs for class S3StorageConfig.

to_dict()

Construct a dictionary based on the attributes provided.

Returns

dict represents the attributes.

Return type

Dict[str, Any]

class sagemaker.feature_store.inputs.FeatureValue(feature_name=None, value_as_string=None)

Bases: sagemaker.feature_store.inputs.Config

FeatureValue for FeatureStore.

Parameters
  • feature_name (str) –

  • value_as_string (str) –

Return type

None

feature_name

name of the Feature.

Type

str

value_as_string

value of the Feature in string form.

Type

str

Method generated by attrs for class FeatureValue.

to_dict()

Construct a dictionary based on the attributes provided.

Returns

dict represents the attributes.

Return type

Dict[str, Any]