Add new SentenceTransformer model

README.md

@@ -7,580 +7,460 @@ tags:
 - sentence-similarity
 - feature-extraction
 - generated_from_trainer
-- dataset_size:3284
+- dataset_size:36
 - loss:MatryoshkaLoss
 - loss:MultipleNegativesRankingLoss
 base_model: Snowflake/snowflake-arctic-embed-m-v1.5
 widget:
-- source_sentence: Does ZenML officially support Macs running on Apple Silicon, and
-    are there any specific configurations needed?
+- source_sentence: What are the abstract methods provided for managing model servers
+    in ZenML's BaseModelDeployerFlavor class?
   sentences:
-  - 'ding ZenML to learn more!
+  - "quired for your GCP Image Builder by running e.g.:zenml service-connector list-resources\
+    \ --resource-type gcp-generic\n\nExample Command Output\n\nThe following 'gcp-generic'\
+    \ resources can be accessed by service connectors that you have configured:\n\
+    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n\
+    ┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE\
+    \ TYPE  │ RESOURCE NAMES ┃\n┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────┼────────────────┨\n\
+    ┃ bfdb657d-d808-47e7-9974-9ba6e4919d83 │ gcp-generic    │ \U0001F535 gcp     \
+    \    │ \U0001F535 gcp-generic │ zenml-core     ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\
+    \nAfter having set up or decided on a GCP Service Connector to use to authenticate\
+    \ to GCP, you can register the GCP Image Builder as follows:\n\nzenml image-builder\
+    \ register <IMAGE_BUILDER_NAME> \\\n    --flavor=gcp \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME>\
+    \ \\\n    --network=<DOCKER_NETWORK> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>\n\
+    \n# Connect the GCP Image Builder to GCP via a GCP Service Connector\nzenml image-builder\
+    \ connect <IMAGE_BUILDER_NAME> -i\n\nA non-interactive version that connects the\
+    \ GCP Image Builder to a target GCP Service Connector:\n\nzenml image-builder\
+    \ connect <IMAGE_BUILDER_NAME> --connector <CONNECTOR_ID>\n\nExample Command Output"
+  - '🧙Installation
 
 
-    Do you support Windows?ZenML officially supports Windows if you''re using WSL.
-    Much of ZenML will also work on Windows outside a WSL environment, but we don''t
-    officially support it and some features don''t work (notably anything that requires
-    spinning up a server process).
+    Installing ZenML and getting started.
 
 
-    Do you support Macs running on Apple Silicon?
+    ZenML is a Python package that can be installed directly via pip:
 
 
-    Yes, ZenML does support Macs running on Apple Silicon. You just need to make sure
-    that you set the following environment variable:
+    pip install zenml
 
 
-    export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
-
-
-    This is a known issue with how forking works on Macs running on Apple Silicon
-    and it will enable you to use ZenML and the server. This environment variable
-    is needed if you are working with a local server on your Mac, but if you''re just
-    using ZenML as a client / CLI and connecting to a deployed server then you don''t
-    need to set it.
-
-
-    How can I make ZenML work with my custom tool? How can I extend or build on ZenML?
+    Note that ZenML currently supports Python 3.8, 3.9, 3.10, and 3.11. Please make
+    sure that you are using a supported Python version.
 
 
-    This depends on the tool and its respective MLOps category. We have a full guide
-    on this over here!
+    Install with the dashboard
 
 
-    How can I contribute?
+    ZenML comes bundled with a web dashboard that lives inside a sister repository.
+    In order to get access to the dashboard locally, you need to launch the ZenML
+    Server and Dashboard locally. For this, you need to install the optional dependencies
+    for the ZenML Server:
 
 
-    We develop ZenML together with our community! To get involved, the best way to
-    get started is to select any issue from the good-first-issue label. If you would
-    like to contribute, please review our Contributing Guide for all relevant details.
+    pip install "zenml[server]"
 
 
-    How can I speak with the community?
+    We highly encourage you to install ZenML in a virtual environment. At ZenML, We
+    like to use virtualenvwrapper or pyenv-virtualenv to manage our Python virtual
+    environments.
 
 
-    The first point of the call should be our Slack group. Ask your questions about
-    bugs or specific use cases and someone from the core team will respond.
+    Installing onto MacOS with Apple Silicon (M1, M2)
 
 
-    Which license does ZenML use?
+    A change in how forking works on Macs running on Apple Silicon means that you
+    should set the following environment variable which will ensure that your connections
+    to the server remain unbroken:
 
 
-    ZenML is distributed under the terms of the Apache License Version 2.0. A complete
-    version of the license is available in the LICENSE.md in this repository. Any
-    contribution made to this project will be licensed under the Apache License Version
-    2.0.
+    export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
 
 
-    PreviousCommunity & content
+    You can read more about this here. This environment variable is needed if you
+    are working with a local server on your Mac, but if you''re just using ZenML as
+    a client / CLI and connecting to a deployed server then you don''t need to set
+    it.
 
 
-    Last updated 3 months ago'
-  - 'Registering a Model
+    Nightly builds
 
 
-    PreviousUse the Model Control PlaneNextDeleting a Model
+    ZenML also publishes nightly builds under the zenml-nightly package name. These
+    are built from the latest develop branch (to which work ready for release is published)
+    and are not guaranteed to be stable. To install the nightly build, run:
 
 
-    Last updated 4 months ago'
-  - 'Synthetic data generation
+    pip install zenml-nightly
 
 
-    Generate synthetic data with distilabel to finetune embeddings.
+    Verifying installations
 
 
-    PreviousImprove retrieval by finetuning embeddingsNextFinetuning embeddings with
-    Sentence Transformers
+    Once the installation is completed, you can check whether the installation was
+    successful either through Bash:
 
 
-    Last updated 21 days ago'
-- source_sentence: How can I change the logging verbosity level in ZenML for both
-    local and remote pipeline runs?
-  sentences:
-  - 'ncepts covered in this guide to your own projects.By the end of this guide, you''ll
-    have a solid understanding of how to leverage LLMs in your MLOps workflows using
-    ZenML, enabling you to build powerful, scalable, and maintainable LLM-powered
-    applications. First up, let''s take a look at a super simple implementation of
-    the RAG paradigm to get started.
+    zenml version
 
 
-    PreviousAn end-to-end projectNextRAG with ZenML
+    or through Python:
 
 
-    Last updated 21 days ago'
-  - 'Configuring a pipeline at runtime
+    import zenml
 
 
-    Configuring a pipeline at runtime.
+    print(zenml.__version__)
 
 
-    PreviousUse pipeline/step parametersNextReference environment variables in configurations
+    If you would like to learn more about the current release, please visit our PyPi
+    package page.
 
 
-    Last updated 28 days ago'
-  - "Set logging verbosity\n\nHow to set the logging verbosity in ZenML.\n\nBy default,\
-    \ ZenML sets the logging verbosity to INFO. If you wish to change this, you can\
-    \ do so by setting the following environment variable:\n\nexport ZENML_LOGGING_VERBOSITY=INFO\n\
-    \nChoose from INFO, WARN, ERROR, CRITICAL, DEBUG. This will set the logs to whichever\
-    \ level you suggest.\n\nNote that setting this on the client environment (e.g.\
-    \ your local machine which runs the pipeline) will not automatically set the same\
-    \ logging verbosity for remote pipeline runs. That means setting this variable\
-    \ locally with only effect pipelines that run locally.\n\nIf you wish to control\
-    \ for remote pipeline runs, you can set the ZENML_LOGGING_VERBOSITY environment\
-    \ variable in your pipeline runs environment as follows:\n\ndocker_settings =\
-    \ DockerSettings(environment={\"ZENML_LOGGING_VERBOSITY\": \"DEBUG\"})\n\n# Either\
-    \ add it to the decorator\n@pipeline(settings={\"docker\": docker_settings})\n\
-    def my_pipeline() -> None:\n    my_step()\n\n# Or configure the pipelines options\n\
-    my_pipeline = my_pipeline.with_options(\n    settings={\"docker\": docker_settings}\n\
-    )\n\nPreviousEnable or disable logs storageNextDisable rich traceback output\n\
-    \nLast updated 21 days ago"
-- source_sentence: How can I autogenerate a template yaml file for my specific pipeline
-    using ZenML?
+    Running with Docker'
+  - ":\n        \"\"\"Abstract method to deploy a model.\"\"\"@staticmethod\n    @abstractmethod\n\
+    \    def get_model_server_info(\n            service: BaseService,\n    ) -> Dict[str,\
+    \ Optional[str]]:\n        \"\"\"Give implementation-specific way to extract relevant\
+    \ model server\n        properties for the user.\"\"\"\n\n@abstractmethod\n  \
+    \  def perform_stop_model(\n        self,\n        service: BaseService,\n   \
+    \     timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,\n        force: bool\
+    \ = False,\n    ) -> BaseService:\n        \"\"\"Abstract method to stop a model\
+    \ server.\"\"\"\n\n@abstractmethod\n    def perform_start_model(\n        self,\n\
+    \        service: BaseService,\n        timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,\n\
+    \    ) -> BaseService:\n        \"\"\"Abstract method to start a model server.\"\
+    \"\"\n\n@abstractmethod\n    def perform_delete_model(\n        self,\n      \
+    \  service: BaseService,\n        timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,\n\
+    \        force: bool = False,\n    ) -> None:\n        \"\"\"Abstract method to\
+    \ delete a model server.\"\"\"\n\nclass BaseModelDeployerFlavor(Flavor):\n   \
+    \ \"\"\"Base class for model deployer flavors.\"\"\"\n\n@property\n    @abstractmethod\n\
+    \    def name(self):\n        \"\"\"Returns the name of the flavor.\"\"\"\n\n\
+    @property\n    def type(self) -> StackComponentType:\n        \"\"\"Returns the\
+    \ flavor type.\n\nReturns:\n            The flavor type.\n        \"\"\"\n   \
+    \     return StackComponentType.MODEL_DEPLOYER\n\n@property\n    def config_class(self)\
+    \ -> Type[BaseModelDeployerConfig]:\n        \"\"\"Returns `BaseModelDeployerConfig`\
+    \ config class.\n\nReturns:\n                The config class.\n        \"\"\"\
+    \n        return BaseModelDeployerConfig\n\n@property\n    @abstractmethod\n \
+    \   def implementation_class(self) -> Type[BaseModelDeployer]:\n        \"\"\"\
+    The class that implements the model deployer.\"\"\"\n\nThis is a slimmed-down\
+    \ version of the base implementation which aims to highlight the abstraction layer.\
+    \ In order to see the full implementation and get the complete docstrings, please\
+    \ check the SDK docs .\n\nBuilding your own model deployers"
+- source_sentence: How can you successfully connect the image builder `gcp-image-builder`
+    to the resources using a connector ID?
   sentences:
-  - "Autogenerate a template yaml file\n\nTo help you figure out what you can put\
-    \ in your configuration file, simply autogenerate a template.\n\nIf you want to\
-    \ generate a template yaml file of your specific pipeline, you can do so by using\
-    \ the .write_run_configuration_template() method. This will generate a yaml file\
-    \ with all options commented out. This way you can pick and choose the settings\
-    \ that are relevant to you.\n\nfrom zenml import pipeline\n...\n\n@pipeline(enable_cache=True)\
-    \ # set cache behavior at step level\ndef simple_ml_pipeline(parameter: int):\n\
-    \    dataset = load_data(parameter=parameter)\n    train_model(dataset)\n\nsimple_ml_pipeline.write_run_configuration_template(path=\"\
-    <Insert_path_here>\")\n\nWhen you want to configure your pipeline with a certain\
-    \ stack in mind, you can do so as well: `...write_run_configuration_template(stack=<Insert_stack_here>)\n\
-    \nPreviousFind out which configuration was used for a runNextCustomize Docker\
-    \ builds\n\nLast updated 21 days ago"
-  - 'Deleting a Model
-
-
-    Learn how to delete models.
-
+  - 'ZenML - Bridging the gap between ML & Ops
 
-    PreviousRegistering a ModelNextAssociate a pipeline with a Model
 
+    Legacy Docs
 
-    Last updated 4 months ago'
-  - 'Load artifacts into memory
 
+    Bleeding EdgeLegacy Docs0.67.0
 
-    Often ZenML pipeline steps consume artifacts produced by one another directly
-    in the pipeline code, but there are scenarios where you need to pull external
-    data into your steps. Such external data could be artifacts produced by non-ZenML
-    codes. For those cases, it is advised to use ExternalArtifact, but what if we
-    plan to exchange data created with other ZenML pipelines?
 
+    🧙‍♂️Find older version our docs
 
-    ZenML pipelines are first compiled and only executed at some later point. During
-    the compilation phase, all function calls are executed, and this data is fixed
-    as step input parameters. Given all this, the late materialization of dynamic
-    objects, like data artifacts, is crucial. Without late materialization, it would
-    not be possible to pass not-yet-existing artifacts as step inputs, or their metadata,
-    which is often the case in a multi-pipeline setting.
 
+    Powered by GitBook'
+  - 'Google Cloud Image Builder
 
-    We identify two major use cases for exchanging artifacts between pipelines:
 
+    Building container images with Google Cloud Build
 
-    You semantically group your data products using ZenML Models
 
+    The Google Cloud image builder is an image builder flavor provided by the ZenML
+    gcp integration that uses Google Cloud Build to build container images.
 
-    You prefer to use ZenML Client to bring all the pieces together
 
+    When to use it
 
-    We recommend using models to group and access artifacts across pipelines. Find
-    out how to load an artifact from a ZenML Model here.
 
+    You should use the Google Cloud image builder if:
 
-    Use client methods to exchange artifacts
-
-
-    If you don''t yet use the Model Control Plane, you can still exchange data between
-    pipelines with late materialization. Let''s rework the do_predictions pipeline
-    code as follows:
-
-
-    from typing import Annotated
-
-    from zenml import step, pipeline
-
-    from zenml.client import Client
-
-    import pandas as pd
-
-    from sklearn.base import ClassifierMixin'
-- source_sentence: How can I create a Kubernetes cluster on EKS and configure it to
-    run Spark with a custom Docker image?
-  sentences:
-  - 'View logs on the dashboard
 
+    you''re unable to install or use Docker on your client machine.
 
-    PreviousControl loggingNextEnable or disable logs storage
 
+    you''re already using GCP.
 
-    Last updated 21 days ago'
-  - "Datasets in ZenML\n\nModel datasets using simple abstractions.\n\nAs machine\
-    \ learning projects grow in complexity, you often need to work with various data\
-    \ sources and manage intricate data flows. This chapter explores how to use custom\
-    \ Dataset classes and Materializers in ZenML to handle these challenges efficiently.\
-    \ For strategies on scaling your data processing for larger datasets, refer to\
-    \ scaling strategies for big data.\n\nIntroduction to Custom Dataset Classes\n\
-    \nCustom Dataset classes in ZenML provide a way to encapsulate data loading, processing,\
-    \ and saving logic for different data sources. They're particularly useful when:\n\
-    \nWorking with multiple data sources (e.g., CSV files, databases, cloud storage)\n\
-    \nDealing with complex data structures that require special handling\n\nImplementing\
-    \ custom data processing or transformation logic\n\nImplementing Dataset Classes\
-    \ for Different Data Sources\n\nLet's create a base Dataset class and implement\
-    \ it for CSV and BigQuery data sources:\n\nfrom abc import ABC, abstractmethod\n\
-    import pandas as pd\nfrom google.cloud import bigquery\nfrom typing import Optional\n\
-    \nclass Dataset(ABC):\n    @abstractmethod\n    def read_data(self) -> pd.DataFrame:\n\
-    \        pass\n\nclass CSVDataset(Dataset):\n    def __init__(self, data_path:\
-    \ str, df: Optional[pd.DataFrame] = None):\n        self.data_path = data_path\n\
-    \        self.df = df\n\ndef read_data(self) -> pd.DataFrame:\n        if self.df\
-    \ is None:\n            self.df = pd.read_csv(self.data_path)\n        return\
-    \ self.df\n\nclass BigQueryDataset(Dataset):\n    def __init__(\n        self,\n\
-    \        table_id: str,\n        df: Optional[pd.DataFrame] = None,\n        project:\
-    \ Optional[str] = None,\n    ):\n        self.table_id = table_id\n        self.project\
-    \ = project\n        self.df = df\n        self.client = bigquery.Client(project=self.project)\n\
-    \ndef read_data(self) -> pd.DataFrame:\n        query = f\"SELECT * FROM `{self.table_id}`\"\
-    \n        self.df = self.client.query(query).to_dataframe()\n        return self.df"
-  - 'e the correct region is selected on the top right.Click on Add cluster and select
-    Create.
 
+    your stack is mainly composed of other Google Cloud components such as the GCS
+    Artifact Store or the Vertex Orchestrator.
 
-    Enter a name and select the cluster role for Cluster service role.
 
+    How to deploy it
 
-    Keep the default values for the networking and logging steps and create the cluster.
 
+    Would you like to skip ahead and deploy a full ZenML cloud stack already, including
+    the Google Cloud image builder? Check out the in-browser stack deployment wizard,
+    the stack registration wizard, or the ZenML GCP Terraform module for a shortcut
+    on how to deploy & register this stack component.
 
-    Note down the cluster name and the API server endpoint:
 
+    In order to use the ZenML Google Cloud image builder you need to enable Google
+    Cloud Build relevant APIs on the Google Cloud project.
 
-    EKS_CLUSTER_NAME=<EKS_CLUSTER_NAME>
 
-    EKS_API_SERVER_ENDPOINT=<API_SERVER_ENDPOINT>
+    How to use it
 
 
-    After the cluster is created, select it and click on Add node group in the Compute
-    tab.
+    To use the Google Cloud image builder, we need:
 
 
-    Enter a name and select the node role.
+    The ZenML gcp integration installed. If you haven''t done so, run:
 
 
-    For the instance type, we recommend t3a.xlarge, as it provides up to 4 vCPUs and
-    16 GB of memory.
+    zenml integration install gcp
 
 
-    Docker image for the Spark drivers and executors
+    A GCP Artifact Store where the build context will be uploaded, so Google Cloud
+    Build can access it.
 
 
-    When you want to run your steps on a Kubernetes cluster, Spark will require you
-    to choose a base image for the driver and executor pods. Normally, for this purpose,
-    you can either use one of the base images in Spark’s dockerhub or create an image
-    using the docker-image-tool which will use your own Spark installation and build
-    an image.
+    A GCP container registry where the built image will be pushed.
 
 
-    When using Spark in EKS, you need to use the latter and utilize the docker-image-tool.
-    However, before the build process, you also need to download the following packages
+    Optionally, the GCP project ID in which you want to run the build and a service
+    account with the needed permissions to run the build. If not provided, then the
+    project ID and credentials will be inferred from the environment.
 
 
-    hadoop-aws = 3.3.1
+    Optionally, you can change:
 
 
-    aws-java-sdk-bundle = 1.12.150
+    the Docker image used by Google Cloud Build to execute the steps to build and
+    push the Docker image. By default, the builder image will be ''gcr.io/cloud-builders/docker''.
 
 
-    and put them in the jars folder within your Spark installation. Once that is set
-    up, you can build the image as follows:
+    The network to which the container used to build the ZenML pipeline Docker image
+    will be attached. More information: Cloud build network.
 
 
-    cd $SPARK_HOME # If this empty for you then you need to set the SPARK_HOME variable
-    which points to your Spark installation
-
-
-    SPARK_IMAGE_TAG=<SPARK_IMAGE_TAG>
-
-
-    ./bin/docker-image-tool.sh -t $SPARK_IMAGE_TAG -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile
-    -u 0 build
-
-
-    BASE_IMAGE_NAME=spark-py:$SPARK_IMAGE_TAG
-
-
-    If you are working on an M1 Mac, you will need to build the image for the amd64
-    architecture, by using the prefix -X on the previous command. For example:
-
-
-    ./bin/docker-image-tool.sh -X -t $SPARK_IMAGE_TAG -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile
-    -u 0 build
-
-
-    Configuring RBAC'
-- source_sentence: How can I configure a pipeline with a YAML file in ZenML?
+    The build timeout for the build, and for the blocking operation waiting for the
+    build to finish. More information: Build Timeout.'
+  - "--connector <CONNECTOR_ID>\n\nExample Command Output$ zenml image-builder connect\
+    \ gcp-image-builder --connector gcp-generic\nSuccessfully connected image builder\
+    \ `gcp-image-builder` to the following resources:\n┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n\
+    ┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE\
+    \ TYPE  │ RESOURCE NAMES ┃\n┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────┼────────────────┨\n\
+    ┃ bfdb657d-d808-47e7-9974-9ba6e4919d83 │ gcp-generic    │ \U0001F535 gcp     \
+    \    │ \U0001F535 gcp-generic │ zenml-core     ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\
+    \nAs a final step, you can use the GCP Image Builder in a ZenML Stack:\n\n# Register\
+    \ and set a stack with the new image builder\nzenml stack register <STACK_NAME>\
+    \ -i <IMAGE_BUILDER_NAME> ... --set\n\nWhen you register the GCP Image Builder,\
+    \ you can generate a GCP Service Account Key, save it to a local file and then\
+    \ reference it in the Image Builder configuration.\n\nThis method has the advantage\
+    \ that you don't need to install and configure the GCP CLI on your host, but it's\
+    \ still not as secure as using a GCP Service Connector and the stack component\
+    \ configuration is not portable to other hosts.\n\nFor this method, you need to\
+    \ create a user-managed GCP service account, and grant it privileges to access\
+    \ the Cloud Build API and to run Cloud Builder jobs (e.g. the Cloud Build Editor\
+    \ IAM role.\n\nWith the service account key downloaded to a local file, you can\
+    \ register the GCP Image Builder as follows:\n\nzenml image-builder register <IMAGE_BUILDER_NAME>\
+    \ \\\n    --flavor=gcp \\\n    --project=<GCP_PROJECT_ID> \\\n    --service_account_path=<PATH_TO_SERVICE_ACCOUNT_KEY>\
+    \ \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME> \\\n    --network=<DOCKER_NETWORK>\
+    \ \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>"
+- source_sentence: How do I finetune embeddings using Sentence Transformers in ZenML?
   sentences:
-  - 'atically retry steps
-
-
-    Run pipelines asynchronouslyControl execution order of steps
-
-
-    Using a custom step invocation ID
-
-
-    Name your pipeline runs
-
-
-    Use failure/success hooks
-
-
-    Hyperparameter tuning
-
-
-    Access secrets in a step
-
-
-    Run an individual step
-
-
-    Fetching pipelines
-
-
-    Get past pipeline/step runs
-
-
-    🚨Trigger a pipeline
-
-
-    Use templates: Python SDK
-
-
-    Use templates: Dashboard
-
-
-    Use templates: Rest API
-
-
-    📃Use configuration files
-
-
-    How to configure a pipeline with a YAML
-
-
-    What can be configured
-
-
-    Runtime settings for Docker, resources, and stack components
-
-
-    Configuration hierarchy
-
-
-    Find out which configuration was used for a run
-
-
-    Autogenerate a template yaml file
-
-
-    🐳Customize Docker builds
-
+  - "nsible for cluster-manager-specific configuration._io_configuration is a critical\
+    \ method. Even though we have materializers, Spark might require additional packages\
+    \ and configuration to work with a specific filesystem. This method is used as\
+    \ an interface to provide this configuration.\n\n_additional_configuration takes\
+    \ the submit_args, converts, and appends them to the overall configuration.\n\n\
+    Once the configuration is completed, _launch_spark_job comes into play. This takes\
+    \ the completed configuration and runs a Spark job on the given master URL with\
+    \ the specified deploy_mode. By default, this is achieved by creating and executing\
+    \ a spark-submit command.\n\nWarning\n\nIn its first iteration, the pre-configuration\
+    \ with _io_configuration method is only effective when it is paired with an S3ArtifactStore\
+    \ (which has an authentication secret). When used with other artifact store flavors,\
+    \ you might be required to provide additional configuration through the submit_args.\n\
+    \nStack Component: KubernetesSparkStepOperator\n\nThe KubernetesSparkStepOperator\
+    \ is implemented by subclassing the base SparkStepOperator and uses the PipelineDockerImageBuilder\
+    \ class to build and push the required Docker images.\n\nfrom typing import Optional\n\
+    \nfrom zenml.integrations.spark.step_operators.spark_step_operator import (\n\
+    \    SparkStepOperatorConfig\n)\n\nclass KubernetesSparkStepOperatorConfig(SparkStepOperatorConfig):\n\
+    \    \"\"\"Config for the Kubernetes Spark step operator.\"\"\"\n\nnamespace:\
+    \ Optional[str] = None\n    service_account: Optional[str] = None\n\nfrom pyspark.conf\
+    \ import SparkConf\n\nfrom zenml.utils.pipeline_docker_image_builder import PipelineDockerImageBuilder\n\
+    from zenml.integrations.spark.step_operators.spark_step_operator import (\n  \
+    \  SparkStepOperator\n)\n\nclass KubernetesSparkStepOperator(SparkStepOperator):\n\
+    \    \"\"\"Step operator which runs Steps with Spark on Kubernetes.\"\"\""
+  - 'Finetuning embeddings with Sentence Transformers
 
-    Docker settings on a pipeline
 
+    Finetune embeddings with Sentence Transformers.
 
-    Docker settings on a step
 
+    PreviousSynthetic data generationNextEvaluating finetuned embeddings
 
-    Use a prebuilt image for pipeline execution
 
+    Last updated 1 month ago'
+  - 'Whylogs
 
-    Specify pip dependencies and apt packages
 
+    How to collect and visualize statistics to track changes in your pipelines'' data
+    with whylogs/WhyLabs profiling.
 
-    Use your own Dockerfiles
 
+    The whylogs/WhyLabs Data Validator flavor provided with the ZenML integration
+    uses whylogs and WhyLabs to generate and track data profiles, highly accurate
+    descriptive representations of your data. The profiles can be used to implement
+    automated corrective actions in your pipelines, or to render interactive representations
+    for further visual interpretation, evaluation and documentation.
 
-    Which files are built into the image
 
+    When would you want to use it?
 
-    How to reuse builds
 
+    Whylogs is an open-source library that analyzes your data and creates statistical
+    summaries called whylogs profiles. Whylogs profiles can be processed in your pipelines
+    and visualized locally or uploaded to the WhyLabs platform, where more in depth
+    analysis can be carried out. Even though whylogs also supports other data types,
+    the ZenML whylogs integration currently only works with tabular data in pandas.DataFrame
+    format.
 
-    Define where an image is built
 
+    You should use the whylogs/WhyLabs Data Validator when you need the following
+    data validation features that are possible with whylogs and WhyLabs:
 
-    📔Run remote pipelines from notebooks
 
+    Data Quality: validate data quality in model inputs or in a data pipeline
 
-    Limitations of defining steps in notebook cells
 
+    Data Drift: detect data drift in model input features
 
-    Run a single step from a notebook
 
+    Model Drift: Detect training-serving skew, concept drift, and model performance
+    degradation
 
-    🤹Manage your ZenML server
 
+    You should consider one of the other Data Validator flavors if you need a different
+    set of data validation features.
 
-    Best practices for upgrading ZenML
 
+    How do you deploy it?
 
-    Upgrade your ZenML server
 
+    The whylogs Data Validator flavor is included in the whylogs ZenML integration,
+    you need to install it on your local machine to be able to register a whylogs
+    Data Validator and add it to your stack:
 
-    Using ZenML server in production
 
+    zenml integration install whylogs -y
 
-    Troubleshoot your ZenML server
 
-
-    Migration guide
-
-
-    Migration guide 0.13.2 → 0.20.0
-
-
-    Migration guide 0.23.0 → 0.30.0
-
-
-    Migration guide 0.39.1 → 0.41.0
-
-
-    Migration guide 0.58.2 → 0.60.0
-
-
-    📍Develop locally
-
-
-    Use config files to develop locally
-
-
-    Keep your pipelines and dashboard clean
-
-
-    ⚒️Manage stacks & components
-
-
-    Deploy a cloud stack with ZenML
-
-
-    Deploy a cloud stack with Terraform
-
-
-    Register a cloud stack
-
-
-    Reference secrets in stack configuration
-
-
-    Implement a custom stack component
-
-
-    🚜Train with GPUs
-
-
-    Distributed Training with 🤗 Accelerate
-
-
-    🌲Control logging
-
-
-    View logs on the dashboard
-
-
-    Enable or disable logs storage
-
-
-    Set logging verbosity
-
-
-    Disable rich traceback output
-
-
-    Disable colorful logging
-
-
-    🗄️Handle Data/Artifacts
-
-
-    How ZenML stores data
-
-
-    Return multiple outputs from a step
-
-
-    Delete an artifact
-
-
-    Organize data with tags
-
-
-    Get arbitrary artifacts in a step'
-  - 'Security best practices
-
-
-    Best practices concerning the various authentication methods implemented by Service
-    Connectors.
-
-
-    Service Connector Types, especially those targeted at cloud providers, offer a
-    plethora of authentication methods matching those supported by remote cloud platforms.
-    While there is no single authentication standard that unifies this process, there
-    are some patterns that are easily identifiable and can be used as guidelines when
-    deciding which authentication method to use to configure a Service Connector.
-
-
-    This section explores some of those patterns and gives some advice regarding which
-    authentication methods are best suited for your needs.
-
-
-    This section may require some general knowledge about authentication and authorization
-    to be properly understood. We tried to keep it simple and limit ourselves to talking
-    about high-level concepts, but some areas may get a bit too technical.
-
-
-    Username and password
-
-
-    The key takeaway is this: you should avoid using your primary account password
-    as authentication credentials as much as possible. If there are alternative authentication
-    methods that you can use or other types of credentials (e.g. session tokens, API
-    keys, API tokens), you should always try to use those instead.
-
-
-    Ultimately, if you have no choice, be cognizant of the third parties you share
-    your passwords with. If possible, they should never leave the premises of your
-    local host or development environment.
-
-
-    This is the typical authentication method that uses a username or account name
-    plus the associated password. While this is the de facto method used to log in
-    with web consoles and local CLIs, this is the least secure of all authentication
-    methods and never something you want to share with other members of your team
-    or organization or use to authenticate automated workloads.'
-  - "━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━┛$ zenml orchestrator connect\
-    \ <ORCHESTRATOR_NAME> --connector aws-iam-multi-us\nRunning with active stack:\
-    \ 'default' (repository)\nSuccessfully connected orchestrator `<ORCHESTRATOR_NAME>`\
-    \ to the following resources:\n┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┓\n\
-    ┃             CONNECTOR ID             │ CONNECTOR NAME   │ CONNECTOR TYPE │ RESOURCE\
-    \ TYPE         │ RESOURCE NAMES   ┃\n┠──────────────────────────────────────┼──────────────────┼────────────────┼───────────────────────┼──────────────────┨\n\
-    ┃ ed528d5a-d6cb-4fc4-bc52-c3d2d01643e5 │ aws-iam-multi-us │ \U0001F536 aws   \
-    \      │ \U0001F300 kubernetes-cluster │ zenhacks-cluster ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┛\n\
-    \n# Register and activate a stack with the new orchestrator\n$ zenml stack register\
-    \ <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set\n\nif you don't have a Service\
-    \ Connector on hand and you don't want to register one , the local Kubernetes\
-    \ kubectl client needs to be configured with a configuration context pointing\
-    \ to the remote cluster. The kubernetes_context stack component must also be configured\
-    \ with the value of that context:\n\nzenml orchestrator register <ORCHESTRATOR_NAME>\
-    \ \\\n    --flavor=kubernetes \\\n    --kubernetes_context=<KUBERNETES_CONTEXT>\n\
-    \n# Register and activate a stack with the new orchestrator\nzenml stack register\
-    \ <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set\n\nZenML will build a Docker image\
-    \ called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME> which includes your code\
-    \ and use it to run your pipeline steps in Kubernetes. Check out this page if\
-    \ you want to learn more about how ZenML builds these images and how you can customize\
-    \ them.\n\nYou can now run any ZenML pipeline using the Kubernetes orchestrator:\n\
-    \npython file_that_runs_a_zenml_pipeline.py"
-datasets: []
+    If you don''t need to connect to the WhyLabs platform to upload and store the
+    generated whylogs data profiles, the Data Validator stack component does not require
+    any configuration parameters. Adding it to a stack is as simple as running e.g.:'
+- source_sentence: What is the purpose of ZenML in the context of ML and Ops?
+  sentences:
+  - "> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS># Register and set a stack\
+    \ with the new image builder\nzenml stack register <STACK_NAME> -i <IMAGE_BUILDER_NAME>\
+    \ ... --set\n\nCaveats\n\nAs described in this Google Cloud Build documentation\
+    \ page, Google Cloud Build uses containers to execute the build steps which are\
+    \ automatically attached to a network called cloudbuild that provides some Application\
+    \ Default Credentials (ADC), that allow the container to be authenticated and\
+    \ therefore use other GCP services.\n\nBy default, the GCP Image Builder is executing\
+    \ the build command of the ZenML Pipeline Docker image with the option --network=cloudbuild,\
+    \ so the ADC provided by the cloudbuild network can also be used in the build.\
+    \ This is useful if you want to install a private dependency from a GCP Artifact\
+    \ Registry, but you will also need to use a custom base parent image with the\
+    \ keyrings.google-artifactregistry-auth installed, so pip can connect and authenticate\
+    \ in the private artifact registry to download the dependency.\n\nFROM zenmldocker/zenml:latest\n\
+    \nRUN pip install keyrings.google-artifactregistry-auth\n\nThe above Dockerfile\
+    \ uses zenmldocker/zenml:latest as a base image, but is recommended to change\
+    \ the tag to specify the ZenML version and Python version like 0.33.0-py3.10.\n\
+    \nPreviousKaniko Image BuilderNextDevelop a Custom Image Builder\n\nLast updated\
+    \ 21 days ago"
+  - "Control caching behavior\n\nBy default steps in ZenML pipelines are cached whenever\
+    \ code and parameters stay unchanged.\n\n@step(enable_cache=True) # set cache\
+    \ behavior at step level\ndef load_data(parameter: int) -> dict:\n    ...\n\n\
+    @step(enable_cache=False) # settings at step level override pipeline level\ndef\
+    \ train_model(data: dict) -> None:\n    ...\n\n@pipeline(enable_cache=True) #\
+    \ set cache behavior at step level\ndef simple_ml_pipeline(parameter: int):\n\
+    \    ...\n\nCaching only happens when code and parameters stay the same.\n\nLike\
+    \ many other step and pipeline settings, you can also change this afterward:\n\
+    \n# Same as passing it in the step decorator\nmy_step.configure(enable_cache=...)\n\
+    \n# Same as passing it in the pipeline decorator\nmy_pipeline.configure(enable_cache=...)\n\
+    \nFind out here how to configure this in a YAML file\n\nPreviousStep output typing\
+    \ and annotationNextSchedule a pipeline\n\nLast updated 4 months ago"
+  - 'ZenML - Bridging the gap between ML & Ops
+
+
+    Legacy Docs
+
+
+    Bleeding EdgeLegacy Docs0.67.0
+
+
+    🧙‍♂️Find older version our docs
+
+
+    Powered by GitBook'
+- source_sentence: How does ZenML facilitate the flow of data between steps in a pipeline?
+  sentences:
+  - "tainer_registry \\\n    -i local_builder \\\n    --setOnce you added the step\
+    \ operator to your active stack, you can use it to execute individual steps of\
+    \ your pipeline by specifying it in the @step decorator as follows:\n\nfrom zenml\
+    \ import step\n\n@step(step_operator=<STEP_OPERATOR_NAME>)\ndef step_on_spark(...)\
+    \ -> ...:\n    \"\"\"Some step that should run with Spark on Kubernetes.\"\"\"\
+    \n    ...\n\nAfter successfully running any step with a KubernetesSparkStepOperator,\
+    \ you should be able to see that a Spark driver pod was created in your cluster\
+    \ for each pipeline step when running kubectl get pods -n $KUBERNETES_NAMESPACE.\n\
+    \nInstead of hardcoding a step operator name, you can also use the Client to dynamically\
+    \ use the step operator of your active stack:\n\nfrom zenml.client import Client\n\
+    \nstep_operator = Client().active_stack.step_operator\n\n@step(step_operator=step_operator.name)\n\
+    def step_on_spark(...) -> ...:\n    ...\n\nAdditional configuration\n\nFor additional\
+    \ configuration of the Spark step operator, you can pass SparkStepOperatorSettings\
+    \ when defining or running your pipeline. Check out the SDK docs for a full list\
+    \ of available attributes and this docs page for more information on how to specify\
+    \ settings.\n\nPreviousKubernetesNextDevelop a Custom Step Operator\n\nLast updated\
+    \ 4 months ago"
+  - "\U0001F5C4️Handle Data/Artifacts\n\nStep outputs in ZenML are stored in the artifact\
+    \ store. This enables caching, lineage and auditability. Using type annotations\
+    \ helps with transparency, passing data between steps, and serializing/des\n\n\
+    For best results, use type annotations for your outputs. This is good coding practice\
+    \ for transparency, helps ZenML handle passing data between steps, and also enables\
+    \ ZenML to serialize and deserialize (referred to as 'materialize' in ZenML) the\
+    \ data.\n\n@step\ndef load_data(parameter: int) -> Dict[str, Any]:\n\n# do something\
+    \ with the parameter here\n\ntraining_data = [[1, 2], [3, 4], [5, 6]]\n    labels\
+    \ = [0, 1, 0]\n    return {'features': training_data, 'labels': labels}\n\n@step\n\
+    def train_model(data: Dict[str, Any]) -> None:\n    total_features = sum(map(sum,\
+    \ data['features']))\n    total_labels = sum(data['labels'])\n    \n    # Train\
+    \ some model here\n    \n    print(f\"Trained model using {len(data['features'])}\
+    \ data points. \"\n          f\"Feature sum is {total_features}, label sum is\
+    \ {total_labels}\")\n\n@pipeline  \ndef simple_ml_pipeline(parameter: int):\n\
+    \    dataset = load_data(parameter=parameter)  # Get the output \n    train_model(dataset)\
+    \  # Pipe the previous step output into the downstream step\n\nIn this code, we\
+    \ define two steps: load_data and train_model. The load_data step takes an integer\
+    \ parameter and returns a dictionary containing training data and labels. The\
+    \ train_model step receives the dictionary from load_data, extracts the features\
+    \ and labels, and trains a model (not shown here).\n\nFinally, we define a pipeline\
+    \ simple_ml_pipeline that chains the load_data and train_model steps together.\
+    \ The output from load_data is passed as input to train_model, demonstrating how\
+    \ data flows between steps in a ZenML pipeline.\n\nPreviousDisable colorful loggingNextHow\
+    \ ZenML stores data\n\nLast updated 4 months ago"
+  - "in the ZenML dashboard.\n\nThe whylogs standard stepZenML wraps the whylogs/WhyLabs\
+    \ functionality in the form of a standard WhylogsProfilerStep step. The only field\
+    \ in the step config is a dataset_timestamp attribute which is only relevant when\
+    \ you upload the profiles to WhyLabs that uses this field to group and merge together\
+    \ profiles belonging to the same dataset. The helper function get_whylogs_profiler_step\
+    \ used to create an instance of this standard step takes in an optional dataset_id\
+    \ parameter that is also used only in the context of WhyLabs upload to identify\
+    \ the model in the context of which the profile is uploaded, e.g.:\n\nfrom zenml.integrations.whylogs.steps\
+    \ import get_whylogs_profiler_step\n\ntrain_data_profiler = get_whylogs_profiler_step(dataset_id=\"\
+    model-2\")\ntest_data_profiler = get_whylogs_profiler_step(dataset_id=\"model-3\"\
+    )\n\nThe step can then be inserted into your pipeline where it can take in a pandas.DataFrame\
+    \ dataset, e.g.:\n\nfrom zenml import pipeline\n\n@pipeline\ndef data_profiling_pipeline():\n\
+    \    data, _ = data_loader()\n    train, test = data_splitter(data)\n    train_data_profiler(train)\n\
+    \    test_data_profiler(test)\n\ndata_profiling_pipeline()\n\nAs can be seen from\
+    \ the step definition , the step takes in a dataset and returns a whylogs DatasetProfileView\
+    \ object:\n\n@step\ndef whylogs_profiler_step(\n    dataset: pd.DataFrame,\n \
+    \   dataset_timestamp: Optional[datetime.datetime] = None,\n) -> DatasetProfileView:\n\
+    \    ...\n\nYou should consult the official whylogs documentation for more information\
+    \ on what you can do with the collected profiles.\n\nYou can view the complete\
+    \ list of configuration parameters in the SDK docs.\n\nThe whylogs Data Validator\n\
+    \nThe whylogs Data Validator implements the same interface as do all Data Validators,\
+    \ so this method forces you to maintain some level of compatibility with the overall\
+    \ Data Validator abstraction, which guarantees an easier migration in case you\
+    \ decide to switch to another Data Validator."
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
 metrics:
@@ -610,49 +490,49 @@ model-index:
       type: dim_384
     metrics:
     - type: cosine_accuracy@1
-      value: 0.1863013698630137
+      value: 1.0
       name: Cosine Accuracy@1
     - type: cosine_accuracy@3
-      value: 0.4794520547945205
+      value: 1.0
       name: Cosine Accuracy@3
     - type: cosine_accuracy@5
-      value: 0.6602739726027397
+      value: 1.0
       name: Cosine Accuracy@5
     - type: cosine_accuracy@10
-      value: 0.7972602739726027
+      value: 1.0
       name: Cosine Accuracy@10
     - type: cosine_precision@1
-      value: 0.1863013698630137
+      value: 1.0
       name: Cosine Precision@1
     - type: cosine_precision@3
-      value: 0.1598173515981735
+      value: 0.3333333333333333
       name: Cosine Precision@3
     - type: cosine_precision@5
-      value: 0.13205479452054794
+      value: 0.2
       name: Cosine Precision@5
     - type: cosine_precision@10
-      value: 0.07972602739726026
+      value: 0.1
       name: Cosine Precision@10
     - type: cosine_recall@1
-      value: 0.1863013698630137
+      value: 1.0
       name: Cosine Recall@1
     - type: cosine_recall@3
-      value: 0.4794520547945205
+      value: 1.0
       name: Cosine Recall@3
     - type: cosine_recall@5
-      value: 0.6602739726027397
+      value: 1.0
       name: Cosine Recall@5
     - type: cosine_recall@10
-      value: 0.7972602739726027
+      value: 1.0
       name: Cosine Recall@10
     - type: cosine_ndcg@10
-      value: 0.47459290361092754
+      value: 1.0
       name: Cosine Ndcg@10
     - type: cosine_mrr@10
-      value: 0.3725994781474232
+      value: 1.0
       name: Cosine Mrr@10
     - type: cosine_map@100
-      value: 0.37953809566266083
+      value: 1.0
       name: Cosine Map@100
   - task:
       type: information-retrieval
@@ -662,49 +542,49 @@ model-index:
       type: dim_256
     metrics:
     - type: cosine_accuracy@1
-      value: 0.18356164383561643
+      value: 1.0
       name: Cosine Accuracy@1
     - type: cosine_accuracy@3
-      value: 0.4876712328767123
+      value: 1.0
       name: Cosine Accuracy@3
     - type: cosine_accuracy@5
-      value: 0.6602739726027397
+      value: 1.0
       name: Cosine Accuracy@5
     - type: cosine_accuracy@10
-      value: 0.7917808219178082
+      value: 1.0
       name: Cosine Accuracy@10
     - type: cosine_precision@1
-      value: 0.18356164383561643
+      value: 1.0
       name: Cosine Precision@1
     - type: cosine_precision@3
-      value: 0.16255707762557076
+      value: 0.3333333333333333
       name: Cosine Precision@3
     - type: cosine_precision@5
-      value: 0.1320547945205479
+      value: 0.2
       name: Cosine Precision@5
     - type: cosine_precision@10
-      value: 0.07917808219178081
+      value: 0.1
       name: Cosine Precision@10
     - type: cosine_recall@1
-      value: 0.18356164383561643
+      value: 1.0
       name: Cosine Recall@1
     - type: cosine_recall@3
-      value: 0.4876712328767123
+      value: 1.0
       name: Cosine Recall@3
     - type: cosine_recall@5
-      value: 0.6602739726027397
+      value: 1.0
       name: Cosine Recall@5
     - type: cosine_recall@10
-      value: 0.7917808219178082
+      value: 1.0
       name: Cosine Recall@10
     - type: cosine_ndcg@10
-      value: 0.47334554819769054
+      value: 1.0
       name: Cosine Ndcg@10
     - type: cosine_mrr@10
-      value: 0.3724179169384647
+      value: 1.0
       name: Cosine Mrr@10
     - type: cosine_map@100
-      value: 0.37931260226095775
+      value: 1.0
       name: Cosine Map@100
   - task:
       type: information-retrieval
@@ -714,49 +594,49 @@ model-index:
       type: dim_128
     metrics:
     - type: cosine_accuracy@1
-      value: 0.18356164383561643
+      value: 1.0
       name: Cosine Accuracy@1
     - type: cosine_accuracy@3
-      value: 0.4684931506849315
+      value: 1.0
       name: Cosine Accuracy@3
     - type: cosine_accuracy@5
-      value: 0.6356164383561644
+      value: 1.0
       name: Cosine Accuracy@5
     - type: cosine_accuracy@10
-      value: 0.7780821917808219
+      value: 1.0
       name: Cosine Accuracy@10
     - type: cosine_precision@1
-      value: 0.18356164383561643
+      value: 1.0
       name: Cosine Precision@1
     - type: cosine_precision@3
-      value: 0.1561643835616438
+      value: 0.3333333333333333
       name: Cosine Precision@3
     - type: cosine_precision@5
-      value: 0.12712328767123285
+      value: 0.2
       name: Cosine Precision@5
     - type: cosine_precision@10
-      value: 0.07780821917808219
+      value: 0.1
       name: Cosine Precision@10
     - type: cosine_recall@1
-      value: 0.18356164383561643
+      value: 1.0
       name: Cosine Recall@1
     - type: cosine_recall@3
-      value: 0.4684931506849315
+      value: 1.0
       name: Cosine Recall@3
     - type: cosine_recall@5
-      value: 0.6356164383561644
+      value: 1.0
       name: Cosine Recall@5
     - type: cosine_recall@10
-      value: 0.7780821917808219
+      value: 1.0
       name: Cosine Recall@10
     - type: cosine_ndcg@10
-      value: 0.46219638130094637
+      value: 1.0
       name: Cosine Ndcg@10
     - type: cosine_mrr@10
-      value: 0.3628680147858229
+      value: 1.0
       name: Cosine Mrr@10
     - type: cosine_map@100
-      value: 0.37047490630037583
+      value: 1.0
       name: Cosine Map@100
   - task:
       type: information-retrieval
@@ -766,55 +646,55 @@ model-index:
       type: dim_64
     metrics:
     - type: cosine_accuracy@1
-      value: 0.2054794520547945
+      value: 0.75
       name: Cosine Accuracy@1
     - type: cosine_accuracy@3
-      value: 0.4767123287671233
+      value: 1.0
       name: Cosine Accuracy@3
     - type: cosine_accuracy@5
-      value: 0.6273972602739726
+      value: 1.0
       name: Cosine Accuracy@5
     - type: cosine_accuracy@10
-      value: 0.7534246575342466
+      value: 1.0
       name: Cosine Accuracy@10
     - type: cosine_precision@1
-      value: 0.2054794520547945
+      value: 0.75
       name: Cosine Precision@1
     - type: cosine_precision@3
-      value: 0.15890410958904108
+      value: 0.3333333333333333
       name: Cosine Precision@3
     - type: cosine_precision@5
-      value: 0.12547945205479452
+      value: 0.2
       name: Cosine Precision@5
     - type: cosine_precision@10
-      value: 0.07534246575342465
+      value: 0.1
       name: Cosine Precision@10
     - type: cosine_recall@1
-      value: 0.2054794520547945
+      value: 0.75
       name: Cosine Recall@1
     - type: cosine_recall@3
-      value: 0.4767123287671233
+      value: 1.0
       name: Cosine Recall@3
     - type: cosine_recall@5
-      value: 0.6273972602739726
+      value: 1.0
       name: Cosine Recall@5
     - type: cosine_recall@10
-      value: 0.7534246575342466
+      value: 1.0
       name: Cosine Recall@10
     - type: cosine_ndcg@10
-      value: 0.46250756548591326
+      value: 0.9077324383928644
       name: Cosine Ndcg@10
     - type: cosine_mrr@10
-      value: 0.37069906501413347
+      value: 0.875
       name: Cosine Mrr@10
     - type: cosine_map@100
-      value: 0.37874559284369463
+      value: 0.875
       name: Cosine Map@100
 ---
 
 # zenml/finetuned-snowflake-arctic-embed-m-v1.5
 
-This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Snowflake/snowflake-arctic-embed-m-v1.5](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Snowflake/snowflake-arctic-embed-m-v1.5](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5) on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
 
 ## Model Details
 
@@ -824,7 +704,8 @@ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [S
 - **Maximum Sequence Length:** 512 tokens
 - **Output Dimensionality:** 768 tokens
 - **Similarity Function:** Cosine Similarity
-<!-- - **Training Dataset:** Unknown -->
+- **Training Dataset:**
+    - json
 - **Language:** en
 - **License:** apache-2.0
 
@@ -862,9 +743,9 @@ from sentence_transformers import SentenceTransformer
 model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m-v1.5")
 # Run inference
 sentences = [
-    'How can I configure a pipeline with a YAML file in ZenML?',
-    'atically retry steps\n\nRun pipelines asynchronouslyControl execution order of steps\n\nUsing a custom step invocation ID\n\nName your pipeline runs\n\nUse failure/success hooks\n\nHyperparameter tuning\n\nAccess secrets in a step\n\nRun an individual step\n\nFetching pipelines\n\nGet past pipeline/step runs\n\n🚨Trigger a pipeline\n\nUse templates: Python SDK\n\nUse templates: Dashboard\n\nUse templates: Rest API\n\n📃Use configuration files\n\nHow to configure a pipeline with a YAML\n\nWhat can be configured\n\nRuntime settings for Docker, resources, and stack components\n\nConfiguration hierarchy\n\nFind out which configuration was used for a run\n\nAutogenerate a template yaml file\n\n🐳Customize Docker builds\n\nDocker settings on a pipeline\n\nDocker settings on a step\n\nUse a prebuilt image for pipeline execution\n\nSpecify pip dependencies and apt packages\n\nUse your own Dockerfiles\n\nWhich files are built into the image\n\nHow to reuse builds\n\nDefine where an image is built\n\n📔Run remote pipelines from notebooks\n\nLimitations of defining steps in notebook cells\n\nRun a single step from a notebook\n\n🤹Manage your ZenML server\n\nBest practices for upgrading ZenML\n\nUpgrade your ZenML server\n\nUsing ZenML server in production\n\nTroubleshoot your ZenML server\n\nMigration guide\n\nMigration guide 0.13.2 → 0.20.0\n\nMigration guide 0.23.0 → 0.30.0\n\nMigration guide 0.39.1 → 0.41.0\n\nMigration guide 0.58.2 → 0.60.0\n\n📍Develop locally\n\nUse config files to develop locally\n\nKeep your pipelines and dashboard clean\n\n⚒��Manage stacks & components\n\nDeploy a cloud stack with ZenML\n\nDeploy a cloud stack with Terraform\n\nRegister a cloud stack\n\nReference secrets in stack configuration\n\nImplement a custom stack component\n\n🚜Train with GPUs\n\nDistributed Training with 🤗 Accelerate\n\n🌲Control logging\n\nView logs on the dashboard\n\nEnable or disable logs storage\n\nSet logging verbosity\n\nDisable rich traceback output\n\nDisable colorful logging\n\n🗄️Handle Data/Artifacts\n\nHow ZenML stores data\n\nReturn multiple outputs from a step\n\nDelete an artifact\n\nOrganize data with tags\n\nGet arbitrary artifacts in a step',
-    "━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━┛$ zenml orchestrator connect <ORCHESTRATOR_NAME> --connector aws-iam-multi-us\nRunning with active stack: 'default' (repository)\nSuccessfully connected orchestrator `<ORCHESTRATOR_NAME>` to the following resources:\n┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┓\n┃             CONNECTOR ID             │ CONNECTOR NAME   │ CONNECTOR TYPE │ RESOURCE TYPE         │ RESOURCE NAMES   ┃\n┠──────────────────────────────────────┼──────────────────┼────────────────┼───────────────────────┼──────────────────┨\n┃ ed528d5a-d6cb-4fc4-bc52-c3d2d01643e5 │ aws-iam-multi-us │ 🔶 aws         │ 🌀 kubernetes-cluster │ zenhacks-cluster ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┛\n\n# Register and activate a stack with the new orchestrator\n$ zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set\n\nif you don't have a Service Connector on hand and you don't want to register one , the local Kubernetes kubectl client needs to be configured with a configuration context pointing to the remote cluster. The kubernetes_context stack component must also be configured with the value of that context:\n\nzenml orchestrator register <ORCHESTRATOR_NAME> \\\n    --flavor=kubernetes \\\n    --kubernetes_context=<KUBERNETES_CONTEXT>\n\n# Register and activate a stack with the new orchestrator\nzenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set\n\nZenML will build a Docker image called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME> which includes your code and use it to run your pipeline steps in Kubernetes. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.\n\nYou can now run any ZenML pipeline using the Kubernetes orchestrator:\n\npython file_that_runs_a_zenml_pipeline.py",
+    'How does ZenML facilitate the flow of data between steps in a pipeline?',
+    '🗄️Handle Data/Artifacts\n\nStep outputs in ZenML are stored in the artifact store. This enables caching, lineage and auditability. Using type annotations helps with transparency, passing data between steps, and serializing/des\n\nFor best results, use type annotations for your outputs. This is good coding practice for transparency, helps ZenML handle passing data between steps, and also enables ZenML to serialize and deserialize (referred to as \'materialize\' in ZenML) the data.\n\n@step\ndef load_data(parameter: int) -> Dict[str, Any]:\n\n# do something with the parameter here\n\ntraining_data = [[1, 2], [3, 4], [5, 6]]\n    labels = [0, 1, 0]\n    return {\'features\': training_data, \'labels\': labels}\n\n@step\ndef train_model(data: Dict[str, Any]) -> None:\n    total_features = sum(map(sum, data[\'features\']))\n    total_labels = sum(data[\'labels\'])\n    \n    # Train some model here\n    \n    print(f"Trained model using {len(data[\'features\'])} data points. "\n          f"Feature sum is {total_features}, label sum is {total_labels}")\n\n@pipeline  \ndef simple_ml_pipeline(parameter: int):\n    dataset = load_data(parameter=parameter)  # Get the output \n    train_model(dataset)  # Pipe the previous step output into the downstream step\n\nIn this code, we define two steps: load_data and train_model. The load_data step takes an integer parameter and returns a dictionary containing training data and labels. The train_model step receives the dictionary from load_data, extracts the features and labels, and trains a model (not shown here).\n\nFinally, we define a pipeline simple_ml_pipeline that chains the load_data and train_model steps together. The output from load_data is passed as input to train_model, demonstrating how data flows between steps in a ZenML pipeline.\n\nPreviousDisable colorful loggingNextHow ZenML stores data\n\nLast updated 4 months ago',
+    'in the ZenML dashboard.\n\nThe whylogs standard stepZenML wraps the whylogs/WhyLabs functionality in the form of a standard WhylogsProfilerStep step. The only field in the step config is a dataset_timestamp attribute which is only relevant when you upload the profiles to WhyLabs that uses this field to group and merge together profiles belonging to the same dataset. The helper function get_whylogs_profiler_step used to create an instance of this standard step takes in an optional dataset_id parameter that is also used only in the context of WhyLabs upload to identify the model in the context of which the profile is uploaded, e.g.:\n\nfrom zenml.integrations.whylogs.steps import get_whylogs_profiler_step\n\ntrain_data_profiler = get_whylogs_profiler_step(dataset_id="model-2")\ntest_data_profiler = get_whylogs_profiler_step(dataset_id="model-3")\n\nThe step can then be inserted into your pipeline where it can take in a pandas.DataFrame dataset, e.g.:\n\nfrom zenml import pipeline\n\n@pipeline\ndef data_profiling_pipeline():\n    data, _ = data_loader()\n    train, test = data_splitter(data)\n    train_data_profiler(train)\n    test_data_profiler(test)\n\ndata_profiling_pipeline()\n\nAs can be seen from the step definition , the step takes in a dataset and returns a whylogs DatasetProfileView object:\n\n@step\ndef whylogs_profiler_step(\n    dataset: pd.DataFrame,\n    dataset_timestamp: Optional[datetime.datetime] = None,\n) -> DatasetProfileView:\n    ...\n\nYou should consult the official whylogs documentation for more information on what you can do with the collected profiles.\n\nYou can view the complete list of configuration parameters in the SDK docs.\n\nThe whylogs Data Validator\n\nThe whylogs Data Validator implements the same interface as do all Data Validators, so this method forces you to maintain some level of compatibility with the overall Data Validator abstraction, which guarantees an easier migration in case you decide to switch to another Data Validator.',
 ]
 embeddings = model.encode(sentences)
 print(embeddings.shape)
@@ -908,89 +789,89 @@ You can finetune this model on your own dataset.
 * Dataset: `dim_384`
 * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
 
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.1863     |
-| cosine_accuracy@3   | 0.4795     |
-| cosine_accuracy@5   | 0.6603     |
-| cosine_accuracy@10  | 0.7973     |
-| cosine_precision@1  | 0.1863     |
-| cosine_precision@3  | 0.1598     |
-| cosine_precision@5  | 0.1321     |
-| cosine_precision@10 | 0.0797     |
-| cosine_recall@1     | 0.1863     |
-| cosine_recall@3     | 0.4795     |
-| cosine_recall@5     | 0.6603     |
-| cosine_recall@10    | 0.7973     |
-| cosine_ndcg@10      | 0.4746     |
-| cosine_mrr@10       | 0.3726     |
-| **cosine_map@100**  | **0.3795** |
+| Metric              | Value   |
+|:--------------------|:--------|
+| cosine_accuracy@1   | 1.0     |
+| cosine_accuracy@3   | 1.0     |
+| cosine_accuracy@5   | 1.0     |
+| cosine_accuracy@10  | 1.0     |
+| cosine_precision@1  | 1.0     |
+| cosine_precision@3  | 0.3333  |
+| cosine_precision@5  | 0.2     |
+| cosine_precision@10 | 0.1     |
+| cosine_recall@1     | 1.0     |
+| cosine_recall@3     | 1.0     |
+| cosine_recall@5     | 1.0     |
+| cosine_recall@10    | 1.0     |
+| cosine_ndcg@10      | 1.0     |
+| cosine_mrr@10       | 1.0     |
+| **cosine_map@100**  | **1.0** |
 
 #### Information Retrieval
 * Dataset: `dim_256`
 * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
 
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.1836     |
-| cosine_accuracy@3   | 0.4877     |
-| cosine_accuracy@5   | 0.6603     |
-| cosine_accuracy@10  | 0.7918     |
-| cosine_precision@1  | 0.1836     |
-| cosine_precision@3  | 0.1626     |
-| cosine_precision@5  | 0.1321     |
-| cosine_precision@10 | 0.0792     |
-| cosine_recall@1     | 0.1836     |
-| cosine_recall@3     | 0.4877     |
-| cosine_recall@5     | 0.6603     |
-| cosine_recall@10    | 0.7918     |
-| cosine_ndcg@10      | 0.4733     |
-| cosine_mrr@10       | 0.3724     |
-| **cosine_map@100**  | **0.3793** |
+| Metric              | Value   |
+|:--------------------|:--------|
+| cosine_accuracy@1   | 1.0     |
+| cosine_accuracy@3   | 1.0     |
+| cosine_accuracy@5   | 1.0     |
+| cosine_accuracy@10  | 1.0     |
+| cosine_precision@1  | 1.0     |
+| cosine_precision@3  | 0.3333  |
+| cosine_precision@5  | 0.2     |
+| cosine_precision@10 | 0.1     |
+| cosine_recall@1     | 1.0     |
+| cosine_recall@3     | 1.0     |
+| cosine_recall@5     | 1.0     |
+| cosine_recall@10    | 1.0     |
+| cosine_ndcg@10      | 1.0     |
+| cosine_mrr@10       | 1.0     |
+| **cosine_map@100**  | **1.0** |
 
 #### Information Retrieval
 * Dataset: `dim_128`
 * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
 
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.1836     |
-| cosine_accuracy@3   | 0.4685     |
-| cosine_accuracy@5   | 0.6356     |
-| cosine_accuracy@10  | 0.7781     |
-| cosine_precision@1  | 0.1836     |
-| cosine_precision@3  | 0.1562     |
-| cosine_precision@5  | 0.1271     |
-| cosine_precision@10 | 0.0778     |
-| cosine_recall@1     | 0.1836     |
-| cosine_recall@3     | 0.4685     |
-| cosine_recall@5     | 0.6356     |
-| cosine_recall@10    | 0.7781     |
-| cosine_ndcg@10      | 0.4622     |
-| cosine_mrr@10       | 0.3629     |
-| **cosine_map@100**  | **0.3705** |
+| Metric              | Value   |
+|:--------------------|:--------|
+| cosine_accuracy@1   | 1.0     |
+| cosine_accuracy@3   | 1.0     |
+| cosine_accuracy@5   | 1.0     |
+| cosine_accuracy@10  | 1.0     |
+| cosine_precision@1  | 1.0     |
+| cosine_precision@3  | 0.3333  |
+| cosine_precision@5  | 0.2     |
+| cosine_precision@10 | 0.1     |
+| cosine_recall@1     | 1.0     |
+| cosine_recall@3     | 1.0     |
+| cosine_recall@5     | 1.0     |
+| cosine_recall@10    | 1.0     |
+| cosine_ndcg@10      | 1.0     |
+| cosine_mrr@10       | 1.0     |
+| **cosine_map@100**  | **1.0** |
 
 #### Information Retrieval
 * Dataset: `dim_64`
 * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
 
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.2055     |
-| cosine_accuracy@3   | 0.4767     |
-| cosine_accuracy@5   | 0.6274     |
-| cosine_accuracy@10  | 0.7534     |
-| cosine_precision@1  | 0.2055     |
-| cosine_precision@3  | 0.1589     |
-| cosine_precision@5  | 0.1255     |
-| cosine_precision@10 | 0.0753     |
-| cosine_recall@1     | 0.2055     |
-| cosine_recall@3     | 0.4767     |
-| cosine_recall@5     | 0.6274     |
-| cosine_recall@10    | 0.7534     |
-| cosine_ndcg@10      | 0.4625     |
-| cosine_mrr@10       | 0.3707     |
-| **cosine_map@100**  | **0.3787** |
+| Metric              | Value     |
+|:--------------------|:----------|
+| cosine_accuracy@1   | 0.75      |
+| cosine_accuracy@3   | 1.0       |
+| cosine_accuracy@5   | 1.0       |
+| cosine_accuracy@10  | 1.0       |
+| cosine_precision@1  | 0.75      |
+| cosine_precision@3  | 0.3333    |
+| cosine_precision@5  | 0.2       |
+| cosine_precision@10 | 0.1       |
+| cosine_recall@1     | 0.75      |
+| cosine_recall@3     | 1.0       |
+| cosine_recall@5     | 1.0       |
+| cosine_recall@10    | 1.0       |
+| cosine_ndcg@10      | 0.9077    |
+| cosine_mrr@10       | 0.875     |
+| **cosine_map@100**  | **0.875** |
 
 <!--
 ## Bias, Risks and Limitations
@@ -1008,22 +889,22 @@ You can finetune this model on your own dataset.
 
 ### Training Dataset
 
-#### Unnamed Dataset
-
+#### json
 
-* Size: 3,284 training samples
+* Dataset: json
+* Size: 36 training samples
 * Columns: <code>positive</code> and <code>anchor</code>
-* Approximate statistics based on the first 1000 samples:
-  |         | positive                                                                          | anchor                                                                              |
-  |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
-  | type    | string                                                                            | string                                                                              |
-  | details | <ul><li>min: 10 tokens</li><li>mean: 22.7 tokens</li><li>max: 48 tokens</li></ul> | <ul><li>min: 17 tokens</li><li>mean: 316.5 tokens</li><li>max: 512 tokens</li></ul> |
+* Approximate statistics based on the first 36 samples:
+  |         | positive                                                                           | anchor                                                                               |
+  |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
+  | type    | string                                                                             | string                                                                               |
+  | details | <ul><li>min: 13 tokens</li><li>mean: 23.92 tokens</li><li>max: 41 tokens</li></ul> | <ul><li>min: 31 tokens</li><li>mean: 321.11 tokens</li><li>max: 512 tokens</li></ul> |
 * Samples:
-  | positive                                                                                                                                                                        | anchor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
-  |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-  | <code>How does ZenML help in integrating machine learning with operational processes?</code>                                                                                    | <code>ZenML - Bridging the gap between ML & Ops<br><br>Legacy Docs<br><br>Bleeding EdgeLegacy Docs0.67.0<br><br>🧙‍♂️Find older version our docs<br><br>Powered by GitBook</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
-  | <code>How can I configure a data integrity check step in ZenML to perform outlier sample detection and string length verification on a dataset with specific conditions?</code> | <code>ks. For example, the following step configuration:deepchecks_data_integrity_check_step(<br>    check_list=[<br>        DeepchecksDataIntegrityCheck.TABULAR_OUTLIER_SAMPLE_DETECTION,<br>        DeepchecksDataIntegrityCheck.TABULAR_STRING_LENGTH_OUT_OF_BOUNDS,<br>    ],<br>    dataset_kwargs=dict(label='class', cat_features=['country', 'state']),<br>    check_kwargs={<br>        DeepchecksDataIntegrityCheck.TABULAR_OUTLIER_SAMPLE_DETECTION: dict(<br>            nearest_neighbors_percent=0.01,<br>            extent_parameter=3,<br>            condition_outlier_ratio_less_or_equal=dict(<br>                max_outliers_ratio=0.007,<br>                outlier_score_threshold=0.5,<br>            ),<br>            condition_no_outliers=dict(<br>                outlier_score_threshold=0.6,<br>            )<br>        ),<br>        DeepchecksDataIntegrityCheck.TABULAR_STRING_LENGTH_OUT_OF_BOUNDS: dict(<br>            num_percentiles=1000,<br>            min_unique_values=3,<br>            condition_number_of_outliers_less_or_equal=dict(<br>                max_outliers=3,<br>            )<br>        ),<br>    },<br>    ...<br>)<br><br>is equivalent to running the following Deepchecks tests:<br><br>import deepchecks.tabular.checks as tabular_checks<br>from deepchecks.tabular import Suite<br>from deepchecks.tabular import Dataset<br><br>train_dataset = Dataset(<br>    reference_dataset,<br>    label='class',<br>    cat_features=['country', 'state']<br>)<br><br>suite = Suite(name="custom")<br>check = tabular_checks.OutlierSampleDetection(<br>    nearest_neighbors_percent=0.01,<br>    extent_parameter=3,<br>)<br>check.add_condition_outlier_ratio_less_or_equal(<br>    max_outliers_ratio=0.007,<br>    outlier_score_threshold=0.5,<br>)<br>check.add_condition_no_outliers(<br>    outlier_score_threshold=0.6,<br>)<br>suite.add(check)<br>check = tabular_checks.StringLengthOutOfBounds(<br>    num_percentiles=1000,<br>    min_unique_values=3,<br>)<br>check.add_condition_number_of_outliers_less_or_equal(<br>    max_outliers=3,<br>)<br>suite.run(train_dataset=train_dataset)<br><br>The Deepchecks Data Validator</code> |
-  | <code>How can I develop a custom data validator in ZenML?</code>                                                                                                                | <code>custom data validator<br><br>📈Experiment Trackers<br><br>CometMLflow<br><br>Neptune<br><br>Weights & Biases<br><br>Develop a custom experiment tracker<br><br>🏃‍♀️Model Deployers<br><br>MLflow<br><br>Seldon<br><br>BentoML<br><br>Hugging Face<br><br>Databricks<br><br>Develop a Custom Model Deployer<br><br>👣Step Operators<br><br>Amazon SageMaker<br><br>Google Cloud VertexAI<br><br>AzureML<br><br>Kubernetes<br><br>Spark<br><br>Develop a Custom Step Operator<br><br>❗Alerters<br><br>Discord Alerter<br><br>Slack Alerter<br><br>Develop a Custom Alerter<br><br>🖼️Image Builders<br><br>Local Image Builder<br><br>Kaniko Image Builder<br><br>Google Cloud Image Builder<br><br>Develop a Custom Image Builder<br><br>🏷️Annotators<br><br>Argilla<br><br>Label Studio<br><br>Pigeon<br><br>Prodigy<br><br>Develop a Custom Annotator<br><br>📓Model Registries<br><br>MLflow Model Registry<br><br>Develop a Custom Model Registry<br><br>📊Feature Stores<br><br>Feast<br><br>Develop a Custom Feature Store<br><br>Examples<br><br>🚀Quickstart<br><br>🔏End-to-End Batch Inference<br><br>📚Basic NLP with BERT<br><br>👁️Computer Vision with YoloV8<br><br>📖LLM Finetuning<br><br>🧩More Projects...<br><br>Reference<br><br>🐍Python Client<br><br>📼Global settings<br><br>🌎Environment Variables<br><br>👀API reference<br><br>🤷SDK & CLI reference<br><br>📚How do I...?<br><br>♻️Migration guide<br><br>Migration guide 0.13.2 → 0.20.0<br><br>Migration guide 0.23.0 → 0.30.0<br><br>Migration guide 0.39.1 → 0.41.0<br><br>Migration guide 0.58.2 → 0.60.0<br><br>💜Community & content<br><br>❓FAQ<br><br>Powered by GitBook</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+  | positive                                                                                                                            | anchor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+  |:------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>How does ZenML integrate Spark step operators for executing individual steps?</code>                                          | <code>Spark<br><br>Executing individual steps on Spark<br><br>The spark integration brings two different step operators:<br><br>Step Operator: The SparkStepOperator serves as the base class for all the Spark-related step operators.<br><br>Step Operator: The KubernetesSparkStepOperator is responsible for launching ZenML steps as Spark applications with Kubernetes as a cluster manager.<br><br>Step Operators: SparkStepOperator<br><br>A summarized version of the implementation can be summarized in two parts. First, the configuration:<br><br>from typing import Optional, Dict, Any<br>from zenml.step_operators import BaseStepOperatorConfig<br><br>class SparkStepOperatorConfig(BaseStepOperatorConfig):<br>    """Spark step operator config.<br><br>Attributes:<br>        master: is the master URL for the cluster. You might see different<br>            schemes for different cluster managers which are supported by Spark<br>            like Mesos, YARN, or Kubernetes. Within the context of this PR,<br>            the implementation supports Kubernetes as a cluster manager.<br>        deploy_mode: can either be 'cluster' (default) or 'client' and it<br>            decides where the driver node of the application will run.<br>        submit_kwargs: is the JSON string of a dict, which will be used<br>            to define additional params if required (Spark has quite a<br>            lot of different parameters, so including them, all in the step<br>            operator was not implemented).<br>    """<br><br>master: str<br>    deploy_mode: str = "cluster"<br>    submit_kwargs: Optional[Dict[str, Any]] = None<br><br>and then the implementation:<br><br>from typing import List<br>from pyspark.conf import SparkConf<br><br>from zenml.step_operators import BaseStepOperator<br><br>class SparkStepOperator(BaseStepOperator):<br>    """Base class for all Spark-related step operators."""<br><br>def _resource_configuration(<br>            self,<br>            spark_config: SparkConf,<br>            resource_configuration: "ResourceSettings",<br>    ) -> None:<br>        """Configures Spark to handle the resource configuration."""</code> |
+  | <code>How can ZenML be used to finetune LLMs for specific tasks or to improve performance and cost?</code>                          | <code>Finetuning LLMs with ZenML<br><br>Finetune LLMs for specific tasks or to improve performance and cost.<br><br>PreviousEvaluating finetuned embeddingsNextSet up a project repository<br><br>Last updated 6 months ago</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+  | <code>How can I develop a custom model deployer in ZenML for efficient deployment and management of machine-learning models?</code> | <code>Develop a Custom Model Deployer<br><br>Learning how to develop a custom model deployer.<br><br>Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our general guide to writing custom component flavors in ZenML. This guide provides an essential understanding of ZenML's component flavor concepts.<br><br>To deploy and manage your trained machine-learning models, ZenML provides a stack component called Model Deployer. This component is responsible for interacting with the deployment tool, framework, or platform.<br><br>When present in a stack, the model deployer can also act as a registry for models that are served with ZenML. You can use the model deployer to list all models that are currently deployed for online inference or filtered according to a particular pipeline run or step, or to suspend, resume or delete an external model server managed through ZenML.<br><br>Base Abstraction<br><br>In ZenML, the base abstraction of the model deployer is built on top of three major criteria:<br><br>It needs to ensure efficient deployment and management of models in accordance with the specific requirements of the serving infrastructure, by holding all the stack-related configuration attributes required to interact with the remote model serving tool, service, or platform.<br><br>It needs to implement the continuous deployment logic necessary to deploy models in a way that updates an existing model server that is already serving a previous version of the same model instead of creating a new model server for every new model version (see the deploy_model abstract method). This functionality can be consumed directly from ZenML pipeline steps, but it can also be used outside the pipeline to deploy ad-hoc models. It is also usually coupled with a standard model deployer step, implemented by each integration, that hides the details of the deployment process from the user.</code>                                                                                                                                                                                                        |
 * Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
   ```json
   {
@@ -1117,7 +998,7 @@ You can finetune this model on your own dataset.
 - `dataloader_num_workers`: 0
 - `dataloader_prefetch_factor`: None
 - `past_index`: -1
-- `disable_tqdm`: True
+- `disable_tqdm`: False
 - `remove_unused_columns`: True
 - `label_names`: None
 - `load_best_model_at_end`: True
@@ -1178,31 +1059,21 @@ You can finetune this model on your own dataset.
 </details>
 
 ### Training Logs
-| Epoch      | Step   | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_384_cosine_map@100 | dim_64_cosine_map@100 |
-|:----------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|
-| 0.3893     | 10     | 1.7142        | -                      | -                      | -                      | -                     |
-| 0.7786     | 20     | 0.4461        | -                      | -                      | -                      | -                     |
-| 0.9732     | 25     | -             | 0.3544                 | 0.3592                 | 0.3674                 | 0.3523                |
-| 1.1655     | 30     | 0.1889        | -                      | -                      | -                      | -                     |
-| 1.5547     | 40     | 0.1196        | -                      | -                      | -                      | -                     |
-| 1.9440     | 50     | 0.0717        | -                      | -                      | -                      | -                     |
-| 1.9830     | 51     | -             | 0.3672                 | 0.3727                 | 0.3728                 | 0.3797                |
-| 2.3309     | 60     | 0.0474        | -                      | -                      | -                      | -                     |
-| 2.7202     | 70     | 0.0418        | -                      | -                      | -                      | -                     |
-| **2.9927** | **77** | **-**         | **0.3722**             | **0.3772**             | **0.3798**             | **0.3783**            |
-| 3.1071     | 80     | 0.0355        | -                      | -                      | -                      | -                     |
-| 3.4964     | 90     | 0.0351        | -                      | -                      | -                      | -                     |
-| 3.8856     | 100    | 0.0276        | 0.3705                 | 0.3793                 | 0.3795                 | 0.3787                |
+| Epoch   | Step  | dim_384_cosine_map@100 | dim_256_cosine_map@100 | dim_128_cosine_map@100 | dim_64_cosine_map@100 |
+|:-------:|:-----:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|
+| 1.0     | 1     | 1.0                    | 1.0                    | 0.875                  | 0.875                 |
+| **2.0** | **3** | **1.0**                | **1.0**                | **1.0**                | **0.875**             |
+| 3.0     | 4     | 1.0                    | 1.0                    | 1.0                    | 0.875                 |
 
 * The bold row denotes the saved checkpoint.
 
 ### Framework Versions
-- Python: 3.12.3
-- Sentence Transformers: 3.0.1
-- Transformers: 4.44.0
-- PyTorch: 2.5.0+cu124
-- Accelerate: 0.33.0
-- Datasets: 2.20.0
+- Python: 3.11.10
+- Sentence Transformers: 3.2.1
+- Transformers: 4.43.1
+- PyTorch: 2.5.1+cu124
+- Accelerate: 1.0.1
+- Datasets: 3.0.2
 - Tokenizers: 0.19.1
 
 ## Citation
@@ -1225,7 +1096,7 @@ You can finetune this model on your own dataset.
 #### MatryoshkaLoss
 ```bibtex
 @misc{kusupati2024matryoshka,
-    title={Matryoshka Representation Learning}, 
+    title={Matryoshka Representation Learning},
     author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
     year={2024},
     eprint={2205.13147},
@@ -1237,7 +1108,7 @@ You can finetune this model on your own dataset.
 #### MultipleNegativesRankingLoss
 ```bibtex
 @misc{henderson2017efficient,
-    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
     author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
     year={2017},
     eprint={1705.00652},

config.json

@@ -19,7 +19,7 @@
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
   "torch_dtype": "float32",
-  "transformers_version": "4.44.0",
+  "transformers_version": "4.43.1",
   "type_vocab_size": 2,
   "use_cache": true,
   "vocab_size": 30522

config_sentence_transformers.json

@@ -1,8 +1,8 @@
 {
   "__version__": {
-    "sentence_transformers": "3.0.1",
-    "transformers": "4.44.0",
-    "pytorch": "2.5.0+cu124"
+    "sentence_transformers": "3.2.1",
+    "transformers": "4.43.1",
+    "pytorch": "2.5.1+cu124"
   },
   "prompts": {
     "query": "Represent this sentence for searching relevant passages: "

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6b8c58ad12b1547c240e2803ab66e9f6a8fa7baaa022ce3c0336bc103cdbb96f
+oid sha256:24b8936c919b841952b05d08c369ac02d906ad7f6e4471b4141ddb8e9799e622
 size 435588776